The Expert Trap - Recognizing and Avoiding Tunnel Vision

I was recently watching ?? Vincent D. Warmerdam 's stellar keynote address at PyData Amsterdam 2023. The speech is an end-to-end master class about challenging your own foundational assumptions on how problems should be solved and concludes with a message I took very much to heart: Familiarity can be a liability. Warmerdam points out that consultants are usually eager to be viewed as experts. To really internalize that vision of ourselves that we present to others, we have to first fully convince ourselves that we know what we are doing. And therein lies the trap—if we have convinced ourselves that we have already figured something out, it becomes a lot harder to apprehend that we might need a new idea.

If we, the so-called expert data consultants, can be honest with ourselves for a minute, how often do we truly take the time to appreciate the unique nature of each client we help? The predominant trend I have seen play out over and over again is, in fact, the exact opposite. There is a huge push in the consulting industry for "turnkey" solutions. And far from this being a dirty little secret, consultants proudly shout from presales rooftops how they can roll out analytical solutions for clients in hours simply because they have some sort of plug-and-play app ready to deploy! What is much more rare to see, unfortunately, is any effort towards reconciling deliverables with a fundamental truth: data & analytics are a means, not an end. It goes without saying that nobody wants to be a one-trick pony. But I would argue that being a ten-trick or hundred-trick pony is almost as bad. The number of tricks up our sleeves is entirely beside the point and defining ourselves by them is a disservice to both ourselves and our customers.

In the broadest terms, a business will invest in data & analytics to better understand itself. Going one level deeper, that can take the form of business intelligence (what happened, the rearview mirror), data science (what might happen, the windshield), or decision science (what is the optimal path forward, the GPS route guidance). But, beyond that, well...the only thing that is universally true is that nothing is universally true. It is very tempting to think, coming into a new project, "hey, I've seen this before, I know exactly what to do!" It is imperative to resist that temptation. No matter how tightly a client fits into the profile of our typical past engagements, it is not identical. In fact, if you cut a cross-section of any two companies, you would be hard-pressed to find a match in any of the most important categories: goals, finances, history, culture, location, etc. On top of that, add the relentless march of progress; a solution that was appropriate for a given company X years ago may no longer even fit that same company!

I do not mean to imply by any of this that experience is not valuable. Far from it. Experience provides two very key things. First, it gives you technical skills; no matter how creative you are, at the end of the day it is not possible to build a data solution without understanding a wide number of technologies. The more expertise you have in the relevant tech stack the faster and better you can put your solutions together. Second, experience can give you a strong starting point when it comes to designing a solution. You might know, for instance, that a given approach is not appropriate in 95% of cases; as long as you remember why (because the whole point is to resist the urge to jump to conclusions) you can save a great deal of trial and error and end up with a better final application.

The key takeaway for me from Warmerdam's speech was this: never stop experimenting. We should use our years of experience as tools in a toolbox rather than wearing them like handcuffs. And, above all, we need to keep our brains in the habit of viewing each problem with fresh eyes and then producing creative solutions, which is absolutely a skill that will atrophy over time if not exercised. To be honest, I fall into the expert trap from time to time myself. There is a great deal of ego involved in being viewed as an expert, as someone who knows everything there is to know on a subject. But here is the key to getting out of that trap: those two things are not the same! What if an expert is not somebody who has learned everything but rather someone who constantly pushes themselves forward? Perhaps that simple redefinition in our own minds would help us get over the ego hurdle of trying new things and approaching new challenges with open minds.

To bring this into more practical terms, I think the simplest thing we can immediately do to apply these principles is develop the habit of asking questions. That can be at both a high level ("Why is this important? What are the business goals behind this analytics initiative?") and a lower grain ("Why do I think this chart is the best choice to help my users find insights? Am I really sure my assumptions are true?"). Notice that I said ask questions—not necessarily answer them. The answers can come from self-reflection, sure, but they can also come from customers, colleagues, mentors, peers, journals, etc. If we start by simply being open to asking questions, we will be more than halfway towards freeing ourselves from the expert trap.