Fallacy of Studying “Best Practices”

I did a full-day workshop in Bangkok last month showing marketers how to make more balanced and scientific decisions with data. No calculations were required, nor were any mathematical equations harmed in the process.

One of the reasons I created this workshop is that even with all the data and analytical tools available today, managers continue to make rudimentary, logical errors. They draw unwarranted conclusions by using unscientific methods and succumbing to biased reasoning.

For instance, remember that last management book or white paper you read, outlining the best practices at various successful companies? Maybe the author argued that these 5 businesses – or maybe these 50 businesses – succeeded wildly, and what they all had in common was a certain set of best practices. Perhaps they all realized that their existing business was in peril, so they stepped up and implemented a “bet the company” strategy. They didn’t look back, they doubled down on innovation, and they succeeded. They succeeded, so the author argues you will too, if you follow this best practice.

But what the book doesn’t tell you is that while these companies succeeded by going all in, another set of companies chose to go all in and they failed, with many of them altogether out of business now. So studying best practices in order to decide what your business should do is a decision-making method that suffers from what experts would call "survivorship bias." Only successes are in your data set, rather than successes and failures, so your data set is not objective.

Truth is, we can all probably learn more by studying failures than we can by studying successes. The only problem is that the managers involved in business failures tend not to be very loquacious, while successful managers welcome all sorts of publicity and easily get featured in white papers and business studies.

The study of best practices (i.e., the study of successes, rather than failures) has become a revered management tool largely because we all have a natural bias to seek out data that confirms our initial hypothesis. And giving in to the confirmation bias is a sure way to make big mistakes. If you want to offset and defeat your own confirmation bias, then you have to will yourself to search carefully for disconfirming data – data that would disprove your judgment or hypothesis.

One way to do this is with an exercise that decision scientist Gary Klein calls a “premortem.” Let’s say you have a big decision to make, and your group has just about finalized its decision. To evaluate your decision more objectively, you should actively seek out any contrary evidence or opinions. So get everyone together in the room and tell them to imagine that we made this decision and went ahead as we’re currently planning, but now it’s two years later and the project failed. So each of us should take five minutes and write down a story explaining why it failed. This exercise gives everyone in the room a license to look for alternatives, and anyone who had any doubts about the decision but was reluctant to speak up can also now be heard.

Another way you can search for disconfirming data is by paying for it. Whenever Warren Buffet acquires another company for Berkshire Hathaway he frequently hires two bankers. One banker gets a bonus if the deal goes through; but the other banker gets a bonus if the deal doesn’t go through.

It’s not that “best practices” aren’t worth thinking about. Just beware of the inherent fallacy in making the assumption that they are truly representative.