Overconfidence: it’s worse than you think (and what to do about it)

Our brains evolved to effortlessly combine past experience and present situations into snap judgements and quick actions. Most of the time, we are stunningly accurate. We can judge aspects of someone’s personality and career success in less than a tenth of a second (Borkenau et al 2009; Todorov et al 2015). Shown a quick video clip from a bike race, our brains effortlessly judge the average speed and heading of the bikers, the average hues and leaf shapes of foliage, and even the family resemblance of bystanders in under a second (Whitney and Lieb 2018).

Yet the last few decades of research have revealed a long and embarrassing list of mental blindspots and biases. If you asked scientists to rank these failures, overconfidence would likely place first as our most persistent and troublesome mental failure (DeBondt and Thaler 1995).

Overconfidence fills the business news. And it's not just Elon Musk's tweets. 70 to 90% of mergers and acquisitions fail (Malmendier and Tate 2015). Yet aquisitions remain frequent, indicating many CEOs believe that they can beat the odds. CEOs often use internal resources to finance these mergers, which suggests they think external investors undervalue and are underconfident in the firm. CEO overconfidence can also be seen in how often they hold stock options until close to their expiration, thus keeping their wealth underdiversified. Overconfidence plagues large projects beyond M&As as well. 90% of large transportation projects have significant cost overruns (Segelod 2017). Disturbingly, planning for these inevitable roadblocks does not seem to be improving. Large projects are just as likely to fail, go over budget, or miss deadlines today as 70 years ago.

The sorry truth is that people and teams have systematic overconfidence. We mistake a clear view of our destination for a simple journey. We mistake a precise projection for a certain outcome. We mistake feeling for knowledge.

Early in my career, I attempted to compensate for this overconfidence by imagining and modeling the most likely, best, and worst case scenarios. Sometimes I would pick the scenarios and numbers, other times I would have the client do it. Either way, comparisons between predictions and reality proved relentlessly humbling. Time and time again, my clients and I narrowed the likely ranges of plausible outcomes, under-appreciating the possibility of extreme values. In statistics, we describe ranges of plausible values for an unknown value with the term confidence interval. So, in statistics jargon, you could say our confidence intervals were too narrow.

In this article, we focus on how we can make broader, more accurate confidence intervals. Looking at overconfidence with this metric allows us to see how bad our problem is. And the lessons we learn from focusing here will apply more broadly.

Quantifying uncertainty with confidence intervals

Confidence intervals come with a percent number that roughly means the probability that the range contains the unknown number of interest. Thus, a 100% confidence interval would contain literally all possible values. This is of course so uninformative as to be useless. By convention, statisticians typically talk about 95% confidence intervals, which should theoretically fail to contain the true value only 5% of the time. However, for making predictions or modeling scenarios we can pick any percent coverage we like. (Technically, there are distinctions between ranges for unknown current values and predicted ranges for future values, but we will ignore this for clarity.)

Many studies have quantified our tendency to make too narrow confidence intervals despite our best efforts. In one experiment, researchers asked people to produce low and high end estimates of the number of eggs produced in the United States (Alpert and Raiffa 1982). Participants were told to estimate a number such that there was only a 1% chance the true value was lower. They were similarly asked to estimate the upper end, the highest possible value, such that there was only a 1% chance the true value was higher. People reliably gave ranges that are too narrow. Rather than covering 98% of likely values, they covered close to 60%. In other words, when asked to give almost the complete range of possibilities, people delivered a myopic theory that covered the truth a little better than half the time

That study used ordinary people. But, as suggested by our earlier discussion of CEOs, experts do not fare noticeably better. For example, Ben-David, Graham, and Harvey (2013) asked corporate chief financial officers to forecast yearly returns for the S&P 500. Despite the CFOs' knowledge about finance and the economy, their 80% confidence intervals contained the true answers only 36% of the time.

What can we do to help ourselves be less overconfident?

One clever quantitative trick for improving estimates is to force yourself to exhaustively consider the entire range of possibile outcomes. In a method called Subjective Probability Interval Estimates, or SPIES, by its inventors (Haran, Moore, & Morewedge, 2010), one divides the entire range of possible outcomes into bins and estimate the probabilities that each bin contains the correct answer. A statistician or data scientist can then use these estimates to calculate any confidence interval desired. This data gathering approach forces us to focus attention on possibilities that we would otherwise overlook. This is great for estimating specific numbers and event possibilities.

A broader and less technical psychological approach to helping us think about overlooked possibilities is the premortem. A premortem is a simulated postmortem in which you imagine the project has failed and you are discussing why it went wrong. This approach is based on the idea of prospective hindsight, in which people can generate more criticisms when they think of an outcome as certain, and was popularized by Gary Klein’s 2007 article in Harvard Business Review. The premortem smartly inverts the process by which dissenting opinions and doubts are gradually suppressed as a project develops. (Atlassian has a nice guide on running a premortem with your team.)

The premortem is great for thinking broadly and uncovering multiple scenarios. The smart move from here is to develop a portfolio of strategies to handle different scenarios. With this portfolio in hand, one helpful technique is to identify the core elements that all the strategies share versus those that are contingent. It can sometimes also be possible to treat the portfolio of strategies like a multi-armed bandit problem, where you dynamically allocate resources based on a metric calculated from the probability of each scenario.

Why is overconfidence such a problem for us?

One possibility is that we dislike uncertainty. Yet there is little evidence people inherently dislike uncertain advice (Gaertig and Simmons 2018). Another possibility is that we find confidence attractive and often cannot distinguish a confident facade from true expertise. Under this theory, we tend to promote the overconfident. Indeed, overconfident candidates may be more likely to be selected for leadership positions (Hayward, Shepherd, & Griffin, 2006) and achieve high status in groups (Anderson et al., 2012). Yet this explanation seems to just move the mystery without explaining it. If we are susceptible to overconfidence because we find confidence attractive, then why do we find confidence attractive?

Isn’t confidence good?

Sure, to be fair, confidence is often helpful and even overconfidence has its upsides. It can sometimes produce great outcomes for the confident and those who believe in them. For example, overconfidence may be useful when people need motivation to succeed (Tenney, Logg, & Moore, 2015). Value in these types of instances could even outweigh the costs in others, which would explain why overconfidence is so prevalent. Overconfident CEOs appear better able to exploit industry growth opportunities and translate them into value for the firm, which may offset their pursuit of acquisitions (Hirshleifer, Low, & Teoh, 2012). One lesson from the literature is that there’s little downside in showing nonverbal expressions of confidence. Appearing confident is always good.

Final thoughts: overconfidence is like getting lost in a story

In the end, there may be no simple explanation for why we humans are so susceptible to overconfidence. The way I see it, the appeal of confidence is closely related to the appeal of a good story. All of those quick impressions and judgements made by our brain ultimately get organized into stories and we spend much of our lives lost in these stories.

Yet there is no one best way to tell a good story. Simplicity can help. So can specific, concrete details. There’s Kenn Adams’ story spine that many movies use. But, for every rule, you can find a brilliant example that breaks it: Narrators can turn out to be unreliable; events can be told out of order; characters may address the audience. What good stories have in common is not a technique but an ability to enthrall and transport us.

This ability to create a shared experience with words is what makes us so susceptible to overconfidence. Finding and embracing good stories is just part of the human condition. It’s something our brains effortlessly do.

If overconfidence is that much a part of who were are, then it’s not a problem we are going to solve. Still, we can be confident that strategies for limiting it will provide us with great and persistent advantages.

Photo by Startaê Team on Unsplash

Mentioned work

Borkenau, P., Brecke, S., M?ttig, C., & Paelecke, M. (2009). Extraversion is accurately perceived after a 50-ms exposure to a face. Journal of Research in Personality, 43, 703–706.

De Bondt, W. F., & Thaler, R. H. (1995). Financial decision-making in markets and firms: A behavioral perspective. Handbooks in operations research and management science, 9, 385-410.

Gaertig, C., & Simmons, J. P. (2018). Do People Inherently Dislike Uncertain Advice?. Psychological science, 29(4), 504-520.

Hirshleifer, D., Low, A., & Teoh, S. H. (2012). Are overconfident CEOs better innovators?. The Journal of Finance, 67(4), 1457-1498.

Klein, G. (2007). Performing a project premortem. Harvard Business Review, pp. 18 –19.

Malmendier, U., & Tate, G. (2015). Behavioral CEOs: The role of managerial overconfidence. Journal of Economic Perspectives, 29(4), 37-60.

Mitchell, D., Russo, J., & Pennington, N. (1989). Back to the future: Temporal perspective in the explanation of events. Journal of Behavioral Decision Making, 2, 25–38.

Segelod, E. (2017). Project Cost Overrun: Causes, Consequences, and Investment Decisions. Cambridge University Press

Tenney, E. R., Meikle, N. L., Hunsaker, D., Moore, D. A., & Anderson, C. (2018). Is overconfidence a social liability? The effect of verbal versus nonverbal expressions of confidence. Journal of personality and social psychology.

Tenney, E. R., Logg, J. M., & Moore, D. A. (2015). (Too) optimistic about optimism: The belief that optimism improves performance. Journal of Personality and Social Psychology, 108(3), 377.

Todorov, A., Olivola, C. Y., Dotsch, R., & Mende-Siedlecki, P. (2015). Social attributions from faces: Determinants, consequences, accuracy, and functional significance. Annual review of psychology, 66, 519-545.

Whitney, D., & Leib, A. Y. (2018). Ensemble perception. Annual review of psychology, 69.