Forecast or Projection

Let's kick off this with a really fun game, that does require a few minutes of thought to complete. It's taken from Kahneman/Sibony/Sunstein's book NOISE and it is the single best example of how most forecasts are done in today's business environment.

Here's the game:

Imagine that you are a member of a team charged with evaluating candidates for the position of chief executive in a moderately successful regional financial firm that faces increasing competition. You are asked to assess the probability that the following candidate will be successful after two years on the job. Successful is defined simply as the candidate’s having kept the CEO job at the end of the two years. Express the probability on a scale from 0 (impossible) to 100 (certain).

Michael Gambardi is thirty-seven years old. He has held several positions since he graduated from Harvard Business School twelve years ago. Early on, he was a founder and an investor in two start-ups that failed without attracting much financial support. He then joined a large insurance company and quickly rose to the position of regional chief operating officer for Europe. In that post, he initiated and managed an important improvement in the timely resolution of claims. He was described by colleagues and subordinates as effective but also as domineering and abrasive, and there was a significant turnover of executives during his tenure. Colleagues and subordinates also attest to his integrity and willingness to take responsibility for failures. For the last two years, he has served as CEO of a medium-sized financial company that was initially at risk of failing. He stabilized the company, where he is considered successful though difficult to work with. He has indicated an interest in moving on. Human resources specialists who interviewed him a few years ago gave him superior grades for creativity and energy but also described him as arrogant and sometimes tyrannical.

Recall that Michael is a candidate for a CEO position in a regional financial firm that is moderately successful and that faces increasing competition.

What is the probability that Michael, if hired, will still be in his job after two years? Please decide on a specific number in the range of 0 to 100 before reading on.

I invite you to share your thoughts later in the comments, but for now, please simply remark upon the number and the reason.

You had much to ponder, and like everyone, you came to this number after a series of trial-and-errors...you likely dismissed anything <5% as too low and above 90% as too high. Based on the weight of some or all of the determining factors, you arrived at you assumption and also, simultaneously prepared all of your arguments to defend your prediction.

The problem is that your prediction is UNVERIFYABLE, and thus does not require a whole lot of rigour on your part to derive it. See, even if you say you are 95% certain he will be, there is the 5% chance he won't, and if he does not survive the two years, it was merely a bad beat. Two problems exist with this that make it a projection and NOT A FORECAST. For one, Michael is not real, so thus cannot be checked. Second is that the assumption is probabilistic, which makes it subject to being just unluckily wrong. People asked to produce these types of projections for you wrap them in the Holiday Paper of valuable insight, but really have no real incentive to be incredibly accurate for you. You should take these assertions and delve just one level below the BIG PICTURE story they are trying to tell. There is only one question a CEO needs to ask in order to test the IMMEDIATE VALUE of any projection or forecast:

What method did you use to come up with that insight?

You see, in the world of projection and forecasts, almost all the insights provided are unverifiable to any extent, either through tech, time, or talent shortfalls. The only method you have to test the value of it is to question how they got to the answer. A path filled with stats models trained over and over and in Test and Control environments should beat tarot cards or tea leaves, regardless of how they have performed in the past. As a business leader, you need not KNOW WHAT the answer is, but knowing the mostly likely path to take to get to the best response is your superpower! Knowing how each input affects the probability is the moonshot.

We can get there. It is in sight.

Jerry M.

PS: I gave Michael Gambardi a 70% chance of staying on based on his arrogance, energy, and assumed his refusal to abandon his commitment.