Sharpen your pencil...

Estimating how long it will take to complete a complex project is extremely difficult. Overruns in public projects are regular headlines and private sector projects often suffer the same fate. Perhaps you have had the same experience? Can you think of a large-scale complex project that delivered to it's original schedule?

This isn't a surprise. Hofstadter's law, named after Douglas Hofstadter, states 'It always takes longer than you expect, even when you take into account Hofstadter's Law'. And yet, we still continue to produce hopelessly optimistic project plans that represent the best-case scenario when everything works perfectly, first time and there are no unexpected delays.

This suspension of disbelief is systematic, embraced by both the project team and the approvers, in a shared planning delusion. Why do we do this to ourselves? Teams are instructed to strip out contingency and 'sharpen their pencil', i.e. revise down their estimates, to hit a top-level aspiration or budget deadline. Realistic estimates, based on real experience (and sometimes even hard metrics from previous projects), are ignored in the pursuit of a best-case work of fiction that we all secretly know is highly unlikely to be met.

One possible explanation for this planning fallacy is known as the authorization imperative. The project team want to get their plan approved and they know that it's easier to get forgiveness (for overruns) than permission (to commence the project if a realistic effort estimate were provided). Such deliberate underestimation was named by Jones and Euske, in their 1991 paper Strategic Misrepresentation in Budgeting, as "strategic misrepresentation".

The incentive to engage in this deception is amplified if there is a competing solution also 'bidding' for the work. This can become a 'race to the bottom' as teams compete to produce a seemingly credible proposal which promises the earliest or cheapest delivery regardless of whether it's actually achievable. In an internal context, with the absence of a commercial penalty for late delivery or cost overruns, this behaviour is entirely rational if a little destructive.

If the decision-makers are unaware, disinterested or simply don't have time to compare the proposals in sufficient detail to detect the optimism-basis then the team with the realistic option loses out, only to watch the competitor secure the work and then slip well past their original, realistic, dates.

Holding teams to account for the accuracy of their estimates is challenging. Firstly, as numerous examples illustrate, the task of estimating in the face of complexity is genuinely hard and the art/science of doing so is still in its infancy. Secondly, and perhaps most significantly, for long-duration complex projects it's rare for the same individuals to see the project all the way through, both team members and approvers. Short-term incentives prevail.

In my experience there are no easy answers to this conundrum. However we can all make a choice and ask ourselves honestly: 'do I really believe this is achievable?' or 'would I bet my <insert prized possession> on hitting this date?'. For those of us endorsing or approving plans we have a duty to take the time to understand and challenge delivery risk, just like we would with other types of risk, and be honest with ourselves for the good of everyone concerned.

(Views in this article are my own)