The inevitability of optimism

Nine out of ten transport infrastructure projects run over on cost. Actual costs are on average around 30% higher than projected (sagging out to over 50% for some categories).

Oil and Gas projects sit at around 20% over on budget on average, 30% on time. However, the spread is large and heavily skewed to the right (with some notable mega-projects more meaningfully measured in ratios of actual to budget, running to 2 to 3).

Why?

Here are few putative explanations. Please add more in the comments.

Planning for success

We all know that the expression "Everything is going according to plan" doesn't imply a 20% cost overrun and a 30% slip in schedule. Our plans are rarely based on past performance and ought all to be prefaced with the implicit caveat "All being well". We build our plans on the basis that nothing goes (substantially) wrong and we manage uncertainty by writing down what could go wrong and what we will do if it does.

The problems of planning confidence are compounded by competition. There is no incentive to make informed allowance for historically reasonable contingency if no-one else in the bidding round is doing the same.

This buoyant optimism is further aided by availability - a bias that bases estimates on the picture of the past most available to our memory. As we set out on new ventures, humans are naturally conditioned to remember the rosy events of the past and to suppress the horrors.

Pernicious portfolio phenomena

Planning optimism means that the probability of coming in under budget is typically quite small. If we take a handful of projects and add the plan costs for each, the probability of reaching this total plan cost quickly becomes vanishingly small. This is because not only is it more likely that individual projects exceed budgets, they are also overwhelmingly more likely to exceed their budgets by a far greater margin than that by which successful projects come under their budget.

We can see this phenomenon if we look at the performance of a single project and compare it to the average performance of five identical projects. We've - not ungenerously - taken the probability of zero over-run for one project to be around 10% (mean over-run around 20%). However when we look at the portfolio together, because more projects run over than run under, and because they run over more than they run under, the probability of the portfolio meeting its target is virtually zero.

This phenomenon is particularly pernicious at a project level. Full project estimates are an aggregation of estimates of a number of constituent phases and activities. If each of these sub-projects is consistently estimated optimistically, then the aggregate is always going to slide further and further away, the more phases we include in the estimate.

This goes some way to explaining why larger projects have proportionally greater chance of over-running (and bigger over-runs) than smaller.

The toil of a fat tail

The portfolio phenomenon is complicated by the fact that projects are rarely independent - especially projects drawing on the same resources - and project phases never are.

The effect of correlations between delays and cost-overruns is two-fold. Like the little girl (with the little curl, right in the middle of her for'head), when projects are good, they're very very good, but when they're bad, they're horrid.

The few good projects (relative to our predictions) do great (which just exacerbates the availability bias), but they are still few, and with over-runs now even worse, they are even less able to save the portfolio.

Probabilistic panacea

A modest measure of uncertainty in costs and in schedule allows a simple scrutiny of past performance to condition our present predictions. Simple stochastic techniques not only furnish better predictions, they show where deviations may originate and provide a framework for leveraging experience (our own and the experience of others) to understand the greatest threats to performance and how best to distribute resources to manage them.