Breaking the curse of numbers

My paper for the European Transport Conference #ETCMilan22 identified three issues with appraisal practice across Europe, based on my experience of working on a range of projects – at various stages in their development - over more than a decade:

·???????Appraisal isn’t suited to so-called ‘transformational projects’. This is called out by Diane Coyle in her book ‘Cogs and Monsters’ (“[cost-benefit analysis] should not be used to assess any investment large enough to have spillover effects”), and recognised in the UK finance ministry's appraisal guidance (the 'Green Book'). The latter now has an Annex that seeks to define transformational projects as those creating permanent change, and with supporting analysis that clearly articulates why non-linearities (e.g. in the relationship between transport infrastructure and economic growth) may emerge from them. But it’s not the case for those schemes that traditional cost-benefit analysis (CBA) will present decision-makers with sufficient information to understand whether they represent value for money.

·???????Appraisal – and particularly numerical analysis - isn’t guided, let alone driven, by outcomes of interest. We proxy the benefits of a scheme using travel-time savings, and test value for money by dividing the value of these by expected scheme costs. The paper includes an example of a public transport scheme CBA where 95% of benefits are derived from travel-time savings. Those deciding whether or not to proceed with that scheme won’t be able to discern whether it’s hitting the objectives they’ve set for it based on this analysis.

·???????Numbers are more salient than qualitative evidence. Decision-makers will remember the ‘benefit-cost ratio of 1.4’, and are less likely to remember the caveats to the analysis, or the fact that it’s based on a single point estimate of the future (in a world of deep uncertainty, a recurring theme at the conference). Indeed, in the words of Daniel Kahneman “What You See Is All There Is”. Not only does the evidence suggest that numbers are more memorable, but also our brains take shortcuts that mean we’ll assume there isn’t something missing in the analysis, even when there may well be.

What do we do about these issues?

I make three recommendations. First, drive analysis right from the start based on the outcomes of interest, and expected major detriments. This may mean appraisal is less comparable across schemes. Second, make sure traffic and appraisal models, created to test suitability against outcomes of interest, can run multiple scenarios, which are used to support decision making – this is a fundamental requirement in a world of deep uncertainty. Third, present information in a more balanced way. There are several accepted approaches to multi-criteria decision analysis (MCDA), but the key to breaking the #curseofnumbers is to present appraisal findings such that all aspects are as memorable as each other.

The paper is available to members of the Association for European Transport (AET) , and I’d be delighted to discuss it with you.