De-risking the Programme Portfolio with Reference Class Forecasting – Thoughts of the Professionals

It is a truism that poor performance in predicting cost, benefit and schedule targets in major programmes are caused as much by psychological issues as by technical and political issues. 

There are a range of clever techniques for overcoming the poor estimation that occurs from these issues with one of the most popular being Reference Class Forecasting (RCF). RCF at its most simplest is a technique where on looks at a reference class of similar programmes to the one about to be embarked on and, using this reference class, works out the statistical probability of where the forecast should reside depending on the risk appetite. Academics, including Bent Flyvbjerg (Professor of Major Programme Management at Oxford University's Sa?d Business School), have found that basing forecasts on actual outcomes of implemented programmes gives an estimate devoid of the psychological issues - biases and strategic misrepresentation - that exist during the planning stages. When combined with the estimate derived from the programmes SMEs this can create a significantly more accurate estimate to then create a realistic business case.

Although a tool of growing popularity for Major Programmes we’ve found no evidence of this tool being used systematically at the Portfolio Level and so Pindy Bhullar and myself took the opportunity of a workshop at the APM’s 5th Annual Portfolio Conference to share how RCF works with a group of Portfolio Practitioners and explore whether it could be implemented and, if so, what the pros and cons could be.

Implementing Reference Class Forecasting at the Portfolio Level

1.      The general consensus was that given the prerequisites of RCF was the capture of standardised time, cost and benefit data both at the beginning and end of a project across a range of projects then the portfolio level was the obvious place to organise that standardisation and data capture.

2.      The topic of data collection was discussed, and to create meaningful reference classes, project data points during the lifecycle would add value by comparing at various stages of the project.

3.      It was also commented that the main value of RCF was at the business case approval stage and in the management of organisational contingency – both traditional Portfolio Level activities. Given the value sat at the portfolio level it was felt the portfolio level would be the best place to consistently drive the RCF activity from.

4.      The ability to predict programmes that potentially could flag red and understand the underlying risk was debated. The assessment of projects in comparison to the reference class could provide the predictability of upcoming issues. Embedding the comparison approach would not only assess future risks and lessons learnt but also validate the health check of the project.

5.      Finally, it was commented that one challenge about RCF was the need to build the reference class which means capturing data over time and often organisations can’t wait for the requisite number of projects to begin and end before getting value from the approach. However, that data set can be acquired by doing a deep dive into projects that have concluded within the organisation – this can radically decrease the amount of time from begin to collect the data to having a workable RCF set. Even starting with one completed project would provide comparative analysis.

Pros and Cons of Reference Class Forecasting at the Portfolio Level 

On the positive side: RCF provides an outside view of the likely contingency need of a specific project. It was felt that the primary value of RCF at the portfolio level, therefore, was to help understand both how large a continency the portfolio director may need to retain (both in time and cost) to provide certainty over portfolio delivery and the level of risk that portfolio was carrying. As a subset of this advantage it also helps the Portfolio Director know where to direct her limited focus (if there is a significant mismatch between the project team estimates and what RCF is implying are necessary then there’s potentially a capability issue in that project). It also provides a valuable extra tool to support both business case drafting and sign-off – traditional areas of importance to the Portfolio Manager. Finally creating an RCF demands a standardisation of information capture across time, cost and benefit estimates – which is a critical discipline for any portfolio office to have mastered. This would require specialist skills to create and maintain the RCF. Portfolio Office’s would need to underpin the implementation of RCF with a robust change management programme.

On the negative side – RCF does require a level of effort and statistical knowledge both to capture the data consistently, across a large enough pool of projects, and to apply the techniques to understand what levels of contingency to apply. The Portfolio does need to be reasonably mature to take advantage of the technique. Where RCF is applied badly it can provide either false comfort or, at the other extreme, drive a need for a larger contingency than is actually needed which locks down resources and reduces competitive advantage.

In conclusion - most of the workshop attendees felt there was a place for Reference Class Forecasting in Portfolio Offices of a certain maturity and that, given a level of maturity in the data, the amount of effort to start applying the tool effectively was relatively light compared to the benefits that can be reaped from considerably better estimation across the portfolio. However, none had seen it applied at the Portfolio Office level. We promised to ask again at next years conference and hope to have received a different answer!

Pindy Bhullar and Adam Skinner