Predictive Reporting - Polishing the Programme's Crystal Ball

If, like me, you’ve enjoyed Carlo Rovelli’s new book ‘The Order of Time’ you’ll know that our perception of time is an illusion and past, present and future are both interchangeable and fundamentally unknowable. If I was to use that excuse at Programme Board, however, to explain why the Programme had just hit an iceberg I suspect I’d get short shrift followed by an invitation to vacate the premises. And yet, as a PMO Manager, people look to me to predict the future or – more precisely – to present an analysis of the information available to me that helps senior stakeholders make a decision that will improve the programme’s future state. I find this challenge endlessly fascinating and, given I’m regularly asked about it, thought I’d share our thoughts on how a good PMO goes about polishing its crystal ball.

If we look at the specific requirement I phrase it as:

‘Analysing accurate information about a programme’s past state to allow the presentation of future scenarios to aid decision making.’ which breaks the problem into 3 manageable chunks:

1)     Analysing – doing something to the data that adds value to it

2)     Accurate information about a programme’s past state – creating the information flow process, tools and assurance activity to capture that accurate view of the past state

3)     To allow the presentation of future scenarios to aid decision making – creating a dashboard or reporting format that easily supports a rapid understanding of the possible scenarios and the decisions that need to be taken through either the formal or informal governance.

Holding a mirror up to the Programme:

All information is, by definition, information about a past state and most of the control machinery of a programme is geared around pulling that information from the programme, through various tools and processes, and assuring its quality and consistency. We refer to this activity as ‘The Engine Room’ in our P2Consulting PMOs and, particularly in the early stages of maturity, it is where a large amount of the effort is expended. There are no silver bullets to reduce this need, and indeed whole manuals can be written on how to do this. I tend to stick to a few top tips:

1)     For each information source understand what, where, how often and, most importantly, why – what question does it answer?

2)     A clear information map showing the flow of information and what processes, tools and templates will be used to gather it are critical both for building the information flow and for the training and stakeholder engagement which will be needed.

3)     A key role of the PMO is assurance of that information to ensure it is complete, compliant, consistent and at 100% coverage which all adds up to the information being correct.

4) Look for 'value multipliers' in the information - where simple integration of data streams (i.e. planning and risk) will deliver radically increased insight that is only available where the information combines.

5)     Keep track of the quality and consistency of reporting from the programme – it helps identify problem areas and show improvement in reporting maturity over time.

and most important of all remember information is both a resource and costs resource to acquire. Don't boil the ocean - focus on the information that really adds value (rather than filling a space in the board pack!) and capture it right! 

Lies, damned lies and statistics – analysing the data:

Well perhaps it’s not that bad but truly analytic predictive reporting does require the use of statistical analysis – which is a useful tool, but a tool nonetheless. How it is used or abused is critical. Once you’ve got your clean and accurate view of the past state then proper analysis can be conducted. I group the analysis I am conducting into 3 types:

1) Heuristic analysis – which is short-hand for using true Subject Matter Experts to review the data and identifying areas of concern. In practice this is why the PMO and Project Manager with years of experience is so valuable – their brains are trained to look for patterns in the data (without even being aware of it) that points to trouble. Never underestimate the power of giving properly refined data to experts and letting them draw conclusions from it.

2) Content-based quantitative analysis – what most people think of as analytics. Usually this involves taking large amounts of data and applying a technique such as Quantitative Risk Analysis or Reference Class Forecasting to present future scenarios and their probability of occurring. These different methodologies have strengths and weaknesses based on the quality of the initial data streams however, the information is more insightful still when combined into a single data set, encoded, and used to inform a machine learning approach to prediction. For example, using approaches such as Random Forest and Neural Networks it is possible to ‘train’ a machine to spot patterns in past programme data, and use these to predict programme issues well in advance of their impact. This also yields important information on the causes of programme issues, which can point the way to preventative improvements in the programme or portfolio. Whatever technique one is using, they are only useful if they can answer 3 questions:

• What is the risk to the current target for the future
• How big is that risk in scale and probability, and
• What decisions does the programme need to make to decrease the likelihood of that risk.

3) Content-free behavioural analysis – potentially the most interesting of the three (especially on extremely large programmes or portfolios where it’s impossible to be a subject matter expert on all areas). This technique looks at the previous predictions of various areas of the change programme/portfolio and ‘scores’ how well they’ve made predictions (i.e. how far out where the resource estimates, how late in the day did the milestone turn ‘red’ etc.). This score is then used to put confidence around future estimates such as resource need or RAG confidence. This can be twinned with a measure of reporting hygiene (i.e. how complete is an area’s reporting/ how often is it late?) to provide some interesting behavioural insight into the health of an area – both pointing out areas for focus/concern and supporting decisions around the size of contingency to allocate to these areas.

Remember, no matter how clever the technique it is only useful if the data it is based on can hold the weight of the analysis and if it can be presented in a way that supports decision making.

Presenting the analysis – the 3Ts

And there are numerous ways of presenting the data – however many are overly complex and take from the conversation rather than adding to it. At P2Consulting we have a rule of thumb for presenting data called ‘the 3Ts’. Essentially all reports should cover 3 things:

1)     Track – Tracking information is historic information that says where we have been. For instance milestones completed, resource utilised or cost spent.

2)     Target – Target information identifies what the current view of success is for the programme. A specific milestone on a specific date, a specific number of points delivered to a specific cost parameter, a specific customer satisfaction score by a specific programme stage.

3)     Trend – Trending information shows the current best estimate of what is likely to happen, against a specific information stream, if the programme continues to current expectations and assumptions.

Critically, therefore, all representations of information relating to the programme should be able to provide the position of all 3 of these information groups in order to inform decision making within the programme and help frame ‘the question’. A worked example using an Agile ‘burn-up’ chart is a helpful illustration:

The ‘track’ shows what has happened, the ‘target’ shows the current success criteria, the ‘trend’ shows the current best prediction. No matter the complexity of the analysis one should be able to present the relevant metric using this approach which focuses all decision making around two questions:

1)     How better to align the ‘trend’ with the ‘target’ (usually a discussion for internal governance), or

2)     Is the ‘target’ in the right place i.e. do we know what 'the right answer is' (usually a discussion for external governance and the business).

Ultimately all Risks can be seen in light of their impact on the ‘trend’ line – i.e. if this risk materialises the trend line will be shifted down by X amount, if we don’t mitigate issue Y then the trend will come in 100 points under the target, these are the options to pull back on target etc.

The Prediction Craft

Obviously the above only scratches the surface of the many approaches to supporting your Programme with Predictive Reporting and there are as many answers as there are programmes with specific and unique needs. The most critical thing to remember, of course, is that prediction is, by definition, an uncertain act. It needs elements of both art/experience and science/statistics – it is, therefore, a craft in the very truest sense and like all crafts constant practice and monitoring ones own performance to ensure improvement over time is the key.

About P2 Consulting:

If you’d like to talk more about predictive reporting or strengthening your PMO please visit www.p2consulting.com or get in touch with Adam or any of the P2 Consulting team on Linkedin. You can also follow P2 Consulting on Linkedin here. 

P2 Consulting is the market leading Project & Programme Management Company. We work in partnership with our clients to turn their business ambitions into reality, bringing a unique blend of leading-edge thinking and hands-on delivery. We bring the drive, the passion and the courage to act leaving your business stronger, fitter and more profitable for having worked with us.

P2 Consulting was formed by a revolutionary team of entrepreneurial leaders and award winning consultants. Since our creation, our passion for delivering business success for our clients has shaped our vision, our company and every member of our P2 Consulting team. We are proud to have delivered some of the largest and most successful programmes in corporate history.