Testing Pearson-Tukey for contingency calculation, reliable or unreasonable?
Pedram Danesh-Mand
President of Risk Engineering Society, Head of Project Controls & PMO at KPMG Australia
Over the years working with our many talented people in Aquenta across Australia, I always thought being with a top national consultancy was our best opportunity to access exciting assignments for supporting clients. Guess what, I was wrong!
Hang on a second before reporting to HR, don’t get me wrong! :)
The exciting opportunities have exponentially increased, not only number wised but also from a wider range of projects and clients, since the acquisition of Aquenta by Jacobs last year. By joining our strong engineering capabilities of Jacobs and integrated project services (cost, time, risk and commercial) capabilities of Aquenta, we definitely took the game to its next level. While we are supporting our clients, we grow, learn and develop together. That’s priceless!
Enough marketing! :)
Last month, we engaged by a major client to undertake an independent review of the cost estimate and the key associated engineering risks as well as the required contingency allowance on one of their major gas pipeline projects at the concept stage of development.
The process below was undertaken by our Jacobs/Aquenta team.
After engineering and cost estimate reviews, and as you expect, we had also a number of discussions (and debates!) when assessing the required contingency allowance considering the project requirements. Factor-Based? Simulation? No-Simulation? Benchmark? Any easy way?
And eventually we decided to put the recommended method of Range-Based (through Johnson modification of the Pearson-Tukey formula) by Risk Engineering Society (RES) Contingency Guideline 2016 to the test. It was a fantastic opportunity to see the performance of this method in action! :)
Based on RES Contingency Guideline, the process for the most common approach of the Range-Based method is presented below:
In assessing the best case, most likely and worst case ranges, a number of factors including estimating judgment, previous experience, historical data, risk appetite and the organisation’s previous performance should be also considered.
The example below illustrates the required steps further:
- For inherent risks: apply a range (best case, most likely and worst case) to individual cost elements (recommended at project WBS level 4), then use any weighted average formula to turn the range into a number. Examples include:
- PERT formula: (BC + 4 X ML + WC)/6
- Johnson modification of the Pearson-Tukey formula for assessing a 50% confidence: (3 X BC + 10 X ML + 3 X WC)/16
- For contingent risks: apply the probability of risk occurring and range of time and cost consequences for residual risks
- Calculate total schedule contingency as the net schedule impact of inherent and contingent risks on the completion date
- Calculate total cost contingency for P50 and P90 levels (some recommendations for P90 estimation by using deterministic formula like twice SD)
We then compared our results with some other industry benchmarks and estimating ranges.
Interested to see the results? Here you go …
- Base Estimate: $412m
- 10% contingency by using the Factor-Based method: $41m
- Recommended P50 contingency by using Range-Based method (Pearson): $20m or 5%
- Recommended P90 contingency by using Range-Based method (Pearson): $89m or 21%
Comparison against AACEi EAR for the status of estimate: -10% to +30%
It was a great experience not only seeing the method in action but also see its quite reasonable performance against other methods of contingency calculation and industry benchmarks. A big tick for RES Contingency Guideline and RES committee!
But what about our final recommendation to this client?
Considering the industry Expected Accuracy Range (EAR) benchmark for a Class 3 Estimate, and as represented below, we recommended either of two decisions below are a reasonable decision for the project cost estimation at this stage of the project:
- The current TOC Estimate, (i.e. Base Estimate + 10% Contingency + Escalation) with an Expected Accuracy Range of -10% and +30%
- Recommended TOC P90 Estimate, i.e. Base Estimate + 21% Contingency + Escalation
Overall, if having an EAR of -20% and +30% is acceptable for the client’s economic modelling at this stage of project development and the project budget allocation purposes, the current estimate is an accurate representation of works. If client would like to proceed with one estimate at 90% confidence level at this stage of project, we recommended the TOC P90 Estimate, i.e. Base Estimate + 21% Contingency + Escalation to be considered.
Final decision? I don’t know! Eventually we are just supporting the decision making process but the final decision should be made not only based on the project characteristics and the phase of investment but also considering other organisation’s internal and external requirements and circumstances.
Final recommendation? Running a sophisticated cost risk model by using Monte Carlo Simulation through 10,000 iterations does NOT necessarily mean the high accuracy of your cost contingency assessment!
Final note? another exciting initiative for RES this year, is the 2nd edition of our RES Contingency Guideline. The document is going through a major development with a number of new methods of cost contingency calculation, practical examples, tips and tricks and lots more. We are expecting the draft document will be published for public consultation mid this year.
I definitely and strongly encourage and invite our members and all other risk and contingency professionals across the world to join and help us shaping and supporting our profession. More fun together!
Your comments and sharing your relevant experiences is welcomed, as always.
Director at Adept Strategic Pty Ltd
5 年Great article Pedram
Senior Business Consultant for AECO
6 年William Lopez Campo might be something you’d like ??
Director at PSOEEng Pty Ltd.
6 年Great article Pedram! Thank you for sharing.
President of Risk Engineering Society, Head of Project Controls & PMO at KPMG Australia
6 年Thanks for your comment, STEVE. For the purpose of this project at the Concept phase, we had a Class 3 estimate with 10% to 40% maturity level of project definition deliverables. We also benchmarked the assumptions against recent as built projects. From contingency calculation purposes and at the early stages of project development, we generally expect a higher contribution of 'Inherent' risks or 'uncertainties' compared to 'Contingent' risks. While a factor based method of contingency calculation, e.g. 10% of the Base Estimate as for this project, might be completely reasonable at the early stage of project development, the decision of 'reasonableness' and 'accuracy' of contingency allowance really depends on the phase of the investment lifecycle, availability and quality of information, quality of the Base Estimate as the most likely scenario, organisation’s maturity and project specific requirements. Our recommendation, from RES point of view, is that although the probabilistic methods of contingency assessment are generally recommended wherever possible especially at the key project investment decision making, e.g. Final Business Case submission, other methods of contingency calculation can, and should, be used successfully. In the 2nd edition of RES Contingency Guideline, which hopefully will be released for public consultation later this year, our recommendation evaluation criteria for selection of the best method of calculation are: - Assumptions and quality of information: what assumptions must be made by the risk analyst in order to implement the method? How strong are these assumptions? How much data is required to support these assumptions? - Technical complexity: is the method well understood by project team? What level of technical expertise is required to implement the method? - Software availability and competency: what types of software are available to implement the method? What level of training is required to implement the software? What are the computational overheads associated with the method? - Decision making process: how useful is the approach to decision makers? Will it consistently produce reliable answers when implemented by di?erent competent analysts?
Senior Quantity Surveyor MAIQS, CQS
6 年Hi Pedram quite interesting scenario you were faced with. One thing not mentioned (and forgive me if I have missed it) is the level of design maturity. Given that the project was in concept phase it was likely to be very low, possibly no more than 5 to15 percent. Given that I'm interested in how you would have carried out an inherent risk analysis and a contingent risk analysis on the base estimate and the project risks when both data sets would most likely be quite immature and lacking detail? It's a great effort to do what you have described but in my experience very rare in concept phase. Maybe the relative simplicity of a pipeline project was a factor?