Reference Class Forecasting

When it is conjectured,?You can't do estimates on software projects. Stop for a second and think about that concept.

Really, "you have no idea whatsoever about how long this will take and how much it will cost?"?I don't want to estimate because it is a bit different and a waste of time. That says you're either too lazy to do the estimating, I need to learn how to do the estimating, or you really want to get going and write software because that's what you do for a living.

Could you take a look at the picture below? Many contributors to the estimating conversation have mentioned?estimating bias?as a source of?bad estimates?(which they want to fix by doing NO estimates)?and Bent Flybjerg. This is how to do?Reference Class Forecasting?for?ANY?project. Now you may not need to do estimates because your project is of a size that you can see the end from the start, it has low enough risk that the variances in cost are not considered a problem, or short in duration so that you can control most of the variables quickly, or your customer doesn't really have that much at risk, so the investment in estimating no matter how small is not considered that useful.

But let's pretend these are not the case. There is some non-trivial amount of money at risk, the?planning horizon?is outside your ability to see the end, your customer has to go ask someone for that money (the CIO, the BOD), or the business actually will depend on staying on or near budget and have your features show up on or near the planned date.?

Here's how to start the process of making a statistically adjusted estimate of time and budget.

Reference Class Forecasting Example

Reference Class Forecasting is a method of predicting the future by looking at similar past situations and their outcomes. This approach to estimating was developed by Daniel Kahneman and Amos Tversky, which earned Kahneman with the Nobel Prize in Economics.

Our long-time friends from California have moved to our neighborhood here in Colorado. Their moving van arrived today.

We brought coffee to them while their old house was being unloaded into the new house. Talking with the moving van owner, he started telling stories about?estimating?the load in pounds. The?agent?makes the first estimate of the weight of the load and issues a quote for the cost of the move. Then the van owner picks up the load and weighs the truck before getting on the road. He told us that the range of precision and accuracy is all over the place, depending on the?agent. Sometimes it's very close. Sometimes it's not.

The estimate's quality depends on the estimator's skill and experience. The?reference class?estimating process is part of that skill and experience.

Suppose we switch to software development. I re-listened to?the Agile for Humans?podcast with Steve McConnell about estimating. Steve's discussion was focused on how to make good estimates. How to put these to work to make business and technical decisions. These themes are based on how to book?Software Estimation: Demystifying the Black Art. This book should?be on the shelf of any credible developer.

After listening again, it was clear those providing a forum for the No Estimates advocates still need to address the fatal flaw of the No Estimates discussion.

There is?NO?principle by which you can decide in the presence of uncertainty without estimating.?

The podcast provided a nice overview of why we should estimate, how to estimate, and what to do with those estimates. We even discussed how to deal with the dysfunctional aspects of management when making estimates. But in the end, we need estimates to provide value in exchange for money credibly.

There can be no consideration for NOT estimating, except on de minimis projects. Just like the moving van owner, he needs an estimate for the weight of the load. From there, he can confirm that the load will fit on the truck. The trailer has a load limit of 33,000 pounds. The mechanics of household goods have a reference class parameter for the size of 33,000 pounds. This comes from empirical?data of moving household goods. Some houses have heavier goods, some have lighter goods. But the estimator?knows how to adjust for that.

It seems those in the software development business who conjecture that estimates are not needed, is the?smell of dysfunction, is evil, and should be stopped, have failed to understand not only the basic need for the estimate when making a decision in the presence of uncertainty but also the basic principles of estimating in the presence of the uncertainties that always exist on projects.

When the idea of No Estimates?is given equal weight to Steve's message, it willfully ignores the decision-making principle. And replacing?that principle?with the practices and processes that attempt to decide without estimates.

This is like the moving van owner saying?I get wild, wrong, and estimates that cause problems with my trailer, tractor, load management, and fee payments for carrying cargo across state lines. So I have an idea. I'll not ask for or use any estimate; I'll start emptying the house in Seal Beach, California, and start planning to move the house to Colorado. What could possibly be wrong with that?

Resources

• "Over Budget, Over Time, Over and Over Again: Managing Major Projects," Bent Flyvjerg, in Chapter 13, The Oxford Handbook of Project Management, pp. 321-344.
• "Reference Class Forecasting for Hong Kong's Major Roadworks Projects," Prof Bent Flyvbjerg, Ir CK Hon, and Ir Dr. WH Fok, ?Proceedings of the Institution of Civil Engineers 169, November, Issue CE6, pp. 17-24
• How Big Things Get Done, ?Bent Flyvbjerg and Dan Gardner, Currency, New York, 2023.
• The Oxford Handbook of Megaproject Management, Edited by Bent Flyvbjerg, Oxford University Press, 2017.
• "Prospect Theory: An Analysis of Decision Under Risk," Daniel Kahneman and Amos Tversky, Econometrica, Volume 47, Number 2, March 1979.
• "Intuitive Prediction: Biases and Corrective Procedures," Danieal Kahneman and Amos Tversky, Defense Advanced Research Projects Agency, Office of Naval Research, Engineering Psychology Programs, June 1977.
• "Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking," Daniel Kahneman, Dan Lovallo, Management Science, Vol. 39, No.1, pp. 17-31, Jan 1993.
• "Procedures for Dealing with Optimism Bias," The British Department of Transport, June 2004.
• "From Nobel Prize to Project Management: Getting Risk Right," Bent Flyvbjerg, Project Management Journal, Vol. 37, No. 3, pp. 5-15, August 2006.
• "Report for the Edinburgh Tram Inquiry," Prof. Bent Flyvbjerg and Dr. Alexander Budzier, Version 14, February 2018.
• "Total Cost Management Framework: An Integral Approach to Portfolio, Program, and Project Management," 1st Edition, Revised, 2012.