The Fallacy of Estimates

From a recent LinkedIn post

That goes on to say ...

Ignoring for the moment the uninformed notion that?estimating is the smell of dysfunction since no dysfunctions have been mentioned in that context, let alone corrective actions, other than to?Not Do Estimates, there is in fact a critical issue about estimating in all domains.

Of course, the last statement also ignores the estimates' time phase. Before the project starts, what's our?risk exposure?for the cost of this project? Let's get started and find out how much this will cost?is like Steve Levitt's?Freakonomics?description of the drug dealers -?here, try this, I give it to you for free.?Now, of course,?Not Estimating?has no connection to getting people hooked on Crack Cocaine, but?let's get started spending your money, and we'll find out later how much you're going to have to commit to this project, sounds a bit like?bait and switch?of the 1970's with?Cal Worthington?in Southern California when he'd advertise a car that didn't exist, get you to come down, then up-sell everything - ah the good olde days.

Just heard the story on National Public Radio about the Spanish firm that will?Stop Work?on the widening of the Panama Canal because they've been overrun by a Billion $.

The Fallacy of Estimating term is used many times without attribution. It starts with Daniel Kahneman and Amos Tversky?and humans' difficulties in estimating. The book,?Think Fast and Slow?is a continuation of their thesis. But the core of the thesis is contained in a few critical papers that must be read before drawing any conclusion, and most importantly, be read before listening to anyone who has read them.

• Anchoring and Adjustment in Software Estimation, Jorge Aranda and Steve Esterbrook, ESEC-FSE '05, ACM.
• Anchoring and Adjustment in Software Project Management: An Experimental Investigation, Timothy Costello, Naval Postgraduate School
• Anchoring Unbound, Nicholas Epley and Thomas Gilovich, Journal of Consumer Psychology, (20) 2010 pp 20-24.
• Assessing Ranges and Possibilities, Strategic Decision and Risk Management Stanford Certificate Program,?Decision Analysis for the Professional, Chapter 12. It is popular to speak about favorite methods as part of?decision-making?without saying what decision-making paradigm is used. The Stanford course is a good start.
• Implications of Attitude Change Theories for Numerical Anchoring: Anchor Plausibility and the Limits of Anchor Effectiveness, Duane T. Wegener, Richard E. Petty,?Brian T. Detweiler-Bedell, and W. Blair G. Jarvis, Journal of Experimental Psychology,?37, 62-69.
• The Anchoring-and-Adjustment Heuristic Why the Adjustments Are Insufficient, Nicholas Epley and Thomas Gilovich, Psychological Research,?Association for Psychological Science, Vol 17, No 4.
• Judgment Under Uncertainty, Amos Tversky; Daniel Kahneman Science, New Series, Vol. 185, No. 4157. (Sep. 27, 1974), pp. 1124-1131.

But another set of knowledge is needed to succeed in the estimating business: the acknowledgment that all estimates are probabilistic. This can't be said enough. The place to start is?Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, Paul R. Garvey. This book is the anchor for everything we do in the cost, schedule, and technical performance estimating business on software intensive programs. Mr. Garvey's work at Mitre is the basis and many of the tools and processes used in our domain.

You must have the last paper on your desk if you're interested in solving the?fallacy of estimating.?

How Did We Get Into This Estimating Fallacy Mess?

The original explanation for the estimating fallacy was that planners focused on the most optimistic scenario for the project rather than using past performance, subject matter experts, or parametric models of how much time similar work would require, given similar conditions. This is the?Optimism?fallacy.?What could possibly go wrong? Well, we know the answer to that now, don't we? This is common in our defense and space procurement world and most other worlds where politics drive high-stakes projects. It is a fundamental axiom?of life.?No Guts, No Glory. When the value at risk is high, conservative actions go by the wayside.?

Another explanation - one found in our domain as well - is the?Authorization Imperative.?If we want to get our program approved, we'd better not tell them how much it will cost. The James Web Telescope and Joint Strike Fighter are good examples of that. JWT is currently at around $7B it started out less than $1B. Similar to JSF.

So What Next?

It's the olde saw?Doctor, Doctor it hurts when I do this? Then stop doing that. This of course, is utter nonsense when it comes to estimating.?Doctor, doctor, we can't make good estimates. Estimates are the smell of dysfunction (with none listed) OK, then stop estimating and start spending. Yea Right!?

Here are a few?facts of life:

• Building products or supplying services for money almost always means spending other people's money. If it's your own money, do as you wish. If it's other people's money they get to say what you do with it. They shouldn't be acting like Dilbert's boss. BTW we can find examples of Dilbert Bosses everywhere. That's trivial. Pointing out those problems is child's play. How about providing solutions? They should understand enough about business management to know that all estimates are probabilistic. All estimates have built-in risks, some reducible, some irreducible, all knowable, and many unknown. If you have?Unknowable?risks, you'd better wait to start the project until you get those back into the?knowable?column.
• When those with the money give it to you, they expect you to spend it wisely. That means you have some notion of what you will spend it on to produce?value?to those who gave it to you. That you also know to some degree of confidence how much money you will need to spend to deliver the expected value the customer has given you the money for.

How Can We Get Better Estimates?

Let's assume for a moment that we understand why we need to estimate how much of other people's money we will spend.?

How do we get better? The answer is simple?-?We're spending other people's money, and they likely want to know how much, when, and what they are?getting.?We start by looking to the tried and true, field-proven, tractable approaches to estimating. This is not a platitude. Do we start by doing our homework, reading books and papers, looking at tools, and asking others?how we do this? We do what any person learning to do something new would do. We look to others first.?

What's Out There to Learn?

There are lots of sources for learning to estimate. But the first - field-proven way - is?Reference Class Forecasting.?It's the basis of most estimating processes in use today.

Bent Flyvbjerg?has lots to say on this. But some care is needed; he tends to overstate the intent of planners and estimators making poor estimates as?Liars. That may be the case at times. I know of a few programs, but it's overstating nonetheless. Calling people Liars is?fight'en words?where I come from in the Texas Panhandle, home of?T. Boone Pickens,?Dog the Bounty Hunter,?and?Randy Matson.

Let's assume you actually want to improve your estimating skills, abilities, and probability of success. Where do you start? First, you start by ignoring those who say it can't be done because it can. Then ignore those who say?estimates are the smell of dysfunction?because estimates are part of any credible business process. OK, if you believe in Unicorns and Pixy Dust, you might believe that making estimates is a dysfunction. Making estimates for the wrong reason is dysfunction. But doing anything for the wrong reason is a dysfunction. Learn to do things for the right reason, and as the poster campaign?at?Rocky Flats?said:

Don't Do Stupid Things On Purpose

By the way, the notion of?drip funding?is fine. But it needs to answer the question,?what is our estimate at complete? Drip Funding, also called?Time Boxed Scheduling?been around for decades.?Here's a small amount of money and a list of things I want you to do.?Do them, come back, and we'll talk more if you did them for?more or less?the money provided well. If not, you've now got information to calibrate the future capacity for work. This is called?Reference Class Forecasting.

You need to estimate credibly to be in business from the lawn care guy who cut our grass every day to the builder of Joint Strike Fighter.

Getting Started

So the first place to start is to inform yourself how others do credible estimating. Let's start with?How to Estimate if You Really?Want To.?But those aren't enough, and you'll need some tools. And before you listen to anyone telling you tools get in the way of innovation and understanding, ask if you can do non-stationary stochastic modeling of Monte Carlo Markov chains of dependent process flows in your head or by exchanging words between people. OK, back to the problem at hand.

There are many starting points for probabilistic estimating, but they all have one thing in common. We need to know what the problem is. For software, we need to know what?capabilities?the customer would like to possess when the project is?DONE. This is called?Capabilities-Based Planning.?Capabilities aren't requirements. Capabilities reveal the requirements, and requirements enable the capabilities to exist. Here's an example of a set of?evolving?capabilities for a health insurance ERP system.

So once we have something like this, we can start to decompose the parts into bite-sized chunks, just like those suggesting?drip funding?needs for success. These are Drips of work.?

Next, you'll need to suspend belief, just like the Unicorns. The suspension goes like this.

For the vast majority of commercial and a whole lot of military and space software systems, there is nothing new under the sun.

You may have yet to experience this new requirement personally. You may not have even heard of such a thing. But help is at hand - Google. Start there. Someone, somewhere, somehow, has built something similar. Find it, ask them,?do your homework, and build a Reference Class. Need help building the Reference Class? OK, then spend some money to build a prototype. Charge the customer for this exploratory effort. This, by the way, is called?agile development. Try a little, learn a little. Try some more, learn some more. Improve your probability of success with direct experience. But make sure you get paid for this. It's part of the project. Exploring like this on other people's money without them knowing it or without them paying you for it is really bad business. That's a true?dysfunction.

A Compendium of Estimating Resources

Here's a collection of resources dedicated to estimating software-intensive systems and their related hardware and work processes when applying traditional and agile product development methods. This setting ranges from cost to schedule, to risk, to technical performance - anything that has uncertainty?associated with the parameters. And since ALL project work has uncertainties - aleatory and epistemic - estimates must be made of these uncertainties to assess their impact on the probability of the project's success.

These Briefings, Papers, Books, Thesis, and other resources - for the most part - are retrieved from public sites and can be found with Google. When memberships are required, anyone spending other people's money should belong to these professional organizations:

IEEE,?IFPUG,?NESMA,?COSMIC,?ACM,?AACE,?ICEAA,?PMI,?APM,?ASPE.??

These professional organizations have estimating resources needed to increase the probability of project success, no matter the domain or development method.

The primary purpose of software estimation is not to predict a project's outcome; it is to determine whether a project's targets are realistic enough to allow the project to be controlled to meet them?— Steve McConnell.

We worked on a program at the Nevada Nuclear Security Site, where I also worked in High School, copying drawings on an ammonia copy machine.

The program manager had a quote applicable to everything we do, from planning to estimating, development, and deployment...

If you lack academic basics and validated experience, your advice is simply an unsubstantiated opinion without any basis in fact or principle.

USAF Col. (Ret), Program Manager,?NNSA Weapons Testing Site

Now For Some Tools to Support Estimate Making

Here's my list of favorites. They're favorites because I use them or know people who do:

• Risky Project
• Risk+?- it's gone, but still one of my favorites
• @Risk for Project
• ACEIT
• DelTek Accumen
• Crystal Ball
• COCOMO?and cousins COSTAR and other derivatives. Steve McConnell's Estimate Pro was like this.
• Seer
• Price Systems