Why Bother with Probability and Statistics?

It is conjectured that uncertainty can be dealt with by ordinary means with open conversation, identification of the uncertainties, and their handling strategies.

Quantitative methods are too elaborate and unnecessary for problems except for the most technical and complicated ones.

When asked what uncertainty means,?the answer many times is?probably?or?very?likely. But not a quantitative measure meaningful to the decision-makers. Since the future is always uncertain in our project domain, making decisions in the presence of uncertainty is a critical success factor [1] for all project work.?

Decision making is one of the hard things in life. True decision-making occurs?not?when we already know the outcome, but when we do not know what to do. When we have to balance conflicting values, costs, schedule, needed capabilities, sort through complex situations, and deal with real uncertainty. To make decisions in the presence of this uncertainty we need to know the possible outcomes of our decision, the possible alternatives and their costs - in the short term and in the long term. Making these types of decisions requires we make?estimates?of all the variables involved in the decision-making process.

What Are Probabilities??

There is a trend in the software development domain to redefine well-established terms in mathematics, engineering, and science - it suits the needs of those proffering that decisions can't be made in the presence of uncertainty.

Probabilities represent our state of knowledge. They are a statement of how likely we think an event might occur or the possible of a value being within a range of values.

These probabilities are based on uncertainty, and uncertainty comes in two forms. Aleatory and Epistemic.?

• Aleatory?uncertainty is the natural randomness in a process. For discrete variables, the randomness is parameterized by the probability of each possible value. For continuous variables, the randomness is parameterized by the probability density function (pdf).
• Epistemic?uncertainty is the uncertainty in the model of the process. This is due to limited data and knowledge. Alternative models characterize epistemic uncertainty. For discrete random variables, the epistemic uncertainty is modeled by alternative probability distributions. For continuous random variables, the epistemic uncertainty is modeled by alternative probability density functions. In addition, there is epistemic uncertainty in parameters that are not random by having only a single correct (but unknown) value.

Both these uncertainties exist on projects. When making good decisions on projects, we know something about these uncertainties and have?handling?plans for the resulting risk produced by the uncertainties.

• For aleatory uncertainty (irreducible risk), we need?a margin. The?margin?protects the project deliverables from unfavorable cost, schedule, and technical performance that is part of the naturally occurring variances.
• Epistemic uncertainty (reducible risk) can be addressed by?buying down?the uncertainty. Paying money to?learn?more.

This is a primary benefit of Agile Software Development, where?forced short-term deliverables provide information to reduce risk. Agile is Not a risk management process; many other steps are needed. But Agile is a means to reveal risk and take corrective action on much shorter time boundaries - reducing the accumulation of risk.

Some Background on Decision-Making in the Presence of Uncertainty?

One way to distinguish good decisions from bad decisions is to assess the outcomes of those decisions. The measurement critical for a?good?or?bad?decision needs some definition itself. There are issues, of course. The results of the decision may not appear for some time in the future, but we need to know something about the possible results before we make the decision. We'd also like to see the results of the alternatives to our decision for the choices that weren't made or rejected.

A fundamental purpose of quantitative decision-making is distinguishing between good and bad decisions. And to provide criteria for assessing the?goodness?of the decision. To do this, we need first to establish what the decision is about.

• When will we be ready to?go live?with the needed capabilities we're paying you to develop?
• If we switch from our legacy systems to an ERP system, how much will we save over the next 5 years with the project's sunk cost?
• On the list of?desirable?features, which ones can we get on the current?need date?if we reduce the budget by 15%?

Making decisions like these in the presence of uncertainty by estimating future outcomes is a normal, everyday, business process. Any suggestion these decisions can be made without estimates is utter nonsense.

Decision analysis starts with defining a decision - the commitment to resources that is irrevocable only at some cost. If there is no cost associated with making the decision or changing your mind after the decision has been made - in the business domain - the decision could have been of more value. This is the?value at risk?discussion.?How much?are we willing to risk if we don't know to some level of confidence what the outcome of our decision is?

The elements of good decision analysis are [2]. So for any good decision and its decision-making process, we'll need answers to the questions on the left, some form of logic to make a decision, the defined actionable steps from that decision, and then an assessment of the outcomes to inform future decisions -?learning?from our decisions.?

Decision support systems that implement the process above are based in part on the underlying uncertainties of the systems under management. Research into the cost and schedule behaviors of these systems is well-developed. Here's one example.

In the end the decision making process will not meet the needs of the decision makes if we don't have alternatives defined, information at hand - and most times this information is probabilities information from condition in the future in the presence of uncertainty, and the value we assign to the outcomes - then making decisions is going to turn out BAD.

We're driving in the dark with the lights off while spending other people's money, and our project will end up like this...

Reference Material for Further?Understanding

1. Strategic Planning with Critical Success Factors and Future Scenarios: An Integrated Strategic Planning Framework,?Technical Report CMU/SEI-2010-TR-037 ESC-TR-2010-102.
2. Decision Analysis for the Professional 4th Edition, Peter McNamee and John Celina,?Smartorg Inc; Fourth edition (October 1, 2001)
3. Real Options: Managing Strategic Investment?in an Uncertain World, Amran, Martha, and Nalin Kulatilaka,??Harvard Business School Press, 1999.?
4. Making Hard Decisions: An Introduction to Decision Analysis, Robert?Clemen,??Duxbury Press, 1996.
5. Software Design as an Investment Activity: A Real Options Perspective, Kevin Sullivan and Prasad Chalasani?
6. Probabilistic Modeling as an Exploratory Decision Making Tool, Martin Pergler and Andrew Freeman, McKinsey & Company, Number 6, September 2008
7. Value at Risk for IS/IT Project and Portfolio?Appraisal and Risk Management, Stefan Koch, Department of Information Business, Vienna University of Economics and BA, Austria.