Managing in the Presence of Uncertainty - Part 2: Measuring Project Success

Risk is Uncertainty that Matters [1]

Risk, created by uncertainty, and its management, is the central paradigm in a wide range of fields, from engineering to medicine, from life sciences to economics.?

Let's look at the Framing Assumptions for managing in the presence of uncertainty that creates a risk. A Framing Assumption is any supposition (explicit or implicit) that is central in shaping cost, schedule, or performance expectations of the outcomes of a project. [2]

Framing Assumptions have a few attributes: [3]

• Critical - significantly affects the expectations of the project
• No workarounds - the consequences cannot be easily mitigated
• Foundational - not derivative of other assumptions
• Project Specific - not generically applicable to all projects

Framing Assumptions drive key decision-making processes and s a result drive key risk attributes. We assume the design is mature when in fact it is not. We Assume the prototype we built is nearly production-ready and production and development can be concurrent. That some attribute of the article being built (its target weight) is known or knowable. And we can now refine the design to be more affordable.

A critical Framing Assumption when managing in the presence of uncertainty is we can determine the probability of a project successfully delivering capabilities needed to accomplish its mission can be determined with integrated data and processes used by project controls and engineering staff to track, and manage technical and programmatic performance, and risks to that performance.?

Measuring Progress to Plan

Measuring progress to plan in the presence of uncertainty, requires probabilistic modeling of the uncertainties of cost, schedule, and technical performance in units of measure meaningful to the decision-makers. This meaningful information of the status of the project during its development is a critical input to decision-making in the presence of uncertainty. The established principles of this process have been around for decades. It’s time these are put back to work.

In the presence of uncertainty, all projects have several important characteristics:

• The development of new, operational systems involves combining subsystems and component technologies to produce a new system's capability. Early in the project, two types of information are available ? the design basis of the general class of systems and some knowledge about newer or improved technologies to be incorporated into the components or subsystems.
• The status of the development can be a measure of uncertainty and progress can be measured by the change in this uncertainty. Physical progress to plan can be measured by the reduction of uncertainty when it is reducible, and by the assessment of the remaining margin when the uncertainty is irreducible. This process has been described ? long ago ? as the sequential purchase of information regarding some object about which the state of knowledge is incomplete. This knowledge is periodically assessed to confirm risk is being bought down at the planned rate, and margin is being consumed according to plan.
• The system’s variables defining the performance characteristics must indicate an increasing mature capability to fulfill a prescribed mission, in the Concept of Operations or Statement of Work and other documents, in units described in the next section.
• Acquiring these measures requires a method of systematically analyzing the implications of this knowledge at the subsystem and component levels of the project.
• Since the development of complex projects is not a random process, but a directed engineering process in the presence of uncertainty, subjective probability distributions are needed for measuring this technological uncertainty. When an objective measurement is possible, it will be used to inform the decision-maker. When this measure is not directly available, a subjective measure must be made with a model containing a degree of uncertainty.

Measures of Progress to Plan

To assess the increasing or decreasing project probability of success for the characteristics described above, units of measure are needed to define this success.?

• Measures of Effectiveness (MoE) ? are operational measures of the desired capability whose success is related to the achievement of the mission or operational objectives, evaluated in the operational environment, under a specific set of conditions.
• Measures of Performance (MoP) ? are measures that characterize physical or functional attributes relating to the system operation, measured, or estimated under specific conditions.
• Technical Performance Measures (TPM) ? are attributes that determine how well a system or system element is satisfying or expected to satisfy a technical requirement or goal.
• Key Performance Parameters (KPP) ? represent the capabilities and characteristics so significant that failure to meet them can be cause for reevaluation, reassessing, or termination of the project.

Each of these measures provides visibility to Physical Percent Complete of the development or operational items in the project.

These measures of "progress to plan" have been developed and applied since the mid-1960’s

References

[1] Hillson, D. and Simon, P., Practical Risk Management: The ATOM Method, Management Concepts, 2007.

[2] "Identifying Acquisition Framing Assumptions Through Structured Deliberation," Mark V. Arena and Lauren A. Mayer, RAND Corporation, 2014

[3] "Framing Assumption (FA)," Defense Acquisition University, Acquipedia

[1] Framing Assumptions (FAs): Lunch and Learn, March 8, 2017, Brian Schultz, Defense Acquisition University