Closing the Loop on Estimates & Cost

A quick study of the above diagram is probably the best continuing education course that project managers, contracts managers, and business owners will get this year. It isn't glamorous - its mundane and details oriented work. It lacks the flash of AI or the buzz of "innovative" - rather it is sound 'meat and potatoes' project management (or business or contract) common sense.

The upshot of the process illustrated is that estimating costs happens throughout the life of the project (no news there) as the scope and or requirements are more carefully elaborated, as estimates turn to reality, and uncertainties become fact. This is all everyday PM life in the trenches.

What's missing is the fact that this knowledge and a careful analysis of what changed, by how much. and why is not immediately apparent. It is (or should be) there as we ask why the initial estimate was wrong and why we didn't think about that factor sooner or in more detail. We call this 'lessons learned' and it is an important part of project management.

More importantly is the project management methodology that enables the analysis throughout the project lifecycle. That methodology - aligned with the Earned Value Management technique or process - depends on and is enhanced by a common organizational structure and data consistency.

When I say data consistency I am saying that the way data is organized and expressed at each phase of the project must present those data in the same format, units, and structure as used in the preceding and subsequent phases. This is an area where I perceive improvement can be made.

Using a large Federal Contract as a model the process starts with a cost proposal expressed using a common structure described in the Federal Acquisition Regulation in FAR 15.408 Table 15-2. This format separates the various elements of cost into direct and indirect costs and includes detail on direct labor, direct material, Overhead, G&A, and so on. But the level of detail is what concerns me.

To be effective as a management tool the data in Table 15-2 should be subdivided or organized according to some other structure - be it by the Contract Line Item Number (CLIN) or the Work Breakdown Structure (WBS) or both. Secondly these data need to be 'time phased' or organized by time period - yearly or monthly (often both).

Such an approach provides deep insight into the plan for accomplishing the work and the projected spending and revenue associated with the planed activities. But while the intent is good - the execution is often flawed. Here's why.

Failure to adopt a standard data format

As I said earlier, there is a lifecycle to the estimate, budget, accounting project management process. And with each phase of the project from planning to execution to close-out different tools are used to capture, record and analyze the project data.

During the planning and pre-award process it's common to see tools like MS Project or other scheduling tools used to plan and estimate work requirements. These tools deal with units of work (labor hours), time (days, weeks, months) and structure (WBS, tasks, milestones). While highly granular if used properly their data does not meet the rigor of the Truthful Negotiation Act (formerly TINA) without some added elements (indirect rates etc.) so these data while explain the basis of estimate do not rise to the level of certifiable cost or pricing data.

What I do (and hopefully many others as well) is import the Project file into a pricing tool such as ProPricer and apply the finishing touches to the planned activity. The advantage of the pricing tool is it organizes the imported data into the Table 15-2 format easily and, if my project schedule has a WBS structure (it should) then that structure is preserved. At some point I also add data to cross reference WBS elements and tasks to their respective CLINs.

At this point I have a detailed cost estimate structured by both WBS and CLIN that is time phased thanks to the project file and my editing. As the proposal continues on the estimate can be refined in Project - perhaps as planning packages are detailed or requirements are better defined.

There comes a point when the planning (proposed) data is negotiated and a contract is awarded. Here is where things start to go off the tracks. Once a contract is awarded the cost and pricing team usually has a celebration of the successful proposal and then goes on to the next proposal. The detailed data they produced may or may not be passed along to the accounting system and even if it is it may be at a summary level.

The next step in the process is usually the 'carving of the turkey' by the program manager. The total contract award value is stripped of the indirect costs and profit leaving direct labor and direct materials cost data. From this remainder a portion is set aside as 'management reserve' and the remainder distributed as project budget to cost account managers(CAMs). And here start to run into trouble.

CAMs hopefully (but not always) use the previous MS project schedule and resource loading as a starting plan - but then the inevitable changes occur. Also, the enterprise accounting system may not establish a 1 to 1 relationship between the CAM's budget, the WBS structure, and the CLIN structure. This makes comparing performance data vs. the planned data nearly impossible.

Secondly, the accounting data is often summarized (i.e., 'rolled up') such that multiple WBS elements end up with their costs being combined in a single account - or dispersed across multiple departments - or both. This again makes meaningful comparisons of actual cost to planned cost difficult if not impossible to do at any significant level of detail.

Even the vaunted Earned Value Management System used by many Federal contractors for large projects is deficient when it comes to backtracking to the proposal. EVMS tracks to a performance baseline - and unless there is a requirement for that baseline to have a clear and easily traceable relationship to the contract cost data submitted by the offeror its value is greatly reduced.

Failure to Maintain Version Control

Change is constant in a project environment. Military planners say a plan seldom survives the initial contact with the enemy - and for good reason. In project (and contract) management the plan is what we hope to achieve to fulfill our contractual obligations. Changes to that plan must be made with consideration to cost, schedule, and contract.

While many organizations have and follow a strict change control process, I perceive a problem whereby the change is not adequately captured in the documentation such that baselines are established and changes are properly annotated such that when a change is made it is readily apparent what has changed and the documentation provides an auditable trail back to the original proposal.

Frequently the changes are made and the plan is updated - but it is done through a process where the baseline is reset in such a manner that the actual changes are hard to discern and missteps or errors are hidden in the revised baseline. This strategy may provide an escape from accountability but it hinders any forensic analysis that may provide valuable lessons learned.

The need for Lessons Learned Analysis

Project data - especially when that data is consistent from inception to completion - provides a valuable input into a post project lessons learned analysis. Direct comparison of plan vs actual with respect to cost and schedule will provide a basis for discussing the why such variances occurred and the insights needed to refine future estimates making them more robust.

The lessons learned should examine the assumptions made and how they played out. The review should also include a grooming of the data to isolate the effects of risks that came into play.

Leveraging the Data

One use for the data - if we can obtain it in a consistent format - is to support informed judgements with respect to uncertainty when planning future projects. While it seems obvious that an analysis of these data would provide insights into the effects of uncertainty on future projects the reality is such data are hard to find and regression analysis associating underrun or overrun amounts with specific attributes like project complexity, technical complexity, experience with similar projects, or any number of uncertainty elements is seriously lacking.

This is an area where I have definite interests and feel research is needed so as to better model project performance expectations based on historical data. While many try to do this through three point estimating, PERT, and Monet Carlo the concern is that lacking detailed lessons learned analysis and consistent cradle to grave cost data such efforts amount to little more than educated guesses analogous to measuring with a micrometer and then cutting with an axe.

Moving Forward

I suggest the first step in the process is integrating the cost and pricing team, their data, and their tools into the project management Integrated Project Team. Project controls should be measuring to the same baseline as the cost and pricing team laid out in the proposal ... apples to apples. This means the proposed costs - once negotiated and updated in the cost model - should be imported directly into the budget and into the accounting system with the pertinent links and relationships intact. This will not only reduce effort - but will ensure data consistency.

Secondly, assumptions concerning uncertainty and risk should be included in the analysis database. But should the baseline reflect contingency or allowances? That is debatable. Contingencies are not allowable costs under FAR Part 31 but allowances are. I contend that a performance baseline using the median values obtained in a PERT analysis would offer certain analytical benefits post execution although it could also create a management impression that the program was performing poorly in highly uncertain projects. On cost type contracts I suggest this is the correct method for tracking performance. On Fixed Price Contracts I suggest that if a contingency has been factored into the cost of the work it be included in the baseline with documentation explains the amount of the contingency and the assumptions and methods used in its estimation.

Thirdly, I recommend each major WBS level be assessed for specific attributes and these be documented pre-execution. Categories should include Project Complexity (size, value, number of stakeholders, number of subcontractors, number of deliverables, etc.) Technical (maturity of technology, experience with the technology, design complexity, etc.) Requirements/Design Definition (i.e., design maturity, number of requirements, percent designed, etc.) and Organizational (defined processes, maturity models, etc.). These attributes will hopefully support regression analysis to correlate their effects on project performance.

This is an area of research I am keenly interested in exploring - If you are interested in sponsoring or collaborating please let me know.

Summary

Closed loop systems as described above claim certain advantages to open loop (i.e., no correcting feedback is present) and have the potential to better define the effects of uncertainty and risk on project execution. To be effective the project data needs to be consistent in structure and units throughout the project. The principal way to accomplish this objective is to integrate the cost estimating group into the project IPT and ensure each revision of the plan is referenced to the initial cost estimate and all changes are tradable to the underlying WBS and CLIN structure.