Capability Maturity Levels and Implications on Software Estimating

An estimate is the most knowledgeable statement you can make at a particular point in time regarding cost/effort, schedule, staffing, risk, and the?...ilities?of the product or service.[1]

Immature versus Mature Software Organizations [3]

Setting sensible goals for improving the software development processes requires understanding the difference between immature and mature organizations. In an immature organization, processes are generally improvised by practitioners and their management during the course of the project. Even if a process has been specified, it needs to be rigorously followed and enforced.

Immature organizations are reactionary, with managers focused on solving immediate crises. Schedules and budgets are routinely exceeded because they are based on something other than realistic estimates. When hard deadlines are imposed, product functionality and quality are often compromised to meet the schedule.

In immature organizations, there is no objective basis for judging product quality or for solving product or process problems. The result is that product quality is difficult to predict. Activities intended to enhance quality, such as reviews and testing, are often curtailed or eliminated when projects fall behind schedule.

Mature organizations possess the organization-wide ability to manage development and maintenance processes. The process is accurately communicated to existing and new staff, and work activities are carried out according to the planned process. The processes mandated are usable and consistent with the way the work actually gets done. These defined processes are updated when necessary, and improvements are developed through controlled pilot tests and/or cost-benefit analyses. Roles and responsibilities within the defined process are clear throughout the project and across the organization.

Let's look at the landscape of maturity in estimating the work for those providing the funding for the work.

1. Initial

Projects are small, short, and while important to the customer, not likely critical to the business's success in terms of cost and schedule.?

• Informal or no estimating
• When there are estimates, they are manual, without any processes, and likely considered?guesses.

The result of this level of maturity could be better forecasting of the cost and schedule of the planned work. And surprise for those paying for the work.

2. Managed

Projects may be small, short, and important. Some form of estimating, either from experience or from the decomposition of the planned work, is used to make linear projects of future cost, schedule, and technical performance.

This past performance is usually not adjusted for the variances of the past, just an average. Also, the linear average usually doesn't consider changes in the demand for work, technical differences in the work, and other uncertainties in the future for that work.

This is the?Flaw of Averages approach to estimating. As well, the effort needed to decompose the work into?same-sized chunks is the basis of all good estimating processes. In the Space and Defense business, the 44-day rule is used to bound the duration of work. This answers the question:?how long are you willing to wait before you find out you're late??For us, the answer is?no more than one accounting period. In practice, project status -?physical percent complete is done every Thursday afternoon.

3. Defined

There is an estimating process using recorded past performance and the statistical adjustments of that past performance. Reference Classes are used to model future performance from the past. Parametric estimates can be used with those reference classes or other estimating processes. Function Points is common in enterprise IT projects where interfaces to legacy systems, database topology, user interfaces, and transactions are the basis of the business processes.?

The notion that?we've never done this before so how can we estimate, begs the question:?do you have the right development team? This is a past performance issue. Why hire a team with no understanding of the problem and then ask them to estimate the cost of the solution? You would only hire a plumber to install a water system if she had done this before somehow.

4. Quantitatively Managed

Measures, Metrics, Architecture assessments - Design Structure Matrix is one we use - are used to construct a model of the future. External Databases are referenced to compare internal estimates with external experiences.

5. Optimized?

There is an?estimating?organization that supports development, starting with the proposal and continuing through project close out. There is also a risk management organization helping inform the estimates about possible undesirable outcomes in the future.

Real-Life Sources of Empirical Data for Project Estimates

In the #NoEstimates conversation, the term?empirical?data is used as a substitute for Not Estimating. This notion of?No Estimates?- that is, making decisions (about the future) with No Estimates, is oxymoronic since gathering data and making decisions about the future from empirical data is actually estimating.

But that aside, for now, the examples in the No Estimates community of empirical data need to be revised for any credible decision-making. Using 22 ?or so data samples with a ±30 variance to forecast future outcomes when spending other people's money doesn't pass the?smell test where I work.?

Here are some sources of actual data for IT projects that can be used to build?Reference Classes that have better statistics.

• Nederlandse Software Metrieken Association
• International Software Benchmarking Standards Group
• Common Software Measurement International Consortium?

Several professional societies guide estimating

• INFORMS
• ICEAA

There are two I participate in.

I have a colleague, Mario Vanhoucke , who speaks at our Earned Value Management conferences and whose graduate studies research project performance management. A recent paper, "Construction and Evaluation of Frameworks for Real Life Project Database ," is a good source of how to apply empirical data to making estimates of outcomes in the future. Mario teaches?Economics and Business Administration at Ghent University and founded OR-AS .?

All of this is to say that using empirical data is necessary but not sufficient. Especially when the data being used if too small a sample size, is statistically unstable, or at a minimum, statistically broad variances. To be sufficient, we need a few more things:

• The correlations between the data samples evolve. This is?a Time Series Analysis.
• Sample sizes are sufficient to draw variances in the assessment of future outcomes.
• A broader Reference Class basis than just the small number of samples in the current work stream. These small samples can be useful IF the future work represents the same class of work. This would imply that the project itself is straightforward, has little emergent risk (reducible or irreducible), and we're confident that everything will stay the same. Those assumptions are necessary for the statistics from those 20 or so samples to be used.

What's Next?

Starting with empirical samples to make estimates of future outcomes is called Estimating. Labeling it as No Estimates seems odd at best.?

With the basic understanding that empirical data is needed for any credible estimating process, look further into the principles and practices of probabilistic estimating for project work.?

This, hopefully, will result in an understanding of sample size calculations to determine the confidence in the forecast as a start.?

Resources

[1]?Improving Estimate Maturity for More Successful Projects , SEER/Tracer Presentation, March 2010.

[2] Estimation Best Practices in CMMI V 2.0

[3] International Function Points User Group

[4] CMMI V 2.0 Model At-A-Glance