What has CMMI? Ever Done for Us?

Quantitative Evidence of the Benefits of CMMI Adoption

One of the most frequently asked questions that you tend to encounter as a consultant working with the Capability Maturity Model Integration? (CMMI?), concerns the benefits that client organizations could realistically be expected to realize. Organizations invest their time, effort and money in adopting this long-established process framework to plan and manage their improvement activities, and understandably want some idea of how they may derive benefit from doing so. It is almost an unanswerable question because, as with most endeavours, the more you put into it, the more benefit you are likely to realize. Yet at the same time, it is obviously not unreasonable for a client to want some credible indication as to the possible return on their investment before making any hard commitments.

ISACA have just released the latest version of their CMMI Performance Report (Full title: “CMMI Technical Report: Performance Results ”) which goes a long way to helping provide an answer to these types of questions.

This document looks at all CMMI appraisals that have been conducted and reported over the period 2019 – 2023 and seeks to identify what benefits organizations are actually achieving against their own stated business objectives through their use of the CMMI. Within a CMMI appraisal, the appraisal team can then determine whether the organization’s choices of how to incorporate CMMI practices into their processes was appropriate in their context and whether or not this helped them progress their objectives. Each identified improvement objective is allocated to a specific category of improvement (e.g. Quality, Productivity etc.) and a more detailed sub-category (e.g. Defect Detection, Schedule performance etc.), and the full appraisal record then records the situation both ‘before’ and ‘after’ the improvement work. From this, an improvement figure for each objective can be calculated.

The performance report aggregates all of this data to identify average improvements for each improvement category to give a picture of where the most benefit is being realized and what the typical magnitude of the improvement is.

Basis for the Report Conclusions

It is important to stress that the Performance Report is underpinned by a sizeable body of data. In the last year alone (i.e. 2023), over 5000 appraisals were reported and the most recent report is based on data from nearly 15,000 appraisals in total over the full period (n.b. The COVID pandemic severely impacted the number of appraisals performed at the start of the report period). These appraisals were undertaken by nearly 14,000 organizations and over 50,000 performance objectives were reported.

Obviously, every organization is different and faces unique challenges of their own, so there are no guarantees as to exactly how much of a difference an individual improvement program will make, but the sample size used in the creation of this report is sizeable and therefore the results can be seen as indicative of what can be achieved within organizations that are using CMMI. Furthermore, all Improvements?are independently validated by CMMI Appraisal teams during the appraisal process so the reported figures are backed up by objective evidence and a degree of independent scrutiny.

So what does the Performance Report tell us?

The first thing to note is the success rate of reported objectives. 86% of all stated improvement objectives were identified as having been ‘Met’ with a further 4% having been marked as ‘Partially Met’.

This is compelling evidence that organizations which use CMMI to set improvement goals have been highly successful in achieving those goals. However, this does not tell us how big these improvements actually are.?For that we have to look at the actual improvement figures.

Where are the biggest improvements being seen?

The Performance Report identifies nearly 100 categories and sub-categories that a performance objective can be assigned to. Of the ones that have been used, the top 5 sub-categories point to impressive average improvements:

These are significant improvements that are being recorded through objectively validated real-world data. And these are just the headline values for the top 5 improvement areas. The main body of the report contains a lot of detail regarding the achieved improvements in all of the main performance categories. As well as average improvements the report includes upper and lower ranges for all popular improvement areas which helps to put these figures into their true context.

But CMMI Is Just for Large Software Organizations isn’t it?

This is a myth that grew up from the early days of CMMI where most of the early adopters tended to be large government contractors. Often these organizations had been actively involved in the development of the initial capability maturity models. So whilst there may have been some basis in fact for this when CMMI was in its infancy, the Performance Report shows clearly that this is certainly not the case today.

In reality, nearly 90% of all appraisals performed over the report period were conducted by organizations with less than 100 FTEs and in fact, nearly 75% of appraisals were performed by organizations with no more than 50 employees. Clearly CMMI is no longer confined purely to large organizations.

Moreover, whilst the initial CMMI sought to unify previously separate Capability Maturity Models for software and systems development, CMMI has now evolved and the latest version defines a total of 8 specialist subject areas where CMMI can be usefully applied. In addition to product development, CMMI now contains best practice information relating to Data Management, People Management, Service Delivery, Supplier Management, Safety, Security and Virtual working. The Performance Report shows data that indicates how the take up of multiple specialist domains is on a steady upward trajectory and the types of organization that are putting CMMI to use continues to expand.

Conclusions

Now in its third main version, CMMI continues to demonstrate that it can give significant benefits to organizations which choose to use it to support their improvement programs. Moreover, these benefits are being reported by organizations of all sizes and across many different industries. The CMMI Performance Report provides evidence of actual benefits achieved through using the CMMI model and shows that the benefits can be substantial and far reaching. With the latest version of CMMI containing 8 specialist domains, the framework can deliver these benefits to a much broader range of industries and organizations.

If you are interested in discussing how CMMI might help your organization, please contact us to arrange an initial consultation. Alternatively visit our website https://excellenceinmeasurement.com/ ? ?

Jonathan Dean (Consultant and Certified CMMI Instructor)

[email protected]

CMMI and Capability Maturity Model Integration are registered marks of ISACA. All rights are hereby recognised and acknowledged. Special thanks to ISACA for permission to use the graphs and artwork from the Performance Report.