Deep Thought: Measuring ROI…Every 10 million Years?

Early in this series we determined the need to implement DevOps based on the “Why” and not the “What”. It was key to determine the Why because we could then link our transformation to tangible business benefit. In asking Why, we can plot the How, which in turn creates the What. However, in order to ensure that our transformation is providing a tangible Return On Investment (ROI), we must measure and compare.

The problem, however, is that the measurement is something we may have found lacking previously. In fact, most measures are typically qualitative, rather than quantitative. For many measures, this must be the case, such as customer satisfaction ratings. These are not based on actual figures you can right down, but based upon how a person or group of people “feel”. This is good to an extent, but a business also needs to see real numbers for their ROI. How much are they saving in costs? How much quicker are we implementing solutions? How many fewer defects are we finding that cause a critical business failure?

I turn again to Douglas Adams with a quote from Marvin in Restaurant at the End of the Universe.

“The first ten million years were the worst," said Marvin, "and the second ten million years, they were the worst too. The third ten million years I didn't enjoy at all. After that I went into a bit of a decline.”

Poor Marvin was not having the greatest of times, but in this example he shows the good and the bad of measurement. On the good side, he has at least broken down his measure into a series of baselines for comparison from one to the other, however, on the bad side, when you look at the results of the comparison, they don’t actually tell you if things got better or worse, that is, if the first10 million years were the worst, how can he state that he later went into a decline? There are some fairly basic rules that we must follow to ensure that any metrics we take are meaningful.

First, start measuring from Day 1. It is vital to always start from a baseline to compare to later and that includes from before any work has actually been done.

Second, always link the measurement to a tangible business benefit you are seeking, one of your reasons why. This ensures that the measure is meaningful to the business.

Thirdly, always breakdown your measures into evenly spaced events so that you can compare baselines over a period of time. The gap between events should be large enough to ensure a distinctive gap, but not so large as they don’t provide a quick enough feedback on progress.

Fourth, ensure that the resulting value is clear, not ambiguous. Saying 10 is twice the size of 5 is clearer than saying something got bigger than it did before.

Fifth and finally, always link the resulting decline or improvement to the actions that have occurred over that same period. It is one thing to have a measure, it is another to be able to use that measure to show what you did to improve, or what you need to reinvent because you didn’t.

Measuring is a key element of any DevOps transformation, so getting it right is a must. Choose the right metrics and ensure they are meaningful, and you will be able to prove that you are providing a real business benefit for the money spent on your efforts. You can prove ROI, but only if you measure and compare, and you won't need 10 million years to do it.