Measuring Begins with Why

One of the central pillars of being Agile is that we base our work on empiricism, which means we take a scientific approach to evolving our products and process, and if we take empiricism seriously we collect data and measure against this data to test our product and process hypotheses and theories. Given that we should absolutely take metrics seriously as part of our adoption of Agile ways of working. However, before we start making judgement calls we actually need to ask ourselves are we using the correct data to assess the theory or hypothesis we have come up with.

An example of this that, unfortunately, came up for me recently was a manager using velocity as the sole indicator of team performance. This anti-pattern to good practice is one of the key attributes of “Dark Scrum” (i.e. Evil Agile), basing the assessment of a team’s performance on velocity. Why is this a problem? To begin with, because velocity is subjective then the team can simply game the system by inflating estimated story points. Voila! If you look at velocity then the Evil Agile Team is now more productive.

That’s the extreme case though. I’d say that most teams are honest and do try to give estimates that are accurate, but even good estimates are going to be inaccurate because knowledge work like software development is not concrete or simple. So velocity, while useful as a way for a team to address capacity to address complexity in a sprint, is not a good indicator on performance. I think of velocity much like a health metric like BMI (Body Mass Index, a measure of weight to height). Its usefulness is contextual and it really only serves to open a conversation about what might be going within the entire process.

Here's a little story about BMI and my younger self to illustrate my point. When I was in the Marines I used to exercise constantly. I packed on some serious muscle. At that time, my BMI was 34 which is considered ‘obese’. I could easily run a 5-minute mile, bench press twice my body weight and had a 29” waist. Obese I was not, despite what my BMI might say. Man, do I miss those days. My BMI today is the same; however, I am now a 49-year-old desk jockey with a much softer midsection who would probably have a major coronary event trying to do what my young Marine self did. So, is BMI a good measurement of overall health? It depends, and you can only answer that by digging deeper and asking questions. As I mentioned, the usefulness is contextual.

So then, what is a good metric to measure how well a team performs? I myself would say that a good team has a set of KPIs or OKRs that they can look at that point to product fit/usability outcomes, and they should also have numbers about post-production defects found in the software. Why? Because value is the intersection of product fit and technical quality. What good is a high velocity if you’re building poor quality crap or a product that people don’t want to use?

But suppose your team is actually in that sweet spot of delivering valuable product to your customers? I’d say have a look at their velocity and ask, “Is there anything that can be done to accelerate your delivery of value?” That may lead to some interesting questions about how to smooth out your CI/CD pipeline or improvements that could be made to reduce overhead around testing. In that context, speaking to them about velocity is positive.

In summary, metrics are absolutely part of good Agile practice, but we need to make sure that the metrics we use are the right ones and are being used properly. If you’re being asked to measure something be sure to use your “Five Whys” to better understand if the metric aligns with the hypothesis you’re testing.

If the metrics you are looking at aren't useful in optimizing your strategy - stop looking at them. - Mark Twain