Good Metrics, Bad Metrics
“What gets measured gets improved.”
That awesome quote is attributed to Peter Drucker; a man of many skills and one who's wisdom I have an enormous amount of respect for. Like all great ideas though, it's a little too abstract to be of practical use without some extra thought and effort.
Without the extra effort, these ideas will inevitably be abused by people in support of their own agendas. It's not their fault of course, we all do that. In this case, I see more and more that we are starting to miss the point in an attempt to measure everything.
I'm going to say that's a really bad idea for three reasons...
- It's not the point of the exercise.
- It's really expensive.
- It causes us to measure what we have and not what we want.
A much better idea is to triangulate your position against and eliminate the things you don't want and measure your progress towards the outcomes that you do want.
1. It's not the point?
Hell no! But I do like the spirit of the idea. We should strive to be scientific in our approach when it's appropriate to do that but here's the thing: Measurement is not the goal!
Let's say you're learning about lean and you decide you want to measure cycle-time. That's a particularly common thing to start measuring for newcomers. Imagine yourself in six months time, looking back at the figures you've collected. What now? What would you have done differently six months ago if you had those numbers already? If you don't know then what are you doing it for?
But "Why not measure everything you can?", you might say. "It doesn't cost anything. Maybe there will be some hidden truth in there". There wont be, or else you would likely have noticed it already, and it absolutely costs! Much more than you might think.
2. It's Really Expensive
When you make the decision to measure something, what you're really saying is "I need this data, so that I can use it to make better decisions." So next time you make a decision, how many of your metrics are you going to consider? The mental burden alone results in a huge amount of waste.
The lean manufacturing guys call that Muri. The west called it 'Paralysis by Analysis'. We seem to intrinsically understand the cost of measurement but we are so swamped by the availability of data these days, we're scared we'll miss something important. That fear comes from not knowing what you are looking for in the first place.
And so, we continue to add our metrics to this invisible burden. Just in case.
Metrics are inventory.
3. Not Bad, is Not Good
Our minds come pre-programmed to naturally prefer simple, quick and easy things. They are so good at it, that we don't have to consciously act in order to do most of it. We walk, breathe, eat and (sometimes) talk without any real mental effort. The development of that skill meant that we were able to run away from danger without having to think too hard about it so it's a useful thing to have. It's that tendency to take the simpler and easier mental route however, that causes us to measure the least valuable things we can.
The easiest things to measure are the things we want to avoid because we probably already have that data. How useful are those things though, really?
Perhaps we decide to measure 'customer satisfaction' because we've heard that's a good thing. Satisfaction isn't a quantitative measure, making it a tough choice at face value. So instead, we find measurement in the data we already have.
Maybe we decide that to get satisfaction we need fewer bugs, so we'll measure that. Seems reasonable. The fatal mistake here though, is that what we are actually measuring, is customer dissatisfaction. That is not the same thing! It's useful, but not enough. Significant, but it's not meaningful.
The number of bugs you have open is only one contributing factor of customer satisfaction. Moving away from something you want to avoid (dissatisfied customers), does not guarantee you are moving towards your desired outcome for the same reason that moving away from Kensal Green doesn't necessarily get you closer to Soho. That is, unless you're currently in Paddington and also moving away from White City and Camden Town at the same time. Even then, you'd only be approximately right, for a while.
See my point? You need to triangulate your position in the map of bad things.
Triangulation, Elimination and Outcomes.
Ok, so measuring things we want to avoid isn't the worst thing in the world. Even when it's done in isolation but the best it can do for you is describe where you are. So let's talk about how you can use that information in some meaningful way to create the knowledge and wisdom to take you where you want to be.
Often a good way to triangulate is to consider the non-essential properties of the system you are building. In software, we more often call them non-functional requirements. NFRs are the majority contributors to qualitative measures and they are the toughest to work with.
Security, reliability, accessibility and performance are just four things in the large list of things you might want to consider whenever you plan on writing a new piece of software. Add to that a few business constraints like cycle-time/lead-time, cost, number of team dependencies and to make things even more complicated, qualitative properties can work in opposing ways. A more secure system, for example, will be less flexible by necessity. How are you supposed to manage that?
You aren't! You're only reasonable option is elimination. The lean guys call that making policies explicit. In the Scrum world, it's the rationale behind the Definition of Done.
So pick the top five or so NFRs for your product/organisation (add others later if you must but be wary of the costs) and define things which are unacceptable. Unacceptable means you will stop other work when one of these things happens. That might mean your penetration testing revealed a major security flaw. It might mean you can't handle 10,000 concurrent users. The important thing is you define it quantitatively and make the commitment to not tolerate slippage. Embed that into your everyday system of work because anything which requires additional effort, will eventually stop happening.
Now you have anchored yourself. You cannot move in the opposite direction to your end goal. You can still move forwards in the wrong direction but you can't slip backwards. All that is front of you now is choice and opportunity.
To get to the desired state though, you have to be even bolder. You have to say what you actually want. Deciding what you want is the topic of a future post so I won't dwell too much on it here but a good starting point is knowing your 'True North'.
When you do know what it is you're after, be specific and honest. It's okay if you want to double your advertising revenue in the next 12 months. If that's what you're looking for, that's easy to understand and easy to measure progress towards.
A word of warning here...It's easy to slip back into old habits sometimes. If you've got something like stakeholder happiness in mind, you'll regularly see cheap, simple and high quality on the list of things they will want but then we're back to measuring things we don't want. We don't want expensive, complicated, poor quality software.
They will need to be able to articulate the specific goals they have in mind. You will need to listen to them and understand their vision.
Summary
Select metrics carefully, always considering the costs. Apply policies to ensure you can't slip backwards and define metrics that determine if you are moving forwards.
Be very explicit about both and measure nothing else.