Why we measure (and punish) speeding...

I was thinking the other day about speeding fines, and what they are for (apart from, obviously, revenue generation). Presumably, the reasoning is that, if people drive slower, there will be fewer accidents, so things will be safer on the roads. Is this true, though? Is it the best way to improve safety?

Well, no - it is easy to stick to speed limits but be be much more dangerous than if you were driving over the speed limit (you can probably tell I cycle in London in the rush hour...); driving too close, using the phone, watching a movie at the same time as driving, falling asleep, being drunk - the list is endless. Conversely, you can drive over the speed limit, but in a very low-risk/safe manner; leaving extra distance from the vehicle in front, for instance, or only driving over the speed limit when the roads are empty. Even, controversially, driving faster if you've had your reflexes measured, and your reaction times are faster than everyone else... So why do we just measure speed? It comes down to two things - it's objective (in the eyes of the law), and, more importantly - it's easy to measure, be it from fixed cameras, speed guns, in-vehicle logging, or any of a raft of potential measurement methods.

But, if we really wanted to make the roads safer, and there were no legal or cost constrains, what would we do? Well, for a start, we'd do what Google are doing and invent computer-controlled cars to remove driver variability and get a step change in safety. If we keep measuring speed in this scenario, though, we'd probably find that the speed limit was being broken more often (unless the cars' speeds were limited), even though safety improved. Indeed, if we kept punishing speeding (non-)drivers, we probably wouldn't get the full benefits of the technology such as reduced congestion, quicker journey times, etc. etc.

So, it turns out that measuring speed was not the best way to achieve the goal of improved safety, but it'll "do for now" as it is easy to measure and the alternative is expensive and requires a lot of effort; much more difficult to define a "safe driving" metric than just point a speed gun at someone.

It occurred to me that we often measure things purely because they are easy to measure, whereas the question we want answering is not answered by the particular metric we are measuring. Such "vanity metrics" might give us a warm feeling that we are measuring something useful, but if they are not measuring what we actually want to know, they may not help at all. Indeed, they may well be downright damaging as they will stop us looking for ways to solve the problem we have rather than looking at ways to optimise a metric that is not measuring our goal.

A very real problem with these vanity metrics are that they can give a false sense of security that things are going well when, in fact, measuring the right thing would not tell this story. As an example, consider cycle time in software development - basically a measure of how long it takes to build something. Putting all our efforts in getting this as low as possible is good right? Well, no, not really. If it takes a year for an idea to reach the development teams, it doesn't really matter if they take 3 days or 3 weeks to build it - it's still taken over a year between idea formulation and the customers seeing it realised. What we should look at is the time from the idea being formed to the customers being able to use it (or, more importantly, the time they can give us feedback...)

Now, some of you may well have already worked out where I'm going with this. What is the single most commonly used metric in Scrum teams? Yes (obviously), good old velocity. It's certainly easy to measure, and it does give us a measure of something, but what, exactly? Imagine 2 scenarios:

1. Team A, who have a velocity over time that has steadily increased, but doesn't change that much from sprint to sprint.
2. Team B, who never hit their "target velocity" (often, incorrectly, synonymous with "sprint goal"). and their velocity varies wildly between sprints.

Usually, the question asked in this scenario is "which is the better team?" and pretty much everyone (particularly in management), eithout hesitation would reply "Team A". My question, however, is this:

Which team delivers the most customer value?

From the two statements above, it is impossible to tell. In my experience, I know which team would typically be held up as the better or "high performing" team, and I know which one would get chastised (a kind of "negative speeding ticket" for going "too slow"?...). It is entirely possible that Team B could be delivering more value, more quickly, to the customer, so measuring the easy thing would be counter-productive, and not answering what we really care about. Interestingly, when I discuss this with some people, they seem unwilling to believe such "Team B"s could deliver more value in the real world! Surely Team A would always be "better". Well, believe it - I've seen it, and it's surprisingly common, indeed, based on my experience, I would venture it is more likely for Team B to be higher performing because they are focussing on what matters and not a vanity metric that can be easily gamed.

As it happens, given the above scenario, I would be extremely suspicious of team A. Can a team really have a stable velocity? Let's think what this means for a moment:

• It means they do the same amount of work - technically, the same amount of complexity (whatever that means) - each sprint, regardless of the scope of the sprint goal, illness, training courses, holidays, team members joining and leaving, environments going down, power cuts, floods, etc...
• It means the level of uncertainty of the work they are doing is always the same; i.e. there is no uncertainty; if not, their estimates would surely vary. (If there is really no uncertainty, why even do Scrum?...)
• It means they do no experiments that fail. Well, unless, they can estimate the success of those as well. Rather unlikely?

All in all, we're only measuring how accurately a team can estimate, or, more accurately, how good they are at gaming estimates, and that is of no interest whatsoever to customers. So what should we measure? Well, it's difficult - how do we define that elusive value measure? It's probably hard, and it's probably context-specific. It could be costly; maybe we need send people out to do interviews with customers and it'll be expensive to interpret the results. Maybe we need to invest in infrastructure to allow us to do A/B testing and market segmentation. Maybe we need to write a whole bunch of code to add tracking to our products and apps. Maybe we need to set up alpha and beta testing strategies. Whatever it is, I suspect it will be harder and more expensive than simply measuring velocity.

But - that should not stop us doing it. Let's measure something that answers the question we are really interested in - if we measure velocity we are reinforcing team behaviours that merely optimise that, whatever our actual goal.

As someone once said (and please let me know the original source if you can):

Some people use metrics like a drunk leaning on a lamp post; for support rather than illumination

Now, some may argue that, even given the above, velocity isn't that dangerous - at least it's measuring something, but, in actual fact, it is highly damaging and reinforces the exact opposite of what we are interested in! Why? because it is forcing the team to get stuff out the door as quickly as possible. Any old stuff - useful or not! And, it is forcing them to prioritise their own stuff over other teams' stuff to meet their velocity, even if collaborating with that other team would be better for the business as a whole and customers.

Further, blindly focussing on velocity stifles innovation; risk-taking and experimentation, by definition, are leaps into the unknown, so it would be a remarkable coincidence if they could be correctly estimated, every time, in advance. I would argue that, if they can be, we are not really pushing our experimental boundaries far enough. If we use the analogy of a high jumper, they only know how high they can jump when they are knocking the bar off most of the time. Similarly, if all our experiments succeed, and take exactly as long as we thought they would, we're not really experimenting, we're just doing "safe stuff" that we know will succeed. And that we can estimate.

Velocity, for me, even goes against the original spirit of Scrum - getting working (or at least potentially shippable) software out every Sprint. Velocity, in my experience, only measures how many stories are "Done" which is typically (in my experience) much less strict than that; how many teams DoD says "the story is live in production, and available for the customer to use"? (Hiding stuff behind a feature flag is not "live", IMHO...) How often would acceptance criteria include things like "We need to find out something concrete about X?" rather "X has been delivered". How many of those "team As" are delivering value to the customer - that the customer most wants - every sprint?

Scaling frameworks can make things even worse - SAFe, for example, has story points as a pretty fundamental bedrock that everything sits on. Story points are used to estimate work in a PI, for tracking against that estimate and so on. However, if (as argued above) a stable velocity is a myth, the whole edifice comes tumbling down; we cannot have any kind of objectivity, particular in tracking and measuring success, if it is based on a purely subjective measure.

Seriously, measure the right thing - however difficult; don't fall into the "velocity as a safety blanket" trap. Don't assume "it's better to measure something rather than nothing".

It will only drive the wrong behaviours.