Four cars – The story of Agile system metrics

Personally I find it hard to deny that Agile has crossed the chasm, especially when one considers the new software product delivery domain. It does not seem to be as revolutionary as it once has been, the chances are that when talking to your top management you will no longer get the extra points just by laying out this paradigm shifting philosophy in front of them. You may hear: “Yeah I’ve read the HBR article, it all seems like a great idea, but how do I know that it’s working in our organisation?”.

“Well Played” one might think. With each passing year we as an Agile community will get more and more of those questions, so how are we to measure the success of our Agile operations?

Being asked this question by my CTO a couple of years ago I embarked on a journey of research, experimentation, failures and epiphanies, let me take you there …

The Velocity Traps

“Burn down chart” the one thing that everyone consistently pointed me to when I ask the candidates for a Scrum Master’s position how they would measure an Agile product delivery system (I probably lost them at “system”). It hurts me to say that a year ago my answer would have been the same. It’s so evident; the velocity has always been a part of the process, and that is the reason why it has been invented in the first place right? Unfortunately, the world is a bit more complicated than a simple graph on the wall.

In my opinion, if you start measuring the burn down it will probably go in one of the two directions:

• a lovely burn down chart,
• or a demotivated team that feels that they cannot make it pretty.

How proud I was installing a RaspberryPI on flat screens across the office with wallboards showing the delivery details among which there has been the dreaded burndown chart. There was nowhere to escape; it was always there looming mercilessly on the walls and as days went by the red line of progress lagged more and more behind the grey one demonstrating the ideal reality. It was not only the teams that have been exposed to it; it was hard to miss by anyone and when it started to flat line a number of dismissive shrugs and eyes rolled always took up sharply.

I have to say that when the “information radiators” got introduced some teams actually took it to their hearts and tried to follow the guideline, in fact, it helped them to build some productivity habits and overall they delivered more business value in their sprints. But now I think that those graphs have just been an impulse of change and not necessarily a useful metric in their own right.

I took them off the walls quite quickly, a bit ashamed, thinking to myself: this is how one learns.

The Lead Time

“At the end of the delivery is the important part”, “screw the burndown” this was my next attitude. At that moment I thought I found the one metric to rule them all: PiLT(js) – The Portfolio Item Lead Time in the function of the Job size (tip: if it’s that complicated you probably missed the point).

In human terms, it’s the number of working days from the decision to commit to the delivery of a Portfolio item (Product, Technical enabler, etc.) until the moment, it starts to deliver its first benefits. We used the typical T-Shirt metric to measure the size. You also probably may want to have the cost going hand in hand with this but let’s not go into the details right now.

As we have been working in an environment where we predominantly delivered new products this metric made a lot of sense though for other environments where you are adding features to the existing products you may want to look at the frequency of releases, after all, if you simplify enough: F=1/T.

At that time, I thought I found the holy grail of the Agile metrics. It worked wonders, especially in confirming the one thing we already knew: it took us too long to deliver anything. “150 Days for a Medium size product?” no one could believe that it took so long, after all, the delivery has been hailed a success.

For all its flaws the metric was definitely an eye opener and helped me a lot in driving the system change when confronting senior stakeholders with the actual data.

So, my recommendation is that if you want to prove that something needs a change, just pull the data. It’s hard for anyone to dispute the power of a calendar especially if you can dangle some improvements in front of them.

But having said that we all know those products that have been rushed market only to leave scars on their brands for years to come. In my experience, the new product delivery rarely is only about the time (and definitely not only about the cost).

I thought that there has to be a more sophisticated approach to measuring an Agile delivery process, and then at a Meetup, Ben Hughe from McKinsey & Company inspired me with a metaphor that I used as a basis of my system.

The four cars

 The muscle car

Have you ever driven a Plymouth Hemi’ Cuda? I have always been a fan of classic cars, and it would seem that the American muscle cars have been built for one thing only: impressive velocity on a straight line.

Petrol was cheap back in the day, and those cars definitely consumed more than their fair share of it, not to mention that it’s not necessarily easy to fit such a monster into a tight corner especially when you managed to gain some momentum with this front engine beast.

You can also shape an Agile team in a similar fashion, set their sights on the goal at the end of a drag strip, fire the starter and keep everyone focused on the stopwatch.

Anytime someone wants to use velocity as a primary (and often only metric) I use this very example.

It’s not that the velocity doesn’t matter, but as some of us do benefit from a sense of urgency, it’s not the case for everyone. And where is the agility when your focus is primary on velocity? Do you actually have the opportunity to inspect and adapt in the never-ending race to deliver the aggressive product roadmap?

The reliable sedan

Some people prefer to buy cars that would not attract unnecessary attention. We can imagine a fleet manager buying a new set of cars for his sales force. You can probably guess what his priorities would be. Economic, reliable and probably quite boring. As long as you can service them after a decent of mileage, you will get the predictability of costs. And predictability is the keyword for these kinds of cars and stakeholders.

A lot of people love predictability; it gives them an illusion of control. Some really liked the good old method of delivering software with the critical paths and waterfall milestones. They resisted this Agile “thing” as much as they could but even they had to yield at some point. They still cling to any notion of control that they can get however and so if they don’t understand how the product delivery teams work they will repeat the word “predictability” flexing it in so many different ways just to get their basic C&C itches scratched.

And again sure, some things can be predictable, Christmas usually happens late December, and the dates of sporting events are generally set well in advance. If your product targets those you may find yourself with a sharp increase in the cost of delay, not to mention the regulatory compliance changes that some of us have to deal with.

There is value in measuring predictability but as with velocity sometimes the responsible adult in the room should stand up and ask “what is the point of delivering a product for Christmas that no one would want to buy and use?”

The 4×4

I do appreciate a good hike, the quiet of nature surrounding gives me some much-needed respite from my day to day in software product delivery. I don’t really understand people taking those cumbersome and loud machines to the wild, but then again in some parts of the world, it’s just a form of transport. The cars like that aren’t always pretty, comfortable, economic nor fast, but they are flexible, and it’s great to be able to drive pretty much anywhere.

You can also have an ultra-flexible Agile team, a cross-functional multi-component feature team, the epiphany of flexibility. Is it the epiphany of agility however? Some would say it is but aren’t we suppose to value the items on the right of the manifesto, even just a bit? Not to go too deep into this subject but I think that there is one takeaway here: as with the velocity and predictability the flexibility of a team is something that you may want to look into measuring, but in my opinion, it shouldn’t be the only metric to use.

The Hippie Van

No, I don’t claim to be that old. The most use I saw from a WV Camper was the mystery machine in Scooby Doo but I do have a vivid imagination. In a Hippie Van the travel is not about the destination nor the journey, it’s all about the experience of happiness and sustainability.

Who said that work couldn’t be fun? Some even say that it is supposed to be if one wants to get the passionate people creating great products. I think that it’s really is a strange strategy to invest time and effort in building an Agile team, and then instead of nourishing them place it in an unsustainable environment.

Obviously, the whole business has to be sustainable and thus we probably would like to steer away from local optimisations of focusing solely on team happiness. Still, I do think it’s worthwhile considering the happiness and sustainability as elements of your metrics.

Conclusion

Velocity, Predictability, Flexibility and Happiness. How do you think your team(s) are doing in these areas?

Of course, it takes some considerable effort to turn a few slogans into a set of practical, self-balancing and gaming proof metrics but then again Rome wasn’t built in a day.

One of the simplest things you can do is to explain the analogies to the team(s) and ask them to rate the areas on a simple 1-10 scale. It’s a handy tool for a team coaching or a retrospective session.

I don’t claim that those are the best metrics there are, but I have to say that the analogy is quite compelling, and I had some success in using it.

After all, it does not matter if you are a Junior quality engineer or a CTO, it’s hard not to turn your head when a fancy sports car is passing by.

Oh, and by the way, my metric seeking journey is not over yet … stay tuned.

This post has originally been published at Adventures With Agile