ODIM :A framework for choosing the right metrics

What are “lamp post metrics”?

If you lose your keys between the restaurant and the car on a dark night, when you go back to look for them, where are you going to look? Of course, you are going to first look where there is light -- under the lamp posts. When you pick a metric because it's easy to gather, that's called a lamp post metric.

Why are they bad?

Let's look at an example.

A few years ago, someone noticed that when Carmelo "Melo" Anthony or Monta Ellis were out sick or injured, their teams had a better chance of winning the game. This was surprising because both players had top ten in the NBA points/game stats for the two years of the analysis. However, points per game is a function of two other metrics: 1) how often you take a shot; and 2) the percentage likelihood that each shot will score. It turns out that these two players had high points per game stats simply because they took more shots. If we were talking about hockey rather than basketball, this would be a good thing. In hockey, good hockey players create more opportunities to take shots but in basketball your team gets a shot essentially every time you go down the court. So, Melo and Ellis were literally stealing shots from other players on the team. That wouldn't be so bad if they had the highest chance of sinking each shot but that's not the case. They both had a lower chance of scoring each shot than the weighted mean of their teammates on the court.

Points per game is a lamp post metric. It's easy to gather and understand. We reward these players literally with fame and fortune based upon this metric. However, what really matters is winning games.

What does this have to do with ODIM?

We do this sort of thing all the time in business. We drive behavior with metrics when those metrics are not good proxies for the desired outcomes. That's why I introduced the ODIM framework -- to help teams and organizations pick metrics that will drive the desired outcomes.

The effect of measurement goes like this:

• Effective measurement and visualization provide better insight.
• Better insight leads to better decisions.
• Better decisions lead to better outcomes.

However, as our NBA example above illustrates, when you are making metrics decisions, it's best to begin with the end in mind. By that, I mean, you should start with the Outcomes you are a trying to achieve and work your way backwards to the Decisions that drive those outcomes, then the Insights that will help you get there, and finally the Metrics and visualization that will give you that insight. That's why the acronym is ODIM rather than MIDO.

Back to the NBA example

We agree that the desired Outcome is winning games. The Decision we want to make is the decision that players make many times a game -- whether to take the shot or pass the ball. The Insight that would best inform this decision are things like, "From this point on the court, when I'm up against a player that is bigger/smaller than me, what's my chance of scoring?" and "What are the chances of the other players from where they are standing or where they could move to?". So, you'd need Measurements (metrics/stats/visualizations) that represent this. From that, you'll look for patterns that can easily be used at game speed.

A simple version of this was used by a number of NCAA teams and those teams that used this went much further on average in the NCAA tournament than they were predicted. Their analysis showed that it was bad to take shots at middle distances from about the distance of the free-throw line to the 3-point line. These shots had a lower chance of going in than ones closer in and they were worth less than shots taken from behind the 3-point line. These NCAA teams coached their players to not take shots from this middle-distance and in looking at the heat map of shots, it's clear they achieved this.

Not all insights are so heuristically simple. You may need more nuanced coaching and it'll take time to change game-time behaviors. One-on-one coaching using retrospective analysis of each shot a player took in the last game where the coaching is informed by these stats will go a long way toward achieving this.

Agile Team Example

Let’s say your team’s desired Outcome is to ship more value, sooner. Each day, maybe several times a day, they make the Decision to either take on more tasks or to help finish the ones that their team has already started. If they have the Insight that lower work in progress (WiP) directly correlates with faster time to market, then the Measurement of the team's current WiP will give them the needed insight to make the best decision between pulling in something new to work on versus doing something to finish work already started... even if that something means swarming on work with other team members.

Join me for a workshop on ODIM... and more

Seats are filling up fast for my workshop, Around the World of Agile Metrics in One Day, on October 17 in downtown Raleigh, NC. If you found this post on ODIM useful, it's only one of many similar pieces of content we explore during the day.

Sign up today