Experience Metrics, Rollercoasters and Rugs: What is The Connection?

Some weeks ago I posted about the infamous use of NPS by a famous Brazilian bank. The post became viral and I realized that measuring things is a big challenge, not only for newbies, but also for experienced professionals.

In this article I will discuss 3 concepts that are very important about metrics:

1. Don't overreact to numbers
2. Look for patterns, not events
3. Not all organizations want to see the dirty under the rug

1. Don't overreact to numbers

One important lesson we learn in Black Belt Lean Six Sigma courses is about the statistical control of processes, using Control Charts. These graphics are useful because they show us that the variation of quality is normal. One day you will get 5 stars, in the other you can get 1, then 4, then 5 again, and so on.

When you don't understand about variation we run the risk of overreacting to the normal behavior of a system. When we look to satisfaction rates (like CSAT, CES, SEQ, ACSI, etc) we see that they vary over the time (over the day, week, month, etc). Some managers panic when they see the number of stars falling, others open a bottle of champagne when the numbers arise. This overreaction is what Marc Graban calls "rollercoaster".

So, what should we do, to avoid overreaction? We should use rules for detecting special causes. Imagine that the chart below demonstrates the different number of stars that people give to a service over the time. SD means standard deviation. When the SD varies under specific conditions, they send a signal that something is happening in the service. The opposite of signal is "noise", that is just the normal variation of the process. Managers overreact when they listen to the noise, instead of paying attention to the signal. If you want to understand more about control charts and how they can be used in services' metrics, you will find this good article here.

2. Look for patterns, not events

When we look for isolated events, we can simply get crazy. If we react to every pain point (late delivery, system outage, weather issues, etc) we will fill our backlog with a long list of "improvements", and we can forget the big picture.

Look the example below:

I took a sequence of TripAdvisor evaluations and plotted in a chart.

Everyday some customers rated the service based on their experiences. Five means "excellent", four means "very good", three is regular and so on. As you can see in the next chart, the different ratings form a trend line.

The pink line represents the trend: the quality of the service is decreasing over the time.

But look what happens when we break into segments, using a box plot chart:

If you look to the variation, it is clear that in the first period analysed, the quality varied between 4 and 5 starts mostly, with 2 outliers. But, over the time, the variation increased, and in quality theory, higher standard deviation from the average means bad service. One day, customers are knocking their socks off, in the other day they are yelling unpronounceable words due frustration.

So, when I asked the owner of the company, if something important happened, between the first and the second period, he simply said: "I fired the manager". So, maybe this could be the cause of the problem. Another possible cause could be located earlier, due to the fact that in systems thinking, many times there is a delay between cause and consequence. You can read more about delays in the 5th chapter of the amazing book The Fifth Discipline, of Peter Senge.

In a nutshell, never look to isolated numbers, because they represent events. If you react to events, you are reactive. If you react to patterns, you can be pro-active.

3. Not all organizations want to see the dirty under the rug

Finally, it is important to ask before measuring performance in your organization: Do the managers want to see what is under the rug?

Don't be innocent. Numbers can show what is wrong. More than this, they can show where is the possible cause of the problem. What is the faulty department. That has a manager. That have no interest on being exposed.

In this context, the NPS is a very convenient metric. It is generic enough for not blaming anyone. The truth usually is hidden in thousands of customer complaints that generate lots of "noise". No signal. No real understanding.

If your organization has real interest on detecting what is really happening, it must be open to lift the rug, see the imperfections and take action. Don't fight people. Fight the problems. Usually the behavior of people is just the consequence of a bad process that is affected by bad strategy and bad shareholders.

Conclusion

Be open to the truth, but don't overreact to noise. Listen to the signals, find the patterns and escape from the rollercoaster. Your life will be easier, trust me.

Ricardo Martins, Professor of Service Design Metrics - SCAD USA