30 Reasons why Net Promoter Score is Broken

It is not empirically clear that Net Promoter Score (NPS) actually predicts growth or financial return. I won’t bore you with those arguments. My focus is on the reckless, lazy, and thoughtless manner the data is collected.

After taking a work flight, I got asked the classic NPS question:

"Based on your experience today, how likely is it you'd recommend us to your friends or family?”

Asking this question during a flight highlights most of the things that are broken with surveys, and especially Net Promoter Score (NPS). NPS has an important place, but it's not the cheap panacea it's made out to be.

1. Context

All customers have a different context of experience so they can’t be intermingled.

I fly for work, it's tax deductible and I fly a full-service airline. That’s a completely different context of experience compared to a friend or family member who’s flying on a budget. I may recommend the airline no matter how good the service is, so I’d answer zero.

2. Wording

The NPS question generalises all friends and family into the same bucket, thus coming irrelevant.

Are we recommending to friends or family? Are we talking about my rich friends? My friends who work for airlines?? My retired friends? My hermit uncle who never flies? My teenage son? The majority of friends or family Just some of my friends and family?

I’ve seen similar questions asked in specialist Doctors surgeries. I wouldn’t recommend a maternity ward to my mother.

3. Lazy

Often the justification for implementing NPS lazy. Just because something is easy, doesn’t make it right.

It’s easier to drive home drunk from a dinner than to get a cab and pick up the car in the morning. Doesn’t make it right though. 

The irony is that explaining the scoring to staff is very difficult. Sure the data is easy to collect, 

4. Analysis

NPS doesn’t answer “why”.

Giving a score of 0/10 gives no insight as to what happened or what needs to be fixed. You cannot use the data for any operational insights. For a flight, was the score attributable to the food? The seat? The staff? All of it? You have no idea and therefore can’t fix the problem.

NPS was meant to be a measure loyalty. Whether it actually does that or not is still in dispute, but it certainly wasn't designed to measure customer service, and fix it.

5. Controllables

NPS can easily combine aspects of customer experience which are not controllable by the staff member

Cabin crew can't control the seats or the food. They can’t control late flights or bumpy rides. A phone support person perhaps cannot help because the process won’t allow it. What if you hang up in frustration? Even the best service agent in the world may get a zero if the customer gets the answer they didn’t want. Similarly, what if the customer had their query answered in the positive, but had to wait on the phone for 20 minutes before being served? Again, the staff member may score 0/10 though it wasn’t their fault.

6. Which part?

All staff are lumped together when the customer interacts with more than one person.

In a long process like a flight or buying a car, what if one little process went poorly and dragged the overall score down? e.g. the beer wasn't quite cold enough because service was delayed by a late departure and turbulence? Interactions occur with lots of people on a flight, from check-in through to walking off at the destination. All staff will be lumped together, or the cabin crew will be lumped with the overall service, even that provided by staff not on the plane.

7. Annoying

You can’t ask NPS questions as often as you should without annoying your customers.

On a flight, ideally the NPS question should be asked at check-in, waiting for the flight, lounge, boarding, in-air, and luggage claim. But this would be overly annoying for the customers. So the company will ask one question, but miss the granularity required.

8. Asking at the wrong time

The NPS question is sometimes asked before the experience has ended, or many weeks after the experience.

I often get asked what I thought of the booking process before it's finished.  e.g. I get asked on the booking website before I've completed the booking or started the flight. At other times I’m asked the NPS question more than one week after the event.

9. Not all interactions are profound

There is a temptation to ask the NPS question at every interaction, even when there is a small interaction with no great loyalty impact.

I was recently on a chat session with my electricity company. We sorted out a minor billing issue in minutes. Was great. They then asked me to complete a survey which asked if my issue was resolved, my satisfaction with the agent, and then the NPS question. The first two questions were relevant, but this was hardly the heady stuff of recommendations as asked in the third question. To make matter worse,, I couldn’t answer the two relevant questions without answering the NPS question.

When the interaction is a tiny part of a long term relationship, the NPS question is irrelevant because a true recommendation would be made only after consideration of all the recent events over your time as a customer, not just the most recent incidental interaction.A side point. Never force the NPS question. I t must be voluntary.

10. Asking unprofitable questions

Customers are blissfully unaware of all the background work which goes into their experience, especially the most profitable activities.

Asking about their experience may measure things which have no impact on profitability.  For example, in a clothing store, asking for the sale and upselling are critical to profitability but have little influence on whether the customer would recommend the service.

11. Assuming the customer knows best

The customer doesn’t always know best, but they judge on the scraps of information they have available.

It still amazes me that passengers get annoyed when a flight is delayed for a technical issue or a storm. They still blame the airline. The airline knows what’s best for the customer, and is in the best position to gauge whether to delay or depart. Yet the customer may mark 0/10 just because the flight was delayed.

The same can happen in a restaurant if the chef wasn’t happy with a meal and started over.

Usually also manifests when the staff member asks a bunch of questions to understand the customer’s needs, but the customer gets annoyed. This especially the case in banking, insurance, real estate, car buying, and other high contact sales. The customer gets annoyed, but the staff member had to ask the questions to do their best job for the customer.

12. Synchroncity

The opposite is not always bad. 

A smile is good, a scowl is bad, fair enough. Getting an upgrade is good (10/10) but not getting an upgrade is not bad. A once off special positive event will affect the NPS score more than just a solid reliable experience. 

13. Fraud

NPS results are easily manipulated by staff.

I was once offered a free tank of fuel from a car dealership if I forwarded my NPS questionnaire to the dealership for them to complete. 

Some stores have put the NPS question on the Credit Card terminal. The customer is asked the question just before entering their PIN number. But I’ve seen staff entering the payment amount and clicking 9/10 for the NPS score, before handing over the terminal to the customer to enter their PIN.

14. Extremities

NPS is ignorant of the silent majority.

The vast majority of customers do not complete their NPS surveys. For many retail stores, receiving ten reports would be seen as a successful campaign. But this ignores the vast majority of customers who did not give any feedback. Furthermore, upset customers and raving fans will give feedback, others will not.

15. Not a proxy for service

Asking the NPS question can be seen as a lazy form of customer service.

One of my friends recently complained that the only time he hears from his supplier representative is when it's time to complete an NPS survey. The representative will contact my friend and ask if he had a chance to complete the survey, and asking for a good score. A survey should not be seen as an alternative to contact.

16. Fatigue

Customers are inundated by requests for surveys.

Survey fatigue is becoming a real issue as more and more companies are adopting lazy NPS surveys. Just because it's only one question, doesn't mean it avoids annoying a customer.

For a recent flight I was annoyingly asked the NPS question at the booking stage, on board, and via email after the flight.

17. Friction

Surveys can create friction which impedes the customer experience.

A lot of the customer service research is moving towards asking customers how difficult it was to deal with the company. Whilst I am not suggesting this as a silver bullet for the silver bullets, it does suggest there is at least one alternative to NPS. More importantly, it suggests that we should make it as easy as possible for a customer to do with us. Asking survey questions only impedes the service experience.

Asking the NPS question at the end of a phone call is one obvious example of this friction. It is annoying to hear that you will be given a survey at the end of the call, and it is annoying to hear it at the end of the call.

18. Ridiculousness

In the quest for gathering data, some companies are gathering information at the most inappropriate and ridiculous places.

Asking someone to take a photo of a QR code in an airport washroom makes the customers uncomfortable and feel like they could get arrested. Asking in the customs hall of an airport could scare someone into answer Yes, or ensure avoidance of the survey altogether.

19. Rewards

Offering customers a reward, such as a free meal to complete a survey can create dysfunctional behaviour.

Some customers just complete the survey with no thought, but simply to collect the reward. Their answers are random, and they may complete more than one response, even if they didn't make a purchase. Sometimes the surveys are even passed on to other family members, so they can collect the reward.

20. Is the information being used?

Sometimes the information is collected but not acted upon. Or it is collected for the wrong reasons.

Some companies simply collect the NPS results because the board or shareholders want it. Sometimes it is used simply to reward staff. When the results are collected nearly for internal purposes rather than to improve the customer experience, they lose their efficacy.

21. Cultural factors

Not all cultures rate service equally. 

A customer in Germany may have different expectations of precision and timeliness than a customer in southern Italy. An Asian customer may place more importance on being treated respectfully than being served quickly. This also applies to cities with different cultural cliques or even tourists. How would a Japanese commuter rate a holiday trip on the NY Subway?

22. Personalities

Different personalities will rate service differently.

I remember driving down the freeway in my 5 year old car, remarking to my colleague that the car impressed me as it was still humming along. Her response? “What’s the big deal, that’s what it’s expected to do!” We’d score the NPS differently.

Some customers will value function over form, some will value speed over price, some will value the personal touch over knowledge. Different people and different personalities will rate the same customer experience differently.

23. Segmentation

NPS needs to measure results from different customer segments separately.

Organisations should cross-reference the responses of different customer segments. The results all the top 20% of customers who give 80% of the Business are probably more important than the 80% of customers who give 20% of the business. But NPS rarely delves to that level of granularity.

24. Too meta.

NPS is often collected at a very meta level, making it almost impossible to use for corrective action. 

Often NPS doesn’t go down to th individual store level, nor identify the staff member. So the data becomes one big blob. Not only is the information usually too meta, but often there is no alternative. For example, flight crew on an airline rarely work in long-term teams. So who owns the service for flight AA1234? No one. The same may apply to any circumstances where there is a high proportion of itinerant workers or where cohesive teams rarely exist (e.g. function centres).

25. Teamwork

Staff cohesion can be affected by poor results.

When teams have multiple staff members, such as in a store, staff may start to play the blame game and start with hunts for poor results, especially when bonuses are involved. Staff who don’t believe they were involved in the poor scores may also resent being caught in the broad brush of results caused by other staff. 

26. Infinitely reducible

All parts of the service experience are ultimately reducible to impractical minutiae.

Should you measure the in-flight experience separate from check-in, boarding and baggage claim? If so, what part of the in-flight experience? Each staff member separately? Facilities like the seats? Movies? Food and drink?

If you really want to get to a level where you can take action on the data, then you need to annoy the customers throughout the experience, and ruin their experience by requesting constant feedback. It’s impractical and destroys the customer experience. The surveys would ultimately become part of the ‘experience’.

27. Animosity

Staff are asked to explain customer behaviour.

The manager of an area scoring 0/10 may be asked to explain why a customer gave that score and what they will do about preventing it in the future. This is a ridiculous question which just creates great animosity due to the unreliable nature of the data. There are instances when a low score is valid, but that is not always the case.

28. Who paid?

The economic buyer may not be the person using the service.

If a parent paid for a child's airline flight, should the NPS question be asked of the child who took the flight but isn’t the economic buyer, or the should we ask NPS of the parent who didn’t experience the service? The same issue applies to corporate travel, or employees driving company cars. For smaller purchases, it may apply if a husband simply popped into a store he’d normally not visit to buy something for his wife?

29. Timings

Customers respond after the event when their memory is affected.

Customers may not receive the survey for many days or weeks after their transaction, thus compromising the data. A customer could easily forget the bad, experiences and overstate the good, or vice versa. Customers could also be affected by buyers regret after the purchase.

30. Just bad design

Is the design of the input giving bad data?

Airports and shopping centres love having iPads and buttons asking NPS style questions. The issue is that anyone can push a button. On a recent trip to Fiji I saw kids running from one terminal to another, randomly pushing buttons, all the while giving false readings.

Ultimately, organisations need to ask two questions:

• How are we going? and
• What should we do?

At best, NPS does a modest job of the first question and does nothing to answer the second.

#nps #netpromoterscore #customerservice #surveys #speaker #mysteryshopping