#8 - An unusual mix: Useless scorecards, unconventional statistics, banking NPS, empathy tests

Welcome to my eighth Customer Strategy Newsletter here on LinkedIn. Here is what I will cover:

• Unfortunately, most companies use survey results exclusively for internal scorecards and are therefore wasting customers' time.
• Basic statistical notions most of us have learned do not apply to customer surveys.
• Why do banks with horrible NPS results simply not go out of business?
• A fascinating look at whether big brands pass the empathy test with their customer communications.

I hope you enjoy the read. As always, please let me know what you think and please pass this along to your friends and colleagues, encouraging them to subscribe.

Let's get on with the insights!

Most companies use surveys to produce inaccurate internal scorecards

Way back in 2015 I started a little experiment. I decided to answer the next 100 survey requests I received by email. It took me over a year. My purpose was to see whether any of the companies involved would get back to me to tell me what they had learned and what they were going to do about it. Just one did. Surprisingly, it was British Airways, who had won my informal prize for the worst survey ever. I even lied when providing responses to some companies, complaining about really bad things that had not actually happened. Nobody got back to me on any of those cases. I felt used. My time was totally wasted.

My point here is that most companies use surveys exclusively to construct internal scorecards. Every department that touches the customer in any way has to get its own score. And they are getting it based on the (unrepresentative) fraction of customers who answer survey requests. Each of these functions would be far better off deciding on its own top operational metrics that cover the things that touch customers directly. After all, they can measure these for 100% of customers, without waiting for surveys. Surveys remain useful as an occasional aid for prioritizing improvement work and determining which operational metrics are most important. I know this is a controversial opinion. All views welcome.

Conventional statistics do not apply to surveys!

And while we are on the subject of surveys, please bear in mind that survey responses are not normally distributed! This means that the basic statistical analysis techniques most CX leaders know and love just do not apply. For example, no, you cannot assume that 30 responses are sufficient to calculate mean and standard deviation numbers that are reasonably representative of the customer population as a whole. And the resulting margin of error calculations are also meaningless.

The reason for all this is that the people who answer surveys are simply not your average customers. In NPS terminology, the main defect is that Detractors are hugely under-represented with typical response rates. They apparently think you have already wasted enough of their time. Reichheld and Markey cover this around page 100 of The Ultimate Question 2.0. They say that Bain experience shows that an NPS of +50 based on a 20% response rate would generally become -22 had 100% of the customers answered. (So, therefore, your own evil plan for CX score improvement could include trying to decrease response rates.)

This is all just another argument in favor of eliminating surveys except as an occasional means of calibrating and prioritizing AI / Machine Learning software or other tools used to monitor operational and other KPIs that affect customers. I have written more on the subject here.

Content from OCX Cognition

• If a Bank Has Poor NPS, Why Doesn’t It Fail? This is the provocative title Richard Owen has given to a great article on LinkedIn. He uses the example of Wells Fargo to ask why the free market has not eliminated the bank, since it has had worst-in-class NPS for some time now. The answers and insights are fascinating. Among other things, Richard points out that “When we first started shifting from survey based NPS measurement to predicting NPS based on machine learning, a funny thing happened. A lot of typical drivers of loyalty didn’t seem to impact NPS outcomes as much as we would have expected.” Sounds interesting? It is. Read the full story here.
• Maybe I was having a down day when I wrote ‘The latest CX developments make me feel stupid...’ here on LinkedIn. Whatever the reason, my feelings were sincere at the time of writing. Read it here.

Insights from other websites

Hannah Moffatt from Schwa just wrote a fabulous article about customer empathy for Neil Davey and his friends at MyCustomer. The title is ‘Do big brands pass the empathy test with their communications?’ They studied eight UK banks in depth. Their intention was to test the validity of their new measure of customer empathy. In this case, validity meant having a strong relationship with customer satisfaction rankings as measured by the UK consumer magazine Which?

I loved the many examples of bad and good empathy. How about the bank whose complaints webpage leads with a statement about how wonderful their service is? Really? You wouldn't be on their complaints page if you agreed with that.

Coming up

I continue to learn more about using operational data rather than surveys to understand customers. I will publish a couple of LinkedIn blog posts on the topic between now and my next newsletter two weeks from now. And please accept my apologies for accidentally sending a blog post to my newsletter distribution list last week.

Notes

Maurice FitzGerald is a retired VP of Customer Experience for HP's $4 billion software business and was previously VP of Strategy and Customer Experience as well as Chief of Staff for HP in EMEA. He and his brother Peter, an Oxford D.Phil in Cognitive Psychology, have written three books on customer experience strategy and NPS, and a fourth book that focuses on Peter's cartoon illustrations for the first three. All are available from Amazon.

OCX Cognition predicts customer futures. Our breakthrough SaaS solution, Spectrum AI, lets enterprises transform what’s possible in customer experience. Reduce your customer risk, break down silos, and drive speedy action – when you can see what’s coming, you can change the outcome. Building on more that 15 years of CX-focused expertise, we’ve harnessed today’s advances in AI, elastic computing, and data science to deliver on the promise of customer-driven financial results. Learn more at?www.ocxcognition.com.

The author can be reached here on LinkedIn or at?[email protected]. Please let me know what you think and what sort of content you would like to see here.