When Numbers Deceive: How the Net Promoter Score Misleads Us [English]

For the German version click here: https://www.dhirubhai.net/pulse/wenn-zahlen-t%C3%A4uschen-wie-uns-der-net-promoter-score-die-torge-tonn-oxhre/

Management Summary:

Question: How accurately does the Net Promoter Score (NPS) describe the current situation of our organization?

The Net Promoter Score (NPS) promises easily understandable solutions for measuring

• Your customer satisfaction and loyalty
• Your growth potential
• And thereby, your success.

By answering a single question and calculating a single score, companies are supposed to gain the ability to increase their revenues and profitability and to improve their growth.

The following article demonstrates from a data analytical perspective why the Net Promoter Score cannot keep this promise and how, due to its scientific veneer, it also lulls you into a false sense of security.

In summary: Since calculating the NPS score no longer allows you to infer how many of your customers have given each point value on the scale of 0 to 10, you also lack information about

• Whether your current value is really a good value
• Whether a value of 50 is really better than a value of 40
• Whether you were better this month than in the previous ones
• And whether the annual results really differ significantly from each other or if the numbers are just randomly aligned.

Why this is the case is explained in this article.

A tip in advance: Use descriptive and inferential statistical methods of data analysis of your scale and do not rely on the interpretations of the NPS score.

Article:

Examining the Effectiveness:

The effectiveness of the Net Promoter Score (NPS) is currently a hot topic. With just one single question and the calculation of a single score, companies are supposed to be able to measure their customer satisfaction and loyalty, determine their growth potential, and thereby increase their success.

Moreover, the NPS meets many companies' desire for data analytics and data science to gain actionable insights from their collected data, create predictive models, and make data-driven decisions.

Therefore, it's crucial that companies only select methods and models that are capable of describing the complex internal and external environment of organizations as realistically as possible. Only those who manage to accurately describe the current situation of their companies or teams can continuously make well-founded and data-based decisions.

Accordingly, the question arises to what extent the NPS is capable of describing the current situation of our companies and organizations, and thus a part of their complex internal and external environment. The examination of whether the NPS can achieve this is divided into the following three steps:

1. We create a randomly generated distribution of survey values between 0 and 10 following a repeatable pattern, so that even if the numbers differ in each iteration, the underlying distribution (i.e., the pattern according to which the numerical values are allocated) remains the same.
2. We calculate various metrics for each randomly generated distribution of survey values, such as the Net Promoter Score, and save them. This involves drawing a random sample from the distribution.
3. We repeat the two steps a certain number of times and then look at the distribution of the calculated metrics.

The expected result: Since the randomly generated distributions of survey values always follow the same pattern, the calculated metrics that are supposed to describe this pattern should only vary slightly (to the extent of the randomly generated deviations). If this is not the case, it suggests that they do not recognize the actual distribution of survey values, thereby not allowing any conclusions about reality.

Technical Interlude: The Model

For our randomly generated distribution and the calculations, we use an R script.

The distribution itself is a randomly generated normal distribution with an expected value of 9 and a standard deviation of 2, with cut-off values at 0 and 10. This means that all generated values falling below or above these cut-off values are assigned a value of 0 or 10, respectively. The resulting distribution is then no longer a normal distribution.

The seed set for the randomly generated values in R is 125.

The modeling is performed 100 times in a row. For each iteration, 400 survey values are generated, from which a sample of 100 values is drawn.        

The Evaluation:

First, we look at the randomly generated distributions of our survey values to check if they indeed approximately follow a repeatable pattern. The following figure exemplary shows this pattern for the first 10 iterations:

We now know that the distributions of our survey values follow a pattern, which we want to describe and evaluate using our metrics. So, in the next step, we calculate the following metrics for each of our 100 iterations:

• The Net Promoter Score
• The mean (arithmetic mean) as a metric for the focal point of the survey values
• And the standard deviation as a metric for the dispersion of the survey values around the mean.

Then, we evaluate the 100 calculated metrics and come to the following result:

Net Promoter Score:

The value of the lowest calculated NPS is 21, the value of the highest NPS is 54. The difference between the lowest and the highest value of the 100 calculations is 33.

The Mean:

The value of the lowest calculated mean lies (rounded to two decimal places) at 8.08 points, the value of the highest mean lies at 8.94 points. The difference between the lowest and the highest value of the 100 calculations is 0.86 points, i.e., less than one scale point.

The Dispersion:

The value of the lowest calculated standard deviation lies (rounded to two decimal places) at 1.19 points, the value of the highest standard deviation lies at 1.78 points. The difference between the lowest and the highest value of the 100 calculations is 0.59 points.

The above figure shows the exact distribution and frequencies of the calculated metrics after 100 calculations. From this, we can see how the metrics distribute across the value ranges and in which value range each metric is located.

The Final Result:

As described, metrics that accurately describe the survey values in 100 measurements of the same distribution patterns should only vary to the extent of the randomly generated dispersion.

In the evaluation, it is evident that while the deviations of the descriptive metrics (mean and standard deviation) are low, the range of the calculated NPS values between 21 and 54, with a difference of 30 points, varies greatly, even though each calculated NPS comes from the same distribution of survey values, and thus should describe the same facts. Because of this variation, we must conclude that with the NPS, we cannot make assertions about

• Whether a value of 50 is really better than a value of 40
• Whether our current value is really a good one, i.e., describes a positive distribution of customer experiences for us
• And whether we were better this month or year than in the previous ones.

The question of whether the NPS accurately describes reality and thus can make reliable statements about the current situation of our companies and teams is therefore clearly answered with a no.

What can we do instead?

Use descriptive and inferential statistical methods of data analysis such as arithmetic means, standard deviations, box plots, histograms, or quartiles and percentiles, and do not rely on the interpretations of the NPS score.

If you want more information on what else to consider when measuring customer satisfaction, loyalty, or purchasing behavior, it is best to read the following article: https://www.dhirubhai.net/pulse/wenn-der-kompass-versagt-teil-1-net-promoter-score-torge-tonn/ (German Version).

#NetPromoterScore #NetPromoterScore #Kundenzufriedenheit #CustomerSatisfaction #Kundenbindung #CustomerLoyalty #Wachstumspotenzial #GrowthPotential #Unternehmenserfolg #BusinessSuccess #Datenanalyse #DataAnalysis #FalscheSicherheit #FalseSecurity #DeskriptiveStatistik #DescriptiveStatistics #Inferenzstatistik #InferentialStatistics #DataAnalytics #DataAnalytics #DataScience #DataScience #Realit?tsnaheMessung #RealisticMeasurement #Entscheidungsfindung #DecisionMaking #ZufallsgenerierteVerteilung #RandomDistribution #KennzahlenAnalyse #MetricsAnalysis #Standardabweichung #StandardDeviation #Mittelwert #MeanValue #StatistischeStreuung #StatisticalVariability #AnalyseMethoden #AnalysisMethods #FundierteEntscheidungen #InformedDecisions