The tyranny of Metrics

In the world of statistical measurement, there is an old adage: “The problem with statistics is that too many people use them like a drunk uses a lamppost; for support instead of illumination”

In the past quarter as companies were finalized 2021 business plans, many business conversations included discussions of OKR’s, KPI’s and other metrics, as companies cascaded their operational plans down through their organizations. As companies drive towards the use of better metrics driven planning and feedback tools, too often I see a tendency for the metric to become the goal instead of a path to the goal.

Do you understand the process before you measure it?

My father started his career in the 1950’s as an industrial engineer at the Steel Company of Canada. One day, he was escorting a senior cost accountant on a plant tour. As they watched steel ingots being produced, the accountant noticed a worker standing a few feet away from a cooling but still pink hot ingot with a tape measure. The worker was estimating the length, width and thickness of the ingot which would weigh a few tons, and then would look to a chart on the wall. Based on the dimensions, the chart specified the weight in pounds to several decimals of accuracy. The worker would then dutifully write down the weight (including several decimals of accuracy) and send the production sheet off to where it would eventually end up in the hands of cost accounting to assign a dollar value (to several decimals of accuracy) to the ingot. As the ingot continued to be processed, it was made into plate, rod, coils, or other products, cut or sliced to smaller dimensions, and possibly treated or coated. At each stage, the cost accountants manually re-calculated its value (to several decimals of accuracy).

Now, as the accountant understood the process, he was horrified thinking about the hours and hours of calculation done by his department (this was in the early days of computers and electro-mechanical calculators) and nearly apoplectic with rage. He demanded to know from my father why the worker was not weighing the ingot on the scale. My father calmly replied: “The scale is broken but even if it wasn’t, it is only accurate to the closest 100 pounds.” Afterall, why would you spend money on better scale accuracy for a product weighing thousands of pounds with a value of less than pennies per pound.

Are your measurements accurate or biased?

An area where I see companies struggle is to determine a success measurement system for the Customer Success organization. While many use Net Promoter Score (NPS) as a measure of customer satisfaction, I remain skeptical of its use as a predictor of future financial results. While the measurement itself may be accurate, does the process to collect it really reflect the client’s true intentions. Throughout the 1970’s and 80’s, IBM was viewed as an arrogant company that bullied customers and didn’t listen. As part of its transformation in the 1990’s, employees’ bonus and managers’ career trajectories were tied to customer satisfaction measurements. At one point, customer satisfaction was measured by a survey of a statistically valid sampling of customers, administered by an independent third party company. By doing so, IBM management believed that the results would be an accurate representation that could be used for analysis and decision-making. By tying the results to employee bonus, it was felt this would inspire correct behaviours.

Unfortunately, the process had flaws. The first flaw was someone had to produce a list of who to phone at each customer. Obviously, those closest to the customer (the account managers) would be the best source of whose opinion at each customer best represents the customer’s company. Since the account managers’ (and their boss’) bonus was tied to the results, the list was subject to biases towards those at the customer who had the most favourable impression of IBM.

In addition to prevent calls to customers who were in the middle of sensitive interactions with the account team such as negotiating a new contract or other “deemed sensitive” events, customer could be flagged by the account team for exclusion from the survey in any given quarter.

Although the list was randomly called each quarter, another flaw was to avoid survey fatigue, customers were never called more than once a year. As a result, managers could quickly isolate the list to subsets of customers each quarter that were likely to be called. This spawned quarterly campaigns to check in with every customer who might be called, to ensure they knew about the process, knew that bonuses were dependent on their answer and to inquire if they had any outstanding issues that could be taken care of for them.

No question, IBM’s customer satisfaction ratings rose but was it because customers were happier or was it because managers were cooking the system.

Does the methodology-used create metrics that predict outcomes?

One of my university professors was Polish. As a young scientist during WWII, he was conscripted by the Nazi to build the V1 and V2 rocket systems. As a conscripted worker, working conditions were hard. His captors were rigorous in checking calculations and any attempt at sabotage was dealt with by summary execution. To thwart them, my professor and his colleagues adopted a different strategy. They applied correct engineering formulae and calculations to the wrong problem. By doing so, failures occurred and progress of developing the rockets was slowed yet the failures could never be tied back to them.

Using a valid methodology to measure something does not ensure the metrics are valid if the measurement methodology is misapplied.

Is the sampling valid?

Many employee performance management systems use a “bell curve” to manage the distribution of performance ratings. In the world of probability and statistics, in certain situations, the Central-Mean Theorem underlies the principle that in a large sample set, a normal distribution (Bell Curve) will result. As a result, managers are told that 5% of your employees are in the bottom tier, 15% are in the next tier, 60% are in the middle, 15% are almost superstars and only 5% can be in the superstar category. While it is likely valid, in the large company with many employees, that the performance ratings of all employees will follow a normal distribution, it is also likely invalid that the performance ratings of a small team of employees will be normally distributed. Yet annually, managers of small teams are told to fit their performance ratings into the Bell Curve.

Are you measuring the “Cause” or the Effect”?

Understanding the process also helps understand and explain the relationships between “cause and effect”. Have you ever heard the argument that the price of our product is too high and therefore the sales team cannot sell their quota so they need to discount more? Is the price the cause of poor sales results or is poorly skilled salespeople, a lousy sales process, or targeting prospects who won’t receive high value from the product really the cause of why the company can’t earn the price they think their product deserves. Don’t confuse “cause” versus “effect”.

Have you validated the business impact of the cause-and-effect relationship?

A few years ago, a client was very hawkish on only hiring candidates who scored high on a particular aptitude test that their CEO liked. One of their senior leaders confessed to me that she wasn't sure that she would have been hired if the test had been in place when she had applied.

I asked a friend, who is a senior employment lawyer, whether a company could ask a prospective employee to take an aptitude test as part of the interview process. His opinion was they probably could not, unless they could demonstrate a validate correlation between the test results and performance of actual employees in their company. If they could not, he would argue that the test may have systemic biases imbedded in it, thus making it discriminatory.

While I often see companies create hypotheses about metrics like

• This employment test is a good indicator of likely hiring success
• This customer success process will result in higher customer retention
• This sales methodology will increase win rates

rarely do I see follow-thru to rigorously confirm the hypothesis is correct or its applicability to their specific situation.

For example, when was the last time you saw metrics like:

• Developers who score above a certain level in a coding employment test are x% more productive than those that fall below that level
• Each additional point in NPS translates to $x of additional revenue in the lifetime value (LTV) of a customer
• Implementing this new sales methodology has increased our win rate by x% while reducing the cost of customer acquisition (CAC) by x% and increased the LTV by x%

Metrics are a great business tool that support the management philosophy of “you can only manage what you measure”, but are your metrics valid measure of the objective or have they become "the objective"?

One final thought. Many software companies measure and celebrate the high NPS scores they receive from clients when they call their Support Line. Is this success? Or, is every time a customer phones a failure… a product quality issue… software that is not intuitive to use… product documentation or help functions that don’t solve the problem… a failure to set the proper expectation during the sales process… or software that was not properly configured during implementation…

If so, why are we celebrating a good NPS score?