Measuring the Wrong Stuff

TLDR: Junk data is more easily obtained, processed and actioned than is nuanced data. Base your business decisions on the former at your peril.

=======

I recently utilized chat tech support to help me through a jam with my personal website. The agent was very helpful and resolved my issue, garnering my appreciation. Upon conclusion of the session, the machine in its wisdom, presented to me the evaluation to which all chat users are now accustomed: "1-5, how did our tech support agent do?"

As you can see, I didn't want to talk about the agent. I wanted to talk about the chat system, which (without going into boring details) was not configured to optimally enable the agent to solve my problem.

I have no insight into the total pile of data with which this particular tech firm works, but I have seen, first hand, organizations do less with less. I eventually gave the ostensibly cheerful agent a 5, but I couldn't help but imagine layers of managers looking at page after page of 3s, 4s and 5s, and tying all that incomplete data to operational and HR decisions. Meanwhile, the machine, as constructed, had no way to capture the fact that the machine itself may have been broken and possibly having a larger impact on the company than the highly scripted activity of its support agents.

This is merely a gentle reminder to always view your data in context. It definitely helps if middle and upper management is in agreement over the end goal, but as anybody who has ever sat at the board table (or, more likely, in those uncomfortable chairs along the wall of the board room) knows, this is not always a given.

And here's a bit of heresy from my own industry to illustrate: It has always been challenging for me to find a statistically significant correlation between instructor ratings and repeat sales. In other words, whenever we endeavored to poke the data for correlations, we found that CPE consumers were purchasing second and third courses from the 3.3s as frequently as they were from the 4.5s.

A big fat NOTE: This is not to detract from the talent of the 4s. They are a joy to watch and they should be rightfully proud of their gift. But if the organization's goal is to maximally monetize its content, a nuanced approach is vital.