VOC: You Get What You Pay For

Voice-of-customer (“VOC”) feedback is always useful, but it’s not always used correctly.

Not long ago I attended a presentation by a senior executive who was charged with collecting voice-of-customer feedback from the business customers served by his very large technology company. He based much of his presentation on some figures that were compiled from more than 100,000 online customer surveys completed over a year’s period.

His VOC feedback program yielded some very helpful insights, particularly in the form of the many verbatim comments collected. Also, it allowed the company to identify and reach out to those who expressed high levels of dissatisfaction. So far, so good.

But then the executive told us that his company was so committed to improving its overall customer experience that senior executives at the firm all had a portion of their incentive pay based on the survey’s results. That’s too bad, I thought, because the way this particular survey was structured meant that it was virtually worthless in terms of understanding the overall level of customer satisfaction.

His VOC survey was based on a highly biased sample. Good for anecdotal feedback and identifying specific problems, but useless for understanding the general level of customer satisfaction within the entire customer base.

Here’s why: All customers were solicited for their inputs virtually every time they completed any kind of transaction on the company’s web site, and there were nearly 4 million such transactions during the year. So 100,000 survey completions, in fact, amounted to a response rate of less than 3%, and with that low a response rate it’s almost certainly the case that those who chose to go to the trouble of completing the survey did so because they were either highly displeased or highly pleased with their interactions.

Remember, these were business customers – busy managers and IT executives who don’t have a lot of spare time to begin with and aren’t likely to dedicate even a few minutes of it to helping some other company figure out how to do its job better. Unless, of course, they were extremely unhappy, and wanted to give the company a piece of their mind, or perhaps extremely pleased, and eager to share their good wishes.

But using such data to infer overall satisfaction levels would be like sticking one hand in ice water and the other in scalding water, and then inferring that, on average, you must be quite comfortable.

It doesn’t matter how large your VOC sample is, if your sample is biased it will be far less useful than a smaller but less biased sample. Rather than soliciting everyone and netting a 3% response (even if that amounts to 100,000 or more respondents), you would do better by soliciting just a couple of thousand participants chosen at random, and offering them some form of compensation or benefit designed to generate a higher participation rate. There will still be some bias in the results, but far less than in a universal survey that suffers from an extremely low participation rate.

With just a few tests you might even be able to find some combination of prizes or fees that would generate an 80% or 90% participation, which would give you much greater assurance that the busy executives whose opinions you were now collecting more fairly represent the entire population you are trying to serve.

The only truly unbiased voice-of-customer feedback, I believe, is the feedback you find "in the wild," that is, by simply observing the comments made by your customers in social media. I wrote about this is a previous post.