Adjusting for non-response bias to get an accurate read

A lot has been written about how the polls failed to predict the latest presidential election results. What we have learnt since then is that there were 2 potential factors at play (although the debate is still going on what influenced the failed prediction the most): 1. Not able to get an accurate signal of how undecided voters were going to vote, particularly in the Rust Belt States and 2. non-response bias from critical segments. When certain segments of the population are not responding to a survey or poll for whatever reason, despite everyone having an equal opportunity to respond and their opinions are different from segments of respondents who did respond, you get non-response bias. This matters a lot when we are doing surveys - a form of polling - to get a representative read for a population at large AND there is also a lot of variation in the responses.

What the results of the election polls demonstrated is that non-response bias is a real problem and can often lead to an inaccurate conclusion. Far too often do I encounter surveys that do not analyze the effect of non-response bias. It's a time consuming exercise to determine all of the different segments, or combinations on which you can dissect a population (gender, income, geography, industry, frequency of usage, education etc.) and often the sample sizes are not large enough to properly understand the effect of non-response bias when multiple variables are involved. Researchers ignore it or don't take the time to understand if non-response bias occurred in the first place and in some cases make random adjustments based on false assumptions that unnecessarily over-complicate things. However, surveys are getting shorter, often conducted within the product through pop-ups which leads to higher sample sizes. There are also better statistical packages from R that make the process of identifying non-response bias in a large data set easier. Companies also have greater access to more data about customers to meaningfully segment a population. These developments should prompt market researchers to pay closer attention to this effect.

The good news is that across the many surveys we have conducted to date, we are starting to see a pattern on which non-response bias is likely to happen, making the process of identifying the right variables to control for easier. The level of engagement (and strength of motivation) to use a product plays a critical role in whether someone is more or less likely to respond to a survey AND the responses between a heavy and light user are typically different.

To illustrate, here is an example with re-scaled figures to preserve confidentiality. Despite the fact that we invited all users of a product to the survey, the heavy users were much more likely to respond than the light users causing the sample to come back with a different distribution than the overall population (see below table) - for example, we ended up with 3 times as many Heavy users than actually exist in the population at large - a result that is common in many of the surveys we do. Heavy users have a greater affinity with the company and the product and are more eager to provide feedback, an effect also found in polls where "decided" voters are more likely to respond than "undecided" or "swing" voters. "NPS" in the below table is the net promoter score for each segment that we captured with the likelihood to recommend question. There is quite a bit of variation in the NPS scores across the three segments. Heavy users have a much higher NPS and are more likely to recommend the product.

If we were to calculate the NPS for the total sample we would get = 30*30% + 10*40% + -20 * 30%    = 7 (column A)

However, if we adjust for non-response bias from light users and map the scores to the actual population distribution it will make the overall NPS = 30*10% + 10*30% + -20*60% = -6 (column B)

These results are significantly different given the sample size and difference in NPS and we can therefore conclude that non-response from light users is a problem and causes bias. If we didn't adjust and simply reported out the overall NPS figure (A) it would create a false sense that NPS is good and positive. Because the difference is driven by underlying segment variations, the conclusions, urgency to act and direction the product team needs to take to improve NPS are inaccurate. For any company to grow its user base which is the ultimate goal of the Product Manager, it's important to understand how to cater to both Heavy and Light users in a balanced and measured way that is an accurate reflection of the total user base. There are different ways to adjust for this effect but our favored approach is to make the adjustment after the fact rather than to stratify the sample and set quotas at the outset to make up for the imbalance. So on your next survey, try and determine if response bias exists and whether an adjustment is needed.