5 Ways Industry Surveys Can Be Misleading
I’ve gotten to read – and write about – lots of survey/study findings over the years, and what I have found, on a surprising number of occasions, is that the “findings” aren’t always that reliable.
As human beings, we’re drawn to perspectives, including surveys and studies that validate our sense of the world. This “confirmation bias,” as it’s called, is the tendency to search for, interpret, favor, and recall information in a way that confirms our preexisting beliefs or hypotheses. It also tends to make us discount findings that run afoul of our existing beliefs.
In its simplest terms then, when you see a headline that confirms your sense of the world, you’ll be naturally inclined to embrace and remember it as a validation of what you already perceive reality to be. Even if the grounds supporting that premise are shaky, sketchy, or (shudder) downright scurrilous.
Here are some things to look for – likely in the fine print – as you evaluate those findings.
There can be a difference between what people say they will (or might) do and what they actually will.
No matter how well targeted they are, surveys (and studies that incorporate the outcome of surveys) must rely on what individuals tell us they will do in specific circumstances, particularly in circumstances where the decision is hypothetical. When you’re dealing with something that hasn’t actually occurred, there’s not much help for that, but there’s plenty of evidence to suggest that, once given an opportunity to act on the actual choice(s), people act differently than their response to a survey might suggest.
For example, people tend to be less prone to action in reality than they indicate they will be – inertia being one of the most powerful forces in human nature, apparently. Also, sometimes survey respondents indicate a preference for what they think is the “right” answer, or what they think the individual conducting the survey expects, rather than what they actually think. That, of course, is why the positioning and framing of the question can be so important (as a side note, whenever possible, it helps to see the actual questions asked, and the responses available).
The bottom line is that when what people tell you they will do, or even what kind of product they would like to buy, if you later find that they don’t, just remember that there may be more powerful forces at work.
There can be a difference between what people think they have and reality.
Since, particularly with retirement plans, there are so few good sources of data at the participant level, much of what gets picked up in academic research is based on information that is “self-reported,” which is to say, it’s what people tell the people taking the survey. The most prevalent is, perhaps, the Survey of Consumer Finance (SCF), conducted by the Federal Reserve every three years.
The source is certainly credible, but the basis is phone interviews with individuals about a variety of aspects of their financial status, including a few questions on their retirement savings, expectations about pensions, etc. In that sense, it tells you what the individuals surveyed have (or perhaps wish they had), but not necessarily what they actually have.
Perhaps more significantly, the SCF surveys different people every three years, so be wary of the trendlines that are drawn from its findings – such as increases or decreases in retirement savings. Those who do are comparing apples and oranges – more precisely the savings of one group of individuals to a completely different group of people… three years later.
The survey sample size and composition matter.
Especially when people position their findings as representative of a particular group, you want to make sure that that group is, in fact, adequately represented. Perhaps needless to say, the smaller the sampling size – or the larger the statistical error – the less reliable the results.
Though you’ll often have to scroll all the way down to the footnotes to see that detail.
Case in point: Several months ago, I stumbled across a survey that purported to capture a big shift in advisors’ response to the Labor Department’s fiduciary regulation. Except that between the two points in time when they assessed the shift in sentiment, they wound up talking to two completely different types of advisors. So, while the surveying firm – and the instrument – were ostensibly the same, the conclusions drawn as a shift in sentiment could have been nothing more than a difference in perspective between two completely different groups of people.
Consider the source.
Human beings have certain biases – and so do the organizations that conduct and pay to surveys and studies conducted. Not that sponsored research can’t provide valuable insights. But approach with caution the conclusions drawn by those that tell you that everybody wants to buy the type of product offered by the firm(s) that have underwritten the survey.
When you ask may matter as much as what you ask.
Objective surveys can be complicated instruments to create, and identifying and garnering responses from the “right” audiences can be an even more challenging undertaking. That said, people’s perspectives on certain issues are often influenced by events around them – and a question asked in January can generate an entirely different response even a month later, much less a year after the fact.
For example, a 2015 survey of plan sponsor sentiment on a topic like 401(k) fee litigation is unlikely to produce identical results to one conducted in the past 30 days, nor would an advisor survey about the fiduciary regulation prior to the publication of the final rule as to its impact. Down in those footnotes about sample size/composition, you’ll likely find an indication as to when the survey was conducted. There’s nothing wrong with recycling survey results, properly disclosed. But things do change, and you need to be careful about any conclusions drawn from old data.
Not to mention the conclusions you might be otherwise inclined to draw from conclusions about old data.
this post originally appeared here.