How to Test Knowledge and Misperceptions on a Survey

It’s not easy to use a survey to measure public knowledge or misperceptions. Why? Because surveys are not quizzes. Respondents expect you to ask questions they can answer truthfully and correctly based on their opinions and experience. As good researchers, we mostly try to reassure respondents: “There are no right or wrong answers! Please give us your honest opinions and feedback.”

Those reassurances do not work if you use survey questions to document levels of public awareness or knowledge. Instead, you need to reassure respondents that their best guess is good enough, and that there is no downside to being wrong. The research on survey cheating we highlighted in another article failed to do that, and the authors (not surprisingly) found that a third of their respondents “cheated” by looking up answers to questions that tested political knowledge.

In our experience, knowledge and misperceptions can be tested via survey questions as long as you shift your thinking and respondents’ thinking by implementing a few important modifications in the survey design. Here is what we have found works best:

• Offer a “don’t know” option so that respondents are not forced to guess (or look up the answer!) if they truly do not know.
• As an alternative, instruct respondents to provide their best guess, even if they are not sure. Let them know that a guess is just as good as a “right” answer.
• Tell respondents what you are doing. Explain that you are trying to understand what people know or do not know about the topic, and that “wrong” answers are just as valuable as right ones.
• Show them the correct answers when they are done. Most people enjoy taking self-assessment quizzes as long as they can see how close they came to getting it right.
• Tell respondents ahead of time that you will show them the correct answers. They will then realize there is no need to “cheat” by looking up the answers.

The research we highlighted last week did none of these things. Their survey design inadvertently encouraged people to look up answers. It’s no wonder they failed to get good measures of political knowledge.

—Joe Hopper, Ph.D.