Excessive Questionnaire Addition Requests? Here's the Fix.

In survey research, a common derailment arises from well-intentioned but sometimes uninformed collaborators who repeatedly request additional questions be added to the questionnaire. Experienced market researchers have all heard requests such as, "Let's add a 'select all that apply' question with these 15 items," or "...as long as we are talking to people in our target market anyway, let's throw in some questions to see if they like our new packaging." What was once a concise, 5-minute survey expands into a 30-minute monstrosity, confusing participants, risking drop-outs, and gathering excessive data that may go unused anyway. Researchers, typically working under tight deadlines, struggle to find a balance between accommodating these requests and preserving research data quality.

The good news is that these requests often mean that our collaborators are engaged and have ideas they want to share. Of course, if the ideas are too far from the primary goals, the additions can result in confusing, onerous surveys that our valued research participants simply won't tolerate. In addition, collaborators have a tendency to suggest specific questions instead of articulating what they are trying to learn, making it difficult to assess. So, how do we get our collaborators back on track?

The Fix: Switch from Questions to Hypotheses

Here is the fix: ask these eager collaborators for the hypotheses behind the questions they are requesting. This forces them to think more clearly, and will quickly reveal if the item is potentially valuable (and relevant) or objectively a candidate for rejection.

The conversation goes something like this:

Collaborator Corey: Since we are surveying our target market anyway, let's ask some questions to see if they like our new packaging.

Researcher Ricky: Is there a specific hypothesis you would like to test about the packaging?

Collaborator Corey: I want to find out if they think it is a big improvement.

Researcher Ricky: OK, so you want to test the hypothesis that "members of the target market feel the new packaging is a big improvement over the previous packaging." Is that correct?

Collaborator Corey: Yes!

Then, with the specificity of a hypothesis identified, there are various likely outcomes; here are two:

Version A: Researcher Ricky: Interesting, and I see how that would be valuable. However, this requires that we have survey participants who 1) have purchased your product before the packaging change and 2) have at least seen the new packaging. Given the screening criteria for the main part of this project (if true), it is unlikely we would have enough respondents who will be able to answer these questions, so I suggest saving this for a separate research effort with appropriately screened people and saving our survey duration time for the primary objectives.

Version B: Researcher Ricky: Interesting, and I think it fits nicely with our current sampling plan (if true). But if this information is only useful if you can ask follow-up questions to find out why (or why not) they feel it is an improvement, we realistically would be adding at least three or four questions—not just one. And that means other content would have to be removed in order to keep to our 10-minute duration. Is that a worthwhile tradeoff for you?

But, if Collaborator Corey had said, "Actually, no, that's not my hypothesis. I just want to hear what they really think about the new packaging."

Researcher Ricky: It sounds like what you really want is to discover a variety of reactions people may be having to the new packaging, and that you want to hear reactions that may be positive, negative, or neutral. If that is the case, adding one question to the survey isn't going to be sufficient. You would be better off using some form of qualitative research to really discover and explore reactions. This will be more open-ended, and you may find packaging perceptions that you didn't expect.

Now, the collaborator feels heard, and the researcher was able to make an objectively sound recommendation.

Using Hypothesis Thinking to Wrangle Collaborators

Our collaborators may not be accustomed to thinking in terms of hypotheses, but it is a great way to get them to think precisely enough so that you, as the research professional, can guide the decision on whether or not to add the additional content to the questionnaire. A great way to coach them into hypothesis thinking is by sharing examples. Here are five examples you can use to help them reframe their questions as hypotheses:

1. Hypothesis: Awareness of our brand is dramatically higher among millennials than baby boomers.
2. Hypothesis: We can charge a premium for Product X if it comes bundled with Product Y.
3. Hypothesis: Sales of New Product A are lackluster because our distribution partners have not had enough training to demonstrate it effectively for the target market.
4. Hypothesis: People with senior dogs are willing to spend more on dog food than people with younger dogs.
5. Hypothesis: People who perceive our brand as trustworthy are more likely than others to have repurchase plans within the next two months.

Constructive Collaboration for Survey Research

Using hypotheses allows the market researcher to objectively assess questionnaire addition requests. This approach helps researchers manage collaborators' suggestions effectively, maintaining the survey's primary objectives and duration limits while considering valuable new ideas.