Survey Design Theory - Review

A very important part of CRO along with A/B testing is doing qualitative research on your visitors and customers to get answers to that "why". Of course, you can figure out everything going wrong in your funnels with a simple Analytics check but would it tell you why some page has a low engagement or high bounce rates? It can't but the people who are bouncing off and not engaging can. That's where a survey comes into play.

By definition, a survey is a sub-sample or cross-sample of the general population. These are pretty effective because they are inexpensive and have an error rate of only +/-5%.

Benefits of Surveys:

1. Understand attitude towards our brand.
2. Stay aware of competitors
3. Evaluate new products
4. Understanding customers

The data collected in these kinds of surveys is both qualitative and quantitative.

Coding Qualitative and Quantitative Questions:

Quantitative questions can be coded easily in close-ended formats.

For coding qualitative questions, a lot of work is required. It can be done structurally through the technique of Zero-sum analysis as explained by the CXL Institute.

Here's how to use zero-sum analysis:

1. Collecting qualitative answers
2. Recoding and organizing based on the commonality of response through some insightful keywords.
3. Doing quantitative analysis on the qualitative data.

Note that no survey data is utterly qualitative. Though surveys are considered a part of the qualitative research methodology, the results are always a mixture of qualitative and quantitative data.

Survey Design Strategies:

There are some rules to follow while designing a survey to come up with quality questions that drive quality answers:

1. Questions must be framed in a close-ended format. For qualitative answers, they should become boxes to comment on once the user has clicked on the answer to expand upon it providing you quality insights.
2. The questions should be non-leading. Eg - "How was it?" instead of "How good was it?"
3. The questions should address the desired information directly without wasting much time on unimportant questions. Ask only what you absolutely want to know!
4. Keep the sample size large. The larger the size, the lesser the chances of getting errors. A sample size of anywhere between 200-250 is good while anything less than 100 responses might come out inaccurate.

Common Mistakes to avoid in Survey Design:

1. Non-intuitive scaling: Sometimes marketers get too creative with their surveys and get inclined to illogical framing of questions like changing the way scaling is to be done. People are used to thinking of the best thing as 5 stars and the worst as 1. Don't try to turn it the other way round in your surveys either. Keep the highest value 5 and the lowest as 1.
2. Mixing questions of behavior with questions of attitude: Always ask behavior questions first and then start asking attitude questions. A behavior question can be "What did you eat?" while an attitude question enquires about the ingredients of that food.
3. Questions that don't communicate: If the questions won't have any relevance to your target group and you use jargon or verbiage that they don't understand, there isn't going to be much quality in the answers that you get.
4. Long Surveys: Whenever your survey is longer than 15 questions, a central tendency kicks in the mind of the user and they would be choosing the options in the center of the scale instead of thinking and answering.
5. Not keeping a Neutral Learning Curve: As users start answering a survey they might become smart with their answers and provide incorrect information. Questions have to be framed with a neutral learning curve to avoid such a tendency.

Conducting In-house Surveys:

In-house surveys stand for the surveys that you conduct with your audience and not with site visitors. These are the people who have done business with you at least once.

Things to remember when conducting in-house surveys:

1. Remember that based on their experience, the existing customers will always be negatively or positively biased towards your brand. Either they like you too much or they don't at all.
2. Keep it short up to 10-15 questions.
3. The sample size should be kept at a minimum of 100 responses.
4. Don't over-assume that if they have done business with you, they are very brand aware and know your business well. Do put questions about their brand knowledge.

Cognitive Biases

Humans are inclined to be biased. Whether it's you who designs the survey or the users you are surveying, one or the other cognitive bias from your end or theirs might affect the accuracy of the results.

In Survey Design:

1. Reading the Room: When the researcher tells the client that what they believe the client wants to hear.
2. Order Bias: Items listed higher in the survey tend to get better scores. To avoid this, you should be mixing up the questions every time it appears in front of a new user. That should be the design and order principle guiding you to avoid such biases.

User's Biases:

1. Sometimes not informing the customers of what they are expecting in the survey might lead to disruption. Debrief them ahead of time 'here's what to expect in the survey'.
2. Write a jargon-free summary of the survey so that they understand everything. Seeing some heavy technical words before the survey has even started will put many people off.
3. Be aware of selective perception. This is the tendency of users to pay more attention to what they already agree with.

Tip: A good way of choosing the language you use in your surveys is to get to know the voice-of-customer. What vernacular they have? What's their verbiage? Use the same language in your surveys and see how people happily people answer it for you.

Conclusion:

A survey is the best way to drive solutions to various complex questions you might have to get to know what's working well on the site and the business and what's not. If there's optimization required, there's no way you can complete your CRO research without conducting some web and email surveys with your customers. I realized that though it might be sometimes difficult to get the most responses and quality insights, with a little bit of effort and the application of these best practices, it's almost possible to get really accurate data on your surveys which drives your optimization strategy further.