So, You Want To Do A Survey...

I have had the pleasure of taking the lead on, and contributing to, a number of rather large and high profile surveys, mainly C-level surveys (CIO, CEO etc.). I learned how difficult it is to do a good survey. I have learned that there is only a small difference between a survey that results in not much useful output, and one that results in a lot of great output. Here are some thoughts on how to do it well. They are not in any way a complete list, just some things I learned along the way. They may be biased towards the types of survey I have been involved with. Nevertheless, I hope you find them useful.

1. Access survey design expertise, don't wing it. If you haven’t done one before, there are lots of subtleties you might not think of. For example rotating the question order, so that the order of questions isn’t a factor in how people answer, and if people do drop off they don’t always miss out the same questions.
2. Access statistical expertise to make sure the way you interpret/ report the results is valid. There are lots of subtleties here too. For example, if you want to be able to ‘slice and dice’ the results, reporting results for particular geographies/ industries etc., how many responses do you need for results to be seen as credible? What confidence level do you need to report results. How do you create an average when you ask respondents to choose from a range (e.g. 1-5%, 5-10%, 10-25%, 25-100%). How do you handle open-ended ranges (e.g. >$1bn) in averages.
3. Start with the end in mind. What are you doing the survey for? What will the outcomes and outputs be. Will you be using the survey to create an interesting report around a theme (e.g. food security, or cloud computing usage), or to make a big investment decision (e.g. whether to launch a new product). Envisage the outcomes and outputs as clearly as possible before starting to design your survey. It is also at this point a good time to reconfirm whether a survey is the best way of achieving your goals. Could focus groups be better? Or, dare I say it, just publishing your opinions without supporting data? Or some other methodology?
4. It is particularly important to be sure whether you are trying to find out/ prove/ something with a high degree of confidence, or just get some directional ideas that you can use/ share. Proving things is very hard, and requires you to be incredibly rigorous, which normally takes a lot of time, effort, numbers of respondents and money. You can probably cover 5-10 times as much ground with a ‘directional’ survey than you can with a ‘proof-focused’ one. But sometimes, you need to be sure.
5. In my experience, a great way to use a survey is to test the validity of a few hypotheses. For example, if you believe the reason that potential customers aren’t buying your products in Africa is because of price alone, you can use a series of questions to check if price is a key issue, and also to rule out other issues. This can be a really effective way of using survey questions. If you are very clever, you can use one question to support multiple hypotheses (survey synergy). The negative opposite of this is a scattergun of unrelated questions – where you are not clear what you will do with the output of each question (other than create a pretty chart).
6. Think about objections in advance. Pretend to be a cynical reader of the survey results. For example, people know that survey respondents often reply aspirationally (i.e. reporting how they would like to behave, not how they actually behave). Pretend to be someone who is negatively affected by the results of your survey, and might try to sue you. Factor these things in to what you believe you can get out of the survey, and hence your survey design.
7. Be clear who you want to participate, then target them. Make sure you are using the right channels to target them. Ensure this includes considerations about geography, industry, size, role, success/failure etc. For example WeChat could be a great tool for surveys in China. If you are targeting CEOs, and send out a request to participate via twitter, there is a good chance that you will get 99% non-CEOs, and will have to reject nearly all people who show interest - which doesn’t leave a great impression, or get a great yield.
8. Make sure that you haven’t got unhelpful biases in the respondent base. If you ask all your happy customers how good you are, you will get a biased result. If you send out a customer survey, you will probably get responses more from customers on the extremes (very happy with you/ very angry with you). If you force people to respond in English, you will bias your audience. If you use big, difficult words, you might intimidate less educated people. If you make a long survey, you might get less successful people with more time on their hands. Etc. etc. It is critical to think this through. It is important in all cases, but particularly if you are going to draw conclusions that might be contentious, e.g. “80% of people hate Ford cars.” (Just an example, not real (as far as I know)).
9. Ensure that there is an appropriate WIIFM (What’s In It For Me?) What do participants get out of it? Could be early sight of the results, or a free copy of a report, or a small gift? Do they value that enough to participate? Will it in any way skew who chooses to participate or how they answer?
10. Signal as clearly as possible to respondents how you are going to use the data, especially if respondents supply their/ their company’s name, to allay respondents’ fears. You may not always be able to tell them exactly how, e.g. if you think it will skew their answers, or if you want to reserve the right to use the results in different ways in future, but tell them as much as you can.
11. Think really hard about whether survey respondents are treated anonymously or not. If they have to give their name or their company's name, and they know you might use it publicly, they might not participate or might be very sensitive to what kind of questions they will answer. If they have to give their name, but trust you won’t reveal it, you can use non-survey data (e.g. company annual reports) to enrich the data. If it is completely anonymous from start to end, respondents might feel free to participate/ give answers, but you have no way of validating their responses, combining with non-survey data, or filtering based on who they are.
12. Don’t make the survey too big. If you know/ feel that your target respondents will gladly give you 10 minutes of their time, don’t create a 20 minute survey, no matter how many good questions you have. If you make your survey too long, or too hard, or too sensitive, you will get lots of poor quality answers, and lots of people dropping out before the end. And it may damage your brand/ ability to do other surveys in future.
13. One way of helping with length is making only a few questions mandatory, and the rest optional. Consider having a small core of questions at the start, then give people the option to answer more questions if they want to/ have time.
14. Multiple choice questions sometimes give you less information, but are much easier for the respondent.
15. Consider forking – i.e. not asking every respondent every question. You can fork randomly (i.e. split the survey into sections A, B, C, and randomly direct each respondent to one of the 3), or you can fork based on responses - e.g. if someone responds that they have a Chief Digital Officer in their company, you can ask them about that; if they don't, you can ask them about something else. This can help with survey length, and also allow you to get some deep insight on specific topics. But remember it will have an impact on how you interpret the survey results, and reduce the numbers who answer each question.
16. To reduce drop-off (respondents failing to complete the survey), leave sensitive questions and questions that require a lot of thought/ textual input towards the end helps too. You can also offer them inducements to finish - e.g. early sight of the results after they finish.
17. Keep the door open at the end with a question like “Anything else you wanted to add?”, “Can we contact you to discuss further?”
18. Test the survey in advance of doing the full launch. This will help you find flaws and calibrate the time it actually takes to fill it out.
19. Make sure you interpret the results correctly. If you are looking to show correlation, don’t forget time lags in the system. Investments this year don’t necessarily map to results this year. And remember that correlation isn’t causality.
20. Ensure that you are saying something interesting in the results. It may not be that interesting to hear that business leaders care about growing revenues and reducing costs. Use the anti-strategy principle to avoid stating the bleeding obvious. (If there isn’t a valid alternative, it isn’t worth saying.)
21. Remember that you are not duty bound to use all the results, and you certainly don't have to use them all in one report/ press release. Sometimes less is more. Infographics with a few big findings can be much more compelling than a detailed, data-heavy report.
22. Remember that everything has an intrinsic value and a signalling value. Doing a survey also sends signals. If respondents know the survey is from you (i.e. you are not conducting it anonymously), the questions you ask will also shape respondents view of you. Asking a question is not neutral.