Survey Says ...

“We conducted a survey with 100 people and got their answers to the following: Name a famous George” That was the first question ever asked on the wildly popular TV game show Family Feud. This show was based on surveys. The purpose was to generate excitement, surprises, and fun.

Surveys date back nearly 6000 years. The first known survey was a census conducted by the Babylonians. They counted not only people, but possessions as well, including livestock, butter, milk, and more. Later came the Roman census. Theirs is considered the most developed of any recorded in the ancient world. In fact, the word census comes from the Latin word “censere” which means “estimate”. 

Even though surveys are the oldest organized technique for understanding people’s opinions in mass, they are still in widespread use today. Age may be in part, one reason why we at D2 have heard prominent business executives, and seen numerous articles that claim surveys are no-good. Further, there are newer and better tools that can and should be used in their place. 

Are surveys useful anymore?

Surveys are usually quick to plan, inexpensive to develop and distribute, and typically don’t take advanced statistical knowledge to analyze. When architected by an experienced behavioral scientist, a custom crafted survey will hit your schedule and budget requirements. In addition, when the survey is deployed to the right demographic with properly engaging questions, valuable information will be gained that will lead to the accomplishment of your original goals.

A survey gives you self-report data from participants, usually in the form of ratings scales and detailed opinions on open-ended questions. They produce simple datasets to help researchers make decisions. Ratings scales give quantifiable data to different questions and can help examine group differences, strengths of correlation, and other useful statistical measures. Open-ended questions help give context to numerical data and help shape questions for future studies.

At D2, we believe that surveys, used properly, are still a valuable tool for listening to your audience. They certainly aren’t the only tool and not always the best one to use when gathering information and predicting behavior. But all newer techniques can trace their lineage back to the mighty survey. If what you are after is a simple, quick, cost-effective, and directional understanding of the opinions of a specific demographic, a survey could very well be the way to go.

So, what exactly is a survey?

That is a simple question. The answer is not so straight forward. Basically, a survey is a method of gathering, compiling, and assessing information from a specific sample group of people with the intention of extrapolating that information to a much larger population. Generally, participants answer questions, giving data in a self-reporting manner. Some of the data is subjective, such as thoughts, feelings, or opinions, which people are reasonably good at estimating. It’s also one of the only ways to get this kind of information. Other data is objective, such as self-reported behaviors and factual data, such as age, location, etc.

There are several types of surveys based on deployment method. Some of these include face-to-face, telephone, paper and pencil, and computer-based. Further, there are three types of surveys based on the frequency of administration. These are cross-sectional, longitudinal, and retrospective surveys. Which type to use depends on the population and topic of interest. These formats facilitate analysis across different groups and/or at different points of time.

What are the limitations of surveys?

Surveys have several real limitations. The self-reporting nature of surveys means that they are completely dependent on the ability of participants to accurately evaluate their own thoughts, feelings, and opinions. While people are reasonably good at that, they are not always perfect. Also, surveys generally have no mechanism to let participants make adjustments after the survey has been completed and submitted.

Surveys are static in nature, meaning that the number of questions and wording of them are fixed at the time of deployment. This is a limitation because participants may answer a question with some new and valuable information, but because of the static nature, the survey cannot explore the new finding further at that time. Finally, surveys don’t measure real behaviors. People, self-reporting on their own behaviors, are not always the best at capturing reality.

Is it too easy to create and launch surveys?

Online survey platforms, such as Survey Monkey, have made the creation and distribution of surveys accessible to everyone. But, just because a tool is cheap and easy to use, does not mean that it will be effective in everyone’s hands. In the Forbes article “7 Deadly Sins of Online Surveys,” Paul Talbot wrote that marketers might be getting to the point where they want to shout, “Get that Survey Monkey off my back.” Poorly conducted surveys can lead people to make bad decisions. But Survey Monkey is not to blame. They provide many tools, tips, and training for users of their survey tool. Unfortunately, Survey Monkey does not prequalify users. Neither, do they check surveys for validity or accuracy. This leaves a wide open, wild west possibility of survey usage. This is what creates the problem. The survey designer needs to be knowledgeable, experienced, and objective. Without knowing who created the survey, who designed the questions, what group was surveyed, and for what purpose the survey was intended, it is prudent to question the validity of that online survey.

You need to do your homework

Don’t be afraid to ask for help! In order to get useful results, it requires the selection of the right research method and the picking of the proper platform for the problem at hand. In order to get useful results, you need to make certain that your research is valid and reliable. This means that your data needs to be accurate and applicable to your population of interest.

It seems everyone knows that the survey-sample-size plays a role in the validity and reliability of your research. While the act of simply increasing your sample size may lower your margin of error, that alone will not guarantee better overall results. In addition to selecting appropriate sample size, you need solid research and that requires someone with expertise to minimize common sources of error, who understands the assumptions with different statistical analyses, and can interpret the data and provide the most likely implications and even provide you with possible alternative explanations.

Done correctly, surveys are a very useful tool for you, even in this modern age. But, if not created properly, by an objective and experienced designer, the information collected can lead you astray, even when you already clearly know the information you are after. Survey questions that are well-written, clear, and focused allow your participants the freedom to think for themselves and give accurate answers. Poorly-written questions can cause confusion or bias participants toward certain answers.

One type of poorly worded question is called “The Leading Question”. Often, this happens accidentally because the survey designer simply doesn’t have the knowledge, skill, or experience to understand how questions and response options should be crafted. However, there are times that leading questions are created on purpose, to steer responses toward the desired outcome. Reasons for doing this might include gaining more budget for a project, affecting policy, or swaying management’s opinion. This type of survey creation is very dangerous. Here is one example of leading survey questions we have seen in a product documentation survey:

Do you want to help the environment and save trees?   (yes or no)

Do you really need a printed instruction manual?          (yes or no)

There are a few major issues with these questions. The initial “Leading Question” incident is the ordering of the questions. The first question doesn’t even belong in that type of survey, but it prompted the vast majority of participants to answer yes, because everyone wants to save trees. By the time participants read the second question, they felt guilty about answering yes to it, even if they did want a printed manual. The second question leads participants by using the exaggerating words “really need.”

At the end of the survey it was reported that well over 90% of participants did not want printed instruction manuals. Those two bad survey questions and the subsequent erroneous report led that company down an unfortunate path for their customers that took years to recover. This example of “the leading question” is just one type of poorly crafted survey question. There are many more.

So, when should you use a survey?

Surveys are a good learning tool to use when you want to reach a lot of people in a short amount of time, and at a lower cost. They are best used to get information from participants that only they would know (e.g. how they think or feel about something). Surveys are also useful for general estimations of behavior, or to have participants predict future choices.

As with most things, a survey is impactful if it’s useful for you and you have an interested audience based on some level of trust. All of that starts with a valid and reliable survey. If you don’t have that, then the results aren’t useful. If good data helps inform your decisions, it’s useful. Sometimes it means sharing the findings with others, either to inform or to market. When distributing results to others, you should try to build trust with the intended audience by being transparent about survey methods, and by describing what alternative explanations there could be for findings.

While surveys can be very useful, they are not a one-size-fits-all tool. They don’t measure actual behaviors. They can’t go into more depth than their original design intent. Finally, other tools measure things that surveys simply can’t.

But using surveys with the right goals in mind and for the right reasons, applied by the right expertise, will provide a very powerful tool in any researcher’s toolkit. That power needs to be put into the right hands. With so much of your future at stake, it is important to involve behavioral scientists who are experts in research, design, deployment, and analysis to ensure that your survey yields valid and impactful results.

We won’t keep you in suspense any longer - back to the first Family Feud question. The six most popular answers to: “Name a Famous George” were:

How many of those Georges would you call famous? Remember, that survey was conducted in the 1976. What would the same survey say today? 

Tony Olson is the founder and president of D2, a customer-focused, solutions-based company. From Startups to Fortune 500 companies, D2 has helped people market, develop, and patent their products in addition to helping develop internal systems and processes to provide a competitive advantage. 

Steven Garelik is a behavioral scientist with a PhD in human factors psychology and nine years of experience in market and academic research. Steven has worked on a variety of research studies for multiple Fortune 500 companies during his five years at D2.

To learn more about what D2 can do for you, click here.