How can you use surveys effectively for data collection?

In the context of science education research, a common mistake researchers make when defining their research question and goals for surveys is: Neglecting the Practical Implications. Researchers sometimes fail to consider how the survey data will inform practical implications or address real-world problems in science education. Researchers can reflect on the practical implications of their research question and goals. They can consider how the survey findings can be used to improve science education practices, curriculum development, or teacher training. By aligning their goals with practical implications, researchers can ensure their survey data has tangible value and impact.

A clear yes/no question has a lower cognitive load to the respondents, which helps to have a higher rate of answers. However, it is not as informative to the researcher as a question containing a wider range of alternatives. For example: 1. Do you drink coffee? a. Yes b. No 2. How many cups of coffee do you drink in a day, approximately? a. None b. 1 c. 2 d. 3 e. 4 f. 5 or more To answer question 2 you need to think a little bit more than question 1 (if you drink coffee), but the researcher can gather more information from the data obtained via question 2 compared to 1. You should consider the tradeoff between more engagement in your answers and a higher granularity of them. A mix of both types could reach better overall results.

When it comes to testing and refining surveys, two common mistakes researchers make, particularly in the context of science education research: 1) Insufficient Pilot Testing and 2) Ignoring Tester Feedback. Researchers may overlook the importance of conducting pilot testing with a small sample of the target population or experts in the field or fail to effectively utilize or address feedback. They may skip this step and directly distribute the survey to a larger audience, missing the opportunity to identify and address potential issues or errors in the survey design. Instead, they could carefully analyze the feedback, identify common themes or concerns, and use that information to make necessary revisions to the survey.

Before starting the field work of collecting get a very experienced statistician to independently examine the collection methodology and discuss your aims and the hypothesis you are testing. He/ she may say for example there is no need to collect so many of one metric or you need to collect just one more to make it statistically sufficient. It is a little frustrating to have collected data and to discover you could have done a little more or much less to achieve the same robustness of result. It must be a full time statistician, someone who lives in the subject. As one statistician said to me once "too many researchers think they know statistics because they have done a course or two. These are the over confident ones who mess up."

Anonymous surveys are more confidential and easy to distribute online. Use them to gather preliminary data for your pilot or minimal cross sectional studies. These surveys give you some understanding of what you want to eventually investigate in a defined population study, in addition to the response rate, quality of responses, timing etc. You can also use anonymous surveys to opt in/opt out potential participants. Once you get your participants enrolled, you should be ready to do a nuanced and focused survey-based study serving your research questions and hypotheses. You also have the opportunity to do a longitudinal study which can inform you in many ways.

You also may want to consider an alternative data collection strategy. Many people have significant “survey fatigue.” Also, be sure to clarify issues around confidentiality and about how the respondents information will be used and stored.