The Importance of Pre-Testing in Evaluation: A Crucial Step for Effective Surveys (Part 1)

Pre-testing in survey design is a critical step that ensures the effectiveness and reliability of the questions posed to respondents. By running the survey with a small test group before distributing it to the entire sample, potential problems can be identified and corrected early. This practice increases the likelihood of gathering valuable information, such as feedback, suggestions, and constructive criticism. Additionally, pre-testing allows researchers to test the validity, reliability, practicality, and sensitivity of the data collection tool intended for the evaluation.

The aim of sharing these insights is to underscore the importance of thorough preparation and iterative improvement in survey methodologies. A well-prepared survey not only enhances the quality of the data collected but also respects the time and effort of participants, ensuring their contributions lead to meaningful results.

This blog delves into the theoretical elements of pre-testing a survey, the benefits it provides, and the practical strategies that can be employed. Part 2 of this blog will explain the application of the QAS 99 survey method, a systematic approach to identifying and resolving potential issues in survey questions.

Why Pre-Testing Matters

Pre-testing is crucial to avoid unforeseen problems during data collection and increase the likelihood of gathering valuable information, such as feedback, suggestions, and constructive criticism. It allows researchers to test the validity, reliability, practicality, and sensitivity of the data collection tool intended for the evaluation. Pretesting is beneficial for participants as well, ensuring their time and effort are not wasted on ineffective surveys.

Benefits of Pretesting a Survey

Pretesting helps:

• Ensure questions are clear, unambiguous, and not offensive or intrusive
• Identify difficulties and potential misunderstandings encountered by respondents
• Adjust the length and structure of the data collection process
• Gain initial insights into the problem based on preliminary responses

Preparing the Pretest Survey

In preparing the pretest survey for your project, focused on several key areas:

• Comprehension: Ensuring respondents understand the survey's purpose and questions
• Logic and flow: Structuring the questionnaire to follow a logical order, aligned with the project's objectives
• Acceptability: Formulating questions that do not offend or touch on sensitive subjects inappropriately
• Length and adherence: Acknowledging that a lengthy survey may cause fatigue but opting for comprehensiveness to gather feedback on its length during the pretest
• Technical quality: Ensuring high standards in all technical aspects
• Introduction and consent: Clearly stating the survey's aim and scope, seeking consent, and ensuring respondent anonymity

Strategies for Pretesting Surveys

There are various strategies for pretesting surveys, involving different actors and conditions. Such methods are:

1. Cognitive interviewing: Conducting face-to-face interviews with a small sample of respondents who think aloud while answering each question. This method helps identify question quality and drop-out risks.
2. Experts’ assessment: Engaging survey methodologists and topic experts to shape the content and form of the survey, ensuring better data quality.
3. Focus groups: Holding semi-structured discussions with 7-15 people to clarify survey concepts and evaluate respondent burden or sensitivity.
4. Experiments: Testing different variations of the survey design on multiple groups to understand the impact of methodological factors. Although ideal, this method was not feasible for our case study due to time and resource constraints.
5. Supplements to conventional pretests: Implementing additional methods like behavior coding, response latency, formal respondent debriefings, and vignettes to enhance conventional pretests.

Data Analysis and Improvements

Analyzing pretest data is crucial for learning and improving the survey instrument. Based on the pretest outcomes, we can refine the survey process and, if necessary, conduct a second round of pretesting to address new questions and doubts.

Conclusion

While not all evaluations pre-test their data collection methods, it is generally understood that anticipating all possible problems during data collection is impossible. Therefore, pre-testing the data collection instruments, such as questionnaires, is essential to minimize errors and ensure reliable and valid data collection.

?

Bibliography

????? Andrews, Frank. 1984. “Construct Validity and Error Components of Survey Measures.” Public Opinion Quarterly 48

????? https://academic.oup.com/poq/article/68/1/109/1855073

????? Baker, Reginald, Scott Crawford, and Janice Swineheart. 2004. “Development and Testing of Web Questionnaires.” In Methods for Testing and Evaluating Survey Questionnaires, ed. Stanley Presser, Jennifer M. Rothgeb, Mick P. Couper, Judith L. Lessler, Elizabeth Martin, Jean Martin, and Eleanor Singer. New York: Wiley.

????? Belson, William. 1981. The Design and Understanding of Survey Questions. London: Gower

????? https://www.betterevaluation.org/methods-approaches/approaches/most-significant-change

????? Dworkin, S.L. Sample Size Policy for Qualitative Studies Using In-Depth Interviews. Arch Sex Behav 41, 1319–1320 (2012). https://doi.org/10.1007/s10508-012-0016-6

????? Graesser, Art, Katja Wiemer-Hastings, Peter Wiemer-Hastings, and Roger Kreuz. 2000. “The Gold Standard of Question Quality on Surveys: Experts, Computer Tools, versus Statistical Indices.” Proceedings of the Section on Survey Research Methods of the American Statistical Association

????? ?Kaplowitz, Michael, Frank Lupi and John P. Hoehn. 2004. “Multiple Methods for Developing and Evaluating a Stated-Choice Questionnaire to Value Wetlands.” In Methods for Testing and Evaluating Survey Questionnaires, ed. Stanley Presser, Jennifer M. Rothgeb, Mick P. Couper, Judith L. Lessler, Elizabeth Martin, Jean Martin, and Eleanor Singer. New York: Wiley

????? ?https://www.qualtrics.com/blog/6-ways-to-pretest-your-survey-before-you-send-it/