When you design a questionnaire, you may want to collect data from different modes, such as online, phone, mail, or face-to-face. However, different modes may affect how respondents understand and answer your questions. How can you ensure that your questions are consistent and comparable across modes? Here are some best practices to follow.
Avoid words or phrases that may imply a specific mode or context, such as "click", "press", "see", or "hear". For example, instead of asking "How often do you click on ads when you browse the internet?", ask "How often do you encounter ads when you browse the internet?". Use simple and clear language that can be easily read or spoken.
If you are making small changes to accommodate the different modes you have to make sure you are not changing the meaning of the question. In the example above, asking "how often do you click on ads when you browse the internet?" and asking "how often do you encounter ads when you browse the internet?" are actually asking about two different behaviors (click vs encounter). The results will not be compatible between modes since they are asking different things.
And, you'll want to check data in the backend before launching to ensure you have consistent variables and codes for the same question across modes, in the event you want to compare results by mode. Make it good practice to always check the variables and how the data is being entered on the back end before launch. Know how you're going to analyze it first.
To design consistent and comparable questions across different modes, adhere to several best practices. Begin by using clear and concise language that is easily understood across various mediums. Ensure questions are phrased identically across modes to maintain consistency in interpretation. Test questions rigorously to identify any mode-specific issues, such as formatting issues in online surveys or pronunciation difficulties in phone interviews. Provide consistent response options and scales across modes to facilitate comparability. Additionally, consider mode-specific features or limitations when designing questions, such as screen size for mobile surveys or audio clarity for phone interviews.
Different modes may have different limitations or advantages for presenting questions and response options. For example, online surveys can use visual aids, such as images or sliders, while phone surveys can use verbal cues, such as repeating or spelling options. Adapt your question format and layout to suit each mode, but keep the meaning and logic of the questions consistent. For example, if you use a Likert scale online, use the same number and labels of points on the phone.
Question order and length may affect how respondents process and recall information, especially in modes that require more cognitive effort, such as phone or face-to-face. To reduce the risk of order effects or memory errors, keep your question order and length consistent across modes, or use randomization or rotation techniques. For example, if you ask about several topics or attributes, use the same order or randomly assign the order for each mode.
Before you launch your questionnaire, test and pilot your questions on each mode to check for any problems or differences in comprehension, response, or quality. Use methods such as cognitive interviewing, usability testing, or split-ballot experiments to identify and fix any issues or biases. For example, you can ask respondents to think aloud while answering your questions, or compare the results of different versions of your questions across modes.
Even after you launch your questionnaire, you may need to monitor and adjust your data collection process to ensure that your questions are consistent and comparable across modes. Use methods such as data quality checks, response rate analysis, or mode preference surveys to track and evaluate any changes or challenges. For example, you can check for any missing or invalid data, or ask respondents why they chose a certain mode or switched modes.
Finally, when you analyze and report your data, you need to account for any potential mode effects or differences that may affect your findings or conclusions. Use methods such as weighting, imputation, or multivariate analysis to adjust or control for any mode-related factors, such as sample composition, measurement error, or nonresponse bias. For example, you can weight your data by mode to reflect the population distribution, or use a mixed-mode model to estimate the mode effects.