What are the limitations of using surveys for security awareness assessments?

1 Bias and validity

One of the main challenges of using surveys for security awareness assessments is to ensure that the questions are unbiased, relevant, and valid. Bias can arise from the wording, format, order, or context of the questions, as well as from the respondents' personal preferences, motivations, or social pressures. For example, a question that asks about the frequency of security breaches may elicit different answers depending on how the term "breach" is defined, or whether the respondents feel comfortable reporting such incidents. Validity refers to the extent to which the questions measure what they are intended to measure, and whether they reflect the actual level of security awareness among the target population. For example, a question that asks about the knowledge of security policies may not capture the actual behavior or attitude of the respondents in practice. To address these issues, surveys should be designed and tested carefully, using clear and consistent terms, avoiding leading or ambiguous questions, and providing adequate options and scales for responses.

2 Response rate and representativeness

Another challenge of using surveys for security awareness assessments is to obtain a sufficient and representative sample of respondents. Response rate is the percentage of people who complete and return the survey, and it can be affected by various factors, such as the length, timing, mode, and incentive of the survey, as well as the interest, availability, and accessibility of the potential respondents. A low response rate can reduce the reliability and generalizability of the survey results, as it may introduce non-response bias, which occurs when the respondents differ from the non-respondents in some relevant characteristics. For example, a survey that is sent by email may exclude those who do not have regular access to the internet, or those who are more likely to ignore or delete such messages. Representativeness is the degree to which the respondents reflect the diversity and distribution of the target population, and it can be affected by the sampling method, size, and criteria. A non-representative sample can also introduce selection bias, which occurs when some groups are over- or under-represented in the survey. For example, a survey that is conducted at a specific location or time may exclude those who work in different shifts, departments, or locations. To address these issues, surveys should be distributed and promoted widely, using multiple and appropriate channels, and offering reasonable and ethical incentives. Moreover, surveys should use random or stratified sampling techniques, and adjust the sample size and weight according to the population characteristics.

3 Data quality and analysis

A final challenge of using surveys for security awareness assessments is to ensure that the data collected are accurate, complete, and meaningful. Data quality can be affected by the respondents' honesty, comprehension, memory, and attention, as well as by the survey administration, processing, and storage. For example, a respondent may provide false, incomplete, or inconsistent answers due to fear, confusion, forgetfulness, or fatigue, or a survey may be lost, damaged, or altered during the collection, entry, or transfer. Data analysis can be affected by the choice, application, and interpretation of the statistical methods, tools, and techniques, as well as by the presentation, communication, and dissemination of the findings. For example, a data analyst may use inappropriate, outdated, or erroneous methods, tools, or techniques to analyze the data, or may present, communicate, or disseminate the findings in a misleading, confusing, or biased manner. To address these issues, surveys should provide clear and concise instructions, definitions, and examples, and use verification and validation checks to detect and correct errors or anomalies. Furthermore, surveys should use appropriate, updated, and reliable methods, tools, and techniques to analyze the data, and present, communicate, and disseminate the findings in a transparent, accurate, and ethical manner.

4 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?