What are some common pitfalls to avoid when writing distractors for educational assessments?

Here are some common pitfalls associated with using implausible distractors: Lack of Relevance: Implausible distractors may be completely unrelated to the question or concept being assessed. This can confuse test-takers and make it challenging for them to discern the correct answer. Failure to Consider Common Misconceptions: Effective distractors should reflect common misunderstandings or misconceptions related to the content being assessed. Ignoring these common errors can lead to a biased assessment of the test-taker's knowledge. Unbalanced Difficulty: Implausible distractors that are significantly easier or harder than the correct answer can bias the results and provide an inaccurate assessment of the test-taker's proficiency.

Base distractors on common student mistakes or misconceptions. Analyze past test data or ask instructors what errors students frequently make. Make sure distractors are plausible but incorrect answers. They should seem reasonable to students who do not know the material well. Avoid outlandish or bizarre options. Distractors should be specific and unambiguous. Vague options like "all of the above" or "none of the above" are poor distractors. Distractors should be homogeneous in content and grammatical structure. They should "fit" the stem stylistically. Avoid giving clues like making one distractor noticeably longer or more specific. Distractors should be equally plausible.

Loss of Validity: Tests with overlapping or contradictory distractors may lack validity, as they do not effectively measure the construct or skill they are intended to assess. This undermines the credibility of the assessment results. Decreased Discrimination: Overlapping or contradictory distractors reduce the ability of the test to discriminate between students with different levels of knowledge or skill. As a result, the test may fail to accurately measure the intended learning outcomes. Ambiguity: Distractors that overlap in meaning or contradict each other can create confusion for test-takers, making it difficult for them to discern the correct answer. This ambiguity can lead to inaccurate assessments of their knowledge or skills.

Ensure each distractor is clearly distinct from the correct answer. Avoid partial truths or overlap. Double check that distractors do not contradict or negate each other. This creates ambiguity. If the question contains "all of the above" or "none of the above" options, verify they are truly applicable. Watch for vague language or broad statements that could overlap multiple options. Make each distractor clearly defined. Don't use opposites as distractors (e.g. "true/false", "yes/no"). Avoid distractors that differ from the correct answer by only one word or phrase. The options should be more substantively different. Have a colleague review the question to catch any unintended overlaps or contradictions you may have missed.

Here are some common pitfalls associated with using clues in distractors: Reduced Validity: Distractors that contain clues or cues may unintentionally guide test-takers towards the correct answer, rather than assessing their actual understanding of the material. This reduces the validity of the assessment results. Compromised Reliability: Assessments with distractors containing clues or cues may have compromised reliability, as they do not consistently measure the intended construct or skill. This undermines the trustworthiness of the assessment results. Unintentional Bias: Distractors that inadvertently provide clues or cues may introduce unintentional bias into the assessment process, favoring certain groups of test-takers over others.

Ensure logical consistency between the stem and each distractor. Illogical distractors are often easily eliminated Do not use absolute words like "never", "always", "impossible" unless justified. They often signal incorrect options Be wary of qualifiers like "usually", "typically", "generally" as they can subtly cue the right or wrong choice Avoid length or formatting clues - e.g. one distractor being a sentence vs a phrase Have a peer review questions and flag unintended cues based on wording, grammar, or logic Read the stem independently from the options first. Then read each distractor independently before comparing them. This can reveal inadvertent clues Avoid cloze-style stems that set up keywords matching with the correct answer

I cannot emphasize enough the importance of avoiding biased or insensitive distractors. In my experience, even subtle stereotypes or assumptions in distractors can negatively impact student performance and violate ethical standards. My advice is to seek input from a diverse group of reviewers who can identify blind spots. I also recommend proactively selecting distractors that counter stereotypes. Assessment is a powerful platform for promoting inclusion. We must be vigilant against unconscious bias creeping into distractors and undermining our ability to support all students equitably. Our role as experts demands we uphold the highest standards for fairness and social justice in assessment.

Avoid assumptions about social roles, backgrounds or interests that pigeonhole certain groups. Use language that is inclusive and respectful. For example, use "they" instead of "he/she". Ensure representations reflect diverse people, perspectives and contributions. Have others review questions to identify potential bias - their feedback can be invaluable. Reference content guidelines to adhere to standards for diversity, equity and inclusion. Remove or rewrite dated options with insensitive, offensive or prejudicial content. Focus distractors on knowledge and skills, not personal attributes irrelevant to the concepts.

Cultural Bias: Distractors that reflect specific cultural perspectives or experiences may disadvantage test-takers from different cultural backgrounds. This can lead to unfair assessment results and contribute to inequities in education. Gender Bias: Distractors that reinforce gender stereotypes or assumptions may unfairly advantage or disadvantage test-takers based on their gender identity. This can lead to biased assessment outcomes and perpetuate gender inequalities in education. Socioeconomic Bias: Distractors that assume a certain level of socioeconomic status or privilege may unfairly disadvantage test-takers from lower socioeconomic backgrounds. This can perpetuate inequalities in education and limit opportunities for advancement.

Too Many Distractors: Overwhelms Test-Takers: Providing too many distractors can overwhelm test-takers and make it difficult for them to identify the correct answer, especially if the distractors are plausible alternatives. Increases Guessing Probability: With more distractors to choose from, test-takers have a higher probability of guessing the correct answer, which undermines the validity of the assessment results. Too Few Distractors: Reduces Test Reliability: Providing too few distractors limits the test's ability to accurately measure the test-taker's understanding or skill level. It reduces the reliability of the assessment results. Decreases Item Discrimination: It becomes easier for test-takers to guess the answer.

3 distractors is often optimal. This provides sufficient options without overly increasing reading demands. 2 distractors may be appropriate for lower level learners or when assessing recall of facts. 4 distractors can work well if they are all equally plausible. More than 4-5 can allow testwise students to eliminate options through cueing or logic. Ensure each distractor plausibly appeals to some students. Avoid non-functioning distractors that are never chosen. Consider item content and complexity when deciding on number of options. Higher order items may need more distractors. Formatting can support additional options (e.g. question clusters, matching columns) without increasing the item reading load.

Inaccuracy: Distractors that contain errors or inaccuracies can mislead test-takers and provide them with incorrect information. This compromises the validity of the assessment and undermines its ability to accurately measure knowledge or skills. Ambiguity: Distractors that are ambiguous or unclear can confuse test-takers and make it difficult for them to discern the correct answer. This leads to unreliable assessment results and may not accurately reflect the test-taker's understanding. Biased Content: Distractors that contain biased or insensitive language may disadvantage certain groups of test-takers and lead to unfair assessment outcomes. Failing to review and revise distractors for bias can perpetuate inequalities in education.