Bad Qs, the bar where your choice always matters!

What do you think the correct answer is?

Welcome to Bad Qs, the bar where your choice always matters! You may have seen this funny multiple-choice question as an ice-breaker within learning geek meetings. I'll come back to the solution at the end of this article. But for now, let's look at a real example.

Prepping for the AWS Certified Cloud Practitioner certification, I ran across this example.

You may not know the answer to the question, so let's simplify this to a general dilemma:

Which of the following options is TRUE?

A) TRUE

B) TRUE

C) TRUE

D) TRUE

E) All the above

What would be your choice? A, B, C, D, or E?

The problems with "All the above"

If a multiple-choice question has 5 options to choose from, and you're not using "all the above," the theoretical odds to make the right choice 1 in 5 (20% chance). In practice, psychometrics has various science-based approaches to determine a more realistic value because not all options are equal :) .

The moment you replace the last option with "All the above," you make this a much easier game. As long as you can identify two TRUE answers, you know that the expected answer is E. As long as you identify one single answer that you know FALSE for sure, you eliminated option E as well.

But, there's even a bigger issue you should be aware of. If this test is a high-stake test in your organization (certificate, legal, etc.), and you allow participants to contest questions, you will be likely to see this argument:

"Selecting a TRUE answer is not the same as NOT selecting a FALSE answer."

What does that mean? As long as the selected option SATISFIES the question stem, it doesn't matter whether there BETTER options. It all depends on how you word the question stem or how explicitly you instructed users to pick the BEST option.!!

Which of the following options is TRUE?

A) TRUE

B) TRUE

C) TRUE

D) TRUE

E) All the above

By selecting A, for example, I can argue that the stem ("Which of the following options is TRUE?") will be perfectly satisfied with the option. Is option A TRUE? Yes. Correct. I'm not saying other options are FALSE, I'm saying option A is a correct answer.

Lessons learned: avoid using all the above unless it's absolutely necessary. But if you do, you must word the question stem to make ONLY ONE of the options the correct by either ranking the correct answers or eliminating some of them.

For example, you may see question stems including words like "most appropriate" answer or "simplest" or "quickest" or "least expensive," etc. The proper wording suggests multiple options may satisfy the question stem, however, out of those, you're looking for one in particular. Context matters!

If you really have multiple correct answers, then just use multiple select. Speaking of multiple select, there are two ways to indicate the number of options to select: explicitly and implicitly. You can explicitly say "SELECT TWO." Or just imply that there are multiple correct answers by saying Select all that apply. There is a huge difference between these!

Let's say there are 5 options and 2 correct answers to select. If you explicitly say to select 2 options, the challenge comes down to 2 out of 5. There 10 different ways to pick 2 out of 5.

If you say "select all that apply:" 0 out of 5 + 1 out of 5 + 2 out of 5 + 3 out of 5 + 4 out of 5 + 5 out of 5. That is 32 different ways. And then you have to think about awarding partial points or not...

Back to our Bad Qs question!

What was your answer? A,B,C,D,E,F, or none of them?

A) Incorrect - It would make all options below correct. It conflicts with option E or F, for example. Also, if A is true, then B is true. And C-F would need to be both true and false at the same time...

B) Incorrect - It would make all options below incorrect. It conflicts with option D, for example.

C) Incorrect - This is more interesting. It claims that either A or B is correct but not both. That contradicts A because option A states both A and B are correct. And it contradicts B, because option B states that C is incorrect.

D) Incorrect - Option D can't be correct because, again, it would make both A and B correct at once. (Just a note: if you ever use "All of the above" or "None of the above," don't shuffle them! The order does matter!)

E) Correct - Check this out for yourself!

F) Incorrect - If option F is correct, then answers A-D are all incorrect. However, if that is true, option E is true as well. Oops. And if option E is incorrect, as stated by option F, then A-D can't be all incorrect.

Conclusion

A) Don't use All of the above and None of the above unless you really need to.

B) If you do use "None of the above," and it is correct. Makes sure in the feedback you tell the user the correct answer.

C) All of the above

PS. If you're interested how to go beyond multiple-choice questions, check out this example: https://www.rabbitoreg.com/examples/aws/story.html