My quest to find the one right answer

Nicola Lindgren posted this screenshot and asked us what we all thought the answer was.

I cannot resist giving my opinion, even when not invited to, so I jumped at the opportunity to weigh in.

Firstly, let's resist the temptation to talk about how this may or may not be a good way to think about testing, whether 100% coverage of equivalence partitions is a valid construct, etc. For this exercise, let's take the question in the spirit it is intended.

In my first attempt, I get 48 total combinations (and therefore 48 test cases to achieve 100% coverage of equivalence partitions):

Hmm... ok. A far cry from any of the multi-choice options in the exam.

But I also have questions about the requirements. I suspect that the requirements about smokers is supposed to come under the "Health Risk Assessment" section. Surely, it's during the Health Risk Assessment that you would elicit whether the employees are smokers or not? Maybe that's where I've gone 'wrong'. Let's see if that changes my answer to a "valid" option:

I get 28 now. Still a far cry from even the largest option, which is 12.

Later while pondering this in the shower (because I have nothing better to do on a sunny Sunday morning) I wondered, "Do they mean 'What are the minimum number of tests you can perform where each variable's option is tested at least once?'"

I finally get a "valid" answer. And checking the answer sheet, it turns out that 4 is indeed the "correct" answer.

This deeply concerns me.

First, academically speaking, I could argue that each possible combination of variables is an equivalence class.

If an equivalence class is "a set of factors that have something meaningful in common", then by extension, if there is a meaningful difference between a set of values, they are not an equivalence class. So if you look at my "48 combo" diagram, each of the 48 paths is a separate calculation that can be performed, therefore, each is its own equivalence class. Furthermore, you wouldn't know they were equivalent or not until you actually tested them! There could be hidden classes we just aren't aware of. This is why we test!

But this argument is the least interesting to me; I'm not interested at this stage to argue what is and isn't an equivalence class because equivalence class partitioning (ECP) is a heuristic that is fallible but sometimes useful depending on how you apply it.

Let's take another angle. Presumably the goal of an exam like this is to offer some kind of assurance to employers that someone who has passed the exam can join a team and support that team to deliver quality by investigating and critiquing a product in order to find things that threaten the value of that product.

My deepest concern is that questions like this teach and reward testers that they should interact with the system as minimally as possible, and should feel confident doing so. Note that four test cases is even less than if you did allpairs combinations, which is what I would consider the bare minimum for any somewhat important system (and if they are paying me to test it, then it must be somewhat important to someone).

And this is even when we take the question as just an academic exercise! A better question might be "You are given these requirements, what questions do you have?" followed by "Assuming you can't ask questions, applying various models of test coverage, what are the different ways you could test these combinations of variables?"

And this is ALSO before we start thinking about real life variables like:

• Are these questions part of a wider flow? If so, do preceding answers affect these questions or the final calculation? Do these answers affect proceeding steps in the flow?
• How are the questions presented to users? Are they readable, understandable, carefully worded to minimise bias?
• What are our target devices and platforms? Desktop, Mobile? If mobile, do we support landscape and portrait? Moving between aspect ratios? What happens if we lose signal mid-transaction?
• Can we legally ask these questions in all locations we are targeting?
• What happens when users enter characters we don't support?
• What happens when users enter large values, small values, decimal values, fractions?
• Can users navigate this questionnaire using keyboard only? Does the tab navigation order comply with WCAG standards and guidelines?
• Are users given sufficient time to complete the questionnaire before it times out?
• etc
• etc
• etc

"OK, but the point of the question is to evaluate whether they understand what ECP is, it's not supposed to represent a real-world example!" I hear you cry.

OK, then. As already shown, ECP is a heuristic, and there's no "one" correct answer depending on which coverage model you're using. So how might I ask this question if my goal was to evaluate their knowledge of ECP and how it applies to models of test coverage?

As alluded to earlier, here is how I might ask this question of my students:

• After reading these requirements, what questions do you have?
• Assuming you can't ask any questions, taking these requirements in isolation and only as written, using equivalence class partitioning:

a) What are all possible combinations of the variables?

b) What are some techniques we might use to prioritise these combinations?

c) What are the fewest combinations where each ECP value still gets tested at least once? What is the risk of doing so?

• What other variables can you think of that aren't specified?
• What are the advantages of ECP?
• In what ways is ECP fallible?

But no. According to the exam, the answer is 4. Three points to you. Next question.

Software testing is a wonderful and messy endeavour to investigate complex systems in order to discover ways it might fail when it gets in contact with the outside world. To do this well, we need to be imaginative and apply judgement and skill. I worry that exams that ask for single 'correct' answers stifle budding testers' imagination and curiousity.