Why thumb rule of using just 5 test users might not be a good idea?

Let’s first understand what the 5-test-users rule is.

In 1993, Jakob Nielsen and Thomas K. Landauer published a study titled "A Mathematical Model of the Finding of Usability Problems." According to their research, the optimal results are achieved by testing no more than 5 users.

You may wonder, how did they arrive at this number? Why not 10, 20, or 50?

After conducting thousands of studies, these researchers pondered whether they could develop a mathematical model to estimate the total number of usability problems that could be identified during the evaluation process. They formulated the following model:

Found(i) = N(1 – (1-A)^i)

Where:

Found(i) represents the number of usability problems found by i evaluators.

i is the number of testers.

N is the total number of usability problems.

A is the probability of finding a usability problem with one additional person, given that it has not been found yet. In simpler terms, it represents the percentage of problems that a single user would find.

Using the equation provided, they created the following graph:

If we assume:

A = 0.31 (31% is what Nielsen found after conducting testing on 11 interfaces using 359 evaluators)

N (Total usability problems) = 100

i (Number of Testers) = 5

Usability Issues Found (by 5) = 100(1-(1-0.31)^5) = 84.35%

This indicates that by employing just 5 evaluators, you would be able to uncover 84.35% of usability issues. Now, let’s examine what happens with the addition of another evaluator.

Usability Issues Found (by 6) = 100(1-(1-0.31)^6) = 89.20%

By increasing the number of evaluators by one, the number of problems found increases by just 5%, which isn't substantial. Hence, the "magic number" 5 emerged.

In his renowned book "Sprint," Jack Knapp states: “They have seen the same phenomenon in their tests. By the time they observe the fifth customer, they are just confirming patterns that showed up in the first four interviews. We tried testing with more customers, but it was not worth it.”

However, my curiosity drove me to delve deeper into their research paper. Here are some important points to consider before adhering to the 5 testers rule:

• The study was conducted 30 years ago.
• In 1993, there were no online user-testing tools available. Moderated user testing was the only option, which was time-consuming and expensive.
• The study was conducted on a limited number of projects within a specific industry.
• The value of A (0.31), which suggests that a single user would be able to find 31 problems out of 100, is based on data from three decades ago. Moreover, this value is influenced by various factors such as the interface itself, interface fidelity, evaluation methods, and the skills of the heuristic evaluators.

When considering a study conducted 30 years ago with numerous constraints, do you believe it's wise to apply it blindly in 2024? I'm not in a position to assert that this study is incorrect. My intention is simply to shed light on certain facts that should be considered before applying it. Now, you might ask, if I don't choose option 5, how should I decide on this number? Providing an answer would be challenging as it depends on various factors such as:

1. Your industry
2. Its type (b2B, B2C, D2C, etc)
3. Your budget and time
4. Complexity of the product
5. Tools you are using to conduct studies, etc

Here are some tips that you can use when you are conducting your next research:

1. Begin by not fixating on a specific number. If utilizing an online tool, conduct as many moderated or unmoderated tests as possible within the allotted budget and time frame.
2. After gathering sufficient data, determine the point at which results became consistent across user tests. This can help determine the appropriate number of tests for future studies.
3. Once you've established the optimal number, let's say it's 20, approach your research iteratively. Instead of testing with all 20 users simultaneously, begin with 10 for the initial test to achieve an 80% result. Then, utilize the remaining 10 to evaluate improved designs and identify any remaining problems. This iterative process ensures comprehensive coverage of potential issues.
4. When working with multiple user personas, ensure each group is represented and tested separately. Conducting studies with only one group may overlook usability problems that other groups might encounter.

To be honest, Research is often considered a luxury, with limited resources available in many companies for conducting comprehensive studies. However, incorporating research into your process can still be achievable by implementing the following strategies:

• Ask for feedback from internal stakeholders, even after design files have been sent for development. This feedback can be integrated into future releases.
• Seek input from friends, family, and online communities for diverse perspectives, which can offer valuable insights, especially from users outside your industry.
• Utilize user-testing tools effectively to streamline the research process. You don’t need time and a big budget for that.
• Establish a database of users to streamline the recruitment process for future tests.
• Determine the optimal number of test users required to uncover the maximum usability problems for your product.

Even more crucially, cultivate a testing mindset. Often, the obstacle to conducting research isn't solely the lack of budget and time but inertia. If research hasn't been a part of your organization's routine for some time, it can be daunting to begin. Overcome this inertia by initiating small tests gradually.

The next time you hesitate to conduct research, remember this:

While bypassing research may yield quick results and a sense of accomplishment, you're losing a significant opportunity to enhance your interface and, consequently, refine your own skill set.