Sample Sizes for Usability Studies: One Size Does Not Fit All

How many participants should you run in a usability study?

How many times have you heard that question?

How many different answers have you heard?

After you sift through the non-helpful ones, probably the most common answer you’ve heard is five. You might have also heard that these “magic 5” users can uncover 85% of a product’s usability issues. Is that true? Are five enough, too few, or too many?

How can you know? Can we really know?

Or are we just resigned to hearing the most dogmatic voices on social media? What are the alternatives?

Perhaps we should average the advice of others, or just make our lives easier by sidestepping the question altogether.

We’ve seen both approaches taken. But is there a better way to find sample sizes?

And is there a single sample size that is right for all usability studies?

Read the full article on MeasuringU's Blog

Summary and Discussion

You probably know the answer: One sample size does not fit all studies. Not much of a surprise there. But there is a way to get to a sample size that doesn’t involve democracy or demagoguery.

The first step in finding a sample size is to define the study type. For the purposes of sample size estimation, there are three types of usability studies: Problem Discovery, Estimation, and Comparison (Table 1) .

So, how many participants do you need for a usability study?

It depends first on the study type. There are three study types—discovery, estimation, and comparison. In contrast to estimation and comparison studies, sample size estimation for discovery studies uses a different mathematical approach.

It still depends within study types. Don’t rely on averaging together recommendations or looking for a single number that will always work even when focusing within a study type such as discovery.

What about the “magic number 5?” The controversial claim based on the research of Nielsen and Landauer that “five is enough” turns out to sometimes be true, but only for a limited range of research contexts.

What about any other magic number? Because the appropriate sample size for discovery studies depends on two factors, no one magic number will be appropriate for all research contexts. In fact, there is no magic number for sample sizes for any type of usability study, formative or summative.

Use the formula for problem discovery. The problem discovery formula can be used to find the sample size based on expected problem occurrences (p) and the likelihood of seeing a problem at least once. You can also use the online calculator .

Parameters have defaults but should be changed when necessary to fit the research needs. The typical parameter for discovering problems is 85%, but this can be increased or decreased depending on the context. The parameter of 31% for the probability of problem occurrence came from an average across datasets from the 1990s. It’s not a bad place to start, but it shouldn’t be the only value for this parameter. Using values of 10%, 20%, and even 5% may make sense depending on how important it is to discover uncommon problems.

If there isn’t a magic number, should we give up on sample size estimation for formative usability studies? Giving up on magic numbers doesn’t mean you have to give up on sample size estimation for formative usability studies (or any other type of discovery study). You just need to be able to make decisions about (1) how rare of an event you need to be able to detect at least once and (2) what percentage of those events you need to discover in the study.

Bottom line: It would be nice if this process were simpler, but unfortunately, one sample size does not fit all research requirements. Fortunately, there is a mathematical model that can guide UX professionals to make reasoned decisions about sample size requirements for formative usability studies.

Read the full article on MeasuringU's Blog

Benchmarking in MUiQ

The MeasuringU Intelligent Questioning Platform (MUiQ ) is an unmoderated UX testing platform geared towards benchmarking the user experience. It has built-in templates that make it easy to set up complex, competitive benchmarks and an analysis dashboard that compares standardized metrics across each condition so your team can quickly gather insights from the research.?

MUiQ supports the following study types:

• Competitive Benchmarking
• Think-Aloud Usability Testing
• IA Navigation (Card Sort, Click Test, Tree Test)
• Large Scale Surveys
• Advanced Surveys (Kano, Top Tasks, MaxDiff)
• Prototype Testing

With the results being presented in an easy to understand analysis dashboard, MUiQ provides all of the tools you need to collect meaningful UX insights.

Reach out today to learn more about MUiQ .