Why We Speak to 5 Customers
Let's dive deeper into why discovery research interviews, common in qualitative research, use small sample sizes, especially compared to the larger sample sizes typical of statistical quantitative research.
Discovery Research and Small Sample Sizes: Why the Big Deal about "Small"?
First off, discovery research interviews are not designed to be representative in the way that surveys or quantitative studies are. The goal is to explore, understand, and uncover insights about how and why people behave in certain ways. This is where the sample size becomes a big (or rather, small) topic of conversation.
Generative/Qualitative Research: In generative or qualitative research, the sample size is often small because you're looking for deep, rich insights rather than numerical generalizations. As the article from UX Design points out, researchers are trying to uncover themes—and often, a small group of participants is enough to generate these insights. When a researcher hears the same themes over and over, this is called reaching "saturation." At this point, more interviews become like pouring water into a full glass—there's no new room for insights, just repetition.
So, Why stop at 5-10 people? What if we miss something? Here’s the trick: discovery research is about depth, not breadth. We aren’t aiming for statistical generalization (i.e., saying 82.4% of people feel this way); we’re exploring the why and how behind specific experiences. If you’re finding recurring patterns after just a few interviews, you can be fairly confident you’ve hit the key points. Like a detective who cracks the case by finding just a few key clues, you don’t need to interview everyone in the town to understand what's going on.
So Why Do Quantitative Research Studies Need Big Samples?
Quantitative research, on the other hand, plays a different ball game. It’s about precision and confidence in your numbers. When running surveys or conducting experiments, researchers aim for results that can be generalized to a larger population. This is where things like sampling error and confidence intervals come into play. The larger your sample size, the more confident you can be that your results represent the population as a whole.
According to MeasuringU, when conducting quantitative research, it's critical to clearly define your target population and ensure that your sample size is large enough to represent that population accurately. Small sample sizes in this context can lead to sampling bias, where the results might skew based on the specific group you've surveyed, rather than reflecting the larger group.
This also ties into how you find participants for these studies. In surveys, ensuring that your participants reflect the larger population is crucial. The sample needs to be representative, not just random. You can’t rely on qualitative discovery rules here; generalization is the goal.
Nielsen Norman Group's 5-User Rule: Less Is More?
Nielsen Norman Group famously argues that just 5 participants in usability testing can uncover 85% of usability issues. While this sounds counterintuitive, the argument here is that most usability problems are found quickly because they are systemic—they impact most users in a similar way. Therefore, after testing a small sample, you’ll likely encounter the same problems repeatedly.
The "5 users rule" is one of the best-known, and also controversial, rules in user research. This rule, popularized by Jakob Nielsen in his famous 2000 article, is based on the observation that most problems are detected during the first sessions of a user study, and that as more participants are added, the number of new problems decreases progressively.
Nielsen was able to calculate the exact impact of including more participants in a study. His conclusions are reflected in a famous graph that shows two axes: the vertical axis represents the percentage of problems present in an interface (from 0% to 100%), while the horizontal axis represents the number of research participants.As you can see in the graph, there is a point, around 85% of the problems, where the line stops growing significantly and begins to flatten out. This point corresponds to 5 participants. According to Nielsen, after 5 people, finding new problems that have not already arisen with previous participants would cost us more and more.
If 5 users can find 85% of the problems, adding another 5 would only allow us to find the next 10%. So doubling our effort would have a very small impact on the research findings.
Does that mean the rest of the users are experiencing different problems? No! It's just that saturation of the same problems pop up early and often. Testing with a smaller sample allows teams to iterate quickly and improve the design without waiting for larger, more expensive tests to take place.
领英推荐
How Does Standard Deviation Affect Sample Size?
But hold on! If you do quantitative research, you might be thinking: "What about variability?" Yes, the more variability there is in your data, the larger your sample size needs to be. MeasuringU has a great breakdown of how standard deviation (SD) affects sample size calculations. Essentially, when your data points are all over the place (high SD), you need a bigger sample size to accurately estimate the population's characteristics.
In contrast, when variability is low (low SD), a smaller sample size will suffice because the data is more consistent. For discovery research, though, we’re not calculating deviations in behavior across thousands of users—we’re just identifying the main issues and getting the broad strokes.
Think of it like this, if quantitative research is a census, asking everyone for their input to draw a conclusion, then qualitative research is more like being a journalist—you’re conducting a handful of interviews with key people to get the story. You're not trying to count every person who experienced an issue, you're just figuring out what the issue is and why it matters.
So the next time someone questions the small sample size in discovery research, remind them that you're not trying to count the number of fish in the sea—you’re just figuring out why they swim in schools and whether they like their water salty! It's not about the 'How Many', it's about the 'Why.'
~Fin
READ THIS STUFF FOR MORE INFORMATION AND DETAILS.
References:
Defining and Finding Participants for Survey Research: https://measuringu.com/defining-and-finding-participants-for-survey-research/
How Do Changes in Standard Deviation Affect Sample Size Estimation? https://measuringu.com/how-do-changes-in-sd-affect-n/
Generative/Qualitative: https://uxdesign.cc/stop-asking-ux-researchers-to-defend-their-sample-size-65aa5c2305d2
Cognition & Human Factors AI/Autonomous Systems - PhD, Associate Prof Human Factors & Cognitive Ergonomics, Research coordinator
2 个月I discussed this some years ago. It is fascinating to see this topic emerging from time to time...but it is not an unknown issue...I have an entire slide deck of potential solutions for lectures in my course ...https://dl.acm.org/doi/10.1145/2506210
Jeff Sauro and I have been working on alternate ways to present discovery curves, which we plan to publish on October 15. We'll show one of them here because it's relevant to the conversation and supports sample sizes in the range of 5-10 for discovery studies (of which formative usability studies are one type). The figure shows that adding additional participants has no effect on the discovery of events (e.g., usability problems available for discovery within the study constraints such as sampled populations of people and tasks) that have a very high likelihood of occurrence (e.g., 75%) and little effect on those that have a very low likelihood of occurrence (1%). The greatest increase in discovery is for problems with moderate likelihoods of discovery. The distance between the lines illustrates the diminishing returns in discovery rates associated with increasing sample sizes. For example, increasing the sample size from five to ten shows significant benefits in the middle of the range of problem probabilities, but the benefit achieved from increasing the sample size from 20 to 25 is much smaller.
User Experience Research Consultant
2 个月As I recall, budget concerns were part of the five people per usability study too. Instead of 15 people in one study he mentioned breaking it up into three studies to possibly gain new insights. With lower costs in unmoderated studies, I recruit different demographics to compare results by age, experience, etc. with 10-15 users. His point is still valid but with today's technology we may have different options.
?? I help innovators build amazing experiences that people love - Service Design | Experience Design | UX Research
2 个月“Test with 5 users” has somehow escaped the effects of inflation these last 25 years, and I’m disappointed by that.