Which research method should I use?

All too often, researchers use incorrect research methods for their projects, which yields inaccurate results leading to ineffective designs. There are numerous articles that explain how to conduct various user research methods as well as descriptions of the benefits of each, but other practitioners have asked me to describe when to use each of the various types of methods.

Your design objective dictates which method is right for your project. Common objectives focus on the degree of improvement and how aggressive the design approach should be. For simplicity, I’ll discuss two ends of the continuum for each of these metrics. The degree of improvement can be defined as somewhere between incremental evolution and revolutionary innovation.

Incremental: This is a good objective when you need to update or evolve an existing product. This objective ensures that your product keeps pace with the competition without a great expense in time, money and effort.

Innovative: When you need a revolutionary new product that disrupts the market, you need to innovate.

Innovation is not just merely coming up with new ideas; it’s about redefining what user need you are trying to solve. So many design teams gather in a conference room and ideate or brainstorm or whatever you want to call it this week, generating lots of ideas. Despite the clever names they call it, it still fails to achieve any useful innovation. These efforts end up creating solutions looking for a problem to solve.

I use innovation to mean redefining the problem you need to solve. Every project I’ve been on initially solved the wrong problem, very well. Good user research should identify an unmet need, a new problem that needs solving. A well-defined problem typically looks very different from your current problem. If it doesn’t, you haven’t done your research right.

Your approach can be defined as being purely reactive, relying on user input to suggest improvements, or more proactive, thinking outside of the box. Edison would never have achieved success by simply iterating on the candle.

Reactive: Getting user reactions to an existing design to identify issues and potential solutions. Users are very good at giving feedback.

Proactive: This perspective helps identify missing elements or a novel design approach. Users are very bad at providing out of the box thinking. Their limited understanding of the current technology often constrains their ability to envision a more advanced product approach.

Research Types

Given that there are many names for the various types of research methods, rather than go through all of them, I’ll just describe common groups of them

Observation: Getting out in the field and observing users in their natural environment with naturally occurring triggers. This includes post observation interviews to clarify specific questions about the observations.

Contextual Inquiry: A blend of observation and interviews. Often the scenarios are contrived. The user research often asks the user to describe how they would perform a task.

Interviews: Asking users about their goals and tasks.

Surveys: Providing a set of questions and asking for constrained or open answers.

Card Sorting: Providing a stack of cards and letting users organize them into meaningful groups.

Usability Testing: This includes A/B testing which is a specific form of usability testing.

Competitive Analysis: An inherent failure of this method is that it assumes that the other guys are solving the right problem, which invariably, they are not.

Weaknesses

Here is a non-exhaustive list of problems with each of the various methods. This is not to suggest that you will experience all of these problems every time, but there is a high probability that you will. The most egregious concern is that you will not likely know what problems you’ve experienced and will think you’ve collected accurate information. It’s one thing to know there’s an issue with your data, it’s another to believe your data is accurate, when it’s not.

Note that all except observation and competitive analysis inherently suffer from self-reporting biases.

When to Use Each Method

Despite their drawbacks, each method has its benefits. As long as you know what objective you are trying to achieve, you can apply the right method to get the best results. Below is graph that roughly illustrates when it’s best to use each method. This is not a quantifiably valid analysis, but gives you and idea of how well each method fits each objective. Basically, these are based on my experiences and opinion and are not scientifically accurate scores.

Different methods work for specific objectives. For instance, despite its apparent value, observation is not the right method for polishing an existing design. Nor is usability testing the right method for identifying innovative design opportunities. Knowing what you want to achieve should help you identify which method to use and help you generate more accurate results.

Again, I must reiterate that this matrix is based on my experiences and opinion and these are not defensible scores. Nonetheless, this matrix should help you avoid solving the wrong problems, very well.