Choosing the Right UX Research Method

Originally published by my friends at UX Mastery.

As more and more organisations become focused on creating great experiences, more teams are being tasked with conducting research to inform and validate user experience objectives.

UX research can be extremely helpful in crafting a product strategy and ensuring that the solutions built fit users’ needs, but it can be hard to know how to get started. This article will show you how to set your research objectives and choose the method so that you can uncover the information you need.

When to do research

The first thing to know is that there is never a bad time to do research. While there are many models and complicated diagrams to describe how products get built, essentially, you’re always in one of three core phases: conceptualising something brand new, in the middle of designing and/or building something, or assessing something that’s already been built.

There’s plenty to learn in each of those phases. If you’re just starting out, you need to focus on understanding your potential users and their context and needs so that you can understand your best opportunities to serve them. In other words, you’re trying to figure out what problems to solve and for whom. This is often called generative or formative research.

Once you’re actively building something, you’ll shift your focus to analysing the solutions that you’re coming up with, and making sure that they address the needs of your users. You’ll want to assess both conceptual fit and specific interactions quality. We usually call this evaluative research.

When you have a live product or service, you’ll want to continue to assess how well you’re serving people’s needs, but you’ll also want to use research to discover how people change and how you can continue to provide value. At this point, you’ll be doing a mix of the generative type of work that is generally in the conceptual phase and evaluative work.

There is no cut-and-dried guide of exactly what methods to employ when, but there should never be a time that you can’t find an open question to investigate.

Determine your specific research objectives

At any given time, your team might have dozens of open questions that you could explore. I recommend keeping a master list of outstanding open questions to keep track of possible research activities, but focusing on answering just one open question at a time. The core goal of a study will determine which method you ultimately use.

If you need help coming up with research goals, consider things like:

• the stage of the project you’re in
• what information you already know about your users, their context, and needs
• what your business goals are
• what solutions already exist or have been proposed
• or where you think there are existing issues.

The questions might be large and very open, like “who are our users?” or more targeted things like “who uses feature x most?” or “what colour should this button be?” Those are all valid things to explore, but require totally different research methods, so it’s good to be explicit.

Once you’ve identified open questions, you and the team can prioritise which things would be riskiest to get wrong, and therefore, what you should investigate first. This might be impacted by what project phase you’re in or what is currently going on in the team. For instance, if you’re in the conceptual phase of a new app and don’t have a clear understanding of your potential user’s daily workflows yet, you’d want to prioritize that before assessing any particular solutions.

From your general list of open questions, specify individual objectives to investigate. For instance, rather than saying that you want to assess the usability of an entire onboarding workflow, you might break down the open questions into individual items, like, “Can visitors find the pricing page?” and “Do potential customers understand the pricing tiers?”

You can usually combine multiple goals into a single round of research, but only if the methods align. For instance, you could explore many different hypotheses about a proposed solution in a single usability test session. Know that you’ll need to do several rounds of different types of research to get everything answered and that is totally OK.

Looking at data types

After determining your research goal, it’s time to start looking at the kind of information you need to answer your questions.

There are two main types of data: quantitative and qualitative.

Quantitative data

Quantitative data measures specific counts collected, like how many times a link was clicked or what percentage of people completed a step. Quantitative data is unambiguous in that you can’t argue what is measured. However, you need to understand the context to interpret the results.

For instance, you could measure how frequently an item is purchased. The number of sales is unchangeable and unambiguous, but whether 100 sales is good or bad depends on a lot of things. Quantitative research helps us understand what’s happening and questions like: how much, how many, how often. It tends to need a large sample size so that you can feel confident about your results.

Common UX research methods that can provide quantitative data are surveys, a/b or multivariate tests, click tests, eye tracking studies, and card sorts.

Qualitative data

Qualitative data is basically every other sort of information that you can collect but not necessarily measure. These pieces of information tend to provide descriptions and contexts, and are often used to describe why things are happening.

Qualitative data needs to be interpreted by the researcher and the team and doesn’t have a precise, indisputable outcome. For instance, you might hear people talk about valuing certain traits and note that as a key takeaway, but you can’t numerically measure or compare different participant’s values. You don’t need to include nearly as many sessions or participants in a qualitative study.

Common UX research methods that can provide qualitative data are usability tests, interviews, diary studies, focus groups, and participatory design sessions.

Some methods can produce multiple types of data. For instance, in a usability study, you might measure things like how long it took someone to complete a task, which is quantitative data, but also make observations about what frustrated them, which is qualitative data. In general, quantitative data will help you understand what is going on, and qualitative data will give you more context about why things are happening and how to move forward or serve better.

Behavioural vs attitudinal data

There is also a distinction between the types of research where you observe people directly to see what they do, and the type where you ask for people’s opinions.

Any direct-observation method is known as behavioural research. Ethnographic studies, usability tests, a/b tests, and eye tracking are all examples of methods that measure actions. Behavioral research is often thought of as the holy grail in UX research, because we know that people are exceptionally bad at predicting and accurately representing their own behaviour. Direct observation can give you the most authentic sense of what people really do and where they get stuck.

By contrast, attitudinal research like surveys, interviews, and focus groups asks for self-reported information from participants. These methods can be helpful to understand stated beliefs, expectations, and perceptions. For instance, you might interview users and find that they all wish they could integrate your tool with another tool they use, which isn’t necessarily an insight you’d glean from observing them to perform tasks in your tool.

It’s also common to both observe behaviour and ask for self-reported feedback within a single session, meaning that you can get both sorts of data, which is likely to be useful regardless of your open question.

Other considerations

Even after you’ve chosen a specific research method, there are a few more things you may need to consider when planning your research methods.

Where to conduct

It’s often ideal to be able to perform research in the context of how a person normally would use your product, so you can see how your product fits into their life and observe things that might affect their usage, like interruptions or specific conditions.

For instance, if you’re working on a traffic prediction application, it might be really important to have people test the app while on their commute at rush hour rather than sitting in a lab in the middle of the day. I recently did some work for employees of a cruise line, and there would have been no way to know how the app really behaved until we were out at sea with satellite internet and rolling waves!

You might have the opportunity to bring someone to a lab setting, meet them in a neutral location, or even intercept them in a public setting, like a coffee shop.

You may also decide to conduct sessions remotely, meaning that you and the participant are not in the same location. This can be especially useful if you need to reach a broad set of users and don’t have travel budget or have an especially quick turnaround time.

There is no absolute right or wrong answer about where the sessions should occur, but it’s important to think through how the location might affect the quality of your research and adjust as much as you can.

Moderation

Regardless of where the session takes place, many methods are traditionally moderated, meaning that a researcher is present during the session to lead the conversation, set tasks, and dig deeper into interesting conversation points. You can tend to get the richest, deepest data with moderated studies. But these can be time-consuming and require a good deal of practice to do effectively.

You can also collect data when you aren’t present, which is known as unmoderated research. There are traditional unmoderated methods like surveys, and variations of traditional methods, like usability tests, where you set tasks for users to perform on their own and ask them to record their screen and voice.

Unmoderated research takes a bit more careful planning because you need to be especially clear and conscious of asking neutral questions, but you can often conduct them faster, cheaper, and with a broader audience traditionally moderated methods. Whenever you do unmoderated research, I strongly suggest doing a pilot round and getting feedback from teammates to ensure that instructions are clear.

Research methods

Once you’ve thought through what stage of the product you’re in, what your key research goals are, what kind of data you need to collect to answer your questions, and other considerations, you can pinpoint a method that will serve your needs. I’ll go through a list of common research methods and their most common usages.

Usability tests: consist of asking a participant to conduct common tasks within a system or prototype and share their thoughts as they do so. A researcher often observes and asks follow up questions.

Common usages: Evaluating how well a solution works and identifying areas to improve.

UX interview: a conversation between a researcher and a participant, where the researcher usually looking to dig deep into a particular topic. The participant can be a potential end user, a business stakeholder or teammate.

Common usages: Learning basics of people’s needs, wants, areas of concern, pain points, motivations, and initial reactions.

Focus groups: similar to interviews, but occur with multiple participants and one researcher. Moderators need to be aware of potential group dynamics dominating the conversation, and these sessions tend to include more divergent and convergent activities to draw out each individual’s viewpoints.

Common usages: Similar to interviews in learning basics of people’s needs, wants, areas of concern, pain points, motivations, and initial reactions. May also be used to understand social dynamics of a group.

Surveys: lists of questions that can be used to gather any type of attitudinal behaviour.

Common usages: Attempting to define or verify scale of outlook among larger group

Diary study: a longitudinal method that asks participants to document their activities, interactions or attitudes over a set period of time. For instance, you might ask someone to answer three questions about the apps they use while they commute every day.

Common usages: Understanding the details of how people use something in the context of their real life.

Card sorts: a way to help you see how people group and categorise information. You can either provide existing categories and have users sort the elements into those groupings or participants can create their own.

Common usages: Help inform information architecture and navigation structures.

Tree tests: the opposite of card sorts, wherein you provide participants with a proposed structure and ask them to find individual elements within the structure.

Common usages: Help assess a proposed navigation and information architecture structure.

A/B testing: Providing different solutions to audiences and measuring their actions to see which better hits your goals.

Common usages: Assess which of two solutions performs better.

Christian Rohrer and Susan Farrell also have great cheat sheets of best times to employ different UX research methods.

Wrapping up

To get the most out of UX research, you need to consider your project stage, objectives, the type of data that will answer your questions, and where you want to conduct your research.

As with most things in UX, there is no one right answer for every situation, but after reading this article you’re well on your way to successfully conducting UX research.

Amanda Stockwell is President of Stockwell Strategy, a UX research practice focused on lean research methods and integrating user knowledge with business goals to create holistic product strategies for organizations of all sorts. She has focused most of the last decade focused on finding innovative ways to understand end users and embed that knowledge into overall process. She’s lead teams that provide research, design, and UX strategy services and frequently writes and speaks about her experience. Check her out on Twitter @MandaLaceyS and view her #LinkedInLearning courses on UX Research Basics, Card Sorting, and Interviewing.

Originally published at uxmastery.com on January 26, 2018.