What's the difference between moderated and unmoderated usability testing tools?

Moderated usability testing is when we watch people using a product or system in a controlled way to see how easy or difficult it is for them. During this type of testing, someone called a facilitator helps the users go through different tasks and situations. The facilitator watches what they do, listens to what they say, and sometimes asks more questions to understand better. The users might also be asked to talk about what they're thinking while they use the product, to give even more insight.

Figma (my preference) or Miro are other great examples of real user testing. Especially Figma allows to create great interfaces and boards for workshops. It allows to interact in real time, make notes, see what users are picking etc. The moderated method is effective as the "moderator" can react and adjust to the situation and the test flow.

Moderated usability testing is a type of usability testing where you interact with your users in real time, either in person or remotely, while they perform tasks with your product or service. You can observe their behavior, ask questions, probe for feedback, and provide guidance if needed. Moderated usability testing tools can help you set up the test environment, record the sessions, and analyze the results. Some examples of moderated usability testing tools are Zoom, UserTesting, and Lookback.

When you as a designer or researcher are a part of the user test, asking questions, posing the tasks and actively communicate with the participant at the time of participating in the test.

Unmoderated usability testing is a type of usability testing where you do not interact with your users during the test, but instead provide them with instructions and scenarios to follow on their own. You can collect data from a larger and more diverse sample of users, at a lower cost and faster speed, than moderated usability testing. Unmoderated usability testing tools can help you create the test script, recruit participants, and generate reports. Some examples of unmoderated usability testing tools are UserZoom, Validately, and UsabilityHub.

When you set up a user test, usually with the help of a tool such as Maze or usertesting.com and then simply distribute that test so that people participate as they fit without any real-time communication between participants and moderators.

Moderated usability testing offers several advantages over unmoderated testing, including: In-depth Exploration of User Thoughts and Feelings: Moderators can ask follow-up questions and probe deeper into users' thoughts and feelings, allowing for a more comprehensive understanding of their experiences. Adaptability: Moderators can adapt the test based on users' responses, allowing for a more flexible and personalized testing experience. Observation of Non-verbal Cues: Moderators can observe users' non-verbal cues, such as facial expressions and gestures, which can provide valuable insights into their experiences.

The disadvantages of moderated usability testing include: Time-consuming and resource-intensive: Moderated usability testing requires more coordination, communication, and facilitation skills, making it more time-consuming and resource-intensive than unmoderated testing. Potential for bias: There is a risk of introducing bias or influencing user behavior during moderated testing, either intentionally or unintentionally. This can affect the validity and reliability of the test results. Difficulty in recruiting and scheduling users: It may be challenging to recruit and schedule users for moderated testing, especially if you are looking for a specific target audience or testing across different locations or time zones.

One of the disadvantages I encountered in a moderated usability test is bias. One time in a small hackathon project we were testing a prototype for an app that included a dog as an assistant for the user. I thought it was not a good idea but I let one of my teammates lead the usability test and she asked the user if she thought the dog was "cute", of course, the user said he was. However, a couple of users later we changed the question frame to like "What do you think of the dog?" and we got much more divided answers. So bias is something that can be an issue in these, just know how to look out for them.

Unmoderated usability testing offers several advantages when compared to moderated usability testing. It enables you to test with a larger and more representative sample of users across different demographics, geographies, and contexts. Moreover, it is faster and cheaper, as you do not need to schedule, attend, or moderate the sessions and can rely on automation for data collection and analysis. Additionally, unmoderated usability testing allows for more realistic testing as it simulates the natural environment and behavior of the users without interference. Finally, it enables you to test more frequently, as you can iterate and validate your design ideas quickly and get feedback at any stage of the development process.

Flexibility and Timing: Moderated usability testing allows real-time adjustments and clarifications, ideal for complex tasks or new interfaces. In contrast, unmoderated testing offers flexibility in scheduling and scalability, accommodating a larger number of participants across different time zones. For example, a moderated session may be beneficial for observing nuanced user behaviors during a website redesign, while unmoderated testing can efficiently gather feedback on multiple iterations of a mobile app interface from a diverse user pool.

Cost and Resources: Moderated testing typically requires dedicated facilitators and specialized facilities, leading to higher costs and resource commitments. On the other hand, unmoderated testing platforms offer cost-effective solutions, requiring minimal supervision and infrastructure. For instance, conducting moderated usability testing in a lab setting may involve expenses for equipment and personnel, while unmoderated testing platforms like UserTesting provide affordable options for remote testing, reducing overhead costs and logistical challenges.

Moderated sessions allow for a richer, more nuanced understanding of user behaviours and attitudes, as I can ask follow-up questions and observe reactions in real time. This is invaluable for exploring complex issues or new features. Unmoderated testing, however, offers efficiency and a broader range of data since it can be conducted at scale and at the user's convenience, making it ideal for gathering initial usability insights.