Lingering Myths Surrounding Usability Testing

User experience testing refers to the process of understanding what users do and why they do it. Good design makes it easier for users to find what they’re looking for easily and quickly and accomplish the tasks they set out to achieve.

In many instances, time and budget constraints either marginalize testing or eliminate it completely. And more often than not, it’s because usability testing is misunderstood or undervalued.

Let’s look at the nine myths about usability testing that have passed their best before date.

Myth No. 1: “We Just Don’t Have the Budget For Testing”

The reality: There is a broad range of options for testing. Lean budgets shouldn’t be an excuse to not learn how real users will interact with your product. With so many ways to approach usability testing, from formal lab-based testing facilities to your local café, there really isn’t an excuse.

Weinschenk, in her white paper Usability: A Business Case, outlines three useful equations for calculating cost savings related to errors, costs of development and maintenance, and productivity. Weinschenk easily demonstrates that avoiding usability testing isn’t a great idea.

The biggest tragedy is when failures are for the most part predictable and avoidable. If product teams don’t catch omissions or issues until final testing –or worse, until after the product has been rolled out—the costs incurred to correct errors will likely be many times greater than if they’d caught the mistake while they were still working on the initial process.

Applying human factors in the initial design can greatly reduce extensive redesign, maintenance, and customer support, which can substantially eat away profits. According to Jakob Nielsen, online shoppers spend most of their time and money at Websites with the best usability.

"Improving user experience can increase both revenue and customer satisfaction while lowering costs."

- Forrester

Myth No. 2: “This Will Take Too Much Time”

The reality: This assumption is common but does not stand up to scrutiny. When done well, usability testing doesn’t slow down the release process, it’s not too hard to do, and it provides valuable information about your product that you simply can’t get in other way. In fact, if usability testing is planned for up-front, going live can be smoother and faster.

Users are tired of dealing with confusing and unintuitive products that force them to either call customer service for help or find another product that is more usable. Consumers are finicky. Consider mobile users, notoriously impatient; if your app confuses them, they will probably move on to something else.

Usability testing provides specific insights, insights that stakeholders of an organization need. For example, senior executives want a condensed version, of “the problem” and to understand what is perceived as the differentiator or premium (for example) that this product offers. A Product Manager or the marketing team seek insights on segmentation, product identity, competitive information, participant reaction to feature sets. Product designers on the other hand want detailed usability feedback to guide enhancements to the product’s interface and behaviour.

Each of these stakeholders need to be given the necessary insights to guide the process. An experienced team keeps product and development teams moving quickly.

Myth No. 3: “You only talked to eight people, how valid can that be?”

The Reality: Even with low numbers, you can learn actionable takeaways from real users. You can ask the testers questions about their pain points, expectations and needs. The truth is that the end result of usability testing is not statistical validity, but verification of insights and assumptions based on behavioural observation.

Usability testing is a qualitative approach that is driven by insights (why users don’t understand or why they are confused) over numbers. Qualitative research uses a different methodology and tactics than quantitative research, so it’s typical that sample sizes are low.

Jakob Nielson and Tom Landauer, show that “the maximum benefit-cost ratio is achieved when using between three and five subjects.” Turns out a small number of testers can result in accurate usability testing.

Myth No. 4: “Shouldn’t you know? You’re the experts”

The Reality: The UX Designer is not your user. The UX Designer does not represent cultural differences, differences in educational levels, prior knowledge of the technology or experience.

When a user interacts with your product, and it delivers the value you promised them in a positive way, then you’ve built a successfully usable product. It’s your UX Designer’s job to learn as much as possible about your users and to keep their experience consistent throughout all phases of your project.

This means your UX Designer needs to be involved in every step of the process from concept to execution to implementation.

Look at testing as an opportunity to engage with your target audience while enhancing your understanding of their needs.

What if you have an existing desktop app, it can be very hard to decide which features to prioritize on mobile. User tests can help you answer that question.

Myth No. 5: “We Think Testing Should Be Done At The End Of The Process.”

The Reality: Test early and test often is the UX designer’s mantra. Testing small portions of a site or items of functionality can be more effective than testing an entire product at once because issues can be pinpointed to specific components in the product. If testing is held off until the end of the design process, a lot of time might be spent going back to fix issues.

Many apps are opened once and then never touched again. You want to know if you have a problem before you ship. You need to know if you are providing features that users actually want, and have you made it easy to get them? User testing throughout the design process helps you learn how your users think and what motivates them.

If you test often, you have the benefit of validating the work you’ve just done. It will also guide what you do next and another bonus: you’ll have actionable feedback.

Myth No. 6: “We’d Like To Use Eye Tracking, We’ve Heard It’s Very Good”

The reality: The devices are expensive, but that’s beside the point. In truth, not every participant can work with an eye tracker. People with a variety of attributes automatically are disqualified from eye tracking. Everything from long eyelashes to contact lenses can become an issue.

Eye trackers take away valuable time to collect data from your users. Preparing the stations and calibrating the device for the user can take time away from valuable learning. Finally, the results are tricky to analyze. What are the outputs actually saying? When someone is gazing at something, is it because they want to look there? Or because the page somehow made them look there? Maybe the users spent more time gazing up and down the page because they couldn’t figure out the navigation.

Myth No. 7: “Without Quantitative Results, Your Results Aren’t Valid.”

The reality: At Usability Matters, we have found that while many research activities have the potential to contribute to the creation of useful and desirable products, the qualitative approach provides the most value to digital product design.

Put simply, the qualitative approach helps answer questions about the product at both the big-picture and functional-detail level with a relatively small amount of effort and expense. No other research technique can claim this.

Quantitative analysis has its place. Geo-demographic techniques can potentially forecast marketplace acceptance of products and services. Statistical information, such as metrics that outline possible correlations between variables can give you data points, but each of these tools provide limited value in assessing the viability of a product.

When you see several people being stumped by the same design element, if it's frustrating users, change it or get rid of it.

It’s easy to get a quantitative study wrong and end up with misleading data. When you collect numbers instead of insights, everything must be exactly right, or you might as well not do the study.

Myth No. 8: “QA Testing Is A Substitute For Usability Testing”

QA is great at identifying technical implementation issues (e.g. system errors, incorrect calculations, etc.) and often issues with front-end design implementation (e.g. CSS misalignment, cross browser differences, etc.). But, QA tests do not focus on the quality of the user experience in regards to usability, affordances, findability, content clarity, or appropriate placements of items with the experience. Nor should it.

Myth No. 9: “Formal Testing Is The Only Way To Go”

The reality: There are many options available for Usability Testing. Each approach needs to be considered based on what you’re trying to achieve. Like all usability tests, you need to decide on your objectives.

When designing for the mass market, it’s easy enough to ask friendly looking strangers if they have a couple minutes to spare. Public spaces and shopping centers present some of the best places to do this, simply because of the sheer amount of foot traffic they receive as well the relaxed nature of the environment. With more specific user sets, however, it’s useful to target subjects based on their context.

Being open to different options may also help allay issues with budget.

In conclusion:

In a perfect world, our understanding of our users and how they work with our websites, our products, our apps, would mean conducting not only a lot more usability testing, but also more interviews, more ethnographic observation, more everything. In our actual world, though, tight budgets and schedules often mean cutting users out of the process, abandoning our empathy for them, and relying on “best-guess design.”

But it doesn’t have to be that way. Set aside a small percentage of your budgets for methods like expert review and heuristic evaluation.

Usability is more than a concept. In many organizations, usability is an important part of product development. And more and more companies are discovering that not only is usability good for users; it’s good business.