Understanding UX Metrics, Part 3

Dear Designer,

This edition of?Beyond Aesthetics?is part 3 of a series on UX metrics. Read?Part 1?or?Part 2.

Q: Do I need tons of users for my UX Metrics to be reliable?

No. You can get great UX metrics with only 5 to 10 users.

But you should understand sample sizes and your margin of error if you want to make generalizations about your full user base.

Last week, I asked if you wanted a 3rd email on UX metrics. 100% of you said you wanted a part 3 on UX metrics.

Yay! Much data! Such experiment!

Whatever you do, don't read the bottom-left corner of that graph.

I hid something there:?only 23 out of 4,000 newsletter readers took the survey.?Schei?e.

That big beautiful 100% is only 23 responses. Labeling that graph 23/23 would have been more truthful.

No matter how pretty the design, my math beneath the graphic isn't sound. Bad math doesn't stink, so it isn't obvious if your math is wrong.

To get a little more confident, I used an?online sample size calculator?to check, and?I needed 67 survey responses?(at a 90% confidence level with a 10% margin of error which is the minimum you want to do).

Of course, that's only if I want to say these survey results reflect all the subscribers. But usually, you want to say that type of thing, so always check your math.

Well, I'm writing part 3, anyway...data be damned! ??

My survey results weren't representative of my full population, but it feels less risky with some data.

This is a great example of how UX metrics might work. You try to gather as much data as possible, but in the end, you have to make decisions based on the (possibly crappy) available metrics.

If you have tons of data, you can measure the behavior or attitudes of every single user. But that sort of user access is rare.

Most of us will sample a slice of our user base.?It's cheaper, and you can use math to see if it represents your whole user base.

But you should still try to get the largest sample size you can manage because that will make your results more accurate. Sample size and confidence are correlated. The higher your sample size, the more confident you can be in your results.

How do you know that you have a large enough sample? 5 isn't always enough.

Wait—didn’t that guy from Nielsen Norman Group say that?I can test with 5 users and be fine??It depends on what you're trying to do.

Let's look at the?3 Ps of UX Metrics.

Performance Metrics

During World War II, military engineers wanted to minimize pilot errors and improve the performance of airplane cockpits. Usability and performance data became a life-or-death situation. These speed and efficiency performance studies led to today’s usability and performance studies.

Performance metrics measure what the user does when doing a task.

The classic usability test uses lots of performance metrics like efficiency or learnability.

Simple Example:?For a simple performance study, you might be troubleshooting big issues with a new design feature. Maybe you’ve got a few wireframes and want to see which has the least issues. Or maybe you’re trying to catch the big stuff before you send your design off to the developers. In that case, you might measure error, error severity, and task success rate.

Advanced Example:?For an advanced performance study, you might optimize details of a critical flow in your product, say a checkout flow. Improving small aspects of a page with high traffic can save millions of dollars. You might look at task time, drop-off rates, and efficiency.

Whatever you’re doing,?performance metrics talk about the user's action. The task provides the scope for the measurement in a usability test, but you could also use an analytics dashboard to look at user performance overall.

You can get away with smaller numbers of participants with performance studies. That’s because you’re testing how people perform with a design, not whether they prefer it.

If you’re new to UX, aim for 5 users in your sample size. If you’re experienced, aim for 8-12.

Here’s the graph that everyone references for these figures:

(from Measuring the User Experience, a very good book on UX Metrics)

NOTE: The classic 5 users rule only applies to troubleshooting a design. It's not going to tell you which design is better. It tells shows you where the problems are. By studying the performance of average users, you can get a pretty good idea where to improve things, but it won't tell you if your users will prefer one design over another.

You will need to test for preference to know which design is more desirable.

Preference Metrics

It’s tough to say that many people prefer one thing over another. But the payoff can be great.

Imagine knowing which design will work best before you launch it. Imagine being able to compare the UX of your product to a standard.

Preference metrics give us a glimpse of the most desirable choice, according to users.

8 to 12 users may help you troubleshoot a design, but you will need at least a hundred to determine user preference.

Take a look at two different preference tests with sample sizes of 115 and 421:

(Graph from an excellent article by Kuldeep Kelkar)

See those orange lines and text on the graph?

That's the?margin of error, often written +-3%. Notice how it goes down with a bigger sample size. In the sample size of 115, those blue bars could vary so much that you can't get a solid answer.

(The 90% Confidence Level talks about the repeatability of the setup, and it's set at 90% for both tests. You can do 90% for quick and dirty UX stuff, but 95% is the level that scientists use so choose one of the two numbers if your tool allows it)

A/B tests also generate preference metrics in an as-close-to-real-world-setting-as-possible. You can also run product experiments or concept tests with prototypes in a lab environment to study the preference of designs.

Whether it’s a survey or a product experiment, preferences can help you avoid designing something nobody wants.

I teach an entire?course on product experiments, where we learn all these new ways designers test with performance and preference.

Perception Metrics

Perception metrics measure feelings and emotions; yes, that's as weird as it sounds.

The most common perception scores are?SUS and SUPR-Q, based on self-reported user perception.?These are usually based on the user's opinions, and they’re often administered before and after a performance study (it’s always a good idea to balance your data collection methods).

Perception metrics get weird when you start using skin monitoring and facial recognition to provide hard data on the user's perception.

Perhaps these tools will get cheaper and more accessible for design teams in the future. Eye-tracking used to require a whole lab, but now you can do it on a smartphone with something like?Eye-Square.

Emerging perception technologies can?provide a nice balance between preference and performance metrics.?Knowing you've designed something well is much easier if the user self-reports that it was a good experience.

Perception Metrics help you track?hard-to-measure things like ease, frustration, surprise, trust, and stress.?

The methodology is complex and unique to the software doing the measuring, so if you're new, start with performance and then move to preference and perception.

If you want to gather perception metrics, find yourself a UX score like SUS or SUPR-Q and learn the ins and outs. Of course, you could always pay for a special tool. Here is a giant map of them from 2022.

In the future, maybe we'll measure future behavior with a simple brain scan. Now that would be trippy...

Well, that's it for today! ??

You just learned Part 3 of UX metrics. ??????????????????????

Until next week, I promise I'll stop talking about UX metrics. ??

-Jeff Humble, Designer & Co-Founder @?The Fountain Institute

Check out my masterclass, How to Lead with UX Metrics. It's completely FREE.

This is your Linkedin edition of?Beyond Aesthetics, a free newsletter for UX and Product Designers from the?Fountain Institute.

For more design education, follow us on?Linkedin,?Instagram, or?YouTube