Machine Learning, Qualitative and Quantitative research in UX Design

1. Introduction

When users hate an app, they uninstall it. A SaaS app depends on keeping subscribers. And internal software products depend on user adoption. User testing, and user - experience research is the holy grail of successful products. But is asking users for input the best, or the only way to do UX research?

If a company's product or service doesn't meet user's expectations, it will fail. That's because users have choices. Users need to trust that the app does what they need it to, and the experience is helpful, not frustrating. A lot of the time, UX designers expect user testing to be the key to successful apps. User interviews are a direct way to study how people might interact with a product or service. In-person, user interviews gauge opinions, and predicts the value of your product/service. But there are problems with direct interviews. People don't always answer truthfully as their self-identity gets in the way. People want to report they are more productive than they are. Or are afraid their boss will read the responses and there will be consequences.

So how do you get good UX research data, to augment user interviews?

If a product like yours is available online, you can use it to gather feedback (from social media pages, blogs, etc.). Indirect methods, like similar websites, bypass the problems of direct interviews. But it's not the only UX research tool. Use this guideline as a starting point when developing new products or services.

2. Machine Learning: Talking about the Future of User Research

Machine learning, or ML for short, is the process by which software learns from data. Websites can have UX research tools implemented in a variety of ways. It is an essential part of the design process for many UX designers. This process is both simple and complex at the same time. For your app, website or SaaS product, placing tracking logs lets you track three things. Heat maps let you see where a user clicks, or what they read. Flow tracking logs which pages a user moves to from a given page. Taken in combination, it can show how users really use a site or app. And skinning can A/B test the visual look of pages to see how the UI impacts user acceptance.

There are two different types of machine learning algorithms: supervised and unsupervised. Supervised machines learn from data sets, or a set of knowledge. Unsupervised algorithms learn from data that is not fixed. In order for a machine to learn about something, it needs to be exposed to lots of different examples. Unsupervised process requires acquiring enough data to make accurate predictions.

3. What if Machines Can do the Work for Us?

For UX designers, there are databases that can predict where a user will click. So you don't even need to do heat-map testing with users anymore. Just load your site into a heat-map tool and get the heat-map. No user interaction required.

I was amazed when I first tried these tools. Corporations loaded all their heat-map data into databases. Machines could then learn where people clicked. Today, we really don't need to run our own eye-tracking, click mapping to get a heat map of our site, app or SaaS pages. Computer, predictive models are faster, cheaper and accurate.

4. The Value of ML in UX Design

A UX designer looks at elements on the page (UI), user behavior on a page (UX) and user flow moving from one page to the next. We now have predictive tools for the UI. But we can enhance that with "skinning" where we make many copies of the same page. Then we can A/B test the pages and track which users accomplish their task faster or "better." And "better" can mean higher user satisfaction (by asking after the user session).

The A/B testing data is saved or logged. We analyze the results, in real time, to see the winning designs. This gives us real data on what design works best.

(from https://weareinnovativeuxdesigns.com/2017/05/08/ray-ganguly-on-machine-learning-for-ux/) Most UX designers depend on user interviews as their only tool to develop apps, websites and SaaS apps.

Even user personas and story cards are derived from user interviews. Machine learning can create break thoughts in UX design, better than user interviews alone. The idea that a user interface is the first thing that people see is a fallacy. The first thing they see is a problem. What people have been taught to do since the days of cave paintings is to look at what they can see. To see the drawing, not the problem the cave dwellers were trying to solve with the paintings. Were they solving boredom (painting as something to do). Or where the paintings instructions on how to hunt (kill food).

5. How to Use ML in UX Design?

Because an app can be simulated as a webpage, we can use a web app to track every user interaction. This includes time on site to complete a task, which page a user moves to, and which controls a user uses first on a page. All this can be tracked, and logged. The logs can then be analyzed.

More importantly, we can try many versions of each page, the page flow, and even the order of controls on a page.

This gives us many variations to test. And even when the app is live, we can continue testing. So the first version does not need to be the last version. That is, we don't have to have the design perfect for the release of version one.

5.1 Data collection and Analysis

A common question from UX designers is “where do I get my data?”. Some data can be borrowed, or leased. That is, existing software (such as eye-tracking data) already exists. That data (known a a corpus), exits in software or SaaS services. Upload your site address, and the software will score your site against the corpus. That will result in a heat-map.

For other data, such as page usage and page flow, a developer can enable tracking and logging. When I create a UX design, I use tools such as Xojo to write server-side tracking. This lets me run A/B testing at the page level, site level, and session level. Every user interaction can be a source of data.

The more users that interact with the site, the greater the number of data points. If I don't have a lot of users for testing, more interactions can supply the data. The more interaction the users have with the site (by volume of users or interaction), the more data we get.

5.2 Data Visualization

What is UX? UX design is one of the most abstractly-defined things in the design toolbox. This field can attract people that don't like numbers, and rigorous details. And that's a problem in a world where computers are really good at getting data.

From these data sets, I like to generate visual representations. This allows all stakeholders to take part in the conversation.

Heat-maps are the simplest of data maps or visualization. On-page flow maps (taking from cumulated time recordings) show page usage. Also, since they are based on time, on-page flows can show user hesitancy. This hesitancy can indicate in-decision about what to do next, or confusion.

Finally, designers can create a visual display map as users move from page-to-page. This map is like on-page flows. Here we look to see hesitancy from the user moving to different pages. Or inconsistent movement. When users are unsure of their path within an app, we can track that. It will show up as users clicking back and forth between pages, with no real input.

Have you ever used an app where you were searching for a setting? Unsure of what to do, or on which page a setting is, users click aimlessly back and forth. We can track this and see the result. Lots of movement between pages, but no entry of data. No changes in any control. And often moving back to the same page.

5.3 Sometimes Generated Data is Better Data

UX design includes models to analyze users' behavior and how they use an interfaces. Eye tracking, heat maps and user acceptance studies are common. But often designers miss the opportunity to use existing data sets, or generate data sets. This is too bad because we don't need to spend all our time re-inventing the wheel. Using data that already exists, or letting a prototype generate the data is often better. Better in that it is more useful, a more authentic representation of how people use the site.

Heat maps show where users spend most time browsing your app or website. For example, on Facebook 60% of users spend time on the homepage; 10% on pages related to friends; 4% on pages about news etc. And we would expect similar data from our site if it's like Facebook. That is, we can often use other site data in place of our site with similar intent.

Eye tracking measures how people move their eyes within an app or website. But it turns out that following mouse movement is as effective as eye tracking. Studies show that people often read articles by using their mouse, as they would their finger. That is, they skim the mouse pointer under the text they're reading. So we can use mouse tracking as a analog for eye tracking.

These are some simple ways we can get hard data on users' behavior. Sometimes even before we design the app. Or at least early in the process, before user surveys.

6. Conclusion

I'm not against user surveys. In fact I think it's super helpful to get feedback from stakeholders. But hard data, numbers often yields faster, better, and more accurate data. Data drives better design decisions. Why? Because we don't need to ferret out user intent from survey data. That can be slow, and often the designer makes assumption.

User testing is a crucial tool in UX design, yet we often overestimate use surveys. Many apps have more than one user type - which complicates the value of surveys. But computers can subdivide user types effortlessly, and infinitely. Then we can assess the potential impact of a particular feature on each user type.

So how do you get around these challenges? There are two main types of user research methods: quantitative and qualitative.

Qualitative research is far more challenging and open - ended than quantitative research methods. This type of research is far more prone to misinterpretation. And can lead to false conclusions which could have negative consequences. Qualitative approaches are referred to as ethnographic. This is because they involve observing real life situations.

User surveys can fall between the two approaches because the “facts” from surveys.

Quantitative research in UX design generally uses data points on a scale. It’s great for quickly understanding the average reactions of users.

The benefit of quantitative methods is bringing objective data into the design process. This allows designers to quickly identify areas that need improvement. It also allows designers to see new features within existing products/services/applications. These new features often improve the overall experience.

Qualitative and quantitative approaches are not equally reliable. But, designers are able to employ both methods equally effectively. And tools exist today to get the data that was elusive in the past.

In the past, survey data was about all there was. Today, big data and machine learning are driving design. Even simple apps can benefit from big data. I'm not saying user surveys are not useful. But sometimes they are - "not useful." Or at least not as useful when they were the only tool in the box.

As a developer turned UX designer, I see first hand the benefit of coding a data tracker into apps, or websites. I am surprised at how often the client was wrong about how users really use their site, or app.

But being wrong in our assumptions often leads to better understanding of how users work, or play. After all, we've learned a lot from the game industry. Successful game companies needed to know what makes a great game. And they turned to data tracking to get that knowledge. I suspect all designers should follow their lead. And follow the data - the hard data that is.

I am a big believer in starting with user surveys but quickly get some data capture into the process. And do it as soon as possible.