A/B testing vs. User Experience Research

Many user experience researchers --aka. UX researchers-- seem to have been taken aback by the rise of analytics. You may, like me, have read multiple articles on the need to embrace analytics in the UX research discipline. One of my favorite researcher in the World, Sam Zaiss, made a talk about how to integrate research and analytics at UXPA 2016, and we have shared any discussions on the topic. I thought I would share a story with other researchers who worry about how their role is impacted by analytics. I believe this story provides another concrete example of how UX research can power better use of analytics.

Update: Note that this is not a discussion on the worth of analytics in itself, but on the complementary value of UX research and analytics.

I had been working in a company for a few month, when I received an email about the results of an A/B test. The test sought to improve conversion of one of the money-making flows by changing the text on the dialog. The results of this test showed no real gain with any of the alternative text, and as I read the alternatives they had tested, I had to admit that I was not surprised.

At that point, my colleague and I had been conducting studies on the target audience for months. I had myself probably observed a few hundred users, and I want to believe I understood them well enough to start making well-informed suggestions. With this understanding, I felt like the text choices were not really well targeted for the audience. So I used what I had learned about the users, and emailed the PM in charge of the test. In the email, I made some suggestions for alternative text that I believed mirrored what a user in this situation might be looking for. I explained my rationale using past observations, and made a case for trying another A/B test. The PM was interested, and ran the experiment.

A few weeks later, the results came back. The new text had significantly increased the chances of the user completing the flow. The change was released to all users. As a thought exercise --and to have numbers in my backpocket to sceptics about the benefits of UX research-- I worked with the PM to make a little back-of-the-napkin calculation: We looked at how much revenue this simple text suggestions was projected to bring back to the company. After looking at the current revenue per user, the increased conversion rate, and assuming a stable target audience, we projected the new text suggestion would increase yearly revenue by more than US$300,000.

Now, I don't know about you, but that's a lot more than my yearly salary. While it was the fruit of months of study, one text change justified the cost of user research, and then some.

What previous A/B tests had failed to impact, user research had impacted in 5 minutes and an email. It could do so because user understanding was able to inform the change, not just evaluate it. User understanding informed meaningful design alternatives, and provided a convincing, user-driven rationale to stakeholders. This is the sort of things user researcher can bring to the table to complement A/B tests and analytics.