Everyone Hates My Site: Breaking-Down A/B Testing

Imagine this: You want to increase the number of visitors to your website who sign up for your company's newsletter. First, you design a flashy web banner to advertise the discount code they'll receive on the mailing list. Second, you design a tiny mascot that dances at the bottom of every screen and grabs people's attention. People like cute things, right? You spend copious energy and resources to ensure everything looks exactly the way you picture it. Finally, you launch your redesigned site.

Over the course of the next few weeks, nothing seems to be happening. It’s flashy! You love it! But no one else does. Where did you go wrong? 

While this is discouraging, the lack of conversion on your website does not indicate your changes are unnecessary. However, the less-than-ideal results may be due to poor research practices when assessing the changes that should be made. If marketers go into the redesign process without a strategy, they are putting the site at an immense risk.

Utilizing A/B Testing

Taking a Hint

A/B Testing is the practice of modifying a webpage, pitting it against the original version, and allowing consumers to utilize one at random. The data collected from each participant gives an idea of each webpage’s performance. One of the best examples of A/B Testing is exemplified by the advertising company, ComScore. After testing three different variations their site, in which they displayed testimonial quotes in different ways, the version which they ended up eventually utilizing was found to increase user engagement by 69%.

When properly executed, A/B Testing is an extremely beneficial way to ensure changes, updates, and improvements to your website are happening in the right places in the right ways. The main areas in which A/B Testing can optimize your website are in the elements of both design and user experience. Design elements include things like visual media, advertisements, and key information. User-experience elements include buttons, page organization, and social media-linked functionality.

Shifting Your Attention

In an A/B test, results may indicate there are no differences between the control and variation versions of your website. If this is the result, A/B Testing enables clients to understand whether their efforts should be concentrated elsewhere. Have you pinpointed the issue, or do you need to keep searching for it?

The digital gaming company, Electronic Arts, commonly referred to as EA, discovered through A/B Testing that the money they were putting towards advertising their new SimCity game was unnecessary. When they removed the advertising messages from their page, their conversions increased by 43.4%. They credited these numbers to their audience already being engaged with the product upon arrival to their site. Customers were already determined to buy the game, they did not need any more encouragement.

Maximizing the Benefits

Keep it Focused

A/B Testing is the best way to begin your strategy when you realize you need to make improvements to your site. This process cuts out the middleman and gets data directly from the people for whom the website is designed: the consumer. The data collected in this type of testing has quality internal validity because the data is coming from consumers and the variables being measured are the exact tasks in which users will engage when browsing your site. Also, the sampling groups are much more representative of the population for which the website is actually serving.

Keep it Fair

One key element of A/B Testing is the practice of randomly assigning participants to a condition in a between-subjects design. Side-by-side comparisons are not the best way to yield results from these studies. For example, in a television commercial for Bing, "random passersby" are asked to make split-second judgments about which search engine they like most after a blind demonstration. Self-identified Google enthusiasts are told they would find Bing's search results more appealing, and if they didn't, they would win a prize. There are multiple issues in the ways the research question, hypothesis, and instructions are revealed to participants. With these research methods, participants are met with an abundance of demand characteristics and expectancy effects. To obtain unadulterated data, it is crucial to avoid issues like these. A/B Testing is a blind experiment, so there is a significant reduction in these researcher effects.

Keep it Simple

All of the best practices for A/B Testing share an emphasis on maintaining focus. For instance, there shouldn’t be too many people in the sample, too many variations in each trial, or variables irrelevant to the hypothesis.

Understanding exactly what makes participants more/less responsive to the trial variation is the goal of an A/B test, so if too many things cloud the findings, they are far more ambiguous. With the well-known ComScore A/B test, they only had three variations and only changed the layout of the quote and the presence of the company logo. If they had manipulated other things, such as the size of the text and the placement among other articles on the page, researchers would be left with even more questions about which changes to implement.

It may be tempting to change multiple variables in the name of convenience. And while making every new “improvement” at the same time may seem like a no-brainer, Optimizely recommends working systematically to keep your sights focused and your research hypothesis clear: use data to create goals, use goals to create hypotheses, and use hypotheses to create variables that relate back to your goals. 

The Bottom Line...

A/B Testing should be utilized to make meaningful changes that last. If companies want to increase their impact with consumers, they must understand what keeps them actively engaged. Not only that, but companies must be able to gather data that is absent from errors and ready to be put into motion. This happens by limiting the number of variables in each variation, promoting proper research practices, and keeping the consumer in mind. A/B Tests are the best way to construe meaningful, concise data.