How do you handle multivariate testing or sequential testing scenarios?

One practical example: Let's assume your company offers great quality for a great price. "Quality" and "Low Price" might be two Independent factors which are both important to your audience. In a multivariate test (MVT) you could create different versions of elements that show a different intensity of quality whily other elements show different price signals. Running an MVT is best practice for identifying tradeoffs between different factors but it rquires high sample sized to eliminate statistical errors.

The best way to understand multivariate testing is by the limitations it overcomes. Let's say you're planning a really big AB test, where you've developed a brand new page and you want to send half of your traffic to it. How do you know what aspect of that new page is responsible for the improvement? And worse - what if one aspect of the page is single-handedly responsible for dragging down the performance of your experiment? These are not things that surface from a normal AB test that you run, but it's where MVTs shine. By breaking down your experiment into it's component parts, you get to point to specific changes as responsible for a proportion of the improvement, and specific changes as dragging the experience down.

Multivariate testing, like great leadership, explores the power of combinations and teamwork. It's not just about testing different elements, but understanding how they interact and complement each other. Like a team with diverse skills, each variation - be it a headline or image - contributes uniquely. The goal is synergy, where the collective impact maximizes outcomes like conversions or engagement. It's a journey of discovery, finding the optimal blend that resonates with your audience, much like finding the right team dynamic in a successful organisation.

Multivariate testing goes beyond A/B testing by simultaneously evaluating multiple elements and their interactions. It's like a science experiment for your website, allowing you to discover the synergistic effects of different design or content variations. Advantages of Multivariate Testing: Unveils hidden interactions: It exposes how elements influence each other, revealing unexpected winners or losers. Optimizes for the whole experience: It goes beyond single elements, finding the best overall combination for your goals. More efficient than individual A/B tests: It tests multiple variations simultaneously, saving time and resources.

Testing approach to take depends on the project requirement and priorities. In general, MVT answers the “what ifs” of creative testing. It lets you test numerous creative elements simultaneously, discovering optimal combinations faster. With MVT, you can explore which headlines work best with specific images and calls to action. Automation makes MVT practical, as it handles the complexity of testing multiple variables at once. Sequential testing is methodical and controlled, but it may miss synergistic effects between elements. This type of testing provides detailed insights but at a lower pace.

Actually, sequential testing isn't about testing one variant at a time. Rather, sequential testing is an approach that moves the decision boundaries every day or so instead of having a fixed one. The decision boundaries are fairly hard to cross early on during a test so that only test with very obvious winners are called. As the test runs, the boundaries get tighter and higher - generally resulting in a shorter runtime than standard frequentist testing.

I think this is a misperception of sequential testing. Sequential testing isn't about testing one variation at a time. It's a way to still use Frequentist statistics (aka p-values that control for false positives) for experimentation without requiring a fixed horizon. That is, you are allowed to "peek" and stop a test early if you have a win or a loss. Sequential testing spreads your error control across each analysis instead of saving it all up for the final analysis at the end. It is designed to guarantee the same error control (ie - if 95% is your goal - you still have 5% guaranteed false positive rate if you "peek" 4 times by checking with 4 analyses each week of a 4-week run time).

Sequential testing, like effective decision-making, focuses on adaptability and responsiveness. It's about evaluating one variation at a time, mirroring how we tackle challenges step by step. The goal is efficiency - to identify what works or doesn't quickly, akin to agile leadership that pivots with changing circumstances. This method, like good leadership, is dynamic, making decisions based on real-time evidence rather than rigid plans. It's a reminder that sometimes the best path forward is to assess, adapt, and act as you go.

Benefits of Sequential Testing: Faster results: No need to wait for a set duration, potentially reducing testing time significantly. Data-driven decisions: Stops based on evidence, not arbitrary timeframes, leading to more informed choices. Resource efficiency: Tests fewer variations in total, saving resources for other optimizations. Challenges to Consider: Increased risk of errors: Early stopping can lead to false positives or negatives if not done carefully. Requires sophisticated analysis: Continuous analysis demands robust statistical methods and expertise. Less interpretable results: Understanding long-term impact might be harder compared to traditional A/B testing.

Sequential testing is a type of experimentation that involves testing multiple variables in a series of steps, with each step building on the previous one. It's used to refine and optimize experimental designs by iteratively testing and refining hypotheses based on the results of previous tests.

Factorial testing makes sense if you have done a radical redesign and want to quickly and efficiently find the most important elements (main effects) on the page for further optimization and where they might interact with each other. It can also be a powerful way to identify interactions with concurrently running experiments if multiple experiments are setup as a factorial design. To be clear, however, factorial design (sometimes called MVTs) are not the opposite of sequential design.

Multivariate testing is like exploring a complex puzzle, where each piece - an element or factor - plays a crucial role. It's about understanding interactions and their collective impact on user behavior. This approach helps in crafting the most resonant design or content, tailored for your audience. However, it demands more - more traffic, more time, akin to a leader needing more resources for a larger vision. Use it when conditions are right: high traffic, a clear hypothesis, and a defined goal. It's a strategic choice, aligning resources with objectives.

Think of your website or app like a sandwich. A/B testing is like testing two different sandwiches—maybe a ham sandwich vs. a turkey sandwich—to see which one people like more. But what if you want to know the effect of changing the bread, meat, and/or sauce, so that you can discover the winning combination of factors (and the best sandwich)? That’s where multivariate testing comes in. Use multivariate testing when you want to test multiple elements at once and understand how they interact to affect your overall goal—like conversions or engagement. Keep in mind though, multivariate testing is often more complex than regular A/B testing.

It is important to ensure the right sample size because each variation adds the possibility for a statistical error. Example: If you test 2 factors with 4 variations, you end up with 16 combinations + control version. All the traffic will be split between these 17 variations so you need a big sample size to identify the statistical differences between the variations. Make sure you use a proper sample size and / or runtime calculator to make sure you don't run into statistical pitfalls.

Use multivariate testing when you want to understand the complex interactions between multiple variables and their impact on a particular outcome. This is particularly useful when you're trying to identify the most effective combinations of variables or when you're dealing with a large number of variables.

You would use sequential testing if you pre-calculated your fixed-horizon runtime and found a runtime that was longer than your available window. Or perhaps you felt strongly that you could beat the estimated lift/MDE. Or - maybe, you just wanted to be able to peek or had an organization that needed to be able to pull the plug quickly if it wasn't working - or wanted to implement quickly if it was a win. All of these are good use cases for sequential design. But to be clear - these are not opposites of factorial / MVT tests. They are as opposed to a fixed-horizon test.

Use sequential testing when you're trying to refine and optimize an experimental design over time. This is particularly useful when you're dealing with a complex problem or when you're trying to test multiple hypotheses.

It's really important in a multivariate test that it's setup correctly in order to be able to interpret the results. Often, I see tests which are intended to be multivariate setup as a univariate test with multiple changes present in a single variant. Instead, you need to make sure you have a variant for each combination of variables. In a test with two different sets of copy and two different images you would need 4 variants. Something to watch out for is that the number of combinations increases very quickly as you add variables and thus your available sample gets diluted across all those combinations. If your available sample is very large, this may not be a problem. However it might cause issues if you tend to have limited samples.

Ultimately, the human is making the decisions and the human will be held responsible for them. So the human has to use the method they can defend best, regardless of the outcomes, for it to actually work.

To handle multivariate testing, use a robust design that accounts for potential interactions between variables, and use statistical methods like regression analysis or machine learning algorithms to identify relationships between variables.

To handle sequential testing, use a phased approach, with each phase building on the previous one, and use techniques like Bayesian updating to refine your hypothesis based on new data.

Begin by establishing a baseline and then systematically introduce variations, monitoring their impact on key metrics. Use statistical methods like Bayesian analysis or sequential hypothesis testing to make informed decisions about when to stop or continue testing each variation.

Something to consider when it comes to multi-variate testing is whether or not it's truly necessary or if a univariate test would deliver the necessary data. For example, if you're redesigning a landing page with a lot of different elements, you could set this up as a multi-variate test with each element as a variable and lots and lots of combinations of variables. The alternative would be to test the new page against the old one as a single variant. You avoid a lot of complexity in your test and still learn if your new page works better. What you would miss out on is understanding why the new page works better. The question is; does it matter?