A/B Testing Statistics issues CROs should ignore with Timothy Chan

In this week's episode of?Experiment Nation: The Podcast, our own?Richard Joe?chats with?Statsig's?Timothy Chan?about A/B Testing Statistics questions that many folks in Conversion Rate Optimization face such as whether you have to worry about interaction effects when running tests at the same time, test isolation, and confidence levels.

Here's a snippet:

"I came to the sort of the conclusion that interaction effects tend to be overestimated. They’re, they’re number one, they’re actually quite rare. And then when they’re found, they’re actually in most cases underwhelming, and in such a way that they generally still produce directionally accurate results. So it wouldn’t change your decision over whether to ship a feature, it would just change your estimate of what that effect. True effect is. So I think in some ways, folks who are really worried about interaction effects are really making too much of a big deal." - Timothy Chan

Check out the full version of this informative session here:?https://lnkd.in/ghBTKDYv