The Cost of False Positive A/B Tests

There has been an ongoing debate in the software industry, with some claims that we should increase the alpha threshold for accepting stat-sig results from 0.05, or run one-tail tests, because 0.05 is too stringent.?

While there are certainly cases where experiments are run for short-term decisions (e.g., headline optimizations), for most experiments the real impact of false positive results is the roadmap—steering the ship into the wrong direction because of some amazing discovery that is wrong: a false positive.

I just listened to a wonderful talk by Ulrich Schimmack (https://videos.files.wordpress.com/9liB1ZFm/princeton.zcurve.22.10.11-1.mp4), where he talks about the sad state of affairs in Psychology, where replication rates are about 37% and much lower in Social Psychology with between-subject designs replication rates of 4%.

One of the examples he shares is a theory of Ego Depletion (https://en.wikipedia.org/wiki/Ego_depletion), where a claim was made in 1998 that we have a limited pool of mental resources that we use up and then lose self-control.? The initial study was held for over 15 years with widespread confidence in the robustness of the effect, including a meta-analysis of 198 independent tests in 2010 (talk about replication) that showed an average effect size of d=0.6.? In hindsight, this showed how much bias there is in accepted publications, where non-significant results are often rejected (the file drawer problem).

In 2016, a major multi-lab replication study failed to find any evidence for the theory.? A subsequent study involving 36 labs also failed to find the effect, which, if exists, was now estimated at d=0.06, an order of magnitude smaller than the initial meta-analysis study.? Uli claims that the original author, Baumeister, finally relented in 2022 and that the theory is now dead, after 24 years.? What a waste of resources!

In A/B testing, replication is cheap and easy: the code already exists. If the p-value is between 0.01 and 0.10 (yes, above 0.05 to reduce false negatives), my recommendation is to do a replication run, ideally at higher power than the original study (e.g., if you ran an A/B/C/D test, now evaluate just the winning version at 50%/50%).? Use meta-analysis to determine the combined p-value, and set the alpha threshold at 0.01 (equivalent to the improvement tail p-value of 0.005). ?