Stop an A/B test the first day it looks significant and false wins pile up. In 20,000 simulated A/A tests (identical versions, 5% sign-up rate, 500 visitors a day per version, 20 days), looking once at the end gave false winners 5.1% of the time; looking every day and stopping at the first p below 0.05 gave 24.6%. Johari, Koomen, Pekelis and Walsh (Operations Research, 2022): “Type I error can easily increase fivefold.” The fix: decide the sample size first, then look once (or use a method built for continuous monitoring).
Full video:
A/B Tests and Statistical Significance, Without the Math Anxiety.
Sources:
Johari, Koomen, Pekelis and Walsh (2022), Always Valid Inference: Continuous Monitoring of A/B Tests, Operations Research
Try QuestionPunk free:
Every number on screen comes from the sources above or is calculated from them.
source