You launch an A/B test on Monday. By Wednesday, the variation is crushing the control — revenue per visitor is up 28%. Your team is ecstatic. Your boss wants to ship it immediately. By Friday, the gap has narrowed to 15%. By the following Wednesday, it is down to 4%. By the end of the month, there is no meaningful difference at all.
What happened? Neither breakage nor diminished effect. You witnessed regression to the mean — one of the most fundamental and misunderstood phenomena in statistics.
What Is Regression to the Mean?
Regression to the mean is a statistical phenomenon where extreme observations tend to be followed by more moderate ones. Sir Francis Galton first described it in the 1880s when he noticed unusually tall parents tended to have children closer to average height. The principle: any measurement with a random component will produce extreme results that naturally move closer to the true average over time. Not because anything changed — but because randomness evens out.
In A/B testing, this is devastating when misunderstood. Early results are dominated by noise. Small sample sizes amplify random variation. A few high-value purchases or a burst of atypical traffic can make one variation look dramatically better. As more data accumulates, the noise averages out, and the true effect — often much smaller or nonexistent — emerges.
The Anatomy of a Disappearing Winner
Consider a common pattern: an A/B test runs for four weeks with four variations against a control, tracking revenue per visitor. Week 1: Variation B leads by $3.50 per visitor. Week 2: The gap is shrinking. Week 3: Variation D has overtaken B entirely. Week 4: No statistically significant difference between any variation and the control. If someone had stopped this test before the full four weeks, they would have drawn a completely wrong conclusion.
Why Early Results Are Especially Misleading
With a small sample, the standard error is large. It decreases proportionally to the square root of the sample size — doubling your sample only reduces uncertainty by about 30%. But human psychology works against patience. We are pattern-recognition machines. When we see an early trend, our brains construct a narrative to explain it. These narratives feel true but are built from insufficient evidence, compounded by confirmation bias.
The Business Cost of Premature Decisions
When teams ship winners based on early results that later regress: the expected lift never materializes, eroding trust in experimentation. Downstream decisions about traffic allocation and marketing spend were predicated on a lift that does not exist. Most insidiously, it creates a culture of overconfidence where future experiments are designed based on learnings that were never real.
How to Protect Yourself
The primary defense: predetermine your sample size and test duration before launching, and do not look at results until the test is complete. Calculate based on your baseline conversion rate, minimum detectable effect, and desired statistical power. Run for at least two full business cycles. In one analysis, 771 out of 1,000 A/A tests reached 90% significance at some point. Over half reached 95%. Peeking at identical pages produces false winners more often than not.
Organizational Countermeasures
Restrict dashboard access during active tests. Designate a single person responsible for calling tests. Create a culture where premature peeking is treated as a methodological violation. Educate stakeholders about regression to the mean before they encounter it — the conversation is much easier when they already understand why early results are unreliable.
The Deeper Lesson
Regression to the mean reminds us that our intuitions about randomness are deeply flawed. We see patterns in noise and construct narratives from insufficient evidence. The organizations that build sustainable experimentation programs trade the thrill of early wins for the reliability of properly concluded tests. Patience is not just a virtue in experimentation — it is the mechanism by which truth separates from noise.