When a result is selected because it is extreme, a repeat measurement often lands closer to the underlying mean. That does not tell you a fixed amount of shrinkage, prove the first result was false, or rule out genuine heterogeneity.
What You'll Learn
- A plain-English definition of regression to the mean (with the famous Galton example)
- The simple decomposition behind noisy A/B test estimates
- Why shrinkage requires a prior or replication evidence, not a lookup table
- How to compare regression, false positives, and genuine heterogeneity
- How to distinguish regression from a false positive (subtle but important)
Quick Stats Reference
Remember these distinctions:
>
- Observed lift = True effect + Noise (the entire concept in one equation).
- Selection matters. Regression to the mean is expected when an observation was selected for being extreme.
- Shrinkage is model-dependent. The amount depends on uncertainty, prior effect sizes, and selection rules.
- Replication informs; it does not automatically identify one mechanism.
The Core Definition
Regression to the mean = the statistical phenomenon where extreme measurements tend to be followed by less extreme ones, closer to the underlying average. Applies to any noisy measurement, including A/B test lifts. The first time you measure an extreme value, part of that value reflects the true underlying signal and part reflects a favorable random fluctuation. The next measurement, on the same underlying system, has a fresh random component — which is on average closer to zero. So the second measurement regresses toward the truth.
This is not a CRO-specific quirk. It operates on heights, baseball averages, stock returns, hospital mortality rankings, MVP awards, and any other domain where you're observing a noisy measurement of a stable underlying signal.
Why This Matters For New Analysts
Three reasons regression to the mean is useful early in your career:
It can explain part of why selected big wins disappoint at rollout. Implementation, population, seasonality, and measurement changes can also contribute. Do not assign the entire gap to one mechanism without evidence.
It changes how you read every case study. Once you understand the mechanism, you stop taking headline numbers at face value. You discount automatically. You read replication failures as the diagnostic signal they actually are.
It gives you the vocabulary to manage stakeholders. When leadership asks why the rollout under-delivered relative to the projection, "regression to the mean" is a defensible, statistically grounded answer. It is also defensible _before_ the under-delivery, which lets you set realistic expectations up front rather than apologizing after.
The Galton Origin Story
The concept comes from Francis Galton in the 1880s, studying heights of fathers and their adult sons.
Galton observed that exceptionally tall fathers tended to have sons who were tall, but on average less tall than the fathers themselves. Exceptionally short fathers had sons who were short, but on average less short. Each generation appeared less extreme than the previous one.
His initial interpretation: some active force was pulling the population toward the average — a biological mechanism shrinking variation over time.
His eventual interpretation, which is correct: the effect is statistical, not biological. Heights are noisy measurements. A person's height equals their underlying genetic predisposition plus a random component (nutrition, growth-spurt timing, childhood illness). When you observe a very tall father, part of his height reflects genetic potential and part reflects a favorable realization of the random component. His son inherits the genetic potential but receives a fresh, independent draw of the random component. The fresh draw is unlikely to be equally favorable. So the son's height is tall but less tall than the father's.
Same dynamic, modern examples:
>
- Athletes who win MVP one year tend to perform below that level the next.
- Films with breakout opening weekends underperform their openings on subsequent weekends.
- Stocks that crashed dramatically tend to recover partially.
- Hospitals at the extreme of mortality rankings drift back toward average on subsequent measurement.
A/B test lifts are noisy measurements. The same dynamic applies.
The Mechanism in One Equation
Every observed lift on an A/B test can be decomposed:
Observed lift = True effect + NoiseTrue effect: the underlying causal impact of your variant — the lift you would see if the test were run infinitely many times under identical conditions.
Noise: everything else. The specific users who arrived during the window, day-of-week mix, traffic-source mix, random variation in conversion behavior, the discrete sample of orders that landed in each arm.
If you run a single test and observe a 25% lift, the observation alone does not tell you how much was true effect and how much was noise. The two components are confounded inside the single number.
You can reason about the decomposition only with a relevant prior or comparable evidence. If your program has a consistently defined historical ledger, model its effect-size distribution and the selection rule. Without that information, a large estimate has wide uncertainty but no defensible fixed “true effect.”
The noise was favorable during your measurement window. This is not a moral or methodological failing — it is a property of how distributions of noisy measurements behave. Extreme observations are extreme partly because the noise component was extreme.
When you rerun under comparable conditions, the new estimate receives fresh sampling noise. If the first observation was selected for being extreme, the expected repeat estimate is closer to the underlying mean. One replication is still noisy and does not automatically become “the truth.”
Build Shrinkage from Evidence
There is no defensible lookup table that maps a reported lift to its “likely true effect.” A shrinkage estimate requires the observation's standard error, a relevant prior effect distribution, and the rule that selected the result for attention.
When those inputs are unavailable, show the confidence interval, include zero in weak planning scenarios, and treat the case study as hypothesis-generation rather than converting it with a universal haircut.
A Worked Example
A pattern that recurs in CRO content and in new-analyst work:
Test 1. A variant ships on Page A. After approximately 10 days, the dashboard reports a low-twenties percent CVR lift. The team declares a winner, projects annualized impact, plans rollout.
Because the team selected and publicized an extreme short-window result, regression to the mean is one plausible contributor. The record does not identify how much of the estimate was effect versus noise.
Test 2. Same variant ships on Page B (comparable surface, overlapping audience). After a comparable window, the dashboard reports a flat result.
This smaller second result is consistent with regression to the mean, a false positive, a context difference, implementation drift, or a combination. Compare designs and intervals before choosing among those explanations.
The team's typical interpretation: "audiences are different."
A disciplined interpretation: the original headline is not a reliable rollout forecast. The second observation updates the evidence, but neither result alone identifies the exact true effect.
Occam's razor: When two explanations fit the data and one requires fewer special assumptions, that one is usually correct. The "different audience" explanation requires a story. The regression explanation requires only basic statistical mechanics.
Test the “Different Audience” Explanation
Heterogeneous treatment effects are real. Sometimes a tactic genuinely performs better for new users than returning users, for paid traffic than organic, or in one product category than another.
Do not assign generic 1-in-5 or 4-in-5 probabilities without a relevant model. If heterogeneity was predicted before the replication and the design can estimate it, test that interaction. If the explanation appears only after an unfavorable result, label it exploratory and downgrade the original generalization.
The dual-standard diagnostic: If the replication had returned at the same magnitude as the first test, what would you have said? Likely: "confirmed; audiences are similar." When it returns flat: "audiences are different." That dual standard is the diagnostic signal. A neutral interpretation has to make the same form of inference in both directions.
Distinguishing Regression from a False Positive
These two concepts are related but distinct, and conflating them is a common error.
A false positive is a finding that is entirely noise. The true effect is zero. On replication, the result is flat because there was nothing real there to begin with. Expected lift on replication: 0.
Regression to the mean operates whether or not the underlying effect is real. Even when there _is_ a real effect (e.g., +5%), an observed lift of +25% contains more noise than signal. On replication, the observed result is closer to the +5% true effect, not flat. Expected lift on replication: positive but smaller than the original.
| False positive | Regression to the mean | |
|---|---|---|
| Underlying truth | No effect | Real but smaller effect |
| Expected on replication | ~0 | Smaller positive value |
| Cause | Test crossed threshold by chance | Original observation in upper tail |
| Operational response | Same: discount, replicate, don't extrapolate |
For practical decision-making, the operational implication is similar in both cases. The appropriate response to an extreme observed lift is to discount it substantially, replicate before scaling, and avoid extrapolating from the headline.
The fix in Early Stopping in A/B Tests addresses the false-positive side. Regression to the mean compounds with that problem: even after peeking is corrected and tests are run with proper discipline, large reported wins will still regress on replication, just less dramatically.
The Remedy: Four Components
1. Shrinkage estimators. Bayesian methods explicitly pull extreme observations toward the prior. If you've run hundreds of tests, the empirical distribution of past lifts is your prior. New observations get shrunk toward that distribution. Large observed lifts shrink substantially; marginal observed lifts shrink less. The reported point estimate is the posterior mean, not the raw observation.
2. Hierarchical models. When multiple related tests are available (multiple pages, multiple campaigns), a hierarchical model jointly estimates a shared effect distribution and individual test effects. Each test borrows statistical strength from the others. Outlier observations are pulled toward the group mean.
3. Replication built into the program. The simplest operational remedy. Replicate every important reported win before generalizing. Run the same change on a second comparable surface or in a second period. If replication produces a materially smaller effect, size the rollout to the smaller estimate. If replication is flat, the original was likely a false positive — do not generalize.
4. Confidence intervals on every reported result. Even without shrinkage or hierarchical models, reporting the point estimate alongside its CI forces the reader to see the uncertainty. A "+25% lift, 95% CI [-2%, 52%]" is a different finding than a "+25% lift" with no interval.
Quick Tips for New Analysts
- Do not apply a universal shrinkage table. Use a relevant prior, interval, and sensitivity range.
- Replicate material findings when feasible. Treat the replication as another estimate, not automatically the truth.
- Read replication failures literally. If a tactic worked once and failed to reproduce, the original is unreliable. Resist the "different audience" reframe.
- Apply the dual-standard test. Would you have explained a confirming replication the same way you're explaining a failing one? If not, you're rationalizing.
- Report confidence intervals on every test result, even if your tool doesn't surface them. Compute them in your reporting layer.
- Set stakeholder expectations on uncertainty before rollout. Show the interval and the assumptions behind any post-launch forecast.
The Behavioral Economics of Resistance
Analysts resist regression to the mean for predictable reasons:
- The headline lift has already become part of the analyst's professional identity.
- The case study has been written.
- The annualized revenue figure has been presented to leadership.
Accepting that an exciting point estimate is uncertain can register as a loss. The response may reflect incentives, identity, or asymmetric reactions to downward updates. Do not assume a universal loss-aversion coefficient or a fixed one-third true effect.
This is the same dynamic that explains why "different audience" is the dominant interpretive move. It preserves the upward update from the first test while explaining away the downward signal from the replication failure.
The trap-avoidance habit: Train yourself to apply the symmetric standard from the start of your career, before the asymmetry becomes habitual. The dual-standard diagnostic is the practical tool — apply it to every replication outcome, in both directions.
FAQ
Does this mean my reported A/B test win was not real?
Not necessarily. The magnitude is likely overstated. A real effect may exist beneath the inflated reported lift. The appropriate response is to replicate, not to discard the result. If replication confirms a smaller effect, you have evidence of a real but more modest win. If replication is flat, the original was likely predominantly noise.
How do I distinguish regression from genuine audience heterogeneity?
Pre-registration. If, before the second test, you documented "this audience is expected to respond differently because of X and Y," and the result matches the prediction, heterogeneity is more credible. If the narrative emerges only after the result, label it post-hoc and test it rather than presenting it as established.
My program reports a 30% win rate. What does that tell me?
Not much until the denominator and decision rule are defined. Audit test mix, power, stopping rules, metric definitions, and whether all outcomes are included. There is no universal mature-program range that makes 30% healthy or broken by itself.
Should I use Bayesian methods to handle this automatically?
In principle, yes. In practice, inspect the tool's model, prior, decision rule, and calibration documentation. A “Bayesian” label alone does not establish hierarchical shrinkage or validity for your stopping policy; you may need a separate reporting-layer model.
What's the rule of thumb when reading any case study?
Do not use a universal percentage discount. Treat the case study as a hypothesis unless it reports enough design and uncertainty detail to support a quantitative prior. If replication data exists, combine the evidence with an appropriate model rather than automatically replacing the first estimate with the second.
Build the Mental Model Early
For new CRO analysts, regression to the mean is a useful guard against treating selected extreme results as forecasts. It gives you a vocabulary for explaining why a rollout estimate should include uncertainty before post-launch evidence arrives.
I built GrowthLayer with shrinkage and replication as first-class concepts. Every reported lift is presented with a confidence interval. Large observed lifts are flagged for replication before being treated as program-level evidence. The platform uses the empirical prior from a program's past tests to shrink new observations into more accurate estimates, so rollouts are sized to the realistic expected effect rather than to the upper tail.
To find experimentation roles where this discipline is the operating standard, explore open positions on Jobsolv.
Or book a consultation for help training a new analyst team in regression-to-mean-aware reporting, or for a methodological audit of an existing program.