CUPED (Variance Reduction)
Controlled-experiment Using Pre-Experiment Data — a variance reduction technique that uses pre-period covariates to shrink metric variance and boost test power.
What Is CUPED?
CUPED, introduced by Deng, Xu, Kohavi, and Walker at Microsoft in 2013, is a technique that removes predictable variance from experiment metrics by adjusting for pre-period behavior. For a valid pre-treatment covariate, simple CUPED variance reduction is tied to the squared correlation with the outcome; the realized improvement must be measured on the company's data.
Also Known As
- Data science: CUPED, variance reduction via covariates, regression adjustment
- Growth: "the technique that cuts test time in half"
- Marketing: pre-period control adjustment
- Engineering: covariate-adjusted estimator
How It Works
You are testing a change to product recommendations. Primary metric is 28-day revenue per user. You compute each user's 28-day revenue before the test (the covariate X). Your adjusted metric is: Y_cuped = Y - theta * (X - mean(X)), where theta = cov(Y, X) / var(X). If pre-period and in-period revenue correlate at r = 0.7, variance drops by r^2 = 49%. A test that would have taken 8 weeks at 0.8 power now takes ~4 weeks.
CUPED requires the covariate be measured before randomization and be uncorrelated with treatment assignment (which is trivially true for pre-period data).
Best Practices
- Use long pre-periods (28–56 days) for heavy-tailed metrics like revenue.
- Re-estimate theta on each experiment — correlations drift.
- Validate with A/A tests that CUPED doesn't introduce bias before rolling out.
- Apply CUPED to continuous metrics first (revenue, sessions, time) where variance is the biggest problem.
- Combine with stratification on high-signal categorical variables (plan tier, region) for additional reduction.
Common Mistakes
- Using a covariate measured during the experiment. This invalidates randomization and biases the estimate.
- Forgetting new users. New users have no pre-period data; handle them with a separate estimator or default-to-mean.
- Applying CUPED to binary conversion when the covariate correlation is near zero. You add code complexity for no benefit.
Industry Context
In SaaS/B2B, CUPED may help on MRR or usage metrics when a valid pre-period measure predicts the outcome. In ecommerce, repeat-buyer behavior may provide useful covariates for GMV or AOV. In lead gen, new leads often lack pre-period behavior, while retention and expansion tests may have suitable data.
The Behavioral Science Connection
Humans see noise as signal — a bias that is formalized in the "hot hand" and gambler's fallacies. CUPED removes one major source of noise (between-user variability) so the remaining signal is more clearly attributable to the treatment. It is a statistical instantiation of the behavioral principle: reduce ambient noise to hear the thing that matters.
Key Takeaway
CUPED can improve sensitivity when a valid pre-treatment covariate predicts the outcome. Its value depends on the realized correlation, data coverage, implementation quality, and decision volume.