Variance Reduction Techniques
Methods that shrink the noise in experiment metrics — including CUPED, stratification, post-stratification, and control variates — to improve sensitivity.
What Is Variance Reduction?
Variance reduction is any valid technique that makes experiment metrics less noisy without changing the estimand. Lower variance can mean tighter confidence intervals and smaller MDEs. Its value depends on covariate quality, engineering cost, metric choice, and decision volume. Common techniques include CUPED, stratification, post-stratification, control variates, and ratio-metric delta methods.
Also Known As
- Data science: noise reduction, sensitivity enhancement, adjustment methods
- Growth: "making tests faster without more traffic"
- Marketing: signal improvement
- Engineering: estimator efficiency improvements
How It Works
Baseline checkout conversion is 6%, variance per user is roughly 0.06 * 0.94 = 0.056. With 20,000 per arm, standard error is ~0.24pp and MDE is ~0.66pp. Apply stratification on device type (desktop converts at 9%, mobile at 4%): the within-stratum variances are smaller, and the pooled estimator is roughly 20% more efficient. Now stack CUPED using 30-day pre-period sessions: another 25% reduction. Combined, your MDE drops to ~0.44pp — a 33% improvement on the same traffic.
Best Practices
- Start with post-stratification on a single high-signal dimension — easy to implement, low risk.
- Layer CUPED on continuous metrics. The largest gains come from user-level behavioral covariates.
- Validate every variance reduction estimator against A/A tests. Bias here silently breaks every readout.
- Document which metrics use which adjustments so readouts are interpretable across the org.
- Do not apply variance reduction to metrics you will only read once — the engineering cost isn't worth it.
Common Mistakes
- Using adjustment variables correlated with treatment. Post-treatment bias destroys the estimator.
- Applying post-stratification without the right weights. Unweighted strata collapse back to naive averages.
- Chasing techniques before fixing basic methodology. CUPED on a program that peeks daily at p-values is putting a spoiler on a car with flat tires.
Industry Context
In SaaS/B2B, variance reduction can help when traffic is scarce and useful pre-treatment covariates exist. In ecommerce, it may improve sensitivity on noisy revenue metrics. In lead gen, ratio-metric methods and stratification by lead source may help when the design assumptions are satisfied.
The Behavioral Science Connection
Attention is a finite resource and variance is an attentional tax. Every point of ambient noise forces stakeholders to squint at results, pattern-match on trends, and argue about interpretation. Variance reduction is less a statistical trick than a cognitive-load reduction — clearer signals produce better decisions and less political debate at the readout meeting.
Key Takeaway
After basic methodology is sound, evaluate variance reduction against other investments using measured sensitivity gains, engineering cost, and the number of decisions it will support.