Variance

What Is Variance?

Variance is the average squared distance of each observation from the mean. It is the square of the standard deviation and is the natural unit of spread for additive math: variances of independent measurements add, standard deviations do not. This makes variance the preferred input when designing experiments or combining uncertainties.

Also Known As

• Data science teams: sigma squared, Var(X), second central moment
• Growth teams: noise, spread
• Marketing teams: variability
• Engineering teams: variance, volatility, dispersion

How It Works

Imagine two A/B tests each with 10,000 visitors per variant. Test A measures conversion rate with a variance of 0.0001 (SD of 1%). Test B measures revenue per visitor with a variance of 1,600 (SD of $40). Because variance is squared, Test B's variance is 16 million times larger than Test A's in raw units; you cannot compare them directly. That is why variance drives sample-size formulas through the ratio of variance to squared effect size.

Best Practices

• Do plan sample sizes using variance, since the sample-size formula scales linearly with variance.
• Do use variance decomposition to understand whether noise comes from users, time, or the treatment.
• Do apply variance reduction techniques (CUPED, stratification) to shrink required sample sizes.
• Do not interpret variance in the same unit as the mean; standard deviation is easier to read.
• Do not mix sample variance (divided by n-1) with population variance (divided by n) in calculations.

Common Mistakes

• Ignoring that revenue variance is typically driven by a tiny fraction of users (whales).
• Assuming variance is constant across segments when it varies dramatically between new and returning users.
• Forgetting that variance decreases with sample size, making peeking extra dangerous.

Industry Context

• SaaS/B2B: MRR and ARR metrics have huge variance; CUPED can reduce it by 30-50% using pre-period data.
• Ecommerce/DTC: Order-value variance is dominated by a few large orders; winsorization commonly halves it.
• Lead gen/services: Lead-to-close variance is enormous; tests often need millions of impressions.

The Behavioral Science Connection

The bias-variance tradeoff in machine learning mirrors the cognitive tradeoff between stereotypes (high bias, low variance) and case-by-case reasoning (low bias, high variance). Kahneman's "System 1" is high-bias and low-variance; "System 2" is the opposite. Good experimenters recognize both failure modes.

Key Takeaway

Variance drives sample-size requirements; reducing variance is the fastest way to speed up your experiment program.