Central Limit Theorem

What Is the Central Limit Theorem?

The Central Limit Theorem (CLT) says that if you take repeated random samples from any population and average them, those averages form a bell-shaped normal distribution as sample size grows. The underlying data can be skewed, binary, bimodal, or wildly irregular; the averages still converge. This is why most A/B testing math works even though conversions are binary and revenue is heavily right-skewed.

Also Known As

• Data science teams: CLT, Gaussian convergence, asymptotic normality
• Growth teams: "the reason A/B tests work"
• Marketing teams: the normality assumption behind significance tests
• Engineering teams: sampling distribution convergence

How It Works

Imagine running an A/B test with 10,000 visitors per variant. Individual conversion outcomes are binary (0 or 1), not normal. But when you compute the mean conversion rate across 10,000 visitors, that mean is one draw from a distribution of possible sample means. If you hypothetically reran the test thousands of times, those 10,000-visitor averages would cluster around the true rate in a tight bell curve. That cluster is what your confidence interval is describing. At 10,000 visitors the CLT is fully in force; at 100 visitors it is wobbly.

Best Practices

• Do require at least 30 events per variant as a minimum before trusting normal approximations, and far more for skewed metrics like revenue.
• Do validate sample-size requirements before launch rather than hoping CLT will save you.
• Do switch to non-parametric or bootstrap methods when you have fewer than a few hundred conversions.
• Do not assume CLT kicks in instantly for revenue-per-visitor where 95% of users contribute zero.
• Do not confuse "data is normal" with "sample mean is normal" — only the latter is what CLT guarantees.

Common Mistakes

• Calling early wins on tests with 50 conversions per arm, where CLT has not stabilized.
• Applying z-tests to revenue metrics without checking whether the mean's sampling distribution has converged.
• Assuming CLT means you can ignore outliers; large outliers delay convergence.

Industry Context

• SaaS/B2B: Low traffic forces careful use of CLT; demo request tests often need weeks to reach stable sampling distributions.
• Ecommerce/DTC: High traffic makes CLT reliable for conversion tests but revenue-per-visitor still needs outlier handling.
• Lead gen/services: Sparse conversions (1-2%) mean you need very large samples before the CLT approximation is trustworthy.

The Behavioral Science Connection

Kahneman and Tversky's "law of small numbers" describes how humans intuitively expect small samples to reflect population properties. CLT is the mathematical rebuttal: only large samples of means converge to normality. The psychological pull to trust small samples is exactly why experimentation programs need the discipline of fixed sample-size calculations.

Key Takeaway

CLT lets you use normal-distribution math on non-normal data, but only once your sample is large enough for the averages to stabilize.