Stratified Sampling
A sampling method that divides a population into distinct subgroups (strata) and samples from each proportionally, ensuring important segments are adequately represented and reducing estimation variance.
What Is Stratified Sampling?
Stratified sampling ensures each important subgroup of your population is represented in proportion to its size (or larger) across treatment and control. Rather than trusting pure randomization to balance segments, you explicitly guarantee balance, which reduces variance and tightens confidence intervals.
Also Known As
- Data science teams: stratified randomization, blocked randomization, stratification
- Growth teams: balanced assignment, pre-balancing
- Marketing teams: segment-balanced testing
- Engineering teams: stratified sampler, blocking
How It Works
Imagine an A/B test with 10,000 visitors per variant where 30% are mobile and 70% are desktop. Pure random assignment might give Variant A 2,850 mobile and Variant B 3,150 mobile by chance — a 300-visitor imbalance on a key segment. Stratified sampling splits assignment within each stratum: exactly 50% of mobile users to each variant, exactly 50% of desktop users to each variant. The resulting experiment has zero imbalance on device, which can reduce variance by 10-30% and shave days off required test duration.
Best Practices
- Do stratify on variables strongly correlated with the outcome.
- Do use stratification when running tests across device, geo, or user-tenure segments.
- Do combine stratification with CUPED for additional variance reduction.
- Do not stratify on too many variables; strata become too small to be useful.
- Do not stratify on post-treatment variables; that introduces bias.
Common Mistakes
- Stratifying on low-signal variables that do not reduce variance meaningfully.
- Forgetting to track strata so analysis cannot verify balance.
- Believing stratification fixes selection bias; it only balances pre-known segments.
Industry Context
- SaaS/B2B: Stratification by account size or plan tier reduces noise dramatically.
- Ecommerce/DTC: Device and traffic-source strata are standard for checkout tests.
- Lead gen/services: Stratification by source channel prevents noisy quality swings.
The Behavioral Science Connection
Stratified sampling addresses what Kahneman calls "the law of small numbers": the false intuition that small samples automatically reflect the underlying population. Stratification turns that intuition into reality by forcing representation explicitly.
Key Takeaway
Stratified sampling is a free variance reduction technique whenever you know a segment predicts your outcome.