Stratified Sampling
A sampling method that divides a population into distinct subgroups (strata) and samples from each proportionally, ensuring important segments are adequately represented and reducing estimation variance.
What Is Stratified Sampling?
Stratified sampling or blocked randomization can enforce balance on selected pre-treatment strata. Its variance benefit depends on how predictive those strata are and how the design and analysis are implemented.
Also Known As
- Data science teams: stratified randomization, blocked randomization, stratification
- Growth teams: balanced assignment, pre-balancing
- Marketing teams: segment-balanced testing
- Engineering teams: stratified sampler, blocking
How It Works
Imagine an A/B test where device type predicts the outcome. Blocked randomization assigns users within the mobile and desktop strata, preventing chance imbalance on that selected variable. Any variance reduction must be estimated from how strongly the stratum predicts the outcome; it is not a fixed percentage or free time savings.
Best Practices
- Do stratify on variables strongly correlated with the outcome.
- Do use stratification when running tests across device, geo, or user-tenure segments.
- Do combine stratification with CUPED for additional variance reduction.
- Do not stratify on too many variables; strata become too small to be useful.
- Do not stratify on post-treatment variables; that introduces bias.
Common Mistakes
- Stratifying on low-signal variables that do not reduce variance meaningfully.
- Forgetting to track strata so analysis cannot verify balance.
- Believing stratification fixes selection bias; it only balances pre-known segments.
Industry Context
- SaaS/B2B: Stratification by account size or plan tier reduces noise dramatically.
- Ecommerce/DTC: Device and traffic-source strata are standard for checkout tests.
- Lead gen/services: Stratification by source channel prevents noisy quality swings.
The Behavioral Science Connection
Stratified sampling addresses what Kahneman calls "the law of small numbers": the false intuition that small samples automatically reflect the underlying population. Stratification turns that intuition into reality by forcing representation explicitly.
Key Takeaway
Stratification can improve precision when a pre-treatment segment predicts the outcome, but it adds design and analysis requirements and does not guarantee a material gain.