Selection Bias
A systematic error that occurs when the sample of users in an experiment is not representative of the target population, leading to results that don't generalize.
Selection bias is the most dangerous threat to experiment validity because it's invisible in the data. Your test can have perfect statistical significance, proper sample size, and clean implementation — and still produce wrong conclusions if the sample is biased.
Common Sources of Selection Bias in CRO
Day-of-week effects: starting a test on Monday and ending on Thursday biases toward weekday behavior. Traffic source shifts: a PR mention during your test floods the site with non-representative visitors. Self-selection: testing a new onboarding flow only on users who opt in means you're testing on the most engaged users.
The Randomization Trap
Proper randomization should eliminate selection bias, but implementation mistakes are common. Server-side randomization is more reliable than client-side. Cookie-based assignment can fail when users clear cookies or switch devices. Deterministic assignment based on user ID is the gold standard for avoiding selection bias.
How to Detect Selection Bias
Check your experiment for Sample Ratio Mismatch (SRM) — if your 50/50 split shows 52/48 or worse, something is wrong with randomization. Also check that pre-experiment metrics (before the treatment was applied) are balanced between control and variant. If conversion rate was already different before the test, your sample is biased.
Practical Application
Always run an SRM check. Always verify that pre-experiment metrics are balanced. Run tests for complete business cycles (full weeks, not partial). And be deeply suspicious of results from tests that started or stopped during unusual traffic periods (holidays, press coverage, outages).