Survivorship Bias
The logical error of focusing on entities that passed a selection process while overlooking those that did not, leading to overly optimistic conclusions and flawed analysis.
What Is Survivorship Bias?
Survivorship bias is the mistake of analyzing only the "survivors" of a filtering process — the customers who converted, the companies that are still in business, the users who reached checkout — while ignoring everyone who did not. The surviving sample is systematically different from the starting population, so any conclusions drawn from it are suspect.
Also Known As
- Data science teams: survivorship bias, conditioning on a collider, sample selection
- Growth teams: funnel bias, converters-only analysis
- Marketing teams: "only looking at customers"
- Engineering teams: post-filter sampling error
How It Works
Imagine a checkout optimization test with 10,000 visitors per variant. You analyze users who reached the payment step and find Variant B increases payment-step conversion by 3%. But wait: Variant B also caused 500 more users to drop off at the shipping step before ever reaching payment. The payment-step survivors in Variant B are now a different population — filtered by a stricter gate — and their higher conversion rate reflects who made it through, not what Variant B actually did. The true end-to-end effect may be negative.
Best Practices
- Do analyze from the top of the funnel, including all randomized users.
- Do use intent-to-treat analysis as the primary evaluation method.
- Do compare dropoffs at each stage, not just success rates.
- Do not restrict analysis to "engaged users" defined post-treatment.
- Do not draw causal conclusions from customer-only surveys.
Common Mistakes
- Studying "what successful startups do" without studying failed ones.
- Reporting payment-step conversion rate as evidence of checkout-flow improvement.
- Analyzing only users who opened the email, ignoring those who did not.
Industry Context
- SaaS/B2B: Customer interview programs overweight survivors; churned users matter equally.
- Ecommerce/DTC: Cart-abandonment analyses that exclude non-starters miss 90% of the story.
- Lead gen/services: Lead-quality studies that exclude disqualified leads produce fantasy conclusions.
The Behavioral Science Connection
The WWII bomber story — armoring where survivors were hit, not where downed planes were hit — is the canonical example. Kahneman's "availability heuristic" explains why: survivors are present to study, so they feel representative. The casualties are invisible and get no weight. Good analysis resists this pull.
Key Takeaway
Always ask what the data you cannot see would tell you; survivorship bias is the silent killer of CRO conclusions.