Confirmation Bias
The tendency to search for, interpret, and recall information in a way that confirms one's preexisting beliefs while ignoring contradictory evidence.
What Is Confirmation Bias?
Confirmation bias is the tendency to seek, interpret, and remember information that confirms what we already believe — and to discount or ignore information that contradicts it. It's less about dishonesty than about automatic pattern matching; we don't consciously decide to filter, we just do.
Also Known As
- Marketing teams: "belief-confirming data" or "cherry-picking"
- Sales teams: "hearing what we want to hear"
- Growth teams: "test-bias risk"
- Product teams: "HiPPO-driven interpretation"
- Behavioral science: Wason's (1960) confirmation bias
How It Works
Confirmation bias can lead a team to overinterpret an ambiguous headline estimate and discount a concerning segment. Pre-registration, uncertainty reporting, and planned follow-up measurement reduce that risk.
Best Practices
- Do pre-register your hypothesis, success metrics, and analysis plan before launching tests.
- Do have one team design tests and another analyze them when possible.
- Do actively seek disconfirming evidence — what would falsify your belief?
- Don't extend tests past planned sample size to "see if it turns significant."
- Don't explain away inconvenient segment results post-hoc.
Common Mistakes
- Stopping tests early when results align with beliefs; running them long when they don't.
- Segmenting data until something significant appears, then calling that the "real result."
- Letting the HiPPO decide the "true" interpretation of ambiguous results.
Industry Context
- SaaS/B2B: Product decisions driven by leadership preference rather than data.
- Ecommerce/DTC: Test interpretation that favors the redesign the team already loves.
- Lead gen/services: Selective use of positive client anecdotes while ignoring negative ones.
The Behavioral Science Connection
Peter Wason documented confirmation bias in the 1960s with his famous 2-4-6 task, showing that participants often sought confirming rather than disconfirming evidence. It connects to motivated reasoning, the HiPPO effect, and hindsight bias, and is a material risk in experimentation.
Key Takeaway
The best experimenters design tests to falsify their beliefs, not confirm them — and institutional safeguards matter more than individual willpower.