Confirmation Bias

What Is Confirmation Bias?

Confirmation bias is the tendency to seek, interpret, and remember information that confirms what we already believe — and to discount or ignore information that contradicts it. It's less about dishonesty than about automatic pattern matching; we don't consciously decide to filter, we just do.

Also Known As

• Marketing teams: "belief-confirming data" or "cherry-picking"
• Sales teams: "hearing what we want to hear"
• Growth teams: "test-bias risk"
• Product teams: "HiPPO-driven interpretation"
• Behavioral science: Wason's (1960) confirmation bias

How It Works

A team has a strong hypothesis: "users want a simpler checkout." They run an A/B test. The simplified version shows a small, non-significant lift, plus a meaningful drop for high-AOV customers. The team celebrates the headline lift, rationalizes the AOV drop as "noise," and ships. Six months later, revenue is down and no one remembers the original warning signs. Confirmation bias worked exactly as it always does.

Best Practices

• Do pre-register your hypothesis, success metrics, and analysis plan before launching tests.
• Do have one team design tests and another analyze them when possible.
• Do actively seek disconfirming evidence — what would falsify your belief?
• Don't extend tests past planned sample size to "see if it turns significant."
• Don't explain away inconvenient segment results post-hoc.

Common Mistakes

• Stopping tests early when results align with beliefs; running them long when they don't.
• Segmenting data until something significant appears, then calling that the "real result."
• Letting the HiPPO decide the "true" interpretation of ambiguous results.

Industry Context

• SaaS/B2B: Product decisions driven by leadership preference rather than data.
• Ecommerce/DTC: Test interpretation that favors the redesign the team already loves.
• Lead gen/services: Selective use of positive client anecdotes while ignoring negative ones.

The Behavioral Science Connection

Peter Wason documented confirmation bias in the 1960s with his famous 2-4-6 task, showing subjects sought confirming rather than disconfirming evidence even when the latter was more informative. It connects to motivated reasoning, the HiPPO effect, and hindsight bias. It's the single most dangerous bias in experimentation, because it corrupts the very process designed to reduce it.

Key Takeaway

The best experimenters design tests to falsify their beliefs, not confirm them — and institutional safeguards matter more than individual willpower.