Type I Error (False Positive)

What Is a Type I Error?

A Type I error occurs when a valid analysis rejects a true null hypothesis. At alpha 0.05, each test has a 5% Type I error rate when its null is true. If all 40 tests in a simple portfolio were true nulls, independent, and tested once without correction, the expected number of false positives would be two; real portfolios mix null and non-null effects and require more careful accounting.

Also Known As

• Data science: alpha error, false positive, FP
• Growth: "we thought we won but we didn't"
• Marketing: phantom lift, ghost winner
• Engineering: false alarm, spurious detection

How It Works

Run an A/A test where both variants are identical and set alpha to 0.05. Under the assumptions of a valid continuous test statistic, the p-value is uniformly distributed under the null. Repeatedly checking an ordinary fixed-horizon p-value and stopping on significance can inflate error above 5%; the amount depends on the monitoring schedule and dependence between looks. Use a valid sequential method or simulate the proposed stopping rule.

Multiplicity compounds this. Ten simultaneous variants at alpha 0.05 have roughly a 40% family-wise error rate.

Best Practices

• Lock alpha before the test and do not lower the bar mid-flight.
• Use Bonferroni or Benjamini-Hochberg corrections for multi-variant or multi-metric tests.
• Adopt sequential testing methods (always-valid p-values, mSPRT) if peeking is unavoidable.
• Track your shipped-winner replication rate as a quality indicator, but do not equate non-replication with a false positive; power, drift, and implementation differences also matter.
• Require a holdout or replication test for high-stakes wins.

Common Mistakes

• Peeking daily and stopping at first significance. This is a common source of avoidable false positives.
• Claiming a "win" on a tertiary metric when the primary was flat. Multiplicity was not accounted for.
• Running dozens of segment cuts post-hoc and reporting the ones that look good.

Industry Context

In SaaS/B2B, false positives are especially costly because you usually cannot ship and iterate rapidly — a bad pricing experiment can take a quarter to unwind. In ecommerce, the volume of tests means false positives accumulate into "optimization debt" where the site is a Frankenstein of non-effects. In lead gen, false positives in the top funnel get amplified by downstream teams who attribute pipeline to "the winning variant."

The Behavioral Science Connection

Confirmation bias makes Type I errors invisible. Teams remember the wins, forget the regressions, and build a narrative of continuous progress that is half fiction. False positives also feed the bandwagon effect — once a team "knows" a change worked, they propagate it, fight against removing it, and treat any evidence against it as a measurement problem.

Key Takeaway

Every program has a false positive rate. The question is whether you know what it is and whether you are controlling it deliberately — or letting peeking, multiplicity, and post-hoc slicing set it for you.