Type II Error (False Negative)
Failing to detect a real effect — the risk is controlled by statistical power and equals 1 minus power.
What Is a Type II Error?
A Type II error is when a truly better variant loses, ties, or is called inconclusive. With power of 0.80 your Type II rate is 0.20 — one in five real winners gets killed. Because teams rarely re-test "losers," Type II errors compound silently: good ideas are buried, stakeholders lose faith in the team's roadmap, and a culture of "nothing ever moves the needle" takes hold.
Also Known As
- Data science: beta error, false negative, FN, miss
- Growth: "we killed a winner"
- Marketing: missed opportunity, buried lift
- Engineering: undetected change, sensitivity miss
How It Works
Suppose a true effect is smaller than the test was designed to detect. A flat estimate may reflect limited precision. A larger pre-registered replication can narrow the interval, but it is not guaranteed to produce a win.
Best Practices
- Choose target power before launch from the smallest effect you would act on, the consequences of missing it, and the available sample.
- Treat flat results as "insufficient evidence," not "no effect." Log the detectable MDE with the result.
- Revisit buried ideas annually using updated traffic and variance assumptions.
- Use variance reduction (CUPED, stratification, pre-exposure baselines) to raise power without more traffic.
- Prefer fewer, larger tests on the roadmap's top bets rather than many underpowered ones.
Common Mistakes
- Calling a flat result a "loss" and removing the idea from the backlog.
- Running short tests on low-traffic pages without a feasibility check, making realistic effects difficult to detect.
- Ignoring power entirely and focusing only on p-value discipline. Controlling Type I while Type II runs wild just means you ship less of everything, including real winners.
Industry Context
Low traffic and noisy downstream metrics can create high Type II risk in any industry. Calculate power for the metric that drives the decision rather than assuming a category-wide error rate.
The Behavioral Science Connection
Type II errors exploit loss aversion in the opposite direction: teams remember failures (shipped losers) more vividly than invisible losses (buried winners). The missing winners do not show up in any dashboard, so they are not grieved. Explicitly logging "effects we could have detected" turns an invisible loss into a visible one and changes investment decisions.
Key Takeaway
False negatives are easy to overlook because missed effects are invisible. Compare the interval with the minimum practical effect: an imprecise result may mean "not yet," while a precise interval can also rule out a worthwhile effect.