Confidence Interval
A range of values that likely contains the true effect of your A/B test variation — more informative than a single point estimate.
A confidence interval tells you the range within which the true effect likely falls. A 95% CI of [+2%, +8%] means you can be 95% confident the real lift is somewhere between 2% and 8%. This is far more useful than a single "5% lift" point estimate.
Why Confidence Intervals Beat p-values
A p-value tells you: "Is there an effect?" A confidence interval tells you: "How big could the effect be?" The second question is always more useful for business decisions.
A test might be statistically significant (p < 0.05) but have a confidence interval of [+0.1%, +0.5%] — meaning the effect, while real, is too small to matter. Conversely, an "insignificant" test with a CI of [-1%, +12%] tells you there might be a huge effect that you're underpowered to confirm.
Reading Confidence Intervals
- CI doesn't cross zero: The effect is statistically significant
- CI is narrow: You have a precise estimate (good sample size)
- CI is wide: You're uncertain about the true effect (need more data)
- CI lower bound is meaningful: Even the worst case is worth shipping
Practical Decision Framework
I recommend shipping changes where the lower bound of the 95% CI exceeds your cost threshold. If the CI is [+2%, +10%] and a 2% lift justifies the engineering cost, ship it — even though the true effect might be closer to 2% than 10%.