Alpha Level

What Is Alpha Level?

Alpha is the long-run probability of rejecting the null when the null is true under the pre-specified analysis procedure. It is a choice, not a law of nature, and must be interpreted together with sidedness, multiplicity, stopping rules, and power.

Also Known As

• Data science: significance level, p-value threshold
• Growth: "the 95% confidence bar"
• Marketing: confidence level (1 - alpha)
• Engineering: decision threshold

How It Works

Run a two-sided test with alpha = 0.05 and baseline 10% conversion. The critical z-value is ±1.96. Any observed z above 1.96 or below -1.96 crosses the bar. At alpha = 0.10 the critical z shrinks to ±1.645 — you ship more winners and accept more false positives. At alpha = 0.01 the bar rises to ±2.576 — fewer false positives, but you need much more sample to clear it.

Alpha interacts with sample size. The additional sample required after changing alpha depends on the baseline, target effect, sidedness, variance, and power, so recalculate the design rather than applying a fixed multiplier.

Best Practices

• Set alpha and power from the consequences of false positives and false negatives, then document the tradeoff.
• Lock alpha in the test doc before launch. Ex-post alpha-shopping is cheating.
• Use alpha 0.025 one-sided only when a directional hypothesis is pre-registered and defensible.
• Apply corrections for multiple comparisons when testing multiple variants or metrics.
• Report the exact p-value, not just "significant / not significant," so future meta-analyses can be done.

Common Mistakes

• Mistaking alpha for the probability the variant is worse. A p-value of 0.04 does not mean 96% chance control is wrong — that is a Bayesian interpretation of a frequentist number.
• Using alpha 0.05 reflexively on everything regardless of stakes or reversibility.
• Changing alpha mid-test because the result is "almost there."

Industry Context

Choose alpha from the decision stakes, reversibility, multiplicity, and downstream economics. No industry-wide threshold or false-positive cost multiple applies.

The Behavioral Science Connection

Humans treat 0.05 as a magic number, a phenomenon called threshold fetishism. A p-value of 0.049 feels meaningfully different from 0.051, but the underlying evidence is nearly identical. Pre-registering alpha protects against the motivated reasoning that kicks in when a stakeholder-favored variant lands at p = 0.06.

Key Takeaway

Alpha is a business decision dressed up as a statistical one. Choose it deliberately, tie it to consequences, and lock it down before you see any data.