Statistical Significance vs Practical Significance
Separate statistical evidence from business value using effect sizes, confidence intervals, implementation costs, and explicit decision thresholds.
- Can control a pre-specified long-run error rate
- Provides a reproducible rule when the design is followed
- Helps distinguish sampling variation from stronger evidence against a null
- Works alongside effect estimates and uncertainty intervals
- Connects experiment results to costs, benefits, and risks
- Makes the action threshold explicit before results are known
- Encourages attention to effect size and interval width
- Can account for implementation effort and reversibility
- Does not measure effect size or economic value
- Depends on sample size, model assumptions, and analysis choices
- A non-significant result does not prove no effect
- A significant result can still be too uncertain or too small to act on
- Thresholds differ across products, surfaces, and organizations
- Revenue translation requires assumptions that should be sensitivity-tested
- Short experiments may not establish durability
- Stakeholders can move thresholds after seeing results unless they are documented
Do not replace statistical significance with a different binary label. Estimate the effect, show its uncertainty, and compare the plausible outcomes with a documented decision threshold. Ship only when the evidence and economics support the action, not merely because a dashboard declares a winner.— Atticus Li
Two Questions, Not Two Competitors
Statistical significance asks a question about evidence under a model. Practical significance asks whether an effect is worth acting on. A sound experiment decision needs both, but they should not be collapsed into one label.
If a frequentist test returns a small p-value, that means the observed data would be relatively unusual under the specified null and analysis assumptions. It does not mean there is a corresponding probability that the variant is better. It also does not say how large the effect is.
Read the Estimate and Interval
Start with the estimand: absolute conversion-rate difference, relative lift, revenue per visitor, retention, or another decision-relevant quantity. Report the point estimate and a confidence interval.
Interpret the interval as a range of effect values compatible with the data and procedure, while remembering its repeated-sampling meaning. If the interval contains meaningful harm and meaningful benefit, the decision is uncertain even if a threshold has been crossed. If the interval is narrow around a trivial effect, more traffic may not make the change worth shipping.
Define Practical Value Before Results
Set a minimum worthwhile effect from the economics of the specific change. Include:
- implementation and maintenance cost;
- expected exposure and the metric's unit value;
- downside risk and reversibility;
- impact on guardrail metrics;
- confidence that the effect will persist after launch;
- opportunity cost relative to other work.
Do not copy one threshold across every test. A reversible copy change and a billing-flow redesign can justify different decision rules without inventing universal risk tiers or dollar cutoffs.
Keep MDE in Its Proper Role
The minimum detectable effect used in sample planning is the effect size a design is set up to detect with specified power and error rate under stated assumptions. It is not a promise that smaller effects do not exist, and it is not automatically the minimum effect worth shipping.
Ideally, the minimum worthwhile effect informs the design target. If available traffic cannot estimate effects near that threshold with useful precision, the honest outcome may be that the experiment cannot answer the business question in the required time.
Make the Decision Auditable
Record the threshold, cost assumptions, analysis plan, and action rule before reading the outcome. After the test, show sensitivity to uncertain inputs rather than presenting one revenue forecast as fact. Then track the implemented change to learn whether the experiment's estimate transferred to production.