Minimum detectable effect, or MDE, is the effect size an experiment is designed to detect with a specified probability, given its sample size and statistical assumptions. An A/B test planned for a 10% relative lift at 80% power has about an 80% chance of detecting that lift if it is real. It can still detect smaller effects, and it can still miss the effect it was designed around.
Choosing MDE means connecting two questions: What improvement would justify the change? And what can the available traffic measure reliably? Those answers may differ. A useful test plan makes that gap visible before anyone launches the experiment.
MDE example: a 10% lift from a 5% baseline
Consider a hypothetical page with a 5% conversion rate. A 10% relative improvement moves the rate to 5.5%. The absolute difference is 0.5 percentage points, not ten points.
With independent visitor-level randomization, equal allocation, a binary conversion outcome, a fixed-horizon analysis, and a two-sided 5% significance level, detecting that change at 80% power needs approximately 31,300 visitors per variant. The sample-size calculation guide shows the formula and assumptions.
The following planning estimates hold those assumptions constant. All figures are illustrative calculations, not company results.
| Relative MDE | Target conversion rate | Approximate sample per variant | What the design asks |
|---|---|---|---|
| 5% | 5.25% | 122,200 | Can we reliably detect a smaller improvement? |
| 10% | 5.5% | 31,300 | Can we reliably detect this intermediate change? |
| 20% | 6% | 8,200 | Can we reliably detect a larger improvement with less traffic? |
The values are rounded upward for planning. Halving the effect from 10% to 5% relative requires roughly four times the sample. That is why a seemingly modest request for more sensitivity can change the delivery schedule substantially.
An MDE calculator reverses this question: given a baseline, sample budget, power, and significance level, what effect can the design detect at that power? Check whether its traffic input means one variant or both, and whether its method matches your planned analysis.
The smallest worthwhile effect and the detectable effect
The smallest worthwhile effect comes from the decision you need to make. A small conversion improvement might justify changing a sentence. The same improvement might not justify a costly checkout rebuild with ongoing maintenance and support costs.
The detectable effect comes from a statistical design. It depends on traffic, variability, allocation, the analysis method, and the error rates you accept. Choosing a larger MDE can shorten recruitment, but it does not make smaller effects commercially irrelevant.
When I evaluate an experiment proposal, I want these two values written separately. If a modest improvement would be worth shipping but the test can only detect a much larger one reliably, a nonsignificant result may leave the important business question unresolved. That limitation belongs in the proposal, not just in the final readout.
Use contribution margin, implementation effort, potential harm, and maintenance costs when judging value. If you translate a hypothetical lift into annual revenue, label that calculation as a scenario. It assumes the effect persists, the future audience resembles the test audience, and downstream customer behavior does not offset the improvement. A scenario is not measured revenue.
Choosing an MDE before launch
Start by naming the decision and primary outcome. “Ship the new onboarding flow if it improves activation without unacceptable retention harm” is more useful than “find a statistically significant result.” Define activation, its observation window, and the relevant guardrails before selecting the effect size.
Next, estimate the smallest improvement that would change the decision. Discuss that threshold with the people responsible for implementation and the business outcome. A reversible copy change and a major pricing change do not have the same costs or downside.
Then estimate traffic using new eligible randomized units, consistent with your analysis. Use the audience that can enter this experiment, not all site visits. Allow for ramp-up, targeting, repeated visitors, and the time needed for outcomes to mature. Compare several plausible MDEs against that traffic budget.
Finally, decide whether the design can answer the question in time. If a worthwhile effect needs months of recruitment and the product will change substantially during that period, the calculation has exposed a planning constraint. It has not supplied a reason to quietly inflate the MDE until the calendar looks convenient.
The useful alternatives depend on the situation: increase the available sample, improve measurement precision with a justified method, test a stronger intervention, or make a research-led decision with a clear account of what remains uncertain. The calculator guide helps compare traffic scenarios; it cannot choose the business tradeoff for you.
Why MDE is not a detection cutoff
Power changes smoothly with the true effect. In the example above, about 31,300 visitors per variant provides roughly 80% power for a 10% relative lift. If the true lift is only 5% relative, the same design has roughly 30% power under the same normal approximation. That is lower sensitivity, not zero sensitivity.
Larger true effects are generally easier to detect in this design, but no individual outcome is guaranteed. Nor does an observed effect larger than the planned MDE retrospectively prove the test was adequately powered: the observed estimate itself contains sampling error.
The statsmodels power documentation describes the two-independent-proportion model used for these illustrative comparisons. Other metrics and methods have different power calculations. For a revenue metric or clustered account experiment, use an analysis that represents that structure.
Historical experiments can inform plausible effects, but compare like with like. An archive containing only successful tests overstates what a typical new test is likely to achieve. Differences in audience, intervention, baseline, and metric also limit comparisons. An optimistic forecast should not quietly replace the minimum effect the decision requires you to understand.
What a nonsignificant result can tell you
A nonsignificant result is not automatically evidence of no effect. It is also not automatically useless. Read the effect estimate and confidence interval against the decision thresholds you set before launch.
For example, imagine a fictional result whose interval includes both meaningful harm and a worthwhile benefit. The result leaves considerable decision uncertainty. An interval concentrated near zero could instead rule out the large improvement needed to justify an expensive rollout, even though it includes zero.
Conversely, a statistically significant positive effect can be too small to justify implementation. The American Statistical Association's statement distinguishes statistical significance from effect size and practical importance. The rollout decision needs all three: the estimated effect, its uncertainty, and the consequences of acting.
If the aim is to show that a new version is no worse than the current one by more than an acceptable margin, plan an appropriate noninferiority design. A failed superiority test does not establish that conclusion. MDE alone does not define every decision rule a business may need.
A useful record of the decision
The experiment plan should explain why the chosen sensitivity is appropriate. Record the baseline and its source, the smallest worthwhile effect, the planned MDE in relative and absolute units, sample per variant, allocation, power, significance level, observation window, and stopping rule.
Also record what the test will not resolve. If the feasible design has low power for small but worthwhile improvements, say how you will interpret an inconclusive result. If downside matters more than upside, explain how the harm threshold and guardrail analysis affect the plan.
Keep these choices in the Experiment Decision Brief, which you can download and share with the people building and reviewing the test. Before trusting the eventual readout, use the SRM diagnostic to check whether the analyzed population remains consistent with the intended assignment.
FAQ
What is a good MDE for an A/B test?
There is no universal percentage. Choose an effect that connects to a business decision, then check the sample needed to detect it at the planned power. If traffic only supports a larger MDE, document the smaller effects the design may miss rather than calling them unimportant.
Should MDE be absolute or relative?
Write both. At a 5% baseline, a 10% relative MDE equals a 0.5-percentage-point absolute change. Calculations use a clearly defined effect scale; showing both prevents a stakeholder's “10% improvement” from becoming an entirely different input.
Can I change MDE after the experiment starts?
Do not revise it in response to observed treatment results and then treat the original fixed-horizon analysis as unchanged. Adaptive designs and certain sample-size re-estimation methods can be valid when appropriately specified. Otherwise, obtain statistical review, document the departure, and distinguish exploratory findings from a confirmatory follow-up. Simply extending the test until a result becomes significant is not a remedy.
Does 80% power mean an 80% chance that the treatment works?
No. It means that, if the assumed true effect exists and the model assumptions hold, the planned procedure rejects the no-difference null about 80% of the time across repeated experiments. It does not assign a probability to the business hypothesis being true.