The Decision That Determines Everything Else
Before you launch an A/B test, you make one decision that shapes the entire experiment: the minimum detectable effect. It determines how long the test runs, how much traffic you need, and what size improvements you can reliably identify.
Get the MDE wrong and everything downstream breaks. Set it too small and the test runs for months, blocking other experiments. Set it too large and you miss real improvements that would have been worth shipping. Most teams either pick an arbitrary number or skip the decision entirely.
MDE deserves more careful thought than any other parameter in experiment design.
What MDE Actually Means
The minimum detectable effect is the smallest true improvement your test is designed to detect with acceptable reliability. If the real effect is equal to or larger than the MDE, your test has enough power to identify it. If the real effect is smaller than the MDE, your test will likely miss it.
This is not the effect you expect or hope for. It is the threshold below which you are willing to accept the risk of a false negative. Everything smaller than the MDE falls into a zone where your test cannot help you.
Think of it as the resolution of your experiment. A microscope with low magnification can see large objects but misses fine detail. Similarly, a test with a large MDE can detect large effects but is blind to smaller ones.
Why MDE Is a Business Decision, Not a Statistical One
Statisticians can tell you the mathematical relationship between MDE, sample size, power, and significance. But they cannot tell you what MDE to use. That is a business judgment.
The right MDE depends on:
The value of the improvement
What is a one-point improvement in conversion worth annually? What about a half-point? A tenth of a point? Translate the MDE into revenue impact. If detecting a particular level of improvement would generate meaningful annual value, it is worth the traffic investment to detect it.
The cost of the test
Every test has an opportunity cost: the traffic could be used for another experiment, and the team could be working on something else. Longer tests for smaller MDEs cost more. At some point, the cost of running the test exceeds the value of detecting the effect.
The cost of implementation
Some changes are cheap to ship. Others require significant engineering effort, ongoing maintenance, or operational complexity. A small effect might justify a trivial implementation but not a major one.
The traffic available
Low-traffic products cannot detect small effects in reasonable timeframes. The MDE must be calibrated to what is physically possible given your traffic level and testing roadmap.
A Framework for Choosing MDE
Step 1: Calculate the revenue impact
For your primary metric, estimate the annual revenue impact of various improvement levels. Map out what different levels of improvement translate to in business terms.
This gives you a menu of MDEs with attached business values. You can now see the trade-off in concrete terms.
Step 2: Estimate the test duration for each MDE
Using your baseline conversion rate and daily traffic, calculate the sample size and duration for each MDE level. Pair this with the revenue impact.
Now you can see: detecting a smaller effect takes a certain number of weeks and is worth a certain annual amount. Detecting a larger effect takes fewer weeks and is worth a different annual amount.
Step 3: Find the sweet spot
The right MDE is the point where the expected value of running the test justifies the cost. Specifically:
- The test duration should be reasonable for your experimentation roadmap — typically between one and four weeks for most product experiments.
- The detectable effect should be large enough that the revenue impact justifies the implementation cost.
- The power should be high enough that you can trust a null result.
For most product teams with moderate traffic, this sweet spot produces MDEs in a range of relative improvements, not the tiny fractions that teams sometimes target.
Step 4: Validate against historical data
Look at the actual effect sizes from your past experiments. If your median effect size is at a certain level, an MDE well below that means most of your real effects will be detectable. An MDE above your median means you will miss more effects than you catch.
Historical effect sizes also serve as a reality check. If no experiment in your program has ever produced a large effect, choosing an MDE that only detects even larger effects means your tests will never show positive results.
Common Mistakes in Setting MDE
Choosing the smallest possible MDE
Teams sometimes set the MDE as small as their statistics allow, reasoning that detecting smaller effects is always better. But the cost is enormous. Halving the MDE roughly quadruples the required sample size. A test that would take two weeks at a moderate MDE takes eight weeks at half that MDE. Few organizations can afford to run two-month tests.
Using the expected effect as the MDE
The MDE is not what you expect the effect to be. It is the smallest effect you want to be able to detect. If you set the MDE at your expected effect, you will only have conventional power to detect it — meaning a meaningful chance of missing it even if it is real. Set the MDE below the expected effect to give yourself a margin.
Ignoring the MDE entirely
The worst mistake is not choosing an MDE at all. Without it, you have no basis for sample size calculation, no way to determine test duration, and no framework for interpreting null results. You are flying blind.
Using the same MDE for every test
Different tests have different stakes, different implementation costs, and different expected effects. A one-size-fits-all MDE is convenient but suboptimal. Calibrate the MDE to the specific decision.
MDE for Different Types of Tests
Landing page tests
Landing pages typically have moderate conversion rates and receive focused traffic. MDEs for landing page tests can often be set at moderate relative improvements because the traffic is concentrated and the metric is straightforward.
Checkout funnel tests
Checkout changes directly affect revenue, making even small improvements valuable. Teams often want smaller MDEs here, but checkout pages may have lower traffic than top-of-funnel pages. The trade-off between sensitivity and duration is especially sharp.
Feature launches
New features often have uncertain effects. Setting a moderate MDE and running a well-powered test is usually better than trying to detect a tiny effect with an underpowered test. You can always iterate after the initial test.
Pricing experiments
Pricing changes can have large effects (positive or negative), and the stakes are high. Use a moderate MDE but ensure high power, because false negatives on pricing changes are especially costly.
Engagement metrics
Metrics like time on site, pages per session, and return visit rate tend to have high variance. MDEs for these metrics usually need to be larger than for conversion rates, or the test durations become impractical.
The Organizational Dimension
Choosing an MDE is inherently a conversation between data, product, and business teams. Data knows what is feasible given the traffic. Product knows what effects are realistic given the change. Business knows what improvements justify the investment.
When these teams do not talk, you get one of two failure modes:
- Data sets the MDE without business input. The result is technically sound but disconnected from business value.
- Business sets the MDE without data input. The result is a test that cannot possibly run long enough to detect the specified effect.
The best organizations treat MDE selection as a collaborative design decision, documented alongside the hypothesis and success criteria before the test launches.
Calculating Minimum Detectable Effect: A Worked Example
Let me make the minimum detectable effect concrete with numbers, because the abstraction is where most teams lose the thread. Suppose your checkout page converts at 4% and you get 60,000 eligible visitors per month. You want to know the smallest lift a one-month test could reliably catch.
Work backward from traffic. Split 60,000 visitors evenly — 30,000 per variant. Plug a 4% baseline, 30,000 per variant, 80% power, and 95% confidence into any sample size calculator and solve for the effect. At that traffic, the smallest detectable lift lands around a 12% relative improvement — meaning your test can reliably distinguish 4.0% from roughly 4.5%, but not 4.0% from 4.2%. Anything under a 12% relative lift falls into the zone where a null result tells you nothing.
That single calculation reframes the whole conversation. If your redesign realistically moves conversion by 5%, a one-month test at this traffic is underpowered before you write a line of code — you would need roughly five months, or a bigger baseline, or a higher-traffic page. Knowing this upfront is the entire point of setting MDE deliberately.
The mechanics matter: MDE scales with the inverse square root of your sample size, so halving the effect you want to detect roughly quadruples the visitors you need. That non-linear relationship is why "let's just detect a 1% lift" quietly turns into a six-month test. To skip the arithmetic, I built a free MDE calculator that takes your baseline rate and available traffic and returns the smallest effect a test of a given length can catch.
How to Use an MDE Calculator in Three Steps
An MDE calculator flips the usual sample size question around. Instead of asking how many visitors you need to detect a 10% lift, it asks the more honest question for most teams: given the traffic you actually have, what is the smallest lift you can detect? Here is how I run it.
Step 1 — Enter your baseline and traffic. Use the real conversion rate of the page you are testing and the number of visitors you can realistically send to each variant over your intended test window. For the checkout example above, that is a 4% baseline and 30,000 visitors per variant across one month.
Step 2 — Set power and confidence. Leave these at the standard 80% power and 95% confidence unless the decision is high-stakes, in which case raise power to 90%. Higher power means the calculator returns a larger, less sensitive MDE for the same traffic.
Step 3 — Read the detectable lift and sanity-check it. The calculator returns a minimum detectable effect — around 12% relative for the checkout example. Compare that to the lift your change realistically produces. If your redesign moves conversion by 5% and the calculator says you can only detect 12%, the test is underpowered before it starts. My free MDE calculator returns this number instantly, so you can screen out unwinnable tests in seconds rather than discovering the problem five weeks in.
FAQ
Can I change the MDE after the test starts?
Technically yes, but it invalidates the original sample size calculation. If you realize mid-test that the MDE is wrong, it is better to stop the test, recalculate with the new MDE, and restart — or accept the revised MDE and extend the test accordingly.
What if my stakeholders demand a very small MDE?
Show them the math. Translate the small MDE into test duration and opportunity cost. Often, when stakeholders see that detecting a tiny effect requires months of testing, they recalibrate their expectations.
How does MDE relate to the effect I actually observe?
The MDE is a design parameter. The observed effect is a data outcome. If the observed effect is larger than the MDE, you had good power to detect it. If it is smaller, you were unlikely to detect it, and a null result is uninformative.
Should I use absolute or relative MDE?
Both are valid. Relative MDE (percentage change from baseline) is more intuitive for stakeholders and easier to compare across metrics. Absolute MDE (raw change in the metric) is more precise for statistical calculations. Use whichever is clearer for communication, but be consistent.
How do you calculate minimum detectable effect?
You calculate minimum detectable effect by fixing your baseline conversion rate, sample size per variant, statistical power (usually 80%), and significance level (usually 95%), then solving for the effect size those inputs can detect. In practice you run it in reverse: enter your available traffic and baseline into a sample size or MDE calculator, and it returns the smallest relative lift the test can reliably catch. MDE scales with the inverse square root of sample size, so detecting smaller effects requires disproportionately more traffic.
What is a good minimum detectable effect?
A good minimum detectable effect is the smallest lift that is both worth shipping and detectable within a reasonable test window given your traffic. For high-traffic pages a 2 to 5% relative MDE is realistic; for lower-traffic flows 10 to 20% is often the floor. There is no universal "good" number — the right MDE is the one where the business value of catching the effect justifies the test duration it demands.
What sample size do I need for a given MDE?
Run the relationship in the other direction with a sample size calculator: fix your baseline, MDE, 80% power, and 95% confidence, and it returns the visitors needed per variant. Because MDE scales with the inverse square root of sample size, tightening the MDE from 10% to 5% roughly quadruples the traffic you need, so small changes in your target lift produce large swings in test duration.
What MDE should I use for a low-traffic site?
On a low-traffic site, let your traffic set the MDE rather than the other way around. Enter a realistic test window — say four weeks of visitors per variant — into an MDE calculator and accept the detectable lift it returns, which is often 10 to 20% relative for smaller flows. If that floor is larger than the effects your changes plausibly produce, A/B testing is not the right tool yet; lean on qualitative research and larger, bolder changes instead.