Before I run any experiment, I calculate the required sample size. Not as a formality — as a genuine feasibility check. At least 30% of the test ideas I've seen would have required six months to run properly given the available traffic. Knowing that upfront saves time, money, and false confidence.
Most teams skip this step or do it wrong. Here's how to do it right.
Why Sample Size Matters: The Underpowered Test Problem
An underpowered test is one that doesn't have enough data to reliably detect the effect you're looking for — even if that effect is real.
Suppose you're testing a new product page design. The new design genuinely improves conversion rate by 8% relative. But your test only reached 40% of the required sample size before someone called it. The test shows no significant result. You kill the variant. You just killed a real winner because you didn't collect enough data.
This is a Type II error — a false negative. You miss real effects. And unlike a false positive (shipping a winner that wasn't), a false negative is invisible. Nobody knows what you lost.
The rate of false negatives is controlled by statistical power. At 80% power and your required sample size, you'll correctly detect a true effect 80% of the time. At 40% of required sample size, your power drops to roughly 50-55%. You're nearly coin-flipping.
**Pro Tip:** Run your sample size calculation before writing a single line of test code. If the test isn't feasible given your traffic, find out now — not after 6 weeks of running it.
The Four Inputs Every Sample Size Calculator Needs
Every credible sample size calculator — Optimizely's, Evan Miller's, AB Testguide — needs exactly four inputs. Mess up any one of them and your calculation is wrong.
1. Baseline conversion rate The current conversion rate of the metric you're optimizing. This must be the actual rate on the specific page being tested, not a site-wide average. If you're testing your checkout page, use checkout page conversion rate — not your overall site CVR.
2. Minimum Detectable Effect (MDE) The smallest relative lift you want the test to be able to detect. See the MDE guide for how to set this correctly — the short version is: set it at the minimum lift you'd actually ship. Check whether your calculator uses relative or absolute MDE.
3. Statistical significance threshold Almost always 95% (alpha = 0.05, two-tailed). Some teams use 90% when running exploratory tests to reduce sample size. Use 99% when the stakes are very high (pricing tests, checkout flow changes on high-revenue pages).
4. Statistical power Almost always 80% (beta = 0.20). Use 90% for high-stakes tests where missing a real effect is costly.
Worked Example with Real Numbers
Scenario: You're testing a new checkout page layout on an e-commerce site.
- Baseline CVR: 3.0%
- MDE: 10% relative (you want to detect a lift from 3.0% to 3.3% or larger)
- Significance: 95% (two-tailed)
- Power: 80%
Running this through a standard two-proportion z-test:
Required sample size: ~26,000 visitors per variation
Now calculate test duration:
- Weekly traffic to the checkout page: 10,000 sessions
- Split: 50/50 (5,000 per variation per week)
- Weeks required: 26,000 / 5,000 = 5.2 weeks
That's a manageable test. You'd plan for 6 weeks to be safe, check all four stopping conditions, and call it in week 6 if sample size and stability conditions are both met.
Now run the same calculation with a 5% relative MDE (you want to detect smaller effects): Required per variation: ~104,000 → 20.8 weeks at the same traffic level. Nearly 5 months. That test almost certainly shouldn't run.
**Pro Tip:** If a sample size calculation produces a test duration over 8 weeks, stop and revisit the MDE. Either raise the MDE to something your traffic can support, find a higher-traffic page to test the same hypothesis on, or reconsider whether the test is worth running at all.
The Two Most Common Mistakes
Mistake 1: Using total site traffic instead of traffic to the tested page.
This is the most expensive mistake in sample size planning. If your site gets 100,000 sessions/week but only 15% of those users visit your checkout page, the effective traffic for a checkout test is 15,000/week — not 100,000. Using the wrong denominator makes tests look feasible when they aren't.
How to fix it: Find the specific page's session count in your analytics tool, filtered to the same audience segment your test will target. That's your weekly traffic figure.
Mistake 2: Not accounting for multiple variations.
Adding variations doesn't just "split the traffic more." It requires you to adjust for multiple comparisons, or your false positive rate inflates.
With two variations (A and B), a 95% significance threshold gives you a 5% false positive rate per comparison.
With three variations (A, B, and C), you have two comparisons (B vs. A, C vs. A). If you use 95% significance for each without adjustment, your family-wise false positive rate rises to approximately 9.75%.
The sample size implication: For a 3-variation test at 95% per-comparison significance with Bonferroni correction, you need approximately 3× the per-variation sample size of a 2-variation test, not 1.5×. Most teams assume they just need more traffic for a 3rd variation — they don't realize the per-variation requirement also increases.
**Pro Tip:** Unless you have a specific reason to run more than two variations, stick to A/B (not A/B/C). Adding a third variation requires roughly 3× the total traffic and significantly extends test duration. The marginal value of testing two variants simultaneously is rarely worth the cost.
MDE Sensitivity: The Number That Changes Everything
Understanding how sensitive sample size is to MDE changes helps you make better tradeoffs.
Using the same baseline (3% CVR, 95% significance, 80% power):
- 5% relative MDE → ~104,000 visitors per variation
- 10% relative MDE → ~26,000 visitors per variation (75% reduction)
- 15% relative MDE → ~11,600 visitors per variation
- 20% relative MDE → ~6,500 visitors per variation
Going from 5% to 10% MDE cuts required sample size by approximately 75%. Going from 10% to 20% MDE cuts it by another 75%. The relationship is roughly quadratic: double the MDE, reduce the sample size by 75%.
This is the most powerful lever in your test design. Use it deliberately.
How to Read and Use the Sample Size Calculator in Optimizely
Optimizely's built-in sample size calculator (accessible in the test setup flow) uses the following defaults: 95% statistical significance, 80% power, and it accepts a relative MDE.
The key output to focus on: unique visitors per variation. This is the number each bucket needs to reach before you have adequate power. Multiply by the number of variations to get total required exposure.
Optimizely also shows a projected test duration based on historical traffic to the target URL. This estimate assumes stable traffic — if you're running a test during a campaign period or a historically volatile traffic window, apply a buffer.
One thing Optimizely's calculator doesn't do by default: adjust for multiple variations with family-wise error rate correction. If you're running 3+ variations, calculate the Bonferroni-corrected per-comparison alpha manually (alpha / number of comparisons) and use that adjusted significance level in the calculator.
**Pro Tip:** Cross-check Optimizely's traffic estimate against your analytics platform. Optimizely uses its own tagged session counts, which may differ from your analytics source of truth if your tag coverage isn't complete.
Minimum Sample Size Per Variant: What the Number Actually Means
Here is the distinction that trips up more teams than any other: the minimum sample size per variant is not the same as the total sample size for your test. The calculator returns a per-variant number — the visitors each version needs to reach adequate statistical power. For a standard two-arm A/B test (control plus one variant), you double that number to get the total traffic required.
Say the calculator returns 25,000 per variant at 80% power, 95% confidence, and a 5% relative MDE. Your control needs 25,000 visitors and your variant needs 25,000 — 50,000 total. Miss this and you will badly underestimate how long the test runs, because you planned for 25,000 total when you actually needed twice that.
The per-variant framing matters even more for multivariate and A/B/n tests. Every arm you add needs its own full allocation. A four-way test (one control, three variants) at 25,000 per variant needs 100,000 sessions — and that is before any correction for multiple comparisons, which pushes the requirement higher still. I have watched teams add a third and fourth variation "since we're already testing" without realizing they just tripled the traffic bill and the timeline.
One rule I apply before adding any variant: re-run the sample size calculation with the new arm count. If the per-variant number times the number of arms exceeds four weeks of realistic traffic, I cut variants until it fits. A clean two-arm test that concludes beats a four-arm test that gets killed at 40% power every time.
What to Do When Your Traffic Is Too Low
You run the calculation. The test would take 6 months. Your options:
Option 1: Increase the MDE. Accept that you'll only detect larger effects. If business context justifies this (a 10% lift would be meaningful), this is the right call.
Option 2: Test a higher-traffic page. If you're trying to validate a hypothesis about value messaging, test it on your homepage (high traffic) before testing on your product page (lower traffic). Use a higher-traffic proxy to validate the direction, then refine on the actual target page.
Option 3: Consider qualitative research instead. If the page gets 500 sessions/week, A/B testing isn't the right tool. Run usability testing, session recordings, or user interviews. Get qualitative signal. Redesign based on that. Then monitor the before/after with analytics. That's not a test, but it's better than running an underpowered test for a year.
Option 4: Run a partial test with a directional hypothesis. If you have 50% of the required sample size, you have ~60% power. You won't reach a definitive conclusion, but you can say "directionally positive at [confidence level]" and use that as input for the product roadmap without claiming a winner.
**Pro Tip:** Be honest with your stakeholders about what low-traffic tests can produce. "We'll run this for 2 weeks and see" on a 500-session/week page is not a test — it's guessing with extra steps. Setting that expectation correctly upfront is better than defending inconclusive results 2 weeks later.
The "We'll Run It for 2 Weeks and See" Fallacy
This is the most common mistake in experimentation practice. "Two weeks" is not a statistical input. It's an arbitrary duration driven by sprint cycles, reporting cadences, and impatience.
A 2-week test might be perfectly calibrated for your traffic and MDE. Or it might give you 30% of the required sample size. The duration doesn't tell you anything on its own. The sample size does.
The correct process: calculate required sample size → divide by available traffic per variation → that gives you required duration. Then round up to the nearest full week to capture weekly cycles. That's your test duration. Not "2 weeks" as a default.
If the calculated duration is longer than you have time for, that's a signal to revisit your MDE — not to run the test anyway and hope for the best.
Common Mistakes
Mistake 1: Not setting the sample size calculation before the test starts. Post-hoc sample size justifications are unreliable. Calculate before, commit to the number, and don't revisit it until the test is over.
Mistake 2: Treating the calculator's duration estimate as a guarantee. Traffic fluctuates. Campaigns change. A test that should take 4 weeks based on historical traffic might take 6 weeks during a slow period. Build buffer.
Mistake 3: Including sessions where the variant wasn't visible. If your test is above the fold and 30% of users don't scroll there, they shouldn't count toward your sample. Filter your analysis to users who were actually exposed to the experiment.
Mistake 4: Not checking the calculator's power assumptions. Some calculators default to 90% power instead of 80%. This isn't wrong, but it means you'll need more traffic. Know what your calculator assumes.
How to Use a Free A/B Test Sample Size Calculator
The four inputs above are all any A/B test sample size calculator actually needs, so using one takes about sixty seconds once you have your numbers. Here is the exact sequence I run before every test.
Step 1 — Pull your real baseline. Open your analytics tool and find the current conversion rate for the specific page or flow you are testing, not the site-wide average. If your checkout converts at 3.0%, that 3.0% is your baseline input.
Step 2 — Set your MDE. Enter the smallest relative lift you would actually ship, say 10%. The calculator will tell you it can detect a move from 3.0% to 3.3% or larger at your chosen power and confidence.
Step 3 — Read the per-variant number, then double it. With a 3.0% baseline and a 10% relative MDE at 80% power and 95% confidence, the calculator returns roughly 26,000 visitors per variant. That is 52,000 total for a clean two-arm test — the single number most teams misread when they plan for the per-variant figure as if it were the total.
Step 4 — Convert to duration. Divide the per-variant figure by your weekly traffic per arm. At 5,000 sessions per variant per week, 26,000 visitors is about 5.2 weeks. If that runs longer than eight weeks, raise the MDE or pick a higher-traffic page. I built a free sample size calculator and a companion MDE calculator at GrowthLayer that do all four steps for you — enter your baseline and traffic, and they return visitors per variant and the detectable lift instantly.
What to Do Next
Run sample size calculations for your next 3 planned tests before building anything. If any of them come back with durations over 8 weeks, use that as the trigger to revisit MDE or page selection.
Frequently Asked Questions
What is the minimum sample size per variant for an A/B test?
The minimum sample size per variant is the number of visitors each version of your test needs to reach adequate statistical power, typically 80% power at 95% confidence. There is no universal number — it depends on your baseline conversion rate and your minimum detectable effect. Detecting a 5% relative lift on a 3% baseline needs roughly 25,000 per variant, while a larger 10% MDE on the same baseline needs closer to 6,500 per variant. Always run the calculation for your own numbers rather than reusing a figure from another test.
How many users do I need for a statistically significant A/B test?
You need enough users to reach the per-variant sample size your calculator specifies, then double it for a two-arm test. Significance is not a fixed visitor count — it is the point at which your observed difference is unlikely to be noise given your sample size and effect. Reaching the pre-calculated sample size at 80% power is what makes a significant result trustworthy; stopping early inflates false positives no matter what the p-value shows.
Do I need to adjust sample size for more than two variants?
Yes. Every additional variant needs its own full per-variant allocation, and running multiple comparisons raises your false-positive risk unless you apply a correction such as Bonferroni or a family-wise error rate adjustment. Both effects increase the total traffic you need, so a four-arm test costs far more than double a two-arm test.
How do I calculate A/B test sample size?
Enter four inputs into a sample size calculator: your baseline conversion rate, your minimum detectable effect, your significance level (usually 95%), and your statistical power (usually 80%). The calculator returns the visitors needed per variant; double that for a two-arm test, then divide by your weekly traffic per arm to get the required duration. Always run this before you build the test, not after — a post-hoc sample size justification is unreliable.
Is a two-week A/B test long enough?
Only if two weeks happens to deliver your required sample size — duration alone tells you nothing. Two weeks on a high-traffic homepage may be well powered, while the same two weeks on a 500-session-per-week page produces a fraction of the data you need. Calculate the sample size first, convert it to weeks at your traffic level, and run for that long, rounding up to whole weeks to capture weekly cycles.
Before you build anything, size the test. I built free tools at GrowthLayer for exactly this: start with the sample size calculator to find how many visitors each variant needs, then run the MDE calculator to check whether the lift you are chasing is even detectable given your traffic. For a complete test design template — sample size planning, stopping rules, and result documentation — see my Optimizely Practitioner Toolkit.