An A/B test's required sample size depends on the conversion rate you start from, the change you want to detect, and how reliably you need to detect it. For a hypothetical test moving conversion from 5% to 5.5%, a conventional fixed-horizon calculation gives approximately 31,300 visitors per variant, or 62,600 across both variants. That is a 10% relative improvement, not a ten-percentage-point increase.

Those numbers describe one design, not a universal minimum. This guide shows the calculation, explains its assumptions, and turns the result into a realistic recruitment period. If you already have your inputs, the sample-size calculator provides a starting estimate; check its method and output units before using the result in a test plan.

The inputs behind an A/B test sample size

The example uses two independently randomized groups, equal allocation, and one binary outcome per visitor: converted or did not convert within a defined window. The planned analysis happens at a fixed horizon. It does not cover revenue distributions, repeated sessions treated as independent people, or accounts randomized in clusters.

InputHypothetical valueMeaning
Baseline conversion rate5%The expected control rate for eligible visitors
Relative minimum detectable effect10%A change from 5% to 5.5%
Absolute difference0.5 percentage points0.005 when rates are expressed as decimals
Significance level5%, two-sidedThe planned false-positive rate under the null, assuming the procedure is followed
Statistical power80%Probability of rejecting the no-difference null if this assumed effect is real
Allocation50:50Half of eligible visitors assigned to each variant

A 5% significance level is not a 5% probability that a particular result is wrong. Similarly, 80% power is not an 80% probability that the treatment works. These describe repeated-test behavior under specified assumptions.

When I evaluate a test plan, I want the proposed decision alongside these inputs. A precise calculation cannot tell us whether the change would be worth maintaining or whether the outcome measures what the team actually cares about. The MDE guide explains how to connect business value to sensitivity before calculating traffic.

The formula for two conversion rates

For equally sized groups, a common normal approximation is:

average rate = (p0 + p1) / 2

A = z(1 − alpha/2) × sqrt(2 × average rate × (1 − average rate))
B = z(power) × sqrt(p0 × (1 − p0) + p1 × (1 − p1))

n per variant ≈ (A + B)² / (p1 − p0)²

Here, p0 is the baseline rate, p1 is the target rate, and z is a standard-normal quantile. For a two-sided 5% significance level, the first z-value is about 1.96; for 80% power, the second is about 0.842. Enter rates as decimals: 5% becomes 0.05.

This approximation uses a pooled rate for the null variance and separate rates for the alternative variance. The statsmodels sample-size documentation describes the closed-form approach and its negligible far-tail approximation in common two-sided designs. Continuity corrections and other methods can produce somewhat different answers.

The denominator explains why small effects are expensive to detect: the absolute difference is squared. Halving that difference requires roughly four times the sample when the variance is similar. The relationship is approximate because the target rate also changes the variance.

Worked example: from 5% to 5.5% conversion

For this fictional landing-page experiment, the baseline is 0.05 and the target is 0.055. The average rate is 0.0525, and the absolute difference is 0.005.

Substituting those values gives:

A ≈ 0.618205
B ≈ 0.265444

n per variant ≈ (0.618205 + 0.265444)² / 0.005²
              ≈ 31,234 visitors, rounded up

For planning, round upward again to about 31,300 per variant. Across control and treatment, that is about 62,600 eligible visitors. Keeping the per-variant label attached to the number matters: allocating only 31,300 visitors across both groups would leave each group with about half its planned sample.

These are hypothetical inputs and calculated estimates. They are not traffic figures or outcomes from a company experiment, and the target rate is an assumption rather than a forecast.

Suppose the page receives about 1,000 new eligible randomized visitors per day in total. The recruitment estimate is:

62,600 total visitors / 1,000 eligible visitors per day ≈ 63 days

The equivalent calculation is 31,300 per variant divided by about 500 visitors per variant per day. Both give the same duration. Dividing the total sample by per-variant traffic would double the estimate incorrectly.

Why recruitment time is not the whole test duration

Sitewide traffic is rarely the correct denominator. Use the flow of new eligible units that will actually enter this experiment, after planned targeting and allocation. Returning visits by the same randomized person do not create independent new visitors. A ramp that exposes only part of the audience also changes the estimate.

The outcome window matters too. If a visitor has seven days to convert, the last recruits need that observation period before their outcomes are complete. Define how you will handle maturation in advance; otherwise, early and late recruits can receive different opportunities to convert.

Cover relevant business cycles when scheduling the test, but do not turn “two full weeks” into a universal sample-size rule. A low-traffic test can run for weeks and still have little precision. Conversely, a high-traffic test can reach its sample quickly while missing weekly behavior or delayed outcomes. The test-duration guide covers the scheduling decision in more detail.

Document the sample target, minimum observation period, analysis date, and stopping rule together. Monitoring instrumentation and safety during a fixed-horizon test is sensible; repeatedly checking ordinary p-values and stopping on the first favorable one changes the inferential procedure.

What changes the required sample

Holding the baseline, power, significance level, and allocation constant, greater sensitivity costs traffic. Higher power and stricter significance thresholds also usually increase the required sample for the same effect.

Baseline comparisons need particular care. For a fixed absolute difference, binomial variance is greatest near a 50% baseline. For a fixed relative lift, a lower baseline also makes the absolute difference smaller. That can require more traffic. “Lower conversion rates need smaller samples” is therefore not a reliable rule.

If the baseline is uncertain, calculate a plausible range of scenarios rather than presenting one exact-looking minimum. Check that the historical denominator, eligibility rules, and conversion window match the proposed experiment. A precise rate from a different audience is a poor planning input.

The calculator guide and lookup table are useful for comparing approximate scenarios. Keep their purpose separate from the final plan: the analysis method used by your testing platform must match the assumptions used to size the experiment.

When this formula is the wrong tool

For revenue per visitor, use a method that accounts for the distribution and variance of that outcome. For account-level randomization, repeated observations within an account are correlated; counting every session as independent overstates the sample. Extra treatments and multiple primary comparisons require a multiple-testing plan.

A sequential design can permit valid interim decisions, but it has its own boundaries and operating characteristics. Do not combine a fixed-horizon sample estimate with an improvised stopping rule and assume the original error rates still hold. There is no universal rule that every sequential design takes more—or less—traffic than every fixed-horizon design.

If the calculated test is impractical, changing the MDE changes the question. A design powered for a large improvement may miss smaller improvements that still matter. Consider a stronger intervention, a defensible broader eligible audience, more observation time, or research-led improvements whose outcomes are reported with appropriate limitations. A more frequent click metric is only useful if it supports the decision; it does not automatically replace purchases or retention.

Before interpreting effects, check assignment and measurement quality with the sample ratio mismatch diagnostic. A large sample cannot repair a biased population. The American Statistical Association's guidance also cautions against basing business decisions on a significance threshold alone.

FAQ

How many visitors do I need for an A/B test?

There is no single minimum. For the hypothetical 5% baseline and 10% relative lift in this article, the stated method needs roughly 31,300 visitors per variant at 80% power and a two-sided 5% significance level. Different baselines, effects, metrics, or analysis methods change that estimate.

Is sample size measured in visitors or conversions?

The formula here counts eligible randomized visitors with a binary conversion outcome, not conversions alone. If randomization occurs at another level, such as an account, the analysis and planning must account for that level and any clustering. Session counts cannot simply substitute for independent visitors.

Does reaching the planned sample guarantee a significant result?

No. The effect might be absent, smaller than assumed, or missed through sampling variation. Even at the assumed effect, an 80%-power design sometimes produces a nonsignificant result. Interpret the effect estimate and uncertainty, and verify the design was implemented correctly.

What should go into the experiment brief?

Record the decision, primary outcome and conversion window, randomization unit, baseline source, relative and absolute MDE, allocation, sample per variant, and stopping rule. Use the Experiment Decision Brief to assemble and download that plan before development starts.

Share this article
LinkedIn (opens in new tab)X / Twitter (opens in new tab)
Atticus Li

Experimentation and growth leader. CXL-certified CRO practitioner, Mindworx-certified in behavioral economics. Led 100+ in-house experiments at NRG in 2025, with project evidence and limits documented in the case studies.