Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating
Meta description: Peeking at a fixed-sample A/B test inflates false positives. Sequential testing lets you check results repeatedly and stop early without cheating.
TL;DR
- Fixed-sample testing assumes you'll wait for a pre-calculated sample size before looking at results. Checking early and stopping the moment you see significance — "peeking" — quietly inflates your real false-positive rate, often far above the 5% you think you're getting.
- Abraham Wald's Sequential Probability Ratio Test (SPRT), developed for wartime quality control, is the mathematically rigorous alternative: a procedure built to be checked repeatedly, with pre-calculated boundaries that keep the false-positive rate honest by construction.
- The difference between the SPRT and peeking isn't willpower — it's that the SPRT's stopping rule is part of the math from the start, so stopping early doesn't cost you anything in error-rate control.
- Sequential design is the right call when traffic is limited, the cost of running a test too long is high, or the business genuinely can't commit to waiting for a fixed horizon — not a substitute for rigor, but a different kind of rigor suited to a different constraint.
- This is a methodology choice, not a shortcut — and it's one input into the broader question of how much certainty a given bet needs, covered in the Confidence Tier Model.
Every experimentation program eventually hits the same moment: a test has been live for four days, the dashboard shows a lift, and someone — a stakeholder, a PM, sometimes you — asks "can we call it?" The honest answer depends entirely on what kind of test you designed, and most teams don't have a clean answer, because most teams designed a fixed-sample test and are now trying to read it like a sequential one.
Those are not interchangeable. Knowing the difference, and choosing deliberately between them before the test starts, is the actual skill — not "wait long enough" or "look for significance," but designing the test so that looking whenever you want doesn't break the statistics.
Why fixed-sample testing assumes a promise you probably won't keep
Classical A/B testing — the kind taught in most CRO courses — works like this: you calculate a required sample size in advance, based on your baseline conversion rate, the minimum effect size you care about detecting, and your desired confidence level. You commit to collecting exactly that much data. Then, and only then, you look at the result and check whether it clears your significance threshold.
The statistics behind that threshold are only valid under one condition: that you looked exactly once, at the pre-agreed sample size. The 5% false-positive rate on a standard significance test is a promise about a single decision point, not about a process of repeated glances.
That promise is almost never what actually happens inside a real company. A test goes live, and a dashboard exists, and someone checks it — not out of malice, but because that's what dashboards are for. The test "looks good" on day three, someone mentions it in a meeting, and the pressure to end it right there builds before the pre-calculated sample size has been reached.
Why peeking inflates false positives — the mechanism, not just the warning
This isn't a hypothetical concern about discipline — it's a well-documented statistical problem, most famously explained in Evan Miller's widely-cited essay "How Not to Run an A/B Test." The mechanism is simpler than it sounds.
A significance test asks: "if there were truly no difference between A and B, how likely is it that random noise alone would produce a gap this large?" At a single, pre-specified sample size, that probability is genuinely 5% (assuming a standard threshold). But random noise doesn't move in one direction and stay there — it wanders. A test's measured lift will drift above and below the "true" effect throughout its run, the same way a coin-flip tally can briefly show more heads than tails even from a fair coin.
If you check that wandering result once, there's a small chance you catch it at a noisy peak. Check it every day for two weeks and you get many chances to catch one — and stopping specifically when you see one is the failure mode. Each additional look is another opportunity for pure noise to cross the significance line, and standard significance math was never built to account for that multiplicity. The stated 5% false-positive rate can realistically climb well past 20-30% for a test checked repeatedly and stopped opportunistically — not because anyone cheated, but because the checking itself changes the odds.
The insidious part is that this happens with zero bad intent. Nobody sets out to defraud their own experimentation program. They just look at a live dashboard the way anyone would, and stop at the first moment it validates what they hoped to see.
What the SPRT actually is
Long before online A/B testing existed, statistician Abraham Wald developed the Sequential Probability Ratio Test in the 1940s for a different problem: quality-control sampling, where the cost of drawing each additional sample was real and testing had to be as fast as it could responsibly be. His procedure was one of the first to formally solve a specific tension — that the _only_ way to be allowed to look repeatedly without inflating your error rate is to build the repeated-looking into the test's design from the start.
Conceptually, the SPRT works like this: instead of a fixed sample size, you pre-calculate two boundaries — an "accept the effect is real" boundary and an "accept there's no effect" boundary — based on the false-positive and false-negative rates you're willing to tolerate. As data comes in, you track a running measure of how strongly the evidence favors one hypothesis over the other, checked after every new data point. The test ends the moment the evidence crosses either boundary — and not a moment before.
The mathematical guarantee is what makes this different from peeking: because the boundaries account in advance for the fact that you'll be checking constantly, crossing one of them means the false-positive rate is still controlled at the level you specified — by construction, not by hoping nobody looked too early. You get to look as often as you want. The looking was priced into the design, rather than bolted on as an afterthought.
Sequential testing versus peeking: same behavior, opposite math
| Peeking at a fixed-sample test | Sequential testing (SPRT-based) | |
|---|---|---|
| When you're "allowed" to look | Only once, at the pre-calculated sample size (in theory) | Continuously, from the start |
| What happens if you stop early on a good-looking result | Real false-positive rate is inflated — often severely | No inflation — the stopping boundary already accounts for repeated looks |
| What the statistical guarantee depends on | Strict discipline never to act on an early look | The test design itself, not the operator's willpower |
| Best suited for | High-traffic tests where a fixed horizon is genuinely affordable | Limited traffic, urgent decisions, high cost of running too long |
| Common failure mode | Stakeholders check the dashboard and stop early anyway | Choosing sequential boundaries so loose they approve weak effects too easily |
The table makes the actual difference clear: it was never about whether you look. It's about whether the math you're using was built to be looked at.
When a business should actually choose sequential over fixed-sample
Sequential testing isn't strictly better — it's a different tool suited to a different constraint. Three situations make it the right call:
Limited traffic. If a fixed-sample calculation says you need six weeks to reach adequate power, and you don't have six weeks of stable traffic, a sequential design lets you stop as soon as the evidence is strong enough — for a genuinely large effect, considerably faster than waiting out a fixed horizon built around detecting a smaller one.
Need to move fast. Some decisions carry real time-value: a pricing test during a narrow competitive window, or a homepage change ahead of a seasonal peak. Waiting for a textbook-perfect fixed sample size while the business context shifts underneath the test can mean the answer arrives after it stopped mattering. Sequential design front-loads the chance to stop early when the true effect is large, without giving up rigor to get there.
High cost of running a test too long. Every day a test runs, some portion of traffic sees a variant that might be worse. If the downside is expensive — a checkout flow, a pricing page, a high-intent conversion point — a method that ends the test the moment sufficient evidence exists reduces exposure to that downside compared to a design that mechanically runs its full pre-calculated course regardless of how the evidence is trending.
None of this is a reason to abandon fixed-sample testing everywhere. A high-traffic page with no urgency and no real cost to a slightly longer runtime is a perfectly reasonable candidate for a classic fixed-horizon design — simpler to explain to stakeholders, and the "wait for the number" discipline is easy to enforce when there's no pressure pushing against it. The choice is about matching the method to the constraint, not defaulting to whichever one sounds more sophisticated.
The senior judgment call underneath this
Choosing sequential over fixed-sample isn't a statistics decision made in isolation — it's a business-constraint decision that happens to have a statistical implementation. The judgment call is diagnosing, honestly, which constraint you're actually under: is it traffic, is it time-value, is it downside risk, or is it none of those and you'd simply prefer not to wait? Only the first three are legitimate reasons to reach for a sequential design. The fourth is impatience wearing a statistical costume, and a good sequential design won't rescue a decision that hasn't earned enough evidence yet — it will just tell you, honestly and a bit faster, that the evidence isn't there.
That's the real value of understanding the SPRT conceptually, even if you never derive the boundary math yourself: it lets you tell the difference between a team that's moving fast because it chose the right tool for its constraint, and a team that's moving fast because it stopped looking as soon as the dashboard agreed with them.
FAQ
Is sequential testing the same thing as a "Bayesian" test?
Not exactly, though they're often discussed together and can be used for similar goals. The SPRT is a classical (frequentist) method built specifically around a likelihood-ratio boundary. Bayesian sequential methods exist too and update a probability distribution rather than tracking a likelihood ratio against fixed boundaries. Both are legitimate ways to handle repeated looking; the SPRT is the historical origin point and the clearest place to build intuition for why sequential design works at all.
Can I just apply a stricter significance threshold to compensate for peeking?
That's a rough patch, not a real fix. A tighter threshold can reduce the damage, but it doesn't carry the same guarantee as a method purpose-built for repeated checking, and it's easy to get wrong in either direction — too loose to help, or so strict the test becomes needlessly slow. If you know you'll want to check repeatedly, design for it from day one rather than retrofitting a fixed-sample test after the fact.
Does sequential testing require more traffic than fixed-sample testing?
Often less, for a real effect — that's much of the appeal. Because the test can stop the moment sufficient evidence accumulates, a genuinely strong effect can be confirmed faster than a fixed-horizon design that's committed to running its full pre-calculated course regardless. For a true null result (no real effect), a well-designed sequential test still runs to a sensible stopping point rather than dragging on forever — that's part of what the pre-calculated boundaries manage.
What's the actual downside of sequential testing, if it sounds strictly better?
Two things. First, it's a more complex methodology to explain and to implement correctly — a poorly-configured sequential design (boundaries set too loosely) can approve weak effects faster than a rigorous fixed-sample test would, which is a real risk if the setup isn't done carefully. Second, some existing tools and stakeholder mental models are built around the "wait for the number" fixed-sample framing, so introducing sequential design also means investing in explaining it well.
How does this connect to deciding how much certainty a bet actually needs?
Sequential testing is one lever for reaching a high-certainty result faster under a real constraint — but it doesn't change how much certainty a given decision requires in the first place. That's a separate question, and it's the one the Confidence Tier Model is built to answer: sequential design can help you earn a Proven-tier result under time or traffic pressure, but it's not a way to skip earning it.
Bottom line
Peeking and sequential testing look identical from the outside — someone checking a dashboard and making a call before a pre-calculated sample size is reached. The difference is entirely in whether the underlying math was built to allow it. Fixed-sample testing makes a promise about looking exactly once; sequential testing, grounded in Wald's SPRT, makes a different and equally rigorous promise about looking as often as you need to. Choosing between them is a constraint-matching decision, not a rigor-versus-speed trade-off — and knowing which constraint you're actually solving for is what separates a fast, disciplined program from one that's just impatient with better vocabulary.
If you're building or auditing an experimentation program and want an outside read on this, get in touch.