Sequential Testing

What Is Sequential Testing?

Sequential testing is a family of statistical methods that let you analyze an A/B test repeatedly as data accumulates without inflating false positive rates. Where classical frequentist testing requires a fixed sample size and a single analysis at the end, sequential methods use adjusted significance thresholds at each look, preserving valid error rates across many peeks. It's the rigorous answer to the most common experimentation sin: stopping when the data looks good.

Also Known As

• Marketing teams often call it always-valid testing or continuous monitoring.
• Growth teams say sequential tests or SPRTs.
• Product teams use continuous analysis or adaptive monitoring.
• Engineering teams refer to always-valid inference or sequential probability ratio test (SPRT).
• Statisticians distinguish between SPRT, group sequential designs, and always-valid confidence sequences.

How It Works

You launch an experiment with a planned sample size of 50,000 users. Repeatedly checking an ordinary fixed-horizon p-value and stopping when it crosses a threshold can raise the false-positive rate above the planned 5%. A pre-specified sequential design allocates error across interim looks or uses an always-valid method so the analysis matches the monitoring behavior. The exact boundaries depend on the chosen method and analysis plan; they are not a universal set of p-value cutoffs.

Best Practices

• Let your experimentation platform handle the math — don't try to compute always-valid p-values by hand.
• Pre-register your maximum sample size even with sequential testing so you have a stopping upper bound.
• Monitor guardrail metrics at every peek, not just the primary metric.
• Communicate the tradeoff clearly: sequential testing costs ~20% more traffic than fixed-horizon testing to reach the same power.

Common Mistakes

• Using classical p-values while peeking daily, effectively running an unprotected sequential analysis.
• Switching between classical and sequential analysis after seeing results, which breaks the error guarantees of both.
• Stopping as soon as significance is reached without checking whether guardrails are stable.

Industry Context

• SaaS/B2B: Useful when low traffic makes every peek tempting — sequential methods let you act on early signal safely.
• Ecommerce/DTC: Revenue-sensitive tests benefit from stopping losers early.
• Lead gen: Moderate value; most lead gen tests have clear fixed-duration windows (weekly pacing).

The Behavioral Science Connection

Sequential testing is a systems-level solution to hyperbolic discounting — our tendency to overvalue immediate outcomes over future ones. Rather than fighting the organizational pressure to peek, sequential methods accommodate it within a rigorous framework. You can check results daily without guilt, because the math has already accounted for your impatience.

Key Takeaway

Sequential testing makes peeking safe — it costs slightly more traffic but removes the single biggest source of false discovery in real-world experimentation programs.