Test Duration

What Is Test Duration?

Test duration is the length of time you let an experiment run before making a decision. It depends on the sample size required for the planned effect and power, traffic volume, exposure patterns, attribution delay, and the business cycles that matter for that specific metric.

Also Known As

• Marketing teams say test runtime or test window.
• Growth teams call it test duration or experiment length.
• Product teams refer to it as experiment window or trial period.
• Engineering teams use runtime or collection window.
• Statisticians distinguish between sample size and duration (the latter being a derivative of traffic).

How It Works

Your page gets 5,000 daily visitors split 50/50. To detect a 10% relative lift on a 4% baseline conversion with 80% power at alpha=0.05, you need ~30,000 per arm (60,000 total). At 5,000/day total, that's 12 days — but you round up to 14 days to cover two full weekly cycles. You launch Monday morning. By Thursday you see a significant lift. You ignore it and keep running — because weekend users behave differently from weekday users, and three days of data doesn't include weekend behavior at all.

Best Practices

• Calculate required sample size before launching, and translate it into days based on your traffic.
• Plan enough calendar time to represent the relevant exposure and purchase cycles; a week is not universal.
• Define a maximum duration before launch, considering identity persistence, seasonality, operational cost, and decision value.
• If the planned duration ends without a decision-ready estimate, report the result as inconclusive or revise the design prospectively rather than extending until significance appears.
• Segment by week if running multi-week tests to check for trend changes.

Common Mistakes

• Stopping at three days because significance showed up — missing all weekend behavior.
• Running tests for three months "to be sure," introducing seasonality and cookie-churn bias.
• Confusing test duration with sample size — you need both correct, not just one.

Industry Context

• SaaS/B2B: Low traffic and delayed revenue outcomes can make tests longer; calculate duration from the actual baseline, effect size, and traffic.
• Ecommerce/DTC: Include the purchase cycles and promotional periods relevant to the question, and pre-specify how seasonal events will be handled.
• Lead gen: Separate the exposure window from the downstream lead-quality attribution window.

The Behavioral Science Connection

Test duration is where hyperbolic discounting wreaks havoc. A three-day 15% lift feels more exciting than the same test's three-week 4% lift, even though the three-week result is dramatically more reliable. Pre-registering duration is the Ulysses contract — you bind your future self before the siren song of early significance arrives.

Key Takeaway

Run every test for at least one full business cycle, and pre-register the duration so you aren't tempted to stop when the numbers look good.