Most teams treat the moment a test hits significance like a gun going off at the end of a race. The experiment reaches p<0.05, someone screenshotting the dashboard fires off a Slack message, and the variant gets shipped before end of day. The instinct makes sense — you ran the test, the numbers moved, you have your answer.
But that framing misunderstands what statistical significance actually tells you.
Significance doesn't mean your result is real. It means your result is unlikely to be explained by chance given the data you've collected so far. Those are not the same thing. One is a destination. The other is a conditional statement about a snapshot in time.
What Novelty Effect Does to Your Data
Here's the mechanism most guides leave out: human behavior shifts when something changes, then it shifts back.
Kahneman's work on attention and cognitive engagement shows that novel stimuli capture disproportionate mental resources early. When you change a button color, a headline, or a checkout flow, the first cohort of users who encounter the change is not a representative sample of future behavior. They're interacting with something new, and novelty triggers heightened attention and deliberate processing. That's System 2 behavior in a funnel that normally runs on System 1 habits.
The practical consequence: early experiment results often show inflated lift for changes that feel different, not changes that work differently. By day seven or ten, users have habituated. The System 2 attention fades. Behavior reverts toward baseline. The lift you captured at peak novelty evaporates.
This is called the peeking problem, and it's one of the most common reasons experiment results fail to hold post-ship. If you stop the moment you hit significance, you may be capturing a novelty spike, not a durable behavioral change.
Loss Aversion Compounds the Problem
There's a behavioral economics angle on why teams stop tests early that rarely gets named directly: loss aversion.
Running a test that's currently "winning" and not shipping the winner feels like leaving value on the table. The prospect of losing that gain — of the result regressing if you wait — is psychologically more painful than the potential upside of confirming the result with more data. Thaler's work on mental accounting explains this pattern well. Teams mentally "book" the win early, and continuing to run the test starts to feel like risking something they already own.
So the decision to stop early isn't usually negligence. It's a predictable response to a psychological setup. The experiment infrastructure creates a context where the rational-looking move (ship the winner) is actually the risky one.
What Good Stopping Rules Actually Look Like
Statistical significance is a threshold, not a signal to stop. What you actually want before ending an experiment:
Minimum runtime, not just minimum sample. A test should run for at least one full business cycle — typically seven to fourteen days depending on your traffic patterns — to smooth out day-of-week effects. Weekday behavior and weekend behavior in most funnels are materially different. A test that hits significance on a Tuesday morning may be reflecting Monday-Tuesday usage patterns only.
Directional stability, not just a significant point estimate. Check whether the lift has been moving toward or away from significance over time. A result that crossed the threshold three days ago and has been weakening since is a different situation than a result that crossed the threshold and has held steady. Most platforms will show you a running confidence interval — use it.
Guardrail metrics, not just success metrics. A variant that lifts your primary conversion metric while degrading session depth, return rate, or downstream activation is not a win. It's a trade you may not want to make. Guardrail metrics are the check on optimization that works in the short term but extracts cost elsewhere.
The Test You Should Run This Week
Pull the last three experiments your team shipped based on a significance threshold. Look at two things:
First, check when they hit significance versus when you stopped them. If the gap is less than forty-eight hours in most cases, you have a peeking culture, not an experimentation culture.
Second, check whether the lift held post-ship. If you have any kind of holdout or post-ship measurement, compare the in-experiment lift to the observed lift in the first month after shipping. A meaningful gap between those two numbers is a signal that novelty effect or early stopping is inflating your results.
Significance is a checkpoint. The finish line is a stable, replicable effect across a full runtime — with guardrail metrics intact.
If you ship before you reach it, you're not optimizing. You're gambling with extra steps.
Related reading: underpowered A/B tests, experimentation governance, and how holdout tests prove incremental revenue. I built GrowthLayer to make experimentation repeatable across a program; for more field notes, subscribe to Lean Experiments.