Most new CRO teams make the same five statistical mistakes in their first year. None of them are about competence. They are the predictable result of joining the field through tool defaults and case studies rather than through the statistical literature. This guide walks through each mistake, why it happens, and what to do instead.
What You'll Learn
- The 5 statistical mistakes that derail most new testing programs
- A 1-minute summary of why each one inflates your "wins"
- A specific fix for each, written as a Monday-morning action item
- The 3 foundational habits that prevent most false positives
- Realistic base rates for win frequency and effect size in mature programs
Quick Stats Reference
The numbers a new analyst should memorize:
>
- 5% = the nominal false positive rate of a 95%-confidence test (run correctly).
- 20-30% = the empirical false positive rate when you peek and stop at peak.
- 10-20% = the realistic win rate at mature programs (Microsoft, Booking, Netflix).
- Single digits = the typical effect size on a real win.
- 50-60% = how much you should discount any reported headline lift before sizing a rollout.
Why These Mistakes Are So Common
Three structural conditions produce the pattern across new analysts and small teams:
Tool defaults teach the wrong protocol. Most consumer A/B testing platforms display a "probability variant B beats A" indicator that updates continuously. The interface implies you should monitor and stop when the probability looks favorable. It does not surface that this protocol invalidates the underlying inference unless the tool explicitly supports always-valid sequential analysis. Most do not.
Case studies select for salient wins. New analysts learn the field through published case studies. Those case studies report wins, often with large lifts on short tests with bundled variants. The model that emerges from this reading is empirically wrong: 20%+ lifts are not common, the typical test does not run 10 days, and methodological diagnostics are not optional.
Organizations reward reported wins over rigor. A new analyst who ships a "winning" test gets credit. One who insists on a 21-day fixed horizon and reports a flat result rarely does. The incentive gradient pushes toward speed and against rigor.
The five mistakes below are the predictable result. None require carelessness or low intelligence. They are the natural protocol that emerges in the absence of explicit training.
"A losing A/B test costs you the test. A flawed winner costs you every decision built on it."
— Atticus Li
---
Mistake 1: Stopping the Test at the First Favorable Observation
Quick definition: _Peeking_ = checking results during a test and stopping when they look favorable. _Fixed-horizon_ = pre-committing to a sample size and running the test until you reach it, regardless of interim observations.
What goes wrong. You launch the test, monitor the dashboard, see the variant pulling ahead, and stop. The reported lift looks like a win. It is largely an artifact of when you chose to stop.
Quick stat. Daily peeking on a 14-day test inflates the false positive rate from a nominal 5% to roughly 25-30%. About one in four "wins" produced this way will be false positives on tests with no underlying effect.
Why your tool is misleading you. The dashboard's "94% probability variant B beats A" indicator is not a valid stopping criterion under continuous monitoring unless the tool is explicitly using always-valid inference. Most are not.
The fix. Pre-commit to a stopping rule before launch:
- Calculate required sample size from baseline CVR, MDE, 95% confidence, and 80% power.
- Convert sample size to days using your daily traffic split.
- Round up to the nearest full week (controls day-of-week effects).
- Document the target end date. Do not act on interim results.
Monday-morning action: Take your three most recent "wins." Note their actual run time. If any ran for fewer than 14 days or did not have a pre-registered sample target, flag them for replication before further action.
Deeper dive: Early Stopping in A/B Tests: A Guide for New CRO Analysts.
---
Mistake 2: Bundling Multiple Changes Into a Single Variant
Quick definition: _Confounded test_ = a test where the variant changes more than one thing at once, making it impossible to identify which change drove any observed effect.
What goes wrong. The "after" version contains four or five simultaneous changes — new hero, copy rewrite, repositioned price, urgency badge, social proof block. The variant wins. You can declare success but cannot identify which change actually drove it.
Quick stat. A bundled variant produces zero generalizable insight even when it wins. You learn that the bundle worked once, on this page, with this audience, in this window. You cannot generalize to "showing savings in dollars works" or "the new hero copy works."
Why it happens. Bundling is the path of least resistance under time pressure, and it matches the design instinct that the changes "belong together" as a cohesive redesign.
The fix.
- Where time permits, prefer one variable per test.
- When bundling is necessary, pre-commit to a decomposition sequence: ship the bundle if it wins, then run follow-up tests that remove each component individually to identify the load-bearing elements.
- Resist the urge to skip the decomposition. The follow-up is where the durable learning lives.
Monday-morning action: Look at your last bundled "win." Identify the 3-5 individual changes inside it. Plan the decomposition tests for the next two sprints. Calendar them now, before the next "interesting" test pushes the autopsy off the roadmap.
Deeper dive: The Confounded-Variable Trap: Why Iteration Beats Big-Bang Redesigns.
---
Mistake 3: Treating the Headline Lift as the Final Estimate
Quick definition: _Regression to the mean_ = the statistical fact that extreme observations tend to be followed by less extreme ones, closer to the underlying average.
What goes wrong. Your test wins with a 25% reported lift. You project annualized impact based on 25%, roll out site-wide, and the realized impact in production turns out to be much smaller — single digits at best, sometimes flat. You attribute the gap to "audience differences at scale" or "implementation drift." The simpler explanation is regression to the mean.
Quick stat. Realistic shrinkage on replication, by reported lift size:
| Reported lift | Likely true effect | Shrinkage to expect |
| ------------- | ------------------ | ------------------------------------ |
| 5-10% | 2-6% | 30-40% |
| 10-20% | 3-8% | 50-60% |
| 20-40% | 4-10% | 70-80% |
| 40%+ | 5-12% | 80%+ (if measurement was even valid) |
Why it happens. Every observed lift = true effect + noise. Extreme observations are extreme partly because the noise component happened to be unusually positive. On replication, the noise component is a fresh draw centered near zero, so the second observation is closer to the true effect.
The fix.
- Discount any reported lift by 50-60% before using it as a planning input.
- Pre-commit to a replication step for any win that will inform a major rollout.
- Report confidence intervals on every result. A "+25% lift, CI [3%, 48%]" reads very differently than a bare "+25% lift."
Monday-morning action: Take the largest "win" your team has shipped in the last 6 months. Re-run the same change on a comparable surface as a controlled replication. The result of that replication is your real estimate of the effect.
Deeper dive: Regression to the Mean: The Statistical Concept Every New CRO Analyst Should Understand.
---
Mistake 4: Skipping the Diagnostic Checklist
Quick definitions: _MDE_ = the smallest effect size you'd consider a meaningful win. _CI_ = confidence interval. _SRM_ = sample ratio mismatch (the test of whether your traffic actually split as configured). _Power_ = the probability the test will detect a real effect of a given size.
What goes wrong. A reported lift with no surrounding context — single number, directional indicator, annualized projection. None of the standard diagnostic elements are surfaced.
Quick stat. A bare point estimate is uninterpretable. The same "25% lift" can mean a real win with tight CI [22%, 28%], or a false positive with CI [-2%, 52%]. Without the interval, you cannot tell.
The minimum diagnostic table for every test:
| Element | What it tells you | Why it matters |
| ------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Power analysis | Required sessions per arm | Without it, the test is implicitly powered for whatever effect happens to materialize |
| MDE | Pre-committed win threshold | Prevents post-hoc rationalization of marginal results as wins |
| Confidence interval | Range of plausible true effects | Distinguishes real wins from false positives at the same point estimate |
| SRM check | Did the split work as configured? | If split is materially off, randomization is broken; rest of analysis is moot |
| Segment cuts | New vs returning, mobile vs desktop, traffic source | Concentrated lifts behave differently from uniform lifts at rollout |
Monday-morning action: Add these five elements to your team's standard test-readout template. Backfill them on the last 5 tests as a calibration exercise. The cost of standardizing is one afternoon. The benefit compounds over every test the team runs from that point forward.
Deeper dive: The Diagnostic Checklist Every New Testing Team Should Standardize.
---
Mistake 5: Reading Case Studies as Evidence Rather Than Hypotheses
Quick definition: _Survivorship bias_ = the systematic distortion that occurs when only successful examples are visible in the data you're learning from.
What goes wrong. A new analyst encounters a published case study reporting a 25% lift from a tactic. They implement the same tactic on their site, expect a similar result, and observe something much smaller, flat, or negative. They conclude they executed poorly.
Quick stat. At companies that publish actual win distributions (Microsoft, Booking, Netflix), the win rate on properly powered, pre-registered A/B tests is 10-20%, with average effect sizes in the low single digits. Most tests are flat or losses. A new analyst forming expectations from curated case studies will feel like they are failing while observing the actual empirical reality of well-run experimentation.
Why it happens. Published case studies are a curated selection by construction. Wins that fit the narrative format get written up; losses, flat results, and ambiguous outcomes typically remain unpublished. The reader sees only the upper tail of the practitioner's actual distribution.
The reader's protocol for new analysts:
- Treat the headline lift as an upper bound, not a central estimate.
- Identify the missing diagnostics. Each missing element downgrades credibility.
- Note any acknowledgment paragraphs about failed replication. Read them literally.
- Reconstruct the corpus. If only wins are visible, the corpus is curated.
- Form a hypothesis to test on your own surface. Do not generalize from the case study.
Monday-morning action: Pick the most recent CRO case study you found compelling. Apply the five-step protocol. Note what is missing. Plan a controlled version on your own surface.
Deeper dive: How to Read CRO Case Studies as a New Analyst (A Reading Guide).
---
The 3 Habits That Prevent Most False Positives
If you adopt nothing else from this guide, adopt these three habits. They close most of the gap between new-analyst work and mature program work.
Habit 1: Pre-register every test. Before launch, document the hypothesis, MDE, sample target, stopping rule, and primary metric. Lock it. The single act of writing it down before the test starts removes the largest source of false positives, because it prevents retroactive rationalization of marginal results.
Habit 2: Report the full diagnostic table on every result. Confidence intervals, sample sizes, SRM checks, and segment cuts. Standardize the template once. Apply it every time. Decisions are then made on actual statistical content, not on bare numbers.
Habit 3: Replicate every win that informs a major rollout. A win on a single page is a candidate finding. A win that holds on a second comparable page is closer to evidence. A win that survives multiple replications can be confidently generalized. Most teams skip this step. Resist the pull.
Tip for CRO managers: Make these three habits non-negotiable in your team's testing template. They are the cheapest possible upgrade to the program's reliability.
---
Quick Tips for New Testing Teams
- Run tests for at least two full weekly cycles. A 10-day test captures only part of the second week and over-weights whichever weekday it terminated on.
- Compute confidence intervals manually if your tool doesn't surface them. The formula takes one cell in a spreadsheet.
- Check SRM first. Before reading any lift, run a chi-squared test that the traffic split came out as configured. If it didn't, your randomization is broken.
- Discount headlines by half. Any reported lift in a case study or LinkedIn post: assume the realized effect on your own site will be 50-60% smaller.
- Document losses with the same care as wins. The corpus you can defend is the one that includes the boring middle of the distribution.
- Look at your win rate. If you're above 25%, something is wrong upstream. Mature programs run 10-20%.
---
FAQ
I'm a new analyst. Which mistake should I fix first?
Mistake 1 (early stopping). It produces the largest single-source inflation of false positives, and the fix (pre-committing a sample target) costs nothing. Once that's standardized, move to the diagnostic table (Mistake 4), then replication (Mistake 3).
My team's tool doesn't surface confidence intervals or SRM checks. What do I do?
Compute them in your reporting layer. A confidence interval is one cell in a spreadsheet given the point estimate and sample sizes. An SRM check is a one-line chi-squared test. Most teams find that adding these manually takes one afternoon and changes the quality of their decision-making materially.
How long should a DTC A/B test actually run?
Long enough to cover two complete weekly cycles, preferably three. Day-of-week effects bias short tests toward the days they happen to span. Beyond that, the duration should be set by the sample size required to detect your chosen MDE at 95% confidence and 80% power.
What's a healthy win rate for a new program?
10-20% on properly powered, pre-registered tests. If you're meaningfully above that, look upstream for peeking, generous interpretation of "win" (counting flat tests with positive point estimates as wins), or selection bias in which tests get reported.
Is bundling multiple changes ever appropriate?
Yes, when time pressure makes one-variable-at-a-time impractical. The discipline is to pre-commit to a decomposition sequence after the bundle ships. Without that follow-up, the bundle is a one-time win, not a repeatable insight.
---
Build the Discipline Early
The five mistakes above are predictable failure modes of CRO work in the absence of explicit training. They are also each fully addressable through pre-commitment, replication, and full reporting. The earlier in your career or your program's life you adopt the discipline, the larger the cumulative compounding effect on the value of your testing program.
I built GrowthLayer to make this discipline the default for new analysts and small testing teams: pre-registered hypotheses, MDE-aware sample sizing, confidence intervals on every result, and a test journal that captures the non-winners alongside the winners.
If you are looking for experimentation roles where this discipline is the operating standard, explore open positions on Jobsolv.
Or book a consultation for help establishing a testing program from scratch, or for a methodological audit of an existing one.