Atticus Li has led enterprise experimentation programs and mentored analysts through data-quality checks, including sample ratio mismatch. This guide explains how to investigate a split rather than treating any fixed percentage as a universal alarm threshold.
I get this question from new analysts. The exact counts change, but the spirit is the same: "The observed split is not exactly what we configured. Is something wrong? Should we restart?"
The percentage alone cannot answer the question. But the fact that they're asking is a good sign: they are paying attention to data quality, which is a habit I want to reinforce. The right next step is a count-based diagnostic, not reassurance from a visual heuristic.
So here's what I tell them.
Random Allocation Is Not a Perfect 50/50 Machine
When you set up a 50/50 A/B test, you're telling the experimentation platform to randomly assign each visitor to one of two groups with equal probability. Equal probability does not mean equal count.
Flip a fair coin a finite number of times and the counts will usually differ. The size of an unsurprising difference depends on how many flips you observed; a percentage alone cannot tell you whether the allocation is unusual.
Your A/B test traffic split works the same way. Random assignment with equal probability will produce slightly unequal groups. This is not a bug. It's basic statistics. The law of large numbers says the ratio will converge toward 50/50 as sample size increases, but at any finite point, there will be variance.
The same observed percentage can be ordinary at one sample size and surprising at another because expected random variation shrinks as the count grows. That is why an SRM check uses observed counts and the configured allocation rather than a fixed percentage-difference rule.
The key concept is expected variance. At any given sample size, a range of split ratios is compatible with fair random assignment. Calculate that compatibility from the counts instead of memorizing example ranges.
Why There Is No Universal Acceptable Range
I do not use a percentage lookup table. The acceptable variation depends on the total count, the planned allocation, the unit of randomization, and the team's pre-specified false-alarm threshold. A test configured 50/50, one configured 90/10, and a cluster-randomized test require different calculations.
The statistical check for whether the observed counts are incompatible with the planned allocation is a Sample Ratio Mismatch (SRM) test. Pre-specify its threshold and investigation policy alongside the experiment design.
The point is: do not diagnose a percentage by eye. Run the planned count-based check.
When to Actually Worry: Sample Ratio Mismatch
Here's where it gets serious. There are situations where a traffic split deviation is not random variance — it's a signal that something in your test setup is broken. This is called Sample Ratio Mismatch, and it's one of the most important diagnostic checks in experimentation.
SRM is flagged when the observed traffic counts are unlikely under the expected allocation at the team's pre-specified threshold. The threshold is a policy choice with a false-alarm trade-off, not a universal 1% rule.
When an SRM check flags a mismatch, something may be systematically affecting assignment, exposure, or logging—or the pre-specified procedure may have produced its allowed false alarm. Investigation distinguishes among those possibilities. Common implementation causes include:
Bot filtering differences. If your bot filter removes more traffic from one variant than the other, you'll see SRM. This happens when the variant triggers different page behaviors that bots interact with differently.
Redirect latency. If one variant requires a redirect and the other doesn't, users who bounce during the redirect are lost from one group but not the other. This is especially common in server-side tests where the control is the default experience.
JavaScript errors. If the variant JavaScript crashes for certain browsers or devices, those users might not get tracked. Your platform thinks they were assigned to the variant, but their events never fire. The split looks off because you're missing data from one group.
Caching issues. CDN caching can serve the wrong variant to users, or cache one variant more aggressively than the other, creating systematic imbalances.
Interaction with other tests. If two tests share traffic and one test's variant affects whether users reach the second test, you can get SRM in the second test.
An SRM flag means the assignment or data pipeline needs investigation before the outcome is interpreted. It does not reveal the cause by itself, and it is not an automatic restart rule. Determine which traffic and metrics were affected, repair the cause, and decide whether the existing data remain usable under the documented policy.
The SRM Diagnostic
For a common independent-assignment design, an SRM check compares observed counts with expected counts using an appropriate goodness-of-fit procedure. Use the procedure and threshold specified in the test plan; clustered assignment, repeated exposure, or very small expected cells may require a different calculation.
I plan an early allocation check once enough observations have accumulated for the diagnostic to be meaningful, then repeat it at the final readout. The checkpoint comes from traffic and operational risk rather than a universal hours-after-launch rule.
For a standard independent-assignment test, GrowthLayer's SRM calculator can compare the expected split with observed counts. A flag tells you to investigate; it does not diagnose the implementation or prove that all outcome data are invalid.
The Mentoring Moment
When a new analyst flags an imperfect split, I do not declare it fine from the percentage. I use it as a teaching moment to explain the full spectrum of split quality.
I walk them through three possible states. First: the counts are compatible with the planned allocation under the pre-specified check. Second: the check flags a mismatch but the cause is not yet known. Third: investigation identifies an assignment, exposure, or logging failure and defines which data were affected. The action follows the diagnosed cause—not an eyeballed percentage.
The lesson isn't "stop worrying about splits." The lesson is "worry about the right thing." Don't worry about normal variance. Do worry about systematic bias. And use a statistical test — not your gut — to tell the difference.
I also tell them something that took me years to internalize: the anxiety about imperfect splits comes from a good place. It means they care about data quality. I want to channel that instinct into a repeatable diagnostic rather than suppress it.
What I Wish Someone Had Told Me
When I started in experimentation, I didn't even know SRM was a concept. I spent weeks agonizing over minor split deviations that were completely normal, and I probably missed at least one real SRM issue because I didn't know what to look for.
SRM is not always covered in introductory material, but it is an important diagnostic in applied experimentation. A flagged test needs investigation before an impressive outcome is used for a decision.
So to every new analyst reading this: your instinct to check the split is correct. Your threshold and follow-up policy need to be pre-specified. A percentage alone is not a diagnosis; an SRM flag is a prompt to investigate.
Plan an allocation check for every randomized test and record the result with the readout. That can prevent a decision from being made on compromised assignment or logging data.
The Split-Quality Takeaway
If you take one thing from this article, it is this: do not spend limited QA time judging small percentage differences by eye. Run the pre-specified allocation diagnostic at meaningful checkpoints and investigate flags before interpreting the outcome.
Perfect splits are a myth. Systematic bias is real. Focus on what matters.
_Run the planned SRM diagnostic on your current independent-assignment tests. GrowthLayer's SRM calculator compares expected allocation with observed counts; investigate any flag before interpreting outcomes._
FAQ
Does a 51/49 split mean the randomization is broken?
Not by itself. Random allocation produces ordinary count variation. Evaluate the observed counts with an SRM check rather than an eyeballed percentage rule.
When should an analyst investigate immediately?
Investigate when the SRM check fails or when allocation differs by browser, device, source, timing, or eligibility in a way the design does not explain.
Can the experiment be trusted after an SRM failure?
Pause the decision, find the implementation or data cause, and assess affected traffic. A strong outcome metric does not repair compromised assignment.
Contact me if your team needs a repeatable traffic-allocation and SRM review checklist.