Sample ratio mismatch (SRM) means an experiment's observed group sizes are unexpectedly different from its intended allocation. A fifty-fifty test does not need identical counts, but a sufficiently unlikely imbalance calls the comparison into question. Before interpreting conversion lift, check the allocation and investigate any SRM warning.
SRM can arise in assignment, delivery, logging, or analysis. It does not identify the cause or tell you whether the treatment helped. The useful question is: where did the observed population stop matching the experiment you planned?
The SRM check, in order
For an ordinary fixed-allocation A/B test, the sequence is:
- Confirm the intended split. Read the saved allocation and ramp history. A planned ninety-ten split should be checked against ninety-ten.
- Count independent randomized units. If assignment happens by person, count distinct people in each variant over the same window. Sessions, pageviews, and conversions are different quantities.
- Compare observed and expected counts. Use the platform's SRM check and its documented alert threshold. A conversion-rate significance test cannot answer this question.
- Investigate a warning before making a rollout decision. Compare assignment logs, recorded exposure, processed data, and the final analysis population to locate the first unexplained discrepancy.
Record the allocation, counting unit, time window, and check result in the experiment brief. That gives the next analyst something reproducible to investigate instead of a screenshot with two percentages.
A worked sample ratio mismatch example
Consider two hypothetical experiments, both configured to assign users equally. Each records a 49:51 split, but one is ten times larger.
| Illustrative experiment | Control users | Treatment users | Expected per group | Chi-square statistic | Approximate p-value |
|---|---|---|---|---|---|
| Smaller sample | 4,900 | 5,100 | 5,000 | 4 | 0.0455 |
| Larger sample | 49,000 | 51,000 | 50,000 | 40 | 0.000000000254 |
These are invented teaching examples, not company results. The same visible percentage gap provides much stronger evidence against the intended allocation in the larger sample.
For each group, subtract its expected count from its observed count, square the difference, and divide by the expected count. Add those contributions:
χ² = Σ (observed − expected)² / expected
In the smaller example, the calculation is 100² / 5,000 + 100² / 5,000 = 4. With two groups and a prespecified split, the reference distribution has one degree of freedom. This is the chi-square goodness-of-fit calculation described by NIST; its approximation requires adequate expected counts.
Microsoft documents an SRM alert threshold of p < 0.0005. Under that rule, only the larger example raises an alert. This is Microsoft's platform practice, not a universal cutoff; use your platform's documented rule, agreed before viewing outcomes. Microsoft's SRM guidance
An SRM p-value measures how unusual the counts would be under the allocation model. It is not the probability that the experiment is broken. Passing the check also cannot certify every part of an experiment: missing data could affect both groups similarly, or an important problem could be too small to detect yet.
The Microsoft experiment whose result changed sign
An MSN team tested a carousel with more cards. Engagement initially appeared to fall. Investigation found that highly engaged treatment users were being mistaken for bots and removed from the analysis. After the filtering problem was resolved, the result became positive. Microsoft's published account shows how a treatment can change which users survive into the dataset. Read the Microsoft case
The practical implication is easy to miss: correct assignment is only the beginning. An analysis can compare biased groups even when the original randomization worked. Checking the final dataset therefore matters as much as checking the assignment service.
Where the mismatch enters the experiment
The cross-company SRM research distinguishes four stages: assignment, execution, log processing, and analysis. Use those stages to structure the investigation.
Assignment
At the assignment stage, reconcile configured allocation with actual buckets. Review identifier stability, eligibility rules, manual overrides, and ramp changes. Keep the denominator consistent: assigned accounts and assigned browsers do not necessarily represent the same population.
Execution
Compare assignment with delivery and recorded exposure. A variant-specific crash, redirect, or loading failure may prevent some assigned users from appearing downstream. A mismatch appearing only after exposure logging points toward this part of the system, although it does not establish which component failed.
Log processing
Compare raw records with the table used for analysis. Inspect delayed ingestion, joins that drop unmatched records, and deduplication rules. Preserve a small set of traceable records from each group so engineering can follow the same units through the pipeline.
Analysis
Check the unfiltered population against each eligibility or trigger filter. Selecting people who clicked a treatment-only component cannot create an equivalent control population. A trigger needs a comparable eligibility definition in both groups, with the necessary logging to identify it.
A focused investigation
A useful investigation produces a timeline and a reconciliation of counts. Start with the stage where the discrepancy first appears, then narrow it using facts that could explain the failure.
Plot the allocation by day or hour and compare changes with releases, outages, targeting edits, and ramp adjustments. Check plausible segments such as browser or app version. Treat those slices as diagnostic leads: searching many slices can produce chance warnings, and a segment that passes is not automatically safe to publish.
Avoid combining a changing allocation into a single fifty-fifty expectation. For planned ramps, preserve each period's allocation and use the platform's supported check. Adaptive allocation requires a method that accounts for how assignment probabilities change; the simple fixed-split example above is insufficient.
For each suspected cause, write down what it predicts. A logging problem might explain clean assignment counts followed by missing exposure records. An eligibility problem might explain a balanced full population and an imbalanced filtered one. Investigate the prediction against source records before changing the analysis.
The outcome should be a documented mechanism, the affected population, and the action required. "The ratio looks better after removing these users" is not enough.
Repairing the analysis versus rerunning
A restricted analysis can sometimes be defensible when the cause is known and affected records are identifiable without selecting on outcomes. The SRM research describes a campaign that forced visitors into one variant; removing those identifiable visitors allowed analysis of the remaining randomized population. The authors also caution that rerunning is often necessary. Authors' research paper
Before accepting a repair, establish why the remaining groups retain a valid randomized comparison, state which population the revised estimate describes, and review its precision. Removing observations can change both the question and the amount of information available.
Rerun after fixing the system when the mechanism remains unexplained, missing records cannot be recovered, or treatment changed who was observed in a way the proposed analysis cannot address. Reweighting group totals does not, by itself, recover those people or explain their absence.
If a rerun is needed, revisit the sample-size calculation using the eligible population. Keep the minimum detectable effect tied to the original decision unless the business question has changed. A smaller retained sample is not a reason to quietly redefine success.
FAQ
Is a 49:51 split always a problem?
No. The intended allocation, total sample, and chosen alert rule all matter. The examples above have the same split but very different p-values. Compare counts with their expected allocation rather than using a fixed percentage-gap rule.
Should SRM be checked only at the end?
No. Monitoring data quality during collection can expose a delivery or logging problem earlier. Follow the platform's monitoring procedure and preserve alerts for investigation. This does not authorize repeatedly checking treatment-effect p-values and stopping as soon as the result looks favorable.
Does a passing SRM check mean the result is trustworthy?
It removes one warning, not every source of bias. Measurement definitions, missing events, conversion windows, interference, and the chosen analysis still need review. If your team cannot reconcile its assignment and measurement, request an experimentation readiness diagnostic before relying on the result for rollout.