Before you trust the results of any A/B test, you should ask a more fundamental question: does your testing infrastructure actually work? An A/A test is the simplest and most powerful way to answer that question. It reveals whether your experimentation platform is measuring accurately, splitting traffic correctly, and producing trustworthy results.
The concept is elegant in its simplicity. You run a test where both variants are identical. There is no treatment, no change, no hypothesis. Both groups see exactly the same experience. If your infrastructure is working correctly, you should find no statistically significant difference between the groups. But what you actually find might surprise you.
What Is an A/A Test and Why Run One?
An A/A test splits your traffic into two groups and shows both groups the exact same experience. No element is changed. The hypothesis, if you could call it one, is that there should be no difference between the groups because there is nothing different to observe.
You would run an A/A test for several reasons. The most important is infrastructure validation. When you set up a new experimentation platform, migrate between tools, or make significant changes to your tracking code, an A/A test confirms that the system is measuring and splitting correctly.
An A/A test also serves as a sanity check on your statistical methods. If you analyze the results using the same methods you use for real A/B tests, you can verify that your analysis pipeline produces sensible outputs. If your A/A test declares a winner with 99% confidence, something is broken.
Finally, A/A tests help calibrate expectations. They show you what "nothing" looks like in your data, which is essential context for interpreting what "something" looks like in a real test.
The Surprising Results of A/A Tests
Here is the result that catches many teams off guard: if you run enough A/A tests, a substantial percentage of them will reach statistical significance. This is not a bug. It is a mathematical certainty.
If you use a significance threshold of p < 0.05, then by definition, 5% of A/A tests will produce a statistically significant result purely by chance. Run 20 A/A tests, and you should expect one false positive. Run 100, and you should expect five. This is the false positive rate working exactly as designed.
Some teams run this exercise and are deeply unsettled by the results. They watch a dashboard declare a "winner" in a test where both variants are identical. The p-value is below 0.05. The confidence interval excludes zero. Every indicator says there is a real effect. And yet there cannot be, because nothing was changed.
This is not evidence that your infrastructure is broken (assuming the test is properly implemented). It is evidence of what the p < 0.05 threshold actually means. It does not mean "this result is real." It means "if the null hypothesis is true, there is less than a 5% probability of observing this result." The null hypothesis is sometimes true, and sometimes you get that 5% outcome.
What A/A Tests Reveal About Significance Testing
The most valuable lesson from A/A testing is not about your infrastructure. It is about the limitations of the statistical framework you use for every experiment.
The Base Rate of False Discoveries
In a typical experimentation program, not every test has a true effect. Some changes genuinely move the needle. Others are inert. If 30% of your tests have a true positive effect and you use p < 0.05, then among your "significant" results, a meaningful fraction are false positives.
The exact proportion depends on your win rate and your significance threshold, but the math is often sobering. In programs where only 10% of tests have a true effect (which is common for mature products), the false discovery rate among significant results can approach 30-40%. Nearly one in three "wins" might be noise.
A/A tests make this abstract concept visceral. You see the false positive with your own eyes, in a context where you know with certainty that the effect does not exist.
The Importance of Effect Size
A/A test results also illustrate why effect size matters more than p-values. A statistically significant result in an A/A test will have a measured effect size, but that effect size is noise. It has no practical meaning. The same principle applies to real A/B tests: a statistically significant 0.05% lift is technically "real" in the statistical sense but commercially meaningless.
Mature programs define a minimum detectable effect before each test. They commit to ignoring results below that threshold, regardless of statistical significance. A/A tests reinforce why this discipline matters.
Infrastructure Problems A/A Tests Can Detect
Beyond the statistical lessons, A/A tests can reveal genuine infrastructure issues. Here are the most common:
Sample Ratio Mismatch (SRM)
If your A/A test assigns 50% of traffic to each group but one group consistently has significantly more visitors, you have a sample ratio mismatch. This indicates a problem with your randomization algorithm, your assignment mechanism, or your data collection. SRM is one of the most serious technical issues in experimentation because it means your groups are not comparable.
Common causes of SRM include bot filtering that disproportionately affects one group, redirect-based implementations where one variant loads faster and captures more events, and cookie-based assignment issues where certain browsers handle the assignment cookie differently.
Tracking Discrepancies
If your A/A test shows different conversion rates between groups at a rate beyond what chance would predict, it could indicate that conversion events are being counted differently. This might happen when the tracking code fires at different points in the page load, when third-party scripts interfere with event recording, or when client-side experiments interact with analytics code.
Temporal Contamination
If users can switch between groups across sessions (assignment is not sticky), your A/A test will show inflated variance and unpredictable results. Proper experimentation requires that a user who is assigned to group A stays in group A for the duration of the test. An A/A test where you track individual user assignments over time can reveal stickiness issues.
How to Run an A/A Test Properly
Running an A/A test seems straightforward, but doing it well requires attention to several details:
Use your production experimentation setup. The point is to validate your real infrastructure, so use the same tools, code, and processes you use for actual A/B tests. Do not create a special A/A test environment. You want to test the same path your experiments follow.
Run it for a full business cycle. At minimum two weeks, ideally four. This captures weekday/weekend variation, beginning/end of month patterns, and other temporal effects that could create artificial differences.
Analyze it with your standard methods. Apply the same statistical tests, significance thresholds, and analysis workflows you use for real experiments. The goal is to validate the entire pipeline, not just the assignment mechanism.
Check multiple metrics. Do not just look at your primary conversion metric. Check secondary metrics, segment breakdowns, and funnel stages. Infrastructure issues sometimes manifest only in specific metrics or segments.
Run multiple A/A tests. A single A/A test has limited diagnostic power. Running several, either simultaneously or sequentially, gives you a better picture of your system's behavior. If more than 5% reach significance, investigate why.
Using A/A Tests to Calibrate Your Program
Beyond validation, A/A tests serve as a calibration tool for your experimentation program. They establish a baseline of "no effect" that helps you contextualize real results.
Track the distribution of p-values across your A/A tests. Under the null hypothesis (which is true in every A/A test), p-values should be uniformly distributed between 0 and 1. If they cluster near 0 (too many significant results), your tests are biased. If they cluster near 1 (too few significant results), your tests are conservative.
Track the distribution of observed effect sizes. In A/A tests, the true effect is zero, so the distribution of observed effects should be centered on zero with a spread determined by your sample sizes and metric variance. This tells you the "noise floor" of your measurement system. Any real A/B test result should produce an effect larger than this noise floor to be considered meaningful.
Some organizations run continuous A/A tests alongside their real experiments as an ongoing system health check. A holdout group that never receives any experimental treatment serves as a persistent control, and monitoring this group over time reveals systematic measurement drift or instrumentation changes.
When Not to Run an A/A Test
A/A tests have costs. They consume traffic that could be used for real experiments. In traffic-constrained environments, dedicating weeks of data to a test with no possible business impact is a significant investment. Weigh this cost against the value of infrastructure validation.
If your experimentation platform has a strong track record of accurate results, regular A/A testing becomes less critical. But you should always run an A/A test after major infrastructure changes: new tracking code, new experimentation platform, site migration, or changes to your data pipeline.
Additionally, if you are already monitoring for sample ratio mismatch and other technical issues in your real experiments, you are getting some of the validation benefits of A/A testing continuously.
The Deeper Lesson
The most important takeaway from A/A testing is epistemic humility. Your experimentation system is not an oracle that delivers truth. It is a measurement instrument with known limitations. False positives are mathematically guaranteed. Infrastructure issues can silently corrupt results. Statistical significance is not certainty.
A/A tests force you to confront these limitations directly. They make abstract statistical concepts concrete and visible. And they build the institutional awareness needed to interpret real experiment results with appropriate confidence, neither overconfident in every significant result nor dismissive of the genuine insights experimentation provides.
Every experimentation program should run A/A tests at least once when setting up and after any significant infrastructure change. The small investment in traffic pays for itself many times over in the trust and reliability of every subsequent A/B test you run.