Experiment Contamination
Any factor that compromises the validity of an experiment by introducing systematic bias, including cross-test interference, bot traffic, data pipeline errors, and external events.
What Is Experiment Contamination?
Experiment contamination is the broad category of problems that corrupt experimental results — making them unreliable for decision-making. Unlike statistical noise (which is random and expected), contamination is systematic: it biases results in a specific direction, often without any visible warning signs.
The most dangerous contamination is invisible. A 2% event-loss bias from a slow data pipeline is enough to flip winners and losers without tripping any alarm.
Also Known As
- Marketing: Campaign cross-contamination, leakage
- Sales: Sales signal contamination
- Growth: Test interference, experiment leakage
- Product: Experiment pollution, cross-test interaction
- Engineering: Tracking bias, data pipeline contamination
- Data: Selection bias, measurement bias, data quality issues
How It Works
A team launches a homepage test. Simultaneously, another team launches an email campaign driving traffic to the homepage. Email-driven visitors behave differently than organic visitors, and the traffic mix shifts during the test window. The homepage test's apparent "winner" is actually an artifact of traffic composition change, not variant effect.
Another team's test on the pricing page reveals a 40% lift — but the variant loads 200ms slower, and bot traffic disproportionately avoids slow pages. Filtering bots reveals the real lift is 3%, within noise range. The team nearly shipped a false winner.
Best Practices
- Run regular A/A tests to verify platform integrity.
- Monitor sample ratios continuously — SRM is the canary for bucketing issues.
- Filter bot traffic before analysis, not after.
- Coordinate concurrent tests to prevent cross-test interference.
- Document external events during test windows.
Common Mistakes
- Ignoring SRM alerts — they indicate contamination, not noise.
- Analyzing without bot filtering — bots can flip conclusions on slow-loading variants.
- Running overlapping tests on the same page without coordination.
Industry Context
SaaS/B2B: Small sample sizes make contamination especially dangerous — a few contaminated conversions can flip statistical conclusions.
Ecommerce/DTC: High bot traffic and frequent concurrent campaigns create rich contamination risk. Rigorous filtering and coordination are essential.
Lead gen: Form submission bots are a persistent contamination source. Honeypot fields and server-side validation are standard defenses.
The Behavioral Science Connection
Experiment contamination exploits our confirmation bias in analysis — teams interpret ambiguous contamination signals as supporting their preferred conclusion. A "too good to be true" result feels exciting, which makes teams less likely to scrutinize it for contamination. Structured contamination checks counter this by requiring proof of cleanliness rather than assuming it.
Key Takeaway
Build contamination detection into your standard analysis workflow — it requires discipline, not intuition, to protect experiment validity at scale.