Interaction Effects

What Are Interaction Effects?

Interaction effects (also called test interference or crosstalk) occur when two or more simultaneous experiments affect each other's results. Test A's measured lift is contaminated by whatever Test B is doing to shared users, and vice versa. This turns efficient parallel testing into a statistical minefield — and the bigger your program grows, the more dangerous it gets.

Also Known As

• Marketing teams call it test conflict or test interference.
• Growth teams say crosstalk or experiment collision.
• Product teams use interaction effects or experiment interaction.
• Engineering teams refer to it as test conflict, overlap, or collision.
• Statisticians call them interaction effects and study them in factorial designs.

How It Works

Test A (pricing page): new discount display. Test B (pricing page): new tier card layout. Both run simultaneously. A user in A-variant + B-variant sees a new discount on new cards; the combination happens to draw the eye differently than either change alone. You measure: Test A shows +5% conversion, Test B shows +4%. You ship both, assuming the lifts will compound. But the real combined lift is +3% — because the changes interact. Users in both variants were counted in both tests' lift, inflating each.

Best Practices

• Maintain a shared experimentation calendar visible to all teams.
• Use mutual exclusion groups for tests on the same surface or decision point.
• For low-risk overlap (different pages, different journey stages), simultaneous testing is fine.
• Run confirmation tests after shipping multiple winners to verify actual compound impact.
• Flag high-risk overlaps (same page, same decision) before tests launch, not after.

Common Mistakes

• Assuming lifts from separate tests simply add up — they often don't, especially on the same page.
• Not tracking which tests a given user was in, so you can't analyze interactions post-hoc.
• Letting teams launch tests independently without coordination, guaranteeing crosstalk.

Industry Context

• SaaS/B2B: Low traffic means few simultaneous tests; interaction risk is usually low.
• Ecommerce/DTC: High traffic enables many simultaneous tests — interaction risk scales fast.
• Lead gen: Usually small teams with single tests; interaction effects are rarely a concern until scale.

The Behavioral Science Connection

Interaction effects expose the "additive fallacy" — our intuition that changes combine linearly. Human behavior rarely works that way. Two nudges can amplify each other (coherent framing) or cancel each other out (cognitive overload). Only factorial designs or explicit interaction analysis reveal which is happening.

Key Takeaway

Interaction effects scale with your test volume — coordinate ruthlessly once you're running more than five simultaneous experiments.