Incrementality Testing

What Is Incrementality Testing?

Incrementality testing uses randomized experiments to measure the causal impact of a marketing activity — the outcomes that would not have occurred without it. Unlike attribution, which distributes credit among observed touchpoints, incrementality tests ask the counterfactual question: how many of these conversions would have happened anyway?

Also Known As

• Marketing team: "holdout test," "lift test," "incremental lift"
• Sales team: "true ROI measurement"
• Growth team: "causal measurement," "incrementality experiment"
• Data team: "RCT," "randomized controlled trial," "causal lift study"
• Finance team: "incremental revenue analysis"
• Product team: "feature lift test"

How It Works

You suspect retargeting ads are driving $500,000/month in attributed revenue. You run a 4-week incrementality test: 80% of your audience sees the ads (treatment), 20% is suppressed (control). At the end, the treatment group generates $2.1M in revenue, the control generates $2.0M pro-rated. Incremental lift = ($2.1M - $2.0M) / $2.0M = 5%. Attributed revenue said $500K; incremental revenue is closer to $105K. The gap (nearly 80%) is revenue that would have happened regardless of the ads.

Best Practices

• Run incrementality tests for your highest-spend channels first — the stakes justify the opportunity cost.
• Pre-register the hypothesis, sample size, duration, and success threshold before the test starts.
• Run for at least one full purchase cycle (4+ weeks for most channels).
• Use geo-based holdouts when user-level suppression isn't feasible.
• Repeat incrementality tests annually — channel economics drift.

Common Mistakes

• Stopping tests early when results look favorable ("peeking") — this inflates false positives.
• Using tiny holdouts (1-2%) that can't detect real effects.
• Not accounting for cross-channel spillover (suppressed users seeing the brand elsewhere).

Industry Context

SaaS and B2B teams run incrementality tests on branded search, retargeting, and review-site placements — channels where cannibalization is suspected. Ecommerce and DTC teams lean on incrementality for paid social and display, where attribution is notoriously inflated. Lead gen operations often use incrementality to validate whether gated-content promotion actually drives qualified pipeline vs. unqualified form fills.

The Behavioral Science Connection

Attribution feeds what Kahneman called "the illusion of understanding" — we see a touchpoint precede a conversion and assume causation. Incrementality testing punctures that illusion by forcing us to measure what we didn't see: the counterfactual world where the marketing activity never happened. It's the operational expression of the difference between correlation and causation.

Key Takeaway

Incrementality testing is the only reliable way to know whether marketing spend is creating value or just taking credit for inevitable conversions.