Mediation Analysis for A/B Tests With a Causal Story

You ran the test. Conversion moved. Now someone asks the question that matters: why?

That question is where a lot of good A/B testing programs go sideways. When a test variant shows a positive total effect on your primary metric, it is easy to assume you know why it happened. However, if I tell myself the wrong story about a winner, I can scale the wrong change, spend more money, and lock in worse product decisions.

This is where understanding the underlying drivers through mediation analysis ab tests earns its keep. It helps me separate the simple fact that a variant worked from the reality of whether it worked through the specific mechanism I intended to change.

Key Takeaways

• Mediation analysis decomposes a test result into the expected path, specifically the indirect effect, and the components that fall outside your hypothesis.
• It matters most when a rollout decision has real revenue, cost, or product risk attached.
• Randomization protects the treatment effect, not the mediator, so weak measurement can ruin the story.
• In product-led growth and startup growth, it helps me avoid scaling proxy metrics that do not create cash.
• If I cannot name the mechanism before launch, I usually should not run the analysis after the fact.

Why lift alone isn't enough in A/B testing

A standard experiment provides what we call a total effect. Variant B raised trial starts by 6%, or checkout completion fell by 2%, or average order value stayed flat. That data is useful, but it remains incomplete.

The problem is simple. Most product changes move more than one thing at once.

For example, a new pricing page serves as the independent variable that influences user behavior, while the subscription rate acts as the dependent variable. A change like this might reduce friction, increase urgency, and attract lower-intent buyers in the same session. A new onboarding flow might improve activation by making one step obvious, while also filtering out users who would have become better long-term customers. If I only look at the headline outcome, I miss the mechanism, and the mechanism is what tells me whether to scale, edit, or kill the change.

This matters more than people admit. In real experimentation programs, a lot of tests do not produce clean wins. GrowthLayer's data-driven insights from A/B test results show how often teams end up with inconclusive outcomes or mixed signals. That is normal. Product behavior is messy.

Mediation analysis gives me a way to ask a better question. Did the treatment work because it changed the user behavior I intended to change, or did it work through some side path I do not fully control?

That distinction affects decision making. If the lift came through a stable, repeatable behavior, I can build on it. If it came from noise, novelty, or a side effect that creates hidden cost, I should slow down.

In growth work, the expensive mistake is not missing a clever insight. It is telling a clean story about a dirty result.

How mediation analysis works in A/B tests

I think about mediation in plain terms.

There is an independent variable, which is the treatment I randomized. There is a mediator variable, which is the internal step I believe the treatment changed on the way to the outcome. Finally, there is the dependent variable, which is the business result I care about.

If I change onboarding copy, the mediator variable might be completed first workspace setup. The dependent variable might be paid conversion within 14 days. If the new copy increases setup completion and that completion explains part of the revenue lift, that indirect effect is the mediated portion of the outcome. The rest of the result is the direct effect, which represents the portion of the outcome that moved without going through that specific measured mediator.

Randomization tells me whether the independent variable changed the dependent variable. Mediation analysis asks how much of that change traveled through the mediator variable.

To visualize these relationships, I often use path diagrams to map out the flow from treatment to mediator and finally to the business result. In traditional statistics, the Baron and Kenny approach was the gold standard for testing these relationships, sometimes supported by the Sobel test to determine significance. While Baron and Kenny laid the foundation, modern causal inference uses metrics like ACME average causal mediation effects to quantify the indirect effect and ADE average direct effects to isolate the direct effect.

That sounds neat on paper. In practice, it gets hard fast.

The treatment is randomized, but the mediator usually is not. That means I get clean identification for the total effect, but I need stronger assumptions to say the mediator caused the outcome. If users who complete setup are also more motivated, better qualified, or less price sensitive, then setup completion may be partly standing in for hidden traits.

This is why mediation analysis in A/B tests is more than a reporting trick. It is a causal claim with extra assumptions attached.

The formal literature is worth reading if your team wants statistical grounding. The ACM paper on direct and indirect effects in online experiments is a strong starting point. If you want the broader framework behind modern causal mediation, Imai's causal mediation paper lays out the assumptions clearly.

I do not need every founder to become a causal inference specialist. I do need them to know this: a mediated story is only as good as the path I have measured and the assumptions I am willing to defend.

When I use mediation analysis, and when I skip it

I do not run mediation analysis on every test. In fact, most teams should not.

If the experiment is low stakes, the user journey is short, and the outcome is already close to the treatment, the extra work is usually not worth it. A button color test on a one-step form does not need a full causal mechanism story.

I reach for mediation analysis when the decision is expensive, the funnel has multiple steps, or the winning metric is a proxy. This is common in product-led growth, where activation stands in for revenue, or in startup growth, where I have to move fast before long-term outcomes fully mature. By evaluating the indirect effect and the direct effect, I can better understand which part of the user journey is actually driving the change.

This is the quick filter I use:

The takeaway is blunt. Mediation analysis is for decisions where the story behind the lift changes what I do next.

That includes pricing flows, onboarding, checkout, trial-to-paid funnels, recommendation systems, and AI features that alter user effort. If you are unsure how messy real outcomes get, it helps to browse real A/B test case studies. A lot of experiments produce results that are directionally interesting but operationally hard to trust.

If I cannot see a clear product or financial decision that depends on the mechanism, I do not bother. Plain A/B testing is enough.

The assumptions that break causal stories

This is the part most teams skip, then regret later.

The biggest risk in mediation analysis is that I mistake a measured event for a true mechanism. To get this right, you need a rigorous causal inference approach rather than just correlating data points. Those are not the same thing.

Say I test a shorter signup form. My mediator is "reached plan selection." My outcome is "paid within seven days." If users who reach plan selection are also the ones with stronger purchase intent, and my treatment affects which users get that far, then the mediator is tangled up with selection bias. Now I am telling a causal story with data influenced by confounding variables.

A significant mediator is not proof of a causal mechanism.

A few failure modes show up over and over.

First, the mediator is measured too late. If I track a post-treatment event after users have already seen several downstream elements, that event may contain the effects of multiple changes. The path is contaminated.

Second, the mediator is too broad. Engagement is not a mechanism. Viewed recommended teammates page might be. Time in app usually isn't.

Third, the treatment changes who gets observed. A common example is when a variant increases early drop-off. Then later-stage events only exist for a selected subset. That can make the mediated path look cleaner than it is, often masking the presence of confounding variables that skew the results.

Fourth, the analytics layer is weak. If event timestamps are off, identities merge poorly, or server-side and client-side logs disagree, the analysis looks sophisticated but rests on sand.

Fifth, the treatment changes multiple plausible mediators at once. That does not kill the method, but it raises the bar. I need to know which path I actually care about, and I need a sufficient sample size to estimate it. When performing this analysis, I often use a bootstrapping method to generate accurate confidence intervals for the mediation effect, ensuring that the results remain stable even when the data is noisy.

The statistical phrase behind a lot of this is "sequential ignorability." I do not use that phrase often in meetings, but I keep the underlying causal assumptions in mind. After randomization, I still need to believe there are no unmeasured confounders between the mediator and the outcome, once I have controlled for the right pre-treatment variables. Without verifying these, your confidence intervals may be misleadingly precise.

That is a strong assumption. In consumer products, it often fails.

So who should ignore mediation analysis? Teams with weak instrumentation, tiny sample sizes, vague theories of user behavior, or a habit of rewriting events mid-test. If that sounds like your environment, fix the basics first. Better plain analytics beats fancy inference built on broken events.

The financial impact of getting the mechanism wrong

This isn't a stats hobby. It is a money problem.

If I scale a winner for the wrong reason, I can create a revenue hole that takes months to uncover.

Take a simple case. Suppose a variant lifts click-through to checkout by 10%, but the final purchase rate only improves 1%. At first glance, it looks like a small win. Mediation analysis may show that while the variant increased checkout entry with statistical significance, it also brought in lower-intent traffic that raised abandonment later. That changes the decision entirely.

If paid acquisition is feeding that page, rolling out the test could increase top-of-funnel volume and ad spend while barely moving the contribution margin. The finance team sees more traffic, and the growth team sees more starts, but cash barely changes. Here, the total effect is positive, but the direct effect of the variant on the purchase outcome is diluted by negative behavior in the middle of the funnel.

I have seen the same issue in SaaS. A new onboarding assistant, often built with applied AI, can raise activation events while shifting work to support, sales, or success behind the scenes. If the mechanism is not that users reached value faster on their own, the indirect effect might be negative or neutral. If the lift does not reflect users finding actual value, it may only move cost from one line item to another.

This is why I care about mediated paths that connect to controllable economics.

If a trial-to-paid lift runs through a first team invite, first integration, or first successful import, I can invest in that path. Those are behaviors a product-led growth team can design around. If the lift mostly runs through a short-term curiosity spike, or through manual intervention, I treat it as fragile.

My decision rule is pretty simple:

• If the total effect is positive and the intended mediator carries a meaningful share of it, I usually keep pushing.
• If the total effect is positive but the indirect effect is weak or backwards, I slow down and audit side effects.
• If the business case depends on a mediator I cannot measure with statistical significance, I do not scale based on belief alone.
• I always look at the direct effect to ensure the lift isn't masking a drop in core user value.

That is where mediation helps with conversion work. It does not replace business judgment. It sharpens it.

Where behavioral science and AI fit into the analysis

A lot of growth changes are really behavior changes in disguise.

When I alter defaults, reorder choices, add progress indicators, surface social proof, or reduce form effort, I am making a bet from behavioral science. I am betting that attention, trust, effort, loss aversion, or commitment will move behavior.

Mediation lets me test whether that behavioral bet was the actual path.

If I add a progress bar to onboarding, I do not only want to know if paid conversion moved. I want to know whether the lift ran through completion of the next setup step, shorter time to value, or more users reaching their first successful outcome. If it did not, I may have created a placebo effect that will not hold once the novelty fades.

In more complex experiments, we often find that the indirect effect of a feature depends on the user group. This is where moderated mediation becomes critical. For example, a copilot might boost engagement for power users but confuse new signups. By applying moderated mediation, I can isolate whether my intervention works differently across distinct user segments. Sometimes, the treatment itself influences the strength of a user characteristic, which is a classic case of mediated moderation.

The same logic applies to AI products.

In applied AI, teams often test summary views, chat copilots, smarter recommendations, or generated onboarding help. Those features do not only change UI. They change cognitive effort, trust, speed, and error rates. The right mediator might be "first useful answer accepted," "time to completed task," or "number of retries before success."

That matters because AI features can create fake wins. A copilot might increase session length because users are confused. A recommendation engine might raise clicks but lower purchase intent. Good analytics can tell me the result. Mediation analysis can help me ask whether the result came through the path I actually wanted.

The research on identifying causal mechanisms from randomized experiments is useful here because modern product systems often have multiple moving parts and nonlinear outcomes.

For founders, the practical point is simple. In a growth strategy built on product behavior, I need mechanism level evidence before I commit headcount, roadmap time, or acquisition budget to scale a winner.

How I would run this on a live experiment

I keep the process tight.

Before launch, I write one sentence: "I believe treatment X will improve outcome Y because it changes mediator M." If I cannot write that sentence cleanly, I do not have a causal story yet; I have a guess.

Then I pick one primary mediator, maybe two. Not six. A long list is a sign I do not know what I am testing.

I instrument the mediator with exact timestamps and stable event definitions. If the event can be fired twice, delayed, or lost client-side, I clean that up first. Mediation analysis is much less forgiving than a basic lift readout.

After the test, I look at four things in order:

1. Did the treatment move the dependent variable at all?
2. Did the treatment move the mediator variable?
3. Is the mediator strongly associated with the outcome after controlling for the right pre-treatment variables?
4. Does the indirect effect line up with the product story I would bet money on?

To quantify these relationships, I often use structural equation modeling. For more rigorous testing, I might employ the lavaan package in R. While older methods like Baron and Kenny or the Sobel test are useful for quick checks, I prefer using the bootstrapping method to ensure the statistical significance of the results.

When evaluating the output of structural equation modeling, I look at whether the data reflects complete mediation, where the mechanism fully explains the change, or partial mediation, where the treatment still exerts a strong direct effect on the outcome. In practice, complete mediation is rare, so I usually expect to see some level of partial mediation.

If any of those steps break, I stop telling a clean mechanism story.

I also compare the story against costs. Did the variant raise support load, discount usage, returns, or fraud review? In conversion work, a path that looks good in product analytics can still be bad for margin.

My short actionable takeaway is this: on your next test, define one mediator before launch and write the rollout rule in advance. If the treatment wins but the mediator does not move, do not scale without a second look.

Frequently Asked Questions

When should I use mediation analysis instead of standard A/B testing?

You should reserve mediation analysis for high-stakes experiments where you need to understand the mechanism behind a lift before making significant investments. It is most valuable when you have complex, multi-step funnels or when your primary winning metric is a proxy, such as activation, that you need to tie back to long-term revenue.

Can I perform mediation analysis on an experiment that has already finished?

You can perform the analysis post-hoc, but it is generally discouraged unless you had a clear hypothesis regarding the mechanism before the test began. If you lack a pre-defined mediator, you risk cherry-picking correlations that do not represent true causal relationships, leading to unreliable conclusions.

What is the biggest risk of relying on mediation analysis?

The primary risk is mistaking correlation for causation due to unmeasured confounding variables or selection bias. Because the mediator is not randomized like the treatment, you must make strong, sometimes untestable assumptions to claim that a specific path truly caused the observed outcome.

How many mediators should I track in a single experiment?

You should focus on one or two primary mediators that align directly with your core product hypothesis. Tracking too many variables increases noise and the likelihood of finding spurious results, making it difficult to distinguish a genuine causal path from random data fluctuations.

Conclusion

When I use mediation analysis in A/B tests, I am not trying to make the result sound smarter. I am trying to avoid an expensive story that feels right but turns out to be wrong.

The true value is clarity. A positive test with a weak mechanism is not the same as a positive test with a strong one. That distinction changes product decisions, budget allocations, and the confidence I have in future experiments. By utilizing mediation analysis ab tests, you gain a far more nuanced understanding of the total effect than you ever could with simple lift figures alone.

If I cannot explain which user behavior carried the lift, I treat the win as provisional. That is usually the safer move, and over time, it is the one that compounds.