Heterogeneous Treatment Effects (HTE)
The observation that treatment effects vary across subgroups — the average effect hides winners and losers underneath.
What Is Heterogeneous Treatment Effects (HTE)?
Average treatment effect (ATE) is the headline number from an A/B test. HTE is the reality underneath: the treatment might lift power users by 15% while hurting trialists by 5%, netting out to a +4% ATE that masks both the real win and the real risk. Ignoring HTE means shipping changes that help the majority while silently degrading critical segments — or killing changes that would have been transformative for a subset.
Also Known As
- Data science: HTE, treatment effect heterogeneity, subgroup effects
- Growth: "it worked for some users, not others"
- Marketing: segment-level impact
- Engineering: conditional effects, interaction effects
How It Works
The overall estimate is +3%, while exploratory cohort estimates differ. If cohorts and decision rules were pre-specified and multiplicity was handled, they may support a segmented decision; otherwise treat them as hypothesis-generating and confirm them in a dedicated test.
HTE is estimated with pre-registered subgroups, causal forests, meta-learners (T, S, X, R learners), or Bayesian hierarchical models. Critical constraint: subgroups must be defined before the test, or inference is exploratory.
Best Practices
- Pre-register 3–5 subgroups in the test doc before launch.
- Correct for multiple comparisons when reporting HTE across many segments.
- Use causal forests or meta-learners for high-dimensional HTE rather than manual subgroup slicing.
- Require replication for any HTE finding before acting on it in a personalization rule.
- Visualize with a forest plot of segment effects to expose heterogeneity at a glance.
Common Mistakes
- Post-hoc subgroup fishing. If you slice 20 ways and report the one that's significant, you've found noise.
- Acting on HTE from underpowered segments. A "win" in a segment of 2,000 users has huge variance.
- Ignoring HTE entirely and treating the ATE as if it described every user equally.
Industry Context
Treatment heterogeneity may exist across plan tier, lifecycle stage, device, customer tenure, traffic source, or other pre-specified segments. Estimate it with adequate power and multiplicity control rather than assuming it.
The Behavioral Science Connection
The ATE summarizes an average effect and may not describe every user. HTE analysis can test whether effects differ across justified segments, but subgroup estimates are noisier and vulnerable to multiplicity.
Key Takeaway
Start with the ATE, then examine pre-specified heterogeneity when the design can support it. Treat post-hoc subgroup patterns as hypotheses until confirmed.