Post-Hoc Analysis

What Is Post-Hoc Analysis?

Post-hoc analysis is any analysis of experiment data that wasn't part of the pre-registered plan — slicing results by segment, checking secondary metrics, or investigating unexpected patterns. It's an invaluable source of hypotheses for future tests but a dangerous source of conclusions in itself. The distinction between confirmatory analysis (pre-planned) and exploratory analysis (post-hoc) is one of the most important epistemological concepts in experimentation.

Also Known As

• Marketing teams call it deep dive, segment analysis, or results exploration.
• Growth teams say exploratory analysis or post-hoc slicing.
• Product teams use ad-hoc analysis or deep dive.
• Engineering teams refer to exploratory analysis or post-hoc slicing.
• Statisticians strictly distinguish confirmatory vs. exploratory analysis.

How It Works

Your test concludes: overall, variant loses. In exploration you slice by 12 segments — device, geo, tenure, traffic source, etc. One segment (new mobile users from paid social) shows a significant lift at p=0.03. You're excited. But with 12 segments at alpha=0.05, you'd expect roughly 0.6 false positives by chance. Your "win" might be real or might be noise. Responsibly, you log it as "hypothesis: variant works for new mobile paid-social users" and add a confirmation test to the backlog.

Best Practices

• Label every post-hoc finding as "hypothesis for future testing," never as a ship decision.
• Apply Bonferroni or false discovery rate corrections when reporting multiple comparisons.
• Be honest about how many segments you checked before finding the interesting one.
• Build confirmation tests for promising post-hoc findings and measure whether they replicate.
• Keep a post-hoc findings backlog separate from confirmatory test results.

Common Mistakes

• Shipping changes based on post-hoc segment analysis without running a confirmation test.
• The "Texas sharpshooter fallacy" — drawing a target around wherever the bullet hit.
• HARKing (Hypothesizing After Results are Known) by claiming the post-hoc finding was the intended hypothesis all along.

Industry Context

• SaaS/B2B: Low-traffic tests are especially prone to post-hoc p-hacking; discipline matters.
• Ecommerce/DTC: High traffic enables more legitimate segment analysis but also more false discoveries.
• Lead gen: Post-hoc lead quality analysis is often more valuable than the original test result.

The Behavioral Science Connection

Post-hoc analysis is where the narrative fallacy is most dangerous. Humans compulsively construct stories to explain any data pattern we observe — "the variant worked better for mobile users because the button was easier to tap" sounds compelling but might be explaining noise. Nassim Taleb's "fooled by randomness" applies directly: the more data we slice, the more "meaningful" patterns we'll find.

Key Takeaway

Post-hoc analysis generates hypotheses, never conclusions — label findings appropriately and confirm with dedicated follow-up tests.