Effect Size (Cohen's d)
A standardized measure of the magnitude of difference between two groups, calculated as the difference in means divided by the pooled standard deviation, independent of sample size.
What Is Effect Size?
Effect size measures how large a difference actually is, stripped of sample size. Cohen's d, the most common version, divides the mean difference by the pooled standard deviation, giving a scale-free number. Conventions call d = 0.2 small, 0.5 medium, and 0.8 large.
Also Known As
- Data science teams: Cohen's d, standardized mean difference, SMD
- Growth teams: lift magnitude (in standardized units)
- Marketing teams: "how big the win actually is"
- Engineering teams: d, g (Hedges' g for small samples)
How It Works
Imagine an A/B test with a precisely estimated but small standardized mean difference. Larger samples increase precision and can detect small effects; they do not manufacture significance. Whether the estimate matters depends on the raw-unit effect, uncertainty, implementation cost, and pre-specified business threshold.
Best Practices
- Do define a minimum detectable effect in Cohen's d terms before launching experiments.
- Do report effect size with a confidence interval for honest communication.
- Do use Hedges' g when samples are small (n < 50 per group); it corrects d's bias.
- Do not celebrate significance without checking whether d is business-meaningful.
- Do not compare Cohen's d across metrics with very different variance structures without care.
Common Mistakes
- Treating small d values as important because p < 0.05.
- Using standardized effect size where a raw dollar or percent lift is more useful to stakeholders.
- Ignoring that effect size conventions are rules of thumb, not universal thresholds.
Industry Context
- SaaS/B2B: Tests often see small d values; cumulative small wins compound meaningfully.
- Ecommerce/DTC: Seasonality produces artificial d fluctuations, making pre-registration critical.
- Lead gen/services: Lead-quality effect sizes are notoriously hard to measure without long follow-up windows.
The Behavioral Science Connection
Effect size directly addresses what Kahneman calls "the planning fallacy" in analytics — treating any positive number as meaningful. By forcing you to ask "how big?" rather than "any difference?", effect size fights the narrative instinct to declare victory at the first green p-value.
Key Takeaway
A p-value summarizes compatibility with a null model under its assumptions. Effect size and its interval describe magnitude and uncertainty. Use both with business consequences.