Power Analysis

What Is Power Analysis?

Power analysis is design-stage math connecting effect size, sample size, alpha, and power. Skipping feasibility and sensitivity planning is one common reason experiments produce imprecise or decision-poor results.

Also Known As

• Data science: sample size calculation, sensitivity analysis
• Growth: test sizing, "how long do we run this?"
• Marketing: audience planning, campaign sizing
• Engineering: load planning for traffic allocation

How It Works

Imagine a signup page at 12% conversion. You want to detect a 10% relative lift (to 13.2%). With alpha = 0.05 (two-sided) and power = 0.80, a standard calculator gives roughly 14,750 users per variant — about 29,500 total. At 2,000 daily signups split 50/50, that is about 15 days. Drop the target to a 5% relative lift and you need roughly 58,000 per variant — a full two-month test.

Power analysis also runs in reverse. Given fixed traffic of 10,000 per variant per week and a two-week window, what lift can you detect? That reverse calculation is how you set realistic expectations with stakeholders.

Best Practices

• Always calculate two ways: required sample size for your target lift, and detectable lift for your available sample.
• Use a recent, representative baseline window and document how seasonality or promotions may affect it.
• Choose power from the consequences of missed effects and the available sample, then document the tradeoff.
• Account for multiple variants — a four-cell test is not just four two-cell tests, multiplicity matters.
• Add a documented buffer based on measured exclusions, attrition, and traffic-quality history.

Common Mistakes

• Using industry benchmarks as your baseline. Use your own representative data rather than substituting a generic market rate.
• Ignoring variance for revenue metrics. Revenue per user is noisier than binary conversion; the same sample size will detect far smaller lifts on conversion than on ARPU.
• Stopping early when you "hit significance." Peeking invalidates the power calculation and inflates false positives dramatically.

Industry Context

In SaaS/B2B, power analysis often reveals that monthly signup volume cannot support testing modest changes — which should redirect teams to qualitative research, usability testing, or bigger bets. In ecommerce, high traffic enables granular tests but revenue variance still limits what is detectable. In lead gen, power must be sized to MQL or SQL, not raw form completion, since downstream quality dominates the business decision.

The Behavioral Science Connection

Power analysis fights planning fallacy and overconfidence. Teams can underestimate how much time or sample a decision requires, and intuitive effect estimates can exceed effects in their own historical data. Writing the assumptions down turns that confidence into a falsifiable plan.

Key Takeaway

Power analysis is the contract between your hypothesis and reality. If your test cannot mathematically detect the effect you care about, no amount of patience or creativity at the readout will rescue it.