Articles exploring statistics through the lens of behavioral science and experimentation. Practical frameworks for growth leaders who measure in revenue, not vanity metrics.
20 articles
Most definitions of statistical significance are wrong — or at least misleading. Here's what p < 0.05 actually means, why 95% confidence is a convention not a law, and how to avoid the significance theater killing your CRO program.
There's no single number. But there is a rigorous framework. Here's how to calculate exactly how long your A/B test needs to run — and why stopping early is the most expensive mistake in CRO.
A practical guide to A/B test sample size calculations, with worked examples, common mistakes, and what to do when your traffic is too low. Covers multiple variations, MDE sensitivity, and how to avoid the '2-week arbitrary duration' trap.
Most teams stop A/B tests for the wrong reasons. This framework gives you four conditions to verify before calling a test — and explains the peeking problem, sequential testing, and what 'results stabilized' actually means.
MDE is the most underrated concept in A/B testing. Get it wrong and you'll run underpowered tests for months. This guide covers MDE calculations, real benchmarks by test type, and how to frame it as a business decision.
A practical comparison of Bayesian and frequentist A/B testing from a CRO practitioner who's run 100+ experiments. Covers the peeking problem, sequential testing, and a decision table for choosing the right approach.
Stopping an A/B test at the wrong time — too early or too late — is one of the most expensive mistakes in experimentation. Here are the rules that mature testing programs actually use.
Seven years running 100+ experiments taught me that test duration is the most violated rule in CRO. Here's the full framework — minimum days, sample size math, and when it's actually OK to stop early.
MDE isn't a calculator input — it's the foundation of your entire experiment design. Set it wrong and you'll either run 6-month tests or miss real effects entirely. Here's the framework, the math, and the business ROI approach that changes how you plan experiments.
Optimizely now offers three statistical engines: Sequential (Stats Engine), Frequentist Fixed Horizon, and Bayesian. Each one changes what you measure, how you decide, and when you can stop. Here's how to pick the right one for your team.
Most practitioners misread statistical significance as 'probability you're right.' It isn't. Here's what it actually means, why 95% is a convention not a law, how peeking kills your results, and how Optimizely's Stats Engine solves the classical problem.
Tuesday your experiment shows 94% confidence. Friday it's 71%. Nothing changed — so what's happening? Here's how to tell normal statistical fluctuation from novelty effect, regression to the mean, and actual drift.
Running 20 tests at 95% confidence means you expect at least one false positive by chance. Here's how Optimizely's FDR control works, what it doesn't protect against, and how to structure your testing program to minimize spurious wins.
Week 4. Your test shows a 6% lift. Optimizely says 'not enough data.' Here are the five reasons experiments stall before reaching statistical significance — and what to do about each one.
Not all A/B tests use the same statistics. Learn which test to use for conversion rates, revenue, count data, and small samples — with a practical decision tree.
Compare Bayesian and Frequentist approaches to A/B testing. Understand the practical differences, when each excels, and why the debate matters less than your fundamentals.
Demystify A/B testing statistics — p-values, confidence intervals, Type I and Type II errors, and one-tail vs two-tail tests explained in plain English with business context.
Two progress bar variants used the same behavioral science principle. One drove $250K-$500K in revenue. The other did nothing. Here's why execution matters more than theory in A/B testing.
Here's something that doesn't get talked about enough in the experimentation world: the idea isn't what wins. The execution is. This is the story of a test where two variants used the exact same psychological mechanism with radically different results.
Sample Ratio Mismatch (SRM) is a critical diagnostic for A/B tests. When variant traffic splits deviate from expectations, it signals broken randomization that can invalidate lift and lead to costly, phantom wins.