Articles exploring statistics through the lens of behavioral science and experimentation. Practical frameworks for growth leaders who measure in revenue, not vanity metrics.
38 articles
Meta-analysis isn't about combining experiments — it's about knowing when you have enough similar tests for the aggregate to tell you something true. The Aggregation Validity Threshold shows how.
Statistical failures compound into credibility damage. The Statistical Trust Deficit framework explains why rigor in SRM detection and false positive control has non-linear ROI.
The Statistical Debt framework shows how underpowered tests and post-hoc metrics compound silently — until one shipped false positive costs the team years of credibility.
CUPED uses pre-experiment data to reduce noise in your A/B tests. Learn how this variance reduction technique works and when it dramatically improves power.
Low traffic does not mean you cannot experiment. Learn proven strategies for running meaningful A/B tests when your sample size is limited.
Prepare for A/B testing interviews with this complete study guide covering statistics, experiment design, business metrics, and behavioral science fundamentals.
Statistical significance and business impact are different things. Learn to translate A/B test results into the financial language that drives decisions.
The majority of A/B tests produce unreliable results due to common statistical errors. Learn the critical mistakes undermining your testing program.
Testing multiple variants, metrics, or segments without correction dramatically increases false discoveries. Learn why this happens and how to control for it.
Checking A/B test results before the planned endpoint is the most common validity threat in experimentation. Learn why it happens and how to prevent it.
Bayesian and frequentist methods answer different questions about your A/B tests. Understand the trade-offs so you can pick the right approach for your program.
MDE is the most important and least understood input to A/B test design. Learn how to set it based on business impact, traffic, and decision context.
Statistical power determines whether your A/B test can detect real effects. Most experiments run underpowered, wasting traffic and producing misleading results.
Running A/B tests without proper sample size calculation wastes traffic and produces unreliable results. Learn the inputs, formulas, and practical trade-offs.
Confidence intervals tell you more than p-values ever could. Learn how to read them, use them for decisions, and avoid the common misinterpretations teams make.
P-values drive every A/B testing decision, but most teams misinterpret them. A clear, jargon-free explanation of what p-values mean and how to use them.
Statistical significance is the most misunderstood concept in A/B testing. Learn what it really measures, why teams misuse it, and how to interpret it correctly.
If your traffic comes in waves, classic A/B testing can feel like driving with fogged-up windows. Monday looks nothing like Saturday. A paid spike hits, then disappears. Your "winner" flips two weeks later.
Nothing burns trust faster than a "winning" test on a page you didn't change. That's why I still use A/A testing when the roadmap is crowded.
Most definitions of statistical significance are wrong — or at least misleading. Here's what p < 0.05 actually means, why 95% confidence is a convention not a law, and how to avoid the significance theater killing your CRO program.
There's no single number. But there is a rigorous framework. Here's how to calculate exactly how long your A/B test needs to run — and why stopping early is the most expensive mistake in CRO.
A practical guide to A/B test sample size calculations, with worked examples, common mistakes, and what to do when your traffic is too low. Covers multiple variations, MDE sensitivity, and how to avoid the '2-week arbitrary duration' trap.
Most teams stop A/B tests for the wrong reasons. This framework gives you four conditions to verify before calling a test — and explains the peeking problem, sequential testing, and what 'results stabilized' actually means.
MDE is the most underrated concept in A/B testing. Get it wrong and you'll run underpowered tests for months. This guide covers MDE calculations, real benchmarks by test type, and how to frame it as a business decision.
A practical comparison of Bayesian and frequentist A/B testing from a CRO practitioner who's run 100+ experiments. Covers the peeking problem, sequential testing, and a decision table for choosing the right approach.
Stopping an A/B test at the wrong time — too early or too late — is one of the most expensive mistakes in experimentation. Here are the rules that mature testing programs actually use.
Seven years running 100+ experiments taught me that test duration is the most violated rule in CRO. Here's the full framework — minimum days, sample size math, and when it's actually OK to stop early.
MDE isn't a calculator input — it's the foundation of your entire experiment design. Set it wrong and you'll either run 6-month tests or miss real effects entirely. Here's the framework, the math, and the business ROI approach that changes how you plan experiments.
Optimizely now offers three statistical engines: Sequential (Stats Engine), Frequentist Fixed Horizon, and Bayesian. Each one changes what you measure, how you decide, and when you can stop. Here's how to pick the right one for your team.
Most practitioners misread statistical significance as 'probability you're right.' It isn't. Here's what it actually means, why 95% is a convention not a law, how peeking kills your results, and how Optimizely's Stats Engine solves the classical problem.
Tuesday your experiment shows 94% confidence. Friday it's 71%. Nothing changed — so what's happening? Here's how to tell normal statistical fluctuation from novelty effect, regression to the mean, and actual drift.
Running 20 tests at 95% confidence means you expect at least one false positive by chance. Here's how Optimizely's FDR control works, what it doesn't protect against, and how to structure your testing program to minimize spurious wins.
Week 4. Your test shows a 6% lift. Optimizely says 'not enough data.' Here are the five reasons experiments stall before reaching statistical significance — and what to do about each one.
Not all A/B tests use the same statistics. Learn which test to use for conversion rates, revenue, count data, and small samples — with a practical decision tree.
Compare Bayesian and Frequentist approaches to A/B testing. Understand the practical differences, when each excels, and why the debate matters less than your fundamentals.
Demystify A/B testing statistics — p-values, confidence intervals, Type I and Type II errors, and one-tail vs two-tail tests explained in plain English with business context.
Here's something that doesn't get talked about enough in the experimentation world: the idea isn't what wins. The execution is. This is the story of a test where two variants used the exact same psychological mechanism with radically different results.
Sample Ratio Mismatch (SRM) is a critical diagnostic for A/B tests. When variant traffic splits deviate from expectations, it signals broken randomization that can invalidate lift and lead to costly, phantom wins.