Data Segmentation for A/B Tests: Finding Hidden Winners
Discover how to uncover segment-level insights hidden within overall A/B test results.
Know what to test, when to trust the result, and what to do next. Practical guides for analysts, growth teams, and founders.
Discover how to uncover segment-level insights hidden within overall A/B test results.
Understand the tradeoffs between client-side and server-side A/B testing architectures.
Inconclusive A/B test results are not failures. Learn how to extract learning from flat tests, distinguish between wrong hypotheses and weak…
Transform your A/B testing program from isolated experiments into a compounding knowledge system.
Move beyond gut-feel prioritization with structured frameworks for ranking A/B test hypotheses.
Discover the external validity threats that can invalidate your A/B test results, from seasonality and sample pollution to the flicker effect, and how to…
Learn what statistical power means for A/B testing, why 80% is the standard, and how underpowered tests lead to costly false negatives that cause you to…
Master A/B test sample size calculation including the relationship between baseline conversion rate, minimum detectable effect, and statistical power to…
Understand what p-values really mean in A/B testing, why common interpretations are wrong, and how to use statistical significance correctly for business decisions.
Understand the difference between one-tailed and two-tailed hypothesis tests in A/B testing, when each is appropriate, and the simple conversion rule between them.
A practical guide to the Bayesian vs Frequentist debate in A/B testing, why it matters less than you think, and what practitioners should actually focus on…
Learn the science behind A/B test duration, why stopping at significance is dangerous, and how to determine the right test length using sample size…
Learn how to interpret confidence intervals and margin of error in A/B test results, why your conversion rate is always an estimate with uncertainty, and…
User testing reveals the gap between how you designed your product and how people actually experience it.
A strong hypothesis is the difference between an experiment that teaches you something and one that wastes traffic.
Heat maps and session replays are seductive but easy to misinterpret. Learn how to use click maps, scroll maps, and form analytics to generate real insights…
Technical bugs and performance issues silently destroy conversion rates.
Most A/B tests fail because they skip the research phase. Learn how conversion research — from heuristic analysis to qualitative methods — builds the…
Quantitative data tells you what is happening on your website. Qualitative research tells you why.
Learn the five-dimension heuristic evaluation framework — relevancy, clarity, value, friction, and distraction — and how to score pages systematically for…
Regression to the mean explains why early A/B test results often look dramatic but fade over time.
Understand how multivariate testing works, when it outperforms A/B testing, the traffic requirements for MVT, and why most programs run roughly ten A/B…
Learn what A/B/n testing is, how traffic splits work with three or more variants, when you need multiple variants, and the tradeoffs compared to simple A/B tests.
Why false positives are the biggest threat to A/B testing programs, how A/A tests prove the problem is real, and why stopping at significance is the number…
Know what to test, when to trust the result, and what to do next. Practical decision guides for analysts, growth teams, and founders. Free. Weekly.
Opens Substack to confirm your subscription · Free · Unsubscribe anytime