Multivariate Testing or a Confounded A/B Test: Which Did You Run?
Four bundled changes in one experiment came back inconclusive, and couldn't have told us anything either way. A confounded-test-design lesson.
Read Article
→
Blog · Tag
Experiment Design
Articles exploring experiment-design through the lens of behavioral science and experimentation. Practical frameworks for growth leaders who measure in revenue, not vanity metrics.
32 articles
Do Website Personalization Examples Help—or Just Add Friction?
See why a customer-selector pop-up can fail, how two first-party chooser tests compare, and how to test useful personalization without adding friction.
Read Article
→
Which Lead Magnet Examples Actually Earn the Download?
See why brochure previews may increase downloads, how to grade the evidence, and how to test lead magnet clarity without mistaking images for proof.
Read Article
→
Color Psychology in Marketing: Does Matching Beat Meaning?
See what a product-color matching test really suggests, what its source omits, and how to test color congruence without relying on folklore.
Read Article
→
How Can You Tell Whether an A/B Testing Case Study Is Trustworthy?
Evaluate any A/B testing case study with a 12-point evidence checklist covering source, sample, metrics, stopping, SRM, limitations, and transfer.
Read Article
→
Which Navigation A/B Test Wins: Visible, Collapsed, or Removed?
Run a cleaner navigation A/B test with visible, collapsed, and removed treatments, precommitted metrics, guardrails, SRM checks, and decisions.
Read Article
→
Should a Landing Page Have Navigation—or Is It Costing Sales?
Should a landing page have navigation? Compare the public evidence, missing methods, intent conditions, guardrails, and a safer A/B test plan.
Read Article
→
Four A/B Testing Examples—and How Much You Should Trust Them
See four A/B testing examples graded by evidence quality, with missing data, limits, transferable lessons, and safer next-test plans.
Read Article
→
Minimum Detectable Effect (MDE): How to Choose the Right One
Minimum detectable effect (MDE) is the most important input to A/B test design. Learn how to calculate and choose the right MDE for business impact and traffic.
Read Article
→
How to Write A/B Test Hypotheses That Actually Hold Up
A practitioner's guide to writing A/B test hypotheses — the structure that survives review, the three failure modes that produce inconclusive tests, and how…
Read Article
→
5 A/B Testing Mistakes That Derail New CRO Teams (And How to Avoid Each One)
Five statistical mistakes that derail new DTC testing teams, with practical checks for sample size, peeking, metrics, and interpretation.
Read Article
→
The Baseline Conversion Trap: Why Your A/B Test Baseline Is Already Wrong
How denominator and time-window mistakes distort experiment baselines, with an illustrative worked example and a framework for sizing under uncertainty.
Read Article
→
Network Effects and A/B Testing: When Users Influence Each Other
Standard A/B tests break when users influence each other. Learn how network effects create interference and the experimental designs that handle it.
Read Article
→
How to Answer 'Design an A/B Test for X' in Interviews
Master the 'design an A/B test' interview question with a structured framework. Learn the step-by-step approach that impresses hiring managers every time.
Read Article
→
Sample Ratio Mismatch: Causes, Checks, and Fixes
Sample ratio mismatch can invalidate an A/B test. Learn how to detect SRM, trace its root cause across the experiment pipeline, and decide whether to rerun.
Read Article
→
Pre-Registration for A/B Tests: Why Documenting Your Plan Prevents Bias
Pre-registration locks in your experiment plan before seeing results. Learn why it prevents p-hacking, metric shopping, and post-hoc rationalization.
Read Article
→
How to Handle Multiple Metrics in A/B Testing
When A/B tests track multiple metrics, statistical complexity increases. Learn frameworks for managing metric conflicts and making sound decisions.
Read Article
→
Guardrail Metrics in A/B Testing: Protecting What Matters
Guardrail metrics prevent A/B tests from causing hidden damage. Learn how to set them up, monitor them, and use them to make better ship decisions.
Read Article
→
Choosing the Right Primary Metric for Your A/B Test
Your primary metric determines whether an A/B test succeeds or fails. Learn how to select metrics that are sensitive, aligned, and actionable.
Read Article
→
How to Design an A/B Test: The Complete Experiment Design Guide
Learn how to design rigorous A/B tests from hypothesis to execution. Covers experiment structure, variable isolation, and common design mistakes.
Read Article
→
Underpowered A/B Tests: The Silent Killer of Experimentation Programs
Underpowered tests waste traffic, miss real wins, and erode trust in experimentation. Learn how to diagnose the problem and fix it before it kills your program.
Read Article
→
What Is Statistical Power and Why Most A/B Tests Don't Have Enough
Statistical power determines whether your A/B test can detect real effects. Most experiments run underpowered, wasting traffic and producing misleading results.
Read Article
→
Sample Size Calculation for A/B Tests: The Complete Guide
Running A/B tests without proper sample size calculation wastes traffic and produces unreliable results. Learn the inputs, formulas, and practical trade-offs.
Read Article
→
Minimum Detectable Effect (MDE): The Most Important Number You're Not Setting Correctly
MDE isn't a calculator input — it's the foundation of your entire experiment design.
Read Article
→
How AI Changes Hypothesis Generation for A/B Tests
Explore how AI and large language models are transforming A/B test hypothesis generation by eliminating confirmation bias, surfacing non-obvious patterns in…
Read Article
→
Statistical Power in A/B Testing: Avoiding False Negatives
Learn what statistical power means for A/B testing, why 80% is the standard, and how underpowered tests lead to costly false negatives that cause you to…
Read Article
→
Sample Size Calculation for A/B Tests: Getting It Right Before You Start
Master A/B test sample size calculation including the relationship between baseline conversion rate, minimum detectable effect, and statistical power to…
Read Article
→
One-Tailed vs Two-Tailed Tests: Does It Actually Matter?
Understand the difference between one-tailed and two-tailed hypothesis tests in A/B testing, when each is appropriate, and the simple conversion rule between them.
Read Article
→
How to Write A/B Test Hypotheses That Actually Generate Insights
A strong hypothesis is the difference between an experiment that teaches you something and one that wastes traffic.
Read Article
→
A/B/n Testing Explained: When and Why to Test Multiple Variants
Learn what A/B/n testing is, how traffic splits work with three or more variants, when you need multiple variants, and the tradeoffs compared to simple A/B tests.
Read Article
→
Anchoring Bias in A/B Testing: How Your Control Variant Shapes What Users See
Anchoring bias silently distorts A/B test results by making the control variant the psychological reference point against which all alternatives are judged…
Read Article
→
The Experiment Brief That Stops Stakeholder Rewrites
How to write experiment briefs that prevent last-minute stakeholder rewrites by building alignment into the document structure.
Read Article
→