A free learning path · Growth & product teams
Run a test.Make a better decision.
Go from an uncertain idea to an experiment you can explain. Six practical lessons connect the planning, tools and judgment that make a result useful.
Free and ungated · Six working templates · No account required
Lesson 01
Can we run a useful test?
Start with the decision and the people eligible for the experiment. A platform can assign variants and analyze outcomes; the information available still depends on your sample and design.
Take away: A traffic and detectable-effect worksheet.
Work through the decision
- Choose the unit you will randomize: a person, an account or another independent unit. Repeat visits are not new independent participants.
- Estimate the baseline for the same eligible population and outcome window. Write down the smallest improvement worth acting on separately from the effect your design can detect.
- Use the sample-size planner with an explicit baseline, detectable effect, allocation, significance level and power. Divide each arm’s target by its eligible daily allocation.
- Check conversion delay, weekly patterns and delivery capacity. If the decision is infeasible, change the question, collect better qualitative evidence or document the remaining uncertainty.
Keep the limit visible. A smaller sample can still constrain plausible effects. It may be unable to distinguish the small improvement you care about. Bayesian or sequential software does not remove that information limit.
Go deeper
Your working notes
Your notes stay in this page. Copy or download them before leaving; they are not submitted or saved.
Download a blank worksheet (.md)Lesson 02
What should we change and measure?
Turn an observed problem into a testable explanation. A useful brief connects the change, the intended audience, the outcome and the decision you will make.
Take away: An experiment brief with a primary metric and guardrails.
Work through the decision
- Record what you actually observed and where it came from. Separate customer evidence, your interpretation and the proposed mechanism.
- Describe control and treatment precisely enough for an implementer to build them. A bundle of changes tests the bundle; it cannot identify which element caused an effect.
- Define one primary decision metric, its denominator and observation window. Add guardrails for outcomes you cannot afford to harm.
- Name an implementation owner and a decision owner. Agree on eligibility, exclusions, stopping and rollout rules before launch.
Keep the limit visible. More clicks on an invitation button do not establish more activated accounts. Verify the downstream outcome and avoid treating a plausible psychological explanation as a measured cause.
Go deeper
Your working notes
Your notes stay in this page. Copy or download them before leaving; they are not submitted or saved.
Download a blank worksheet (.md)Lesson 03
Which tool fits our team?
Write the requirements before comparing brands. Your experiment surface, implementation owner and measurement setup determine which products belong on the shortlist.
Take away: A two-tool shortlist with eligible plans and trade-offs.
Work through the decision
- Specify native mobile, product code, an existing website or a landing-page builder. These are different implementation decisions.
- Record who owns installation, exposure events, analysis and ongoing QA. Feature flags alone do not establish a complete experimentation workflow.
- Check the actual plan supporting the experiment, its billing unit, allowances, overages, seats and annual commitment. Distinguish a free tier from a trial or self-hosted software.
- Compare two eligible candidates using meets, partly meets, does not meet or not verified. Keep evidence confidence separate and record a reason to reject each option.
Keep the limit visible. The linked comparisons are researched evaluations, not claims that every product was personally tested. Verify decisive features and current commercial terms before purchase.
Go deeper
Your working notes
Your notes stay in this page. Copy or download them before leaving; they are not submitted or saved.
Download a blank worksheet (.md)Lesson 04
How do we launch reliably?
A clean implementation gives the result meaning. Trace one eligible unit from assignment through the outcome before sending the full audience into the experiment.
Take away: An assignment, exposure and tracking checklist.
Work through the decision
- Check eligibility and stable assignment across relevant devices, sessions or colleagues. Record expected allocation and who can change it.
- Verify when an exposure is logged, whether it duplicates, and whether the same definitions apply to both variants. Preserve assignment and exposure counts separately.
- Reconcile outcomes, timestamps, exclusions and missing events against a trustworthy source. Test errors, loading behavior and the rollback path.
- Investigate sample-ratio mismatch before trusting a lift. An A/A test can reveal problems, but one significant result can occur under a correctly calibrated procedure.
Keep the limit visible. An SRM warning is an investigation trigger, not a diagnosis of one specific cause. Monitoring operational harm is distinct from repeatedly checking a fixed-horizon p-value and stopping when it looks favorable.
Go deeper
Your working notes
Your notes stay in this page. Copy or download them before leaving; they are not submitted or saved.
Download a blank worksheet (.md)Lesson 05
Can we trust the result?
First establish what was measured and whether the design remained valid. Then compare the plausible effects with the business decision, including the cost of getting it wrong.
Take away: A worked readout with uncertainty and data-quality findings.
Work through the decision
- Record actual allocation, data-quality findings, sample, outcome window and deviations from the original plan.
- Report the primary effect in its original units with an interval appropriate to the preselected method. Distinguish absolute percentage-point differences from relative changes.
- Examine guardrails and preplanned subgroup analyses. Label exploratory segment findings and account for multiplicity before treating them as separate discoveries.
- Describe what the interval rules out and what remains plausible. A non-significant result does not prove equivalence, and statistical significance does not establish commercial value.
Keep the limit visible. That interval assumes independent binary outcomes and an appropriate fixed analysis. Repeated observations, account clusters, small counts or sequential monitoring require methods that match those conditions.
Go deeper
Your working notes
Your notes stay in this page. Copy or download them before leaving; they are not submitted or saved.
Download a blank worksheet (.md)Lesson 06
Ship, retest or stop?
Close the loop by stating the action, the uncertainty you accept and what you learned. The record should help a future team understand the decision without reconstructing a dashboard.
Take away: A decision record your team can reuse.
Work through the decision
- Apply the precommitted business and statistical rule. Separate evidence of benefit, evidence of harm and uncertainty that still affects the decision.
- Consider delivery cost, reversibility and guardrails. If you ship for a non-experimental reason, say so rather than relabeling the test a causal success.
- Retest only when new information, a changed treatment or a justified new design makes the next study useful. Extending a test solely to obtain significance changes its error behavior.
- Save the brief, implementation notes, readout and decision together. Include the population and conditions where the result may not transfer.
Keep the limit visible. A test of a bundled treatment supports a decision about that bundle under the studied conditions. It does not establish a universal psychological rule or predict the lift for another company.
Go deeper
Your working notes
Your notes stay in this page. Copy or download them before leaving; they are not submitted or saved.
Download a blank worksheet (.md)Put the learning to work
Keep the reasoning with the result.
Bring your brief, readout and decision into the same team conversation. GrowthLayer, a product I’m building, provides experiment templates and a place to preserve that knowledge.
For occasional research and practical decision notes, subscribe to the existing newsletter.
Interested in a clearly labeled placement or an educational resource? Explore partnerships.