Pre-Registration (Experiment Protocol)
The practice of formally documenting an experiment's hypothesis, metrics, sample size, and analysis plan before the test begins, preventing post-hoc rationalization of results.
What Is Pre-Registration?
Pre-registration is the practice of formally documenting your experiment's hypothesis, primary metric, sample size, duration, guardrails, and decision criteria before you launch — and ideally before you've even looked at any preliminary data. It's borrowed from clinical research, where it was introduced to combat p-hacking and publication bias. In A/B testing, it prevents the unconscious (or conscious) cherry-picking of favorable results after the fact.
Also Known As
- Marketing teams call it the test brief, test plan, or experiment doc.
- Growth teams say experiment brief or test plan.
- Product teams use experiment brief, spec, or protocol.
- Engineering teams refer to it as the test doc or experiment spec.
- Data science teams call it pre-registration, experiment protocol, or analysis plan.
How It Works
Before launching, you write a one-page document: "Hypothesis: adding a progress bar to checkout will increase completion rate by 5%+ because the Zeigarnik effect creates psychological momentum to finish started tasks. Primary metric: completion rate, measured as checkouts divided by initiated checkouts. Sample size: 40,000 per arm. Duration: 14–28 days. Guardrails: refund rate, AOV, support tickets. Stopping rule: stop when sample size and 14 days reached, or at 42 days. Ship criterion: p<0.05 AND relative lift ≥ 3% AND no guardrail degradation ≥ 2%." Share this with your team, timestamp it, launch.
Best Practices
- Write the brief in a shared document that captures a timestamp (Notion, Google Docs, or a version-controlled file).
- Include decision criteria, not just metrics — specify what constitutes ship, don't-ship, and inconclusive.
- Review the brief at test conclusion against actual results — don't retroactively rewrite.
- Keep briefs short and structured — one page forces clarity.
- Build a public experiment log so everyone can see the commitment.
Common Mistakes
- Writing the brief after seeing preliminary data — defeats the entire purpose.
- Using vague language ("we'll analyze the results and decide") that provides no pre-commitment.
- HARKing (Hypothesizing After Results are Known) when the test shows something unexpected and you pretend that's what you were testing.
Industry Context
- SaaS/B2B: Essential because long tests invite scope creep — pre-registration locks the plan.
- Ecommerce/DTC: Valuable for keeping merchandising and marketing aligned on test criteria.
- Lead gen: Often skipped because tests are "obvious" — don't skip; pre-registration catches HARKing even on obvious tests.
The Behavioral Science Connection
Pre-registration leverages the commitment and consistency principle (Cialdini, 1984). Once you've publicly committed to a plan, you're psychologically motivated to follow through. Public commitments are stronger than private ones, which is why sharing the brief with teammates matters.
Key Takeaway
One-page experiment briefs, shared before launch, eliminate more bad decisions than any statistical technique.