How to Fund Experimentation on Evidence, Not Arguments

The same experiment result can get a growth program funded for another year or gutted at the next planning cycle — and which one happens often has nothing to do with the number itself. It has to do with who decides what the number means, and when.

TL;DR

• This is a decision-governance problem, not a reporting one. An honest, unspun result can support opposite investment calls depending on the reference point the room adopts — and the reference point is chosen, usually by whoever frames it first.
• Science already solved a version of this. Clinical trials pre-register their outcomes precisely so researchers can't decide what "worked" after seeing the data. Growth teams almost never do the equivalent.
• The leader's move is to pre-register the decision rule — commit, before the data arrives, to what result triggers what call — so an honest number can't be retrofitted into whatever conclusion someone wanted.
• Audience note: this is written for the person who funds or cuts experimentation — a founder, growth exec, or the manager deciding whether the function earns its budget — not for the analyst running the test.

Same figures, both columns. The number doesn't decide the budget. The reference point does — and if nobody set it in advance, it goes to whoever argues best.

This is not the spin problem — the number is honest

It's worth separating this from the failure mode most people picture. The politics of A/B testing is about results getting misreported — cherry-picked, selectively segmented, dressed up. That's a practitioner problem, and the fix is rigor and a single source of truth for how numbers get computed.

This is the opposite situation: the number is computed correctly, reported honestly, and still justifies two contradictory decisions. Nobody lied. A 15% win-rate is genuinely "85% of tests failed" and genuinely "we found real, validated improvements at low cost per attempt." Both readings are defensible. Which one the room believes depends on the comparison point it adopts — and comparison points are a choice, not a fact. Reference-dependence is one of the most robust findings in behavioral economics: a number has no meaning until you fix what it's being measured against.

So the person who establishes the reference point has effectively made the budget decision before anyone looks at a second data point. In most organizations, that person is whoever speaks first and frames most confidently — which is not the same as whoever is right.

What clinical trials know that growth teams don't

Medicine ran into this decades ago and built a fix into the system. A clinical trial has to register its primary outcome before collecting data — you declare, in advance, exactly what you're measuring and what would count as success. The reason is precise: if researchers get to choose which outcome to emphasize after seeing the results, they'll unconsciously (or consciously) pick whichever one looks best. That's p-hacking, and pre-registration exists to make it impossible.

How badly does the after-the-fact version distort things? Ben Goldacre's COMPare project checked 67 trials in top journals against their registered protocols and found 301 pre-specified outcomes that went unreported, and 357 new outcomes silently added without justification (COMPare Trials). Even with registration, "outcome switching" was rampant — which is why the stronger safeguard, Registered Reports, locks the analysis plan in via peer review before any data exists (overview of outcome switching).

The scientific insight is simple and transferable: if you decide what a result means after you see it, you will decide in favor of what you already wanted. The only defense is to decide before.

Growth teams run experiments with real rigor and then throw all of it away at the decision layer, because the interpretation — what this result should trigger — is left wide open until after the number lands. Pre-registration is standard practice for the test and nonexistent for the decision the test was supposed to inform.

The leader's move: pre-register the decision rule

The fix isn't a better dashboard. You cannot out-argue reference-dependence in the moment, because both sides already hold the same correct number. The only durable move is to remove the ambiguity before the number exists — the executive version of pre-registration.

Concretely, before the quarter or the test window, the leader commits the program to a decision rule in writing:

1. Pre-register the outcome-to-action map. "If the program produces at least N validated winners worth at least $X in incremental margin, it's funded at this level; below that, we cut scope." Now the win-rate can't be reframed after the fact, because the bar and the consequence were fixed while the outcome was still unknown.
2. Pre-agree the reference point. Establish what a normal result is for your context — a 15% win-rate is normal, not low — before results are in, so the number arrives with its meaning already attached instead of up for grabs.
3. Judge on a small basket, not one metric. Realized incremental margin, the option value of the ambitious bets taken, learning that redirected strategy. The moment a single number becomes the budget target, it gets gamed — teams run safe, unambitious tests to inflate a win-rate, and the metric rots. That's Goodhart's law in action: when a measure becomes a target, it stops being a good measure.
4. Report in incremental dollars, not proxies. A win-rate is a proxy; incremental revenue is the metric that isn't a vanity number. The more the funding conversation runs on realized dollars agreed in advance, the less room there is to win it on framing.

The through-line: a result's meaning should be settled while it's still hypothetical and nobody knows which way it'll cut. Decide it after, and meaning goes to whoever frames fastest.

Why this is the leader's job, not the analyst's

The analyst can compute an impeccable number and still watch it get weaponized upward or downward in a room they don't control. Pre-registering the decision rule is an act of governance, and governance is the leader's job. The most durable experimentation programs I've built started exactly here: agreeing the decision rule with finance before the test ran — projecting the revenue impact and how it would be valued — so that when the result landed, its meaning was already settled and the conversation happened in dollars everyone had pre-committed to. The programs that struggled skipped that step and let each honest result arrive as a blank slate for whoever framed it first to fill.

This is also what separates a growth function that survives budget scrutiny from one that doesn't. Across a large portfolio, the discipline that holds up is tying every test to a pre-agreed dollar value rather than defending a contestable proxy after the fact. A leader who pre-registers decisions is running the function like an evidence-based investment portfolio. A leader who doesn't is running it like a debate club, where the best rhetorician allocates the capital.

FAQ

How is this different from just having good reporting?

Good reporting fixes how the number is produced — it stops cherry-picking and drift. It does nothing about what the number means for a decision, because an honestly-produced number still supports multiple conclusions depending on the reference point. Pre-registration operates one layer up: it fixes the decision rule and the comparison point in advance, so the interpretation isn't contestable after the fact. You need both — clean reporting and a pre-committed decision rule.

Isn't setting the decision rule in advance just gaming it early in your favor?

It's the opposite, because you commit before you know which way the result will fall. Gaming is choosing the frame after you see the outcome to reach the conclusion you wanted. Pre-registration binds you to a rule while the outcome is still unknown, so it constrains you regardless of how the number lands. The integrity comes precisely from committing under uncertainty — the same reason clinical-trial registration works.

What if my stakeholders won't agree to a decision rule up front?

Refusal to pre-commit is itself the finding. It usually means someone wants to preserve the option to reframe the result later. Surfacing that turns an implicit framing fight into an explicit conversation about what the program is actually for — which is a conversation a leader should want to have before spending a quarter's budget. Even a rough pre-agreed bar removes most of the post-hoc weaponization.

I'm evaluating a growth leader to hire — what should I look for?

Ask how they decide whether an experimentation program is working. A weaker candidate describes the metrics they'd report. A stronger one describes the decision rule they'd set before running anything, and how they'd keep a single proxy from becoming the target. The difference is whether they think like a reporter of results or a governor of decisions — and only one of those scales a function without it rotting.

Bottom line

The dangerous budget fights aren't over wrong numbers — those you win with better data. They're over right numbers, where the same honest result justifies opposite calls depending on a reference point that goes to whoever frames it first. Medicine solved this with pre-registration: decide what a result means before you see it, because deciding after means deciding in favor of what you already wanted. Growth leaders should borrow the discipline directly — pre-register the decision rule, pre-agree the reference point, judge on a basket of pre-committed measures, and report in incremental dollars. Do that and your experimentation budget is governed like evidence. Skip it and it's governed by whoever argues best.

If you're building or funding an experimentation function and want it to run like an investment portfolio rather than a debate, that's the work I do — and I built GrowthLayer to make this kind of decision discipline repeatable. For more field notes on the governance of experimentation, subscribe to Lean Experiments.