Atticus Li leads Applied Experimentation at NRG Energy (Fortune 150), where he runs 100+ experiments per year and generated $30M in verified revenue impact in 2025. He writes about the operational reality of building experimentation programs that survive contact with organizational politics.
The most dangerous question a VP can ask about your experimentation program is: "What percentage of your tests win?"
Not because it's a bad question. Because the answer — if you're honest — sounds like failure to anyone who doesn't understand how experimentation works. And most executives don't.
Industry win rates hover between 15% and 30%. That means 70-85% of tests produce no statistically significant positive result on the primary metric. If you present that number without context, you've just given your leadership team a reason to question whether your program is worth funding.
But here's the thing: a 15-30% win rate is exactly what a healthy experimentation program looks like.
Why Most Tests Lose (And That's Fine)
Experimentation is a search process. You're systematically testing hypotheses about what will change customer behavior. Most hypotheses are wrong. That's not a flaw in the program — that's the nature of trying to change human behavior in complex systems.
Think about it from first principles. If you could reliably predict which changes would improve conversion rates, you wouldn't need to test. You'd just implement them. The whole point of experimentation is that you don't know. You have informed guesses, but you're operating in uncertainty.
A 15% win rate means that 15% of the time, your informed guesses are correct and produce a measurable positive impact. The other 85% of the time, you learned something valuable: this particular change doesn't work in this context for these users. That information is worth money because it prevents the company from shipping changes that don't help — or actively hurt.
The cost of not testing is shipping every idea that sounds good in a meeting. And I guarantee you, if you shipped every idea without testing, your overall conversion rate would go down, not up.
Why Stakeholders Misunderstand Win Rates
The misunderstanding comes from a reasonable but incorrect mental model. Stakeholders think of experiments like projects. Projects have a success rate. If 85% of your projects failed, you'd have a serious execution problem.
But experiments aren't projects. They're questions. And in science, most questions yield null results. That's not failure — that's the process of elimination that makes the eventual discoveries valuable.
The other mental model problem is comparison to marketing channels. If your paid ads have a positive ROAS, that feels like a "win rate" close to 100%. So why can't experimentation do the same? Because paid ads operate on known mechanics — you're buying attention, not discovering new knowledge. Experimentation is research and development. The uncertainty is the whole point.
I've found that the most effective reframe for executives is this: "We're not trying to win every test. We're trying to find the 15-25% of ideas that actually move the needle and prevent the 75-85% that don't from getting shipped."
Portfolio Thinking: The Right Frame for Leadership
The way to present win rates to leadership is through portfolio thinking. Don't talk about individual test outcomes. Talk about the cumulative impact of the portfolio.
Here's how I frame it at NRG. In 2025, we ran over 100 experiments. Our win rate was 24% — above the industry average. Those winning tests generated $30M in verified revenue impact. The losing tests cost us virtually nothing beyond the testing resources, because we didn't ship them.
Now compare that to the alternative. If we'd skipped testing and shipped every idea, we would have implemented roughly 100 changes. Based on our data, about 24 would have been positive, roughly 10 would have been actively harmful, and the rest would have been neutral. The net impact of shipping everything without testing would have been a fraction of the $30M — possibly negative when you factor in the harmful changes.
The portfolio math is overwhelmingly in favor of testing, even with a low win rate. Especially with a low win rate. Because the low win rate is what protects you from shipping the bad ideas.
Present the annual revenue impact. Present the cost avoidance from tests that prevented bad changes. Present the portfolio return on investment. Win rate becomes a footnote, not the headline.
The 24% Win Rate at NRG: Above Average Because of Research, Not Luck
Our 24% win rate in 2025 was meaningfully above the industry average of 15-20%. I want to be transparent about why, because it's not because we're smarter.
It's because we invest heavily in research before testing. Every experiment at NRG starts with a research phase: customer data analysis, session recordings, qualitative feedback, competitive analysis, and cross-referencing with previous test results. By the time we write a hypothesis, we've already eliminated the obviously wrong ideas.
The research investment filters the test pipeline. We're not testing random ideas from brainstorm sessions. We're testing hypotheses that have survived scrutiny. That pre-filtering is what pushes our win rate above average.
This is why I push back hard on teams that want to "just run a quick test" without research. Yes, you'll run more tests. But your win rate will drop, your revenue per test will drop, and your program will look less efficient to leadership. Volume without quality is a trap.
The programs with the highest win rates aren't the ones with the best testers. They're the ones with the best research processes feeding the test pipeline.
Why a 100% Win Rate Should Scare You
If someone tells me their experimentation program has a 90% or 100% win rate, I know exactly what's happening. They're not testing anything risky. They're running experiments where the outcome is already known — button color changes where the treatment is obviously better, fixing broken UI elements that were clearly hurting conversion, testing changes that are so conservative they're guaranteed to win.
A high win rate means you're leaving money on the table. The biggest wins come from bold hypotheses — the ones where you genuinely don't know what will happen. Those tests have lower individual probabilities of success, but the ones that win produce outsized impact.
A portfolio with a 15% win rate that includes moonshot tests generating $5M each is worth dramatically more than a portfolio with an 80% win rate where every win generates $50K.
The math is simple. Ten tests at 15% win rate with $5M average impact: $7.5M. Ten tests at 80% win rate with $50K average impact: $400K. The "worse" program generates nearly 19x more revenue.
I intentionally keep a portion of our test pipeline reserved for high-risk, high-reward experiments. They drag the win rate down. They also generate the headline numbers that secure the program's funding.
How to Communicate This to Your Organization
Here's the talk track I've refined over years of presenting to leadership.
Start with the portfolio result. "The experimentation program generated $30M in verified revenue impact in 2025." Lead with the number that matters.
Explain the mechanism. "We tested 100+ hypotheses. About a quarter produced significant positive results. Those winners are responsible for the full $30M impact. The other 75% of tests prevented us from shipping changes that wouldn't have helped — saving development resources and protecting the customer experience."
Reframe the win rate. "Our 24% win rate is above the industry benchmark of 15-20%. It reflects our research investment — we're testing better hypotheses, not more hypotheses."
Address the implicit question. "A higher win rate wouldn't mean a better program. It would mean we're not testing bold enough ideas. The biggest wins come from the tests where we genuinely didn't know the outcome."
End with the ROI. "For every dollar invested in the experimentation program, we generated [X] in revenue impact. That's a return that very few functions in the company can demonstrate with this level of rigor."
Win rate is a process metric. Revenue impact is the outcome metric. Keep the conversation on outcomes, and the win rate takes care of itself.
---
_Want to make sure your experiments are properly sized to detect real winners? GrowthLayer's sample size calculator helps you plan tests that won't miss the wins that matter._