The Problem Nobody Talks About
Here is a real result set from a CRO program:
Winner. Winner. Winner. Loser. Loser. Winner. Inconclusive. Loser. Inconclusive. Loser. Loser.
Eleven tests. Three labels. And almost certainly, at least three of those classifications are wrong.
The issue is not the math. The issue is that most experimentation teams are collapsing three distinct concepts into a single label: the statistical outcome (what the data actually says), the decision framework (what you do about it), and the business impact (whether it matters to the P&L). When you flatten these layers, you get confusion like "it's flat but significant" or "it's not significant but looks positive" — phrases that should not exist in a well-run program but show up in nearly every experiment review I have seen.
This is not a statistics problem. It is a decision architecture problem. And it has direct consequences for EBITDA.
Every Test Design You Will Actually Use
Before we talk about results, we need to talk about what you are running. Each test design produces data differently, and conflating designs leads to misinterpretation.
Standard A/B test. One control, one variant. The workhorse. You are testing whether a single change produces a measurable difference on a primary metric. This is what most teams run 80% of the time.
A/B/n (multi-treatment) test. One control, multiple variants. You are comparing several ideas simultaneously. The statistical burden is higher because you need to account for multiple comparisons — otherwise you inflate your false positive rate. Most tools handle this with corrections, but teams often forget that more variants means you need more traffic or more time.
Multivariate test (MVT). You are testing combinations of element changes — for example, three headlines crossed with two button colors gives you six combinations. This is not the same as A/B/n. MVT isolates interaction effects between elements. It requires substantially more traffic and is only worth running when you have reason to believe elements interact with each other.
Split-URL (redirect) test. Entirely different pages served at different URLs. Useful for testing fundamentally different page architectures, not just element-level changes. The measurement is the same, but the implementation and the scope of what you learn are different.
Non-inferiority test. The goal is not to prove the variant is better. The goal is to prove it is not meaningfully worse. You set a margin — say, negative 2% — and test whether the lower bound of your confidence interval stays above that margin. This is the most underused test design in CRO.
Equivalence test. Stricter than non-inferiority. You are proving the variant performs within a two-sided tolerance band — not just "not worse" but "practically the same in both directions." This matters when you are replacing systems, redesigning for maintainability, or simplifying UI while preserving performance.
Holdout test. You withhold a feature or change from a percentage of users for an extended period to measure cumulative or program-level impact. This is how you answer "is our experimentation program actually moving the needle?" rather than "did this one test win?"
Multi-armed bandit. A dynamic allocation mechanism that shifts traffic toward better-performing variants during the test. Bandits optimize for reward during the experiment but produce weaker causal inference than fixed-horizon tests. They are excellent for short-lived promotions or personalization. They are not a replacement for rigorous A/B testing when you need clean learning.
Each of these designs can produce the same set of result categories. The design determines how you collect evidence. The result taxonomy determines what that evidence means.
The Seven Result Categories That Actually Exist
Most teams use three labels: Winner, Loser, Inconclusive. That is like a doctor using only "healthy," "sick," and "not sure." It misses the nuance that drives correct treatment — or in our case, correct business decisions.
Here is the complete taxonomy.
1. Winner
The variant beats control on the pre-registered primary metric under whatever decision framework you chose before launch, and the effect is large enough to matter economically or strategically.
In frequentist terms: the confidence interval excludes zero in the positive direction, and the observed lift exceeds your minimum meaningful effect threshold. In Bayesian terms: your pre-registered shipping criteria were met — not merely that the dashboard shows a flattering probability number.
This is the only category that justifies shipping with confidence.
Example: +3.5% lift in enrollment starts. Confidence interval: +1.2% to +5.8%. Your minimum meaningful lift was 2%. This is a clear Winner — real signal, meaningful impact.
The behavioral science note: Winners often work because they align with how the brain actually processes decisions. A checkout flow that reduced steps from five to three did not win because "fewer steps." It won because it reduced the cognitive load at the point of highest decision fatigue — the same principle behind why supermarkets put impulse items at checkout rather than aisle seven. The mechanism matters because it tells you whether the win will replicate in adjacent contexts.