An experiment review can end with everyone agreeing that a test was “inconclusive” while disagreeing completely about what happens next. One person means the variant probably did nothing. Another means the sample was too small. A third has found a tracking problem. Those are different situations, and giving them the same label makes the next decision harder.

When I review an experiment, I want the result to answer three separate questions: can we trust the comparison, what effects remain plausible, and what action does the evidence support? A single winner-or-loser field cannot carry all three answers. The seven categories below are a practical reporting framework, not an exhaustive statistical standard or a claim about how frequently teams make mistakes.

Separate the evidence from the shipping decision

The statistical result describes an estimate and its uncertainty under the chosen analysis. The business decision also depends on implementation costs, customer outcomes, reversibility, and the cost of waiting. Keep both visible.

The American Statistical Association's statement on p-values explains why statistical significance alone cannot establish an effect's size or business importance. A small improvement can be precisely measured and still fail to cover its maintenance cost. A result that does not pass a significance threshold can still leave a substantial loss entirely plausible.

Before launch, specify the primary outcome, the smallest worthwhile benefit, acceptable harm limits, guardrails, analysis method, and stopping rule. These choices establish what the result must resolve. They should not be adjusted afterward to fit the most flattering number on the dashboard.

Record absolute and relative changes separately. If conversion rises from 10% to 10.3%, that is an increase of 0.3 percentage points and a 3% relative lift. Confusing those units can distort an otherwise sensible business case.

Check the design before assigning a result

A standard A/B test compares one variant with a control. An A/B/n test compares several treatments and needs an analysis that accounts for the comparisons being made. A factorial design can estimate interactions between deliberately crossed factors; a collection of unrelated variants cannot automatically do that.

A redirect test changes where users land, making assignment, loading behavior, and cross-domain tracking particularly important. A holdout follows a reserved comparison group over time. Adaptive allocation methods, including bandits, require inference suited to their allocation and stopping behavior; ordinary fixed-sample calculations are not automatically valid.

Non-inferiority and equivalence describe different questions. Non-inferiority asks whether a loss larger than a predefined margin can be excluded. Equivalence asks whether effects outside a predefined two-sided tolerance can be excluded. Neither is established simply because a superiority test failed to find a difference.

Use the experiment design approach to decide which question matters before building the test. A system replacement may need evidence of acceptable performance plus a documented operating-cost saving, rather than proof that customers convert more often.

Seven useful result categories

Treat these as a reporting sequence. Check validity first, then material conflicts across important outcomes, then the primary effect and its uncertainty. Add secondary descriptors where needed: an experiment can be equivalent on conversion while still having an unresolved accessibility problem.

1. Meaningful improvement

The evidence meets the planned standard for improvement and supports the business threshold that justified the test. Guardrails remain within their acceptable limits, and the expected benefit is worth the implementation cost.

For a hypothetical example, suppose the minimum worthwhile relative lift is 2%, and the relevant confidence interval runs from +2.4% to +5.8%. That provides stronger support for exceeding the business threshold than an interval from +0.2% to +5.8%. Both may exclude zero, but only the first excludes effects below the stated threshold at that interval's confidence level.

The action is a controlled rollout with monitoring. Keep the measured outcome separate from an explanation of why it changed; a successful treatment does not by itself prove the proposed psychological mechanism.

2. Meaningful deterioration

The comparison provides evidence of a loss that matters under the predefined decision rule, or a critical guardrail crosses its harm threshold. A positive engagement metric does not cancel that loss.

For example, a redesign may increase interactions while reducing completed purchases. Whether those interactions are attractive or intuitive is secondary to the customer and business outcome the test was designed to protect.

The action is to retain control, investigate the cause, and decide whether a narrower follow-up is justified. Record what failed instead of converting a negative result into a general claim that the underlying idea never works.

3. Detectable movement with limited practical value

The analysis detects a change, but the evidence places it inside a range too small to justify the proposed investment. This is a different result from an imprecise estimate.

Consider a hypothetical relative lift interval of +0.1% to +0.4% when a 2% improvement is needed to cover development and operating costs. The direction is encouraging, but it does not satisfy that business case. A tiny negative effect likewise deserves a cost comparison rather than an automatic declaration of disaster.

The action depends on total value. A nearly costless simplification could still be worthwhile; an expensive new dependency may not be. Document the tradeoff without advertising the result as a meaningful conversion win.

4. Practical equivalence established

The experiment was designed and analyzed to test a predefined equivalence range, and the appropriate interval or test satisfies that criterion. The conclusion is bounded: the outcomes are equivalent within that tolerance, for the population and period studied.

This can support removing a costly component, simplifying a flow, or replacing an expensive system. The value may come from reduced operating effort while customer performance stays acceptably similar.

The action is to verify the separate cost or usability benefit and review guardrails. Do not describe the treatments as identical, and do not manufacture equivalence by choosing a wider tolerance after seeing the data.

5. Non-inferiority established

The test supports ruling out a loss larger than the predefined acceptable margin, using the planned non-inferiority analysis. Unlike equivalence, the question is one-sided: the variant could still be substantially better.

A hypothetical migration might be acceptable if conversion does not fall by more than a narrowly justified margin and infrastructure costs fall. The acceptable margin must come from business and customer consequences, not the sample size that happens to be affordable.

The action can be to ship for that independent benefit. Record “non-inferior within the agreed margin” rather than “winner.” If equivalence was also the prespecified goal and established, use that more specific description.

6. Inconclusive evidence

The result has not resolved the decision. The uncertainty may include meaningful benefit, meaningful harm, or both; alternatively, it may distinguish the direction without establishing whether the effect clears the business threshold.

This category includes the common case where the point estimate looks positive but the interval is too wide to justify the intended claim. It also includes a failed non-inferiority test: failure to establish acceptable performance is not proof of unacceptable performance.

The action is to compare the value of more information with its cost. Run a properly planned follow-up, retain control, or make a reversible product decision that explicitly acknowledges uncertainty. Do not keep extending a fixed-horizon test until it becomes significant.

7. Not decision-ready

Some results cannot yet support the proposed shipping decision. Distinguish two reasons inside this category: an invalid comparison, and a valid comparison with unresolved tradeoffs.

Invalid comparisons include broken assignment, differential event loss, or an unexplained sample ratio mismatch. Microsoft's guidance on diagnosing sample ratio mismatch shows why missing or unevenly counted users can distort the apparent effect. Diagnose the cause before interpreting the lift.

A valid but conflicted result is different: conversion might improve while refunds increase, or downstream outcomes may look better while the planned primary remains uncertain. Preserve the usable estimates and resolve the tradeoff. The flat-primary decision framework addresses that second situation.

Turn the classification into a useful decision record

A useful readout needs the result category, the primary estimate and interval, important guardrails, any data-quality concerns, and an explicit decision with an owner. Attach the hypothesis and original stopping rule so a later reviewer can distinguish planned analysis from exploration.

Then write what would change the decision. That might be a narrower uncertainty interval, evidence that refunds have matured, or a follow-up focused on mobile visibility. “More data” is too vague to guide a team; name the unresolved question.

The financial estimate deserves its own line. Use eligible traffic, the observed outcome, contribution per outcome, implementation cost, and uncertainty. A projected gain is a model, not realized revenue. This distinction helps an experimentation program prioritize useful work without turning every positive dashboard cell into a forecast.

FAQ

Does a nonsignificant result mean the variant had no effect?

No. It means the analysis did not meet that test's rejection criterion. Inspect the interval and the original design to see which benefits and harms remain plausible. Equivalence requires its own predefined criterion.

Can a test ship without being a conversion winner?

Yes. A change might meet a non-inferiority criterion while delivering verified cost savings, or solve a necessary product problem through a reversible rollout. Document the reason and the remaining uncertainty instead of changing the result label to justify the decision.

Should every experiment fit exactly one category?

Use one headline category to support communication, then retain qualifiers. Statistical evidence, validity, guardrails, and the shipping decision are distinct dimensions. The framework makes them visible; it does not replace the analysis.

Apply these categories to your next readout before deciding whether the experiment succeeded. A useful label should make the next action clearer and preserve what the data has not yet established.

Share this article
LinkedIn (opens in new tab)X / Twitter (opens in new tab)
Atticus Li

Experimentation and growth leader. CXL-certified CRO practitioner, Mindworx-certified in behavioral economics. Led 100+ in-house experiments at NRG in 2025, with project evidence and limits documented in the case studies.