The Test That Defies a Clean Label
A major energy provider tested clearer copy for their satisfaction guarantee on the plan comparison page. The guarantee allowed customers to change their plan within a set period without a fee. The existing copy mentioned this, but the team worried it was not clear enough — potentially causing hesitation in plan selection.
The result was one of the most common and most debated patterns in experimentation: the primary metric was essentially flat (fractional negative movement, not significant), but downstream metrics moved positively. Date selection improved by a low single-digit percentage, and enrollment confirmations improved by a slightly higher margin. Neither was statistically significant individually. But the direction was consistent, and behavioral analytics told a supporting story: higher attractiveness scores, lower time to first interaction, and higher plan engagement in the variant.
The team shipped it to 100% of traffic. Was that the right call?
Why the Primary Metric Was Flat
The guarantee is a risk-reduction mechanism. It does not make the plans more attractive at first glance — it makes the decision to commit less frightening. That distinction explains why the primary metric (initial plan engagement) did not move while downstream metrics (where commitment actually happens) improved.
Loss aversion — the principle that potential losses feel roughly twice as painful as equivalent gains — is most active at the moment of commitment, not the moment of exploration. Users browsing plans are not yet in a loss-averse state. Users about to select a plan and provide personal information are.
The guarantee reduced perceived risk at the commitment point. It did not change the exploration behavior that feeds the primary metric.
The Visibility Problem
Desktop showed notably larger downstream improvements than mobile — nearly reaching statistical significance on secondary metrics. Mobile showed minimal effect.
This is not random noise. It is a visibility signal. On desktop, the guarantee element was prominent and hard to miss. On mobile, it was easy to scroll past without reading.
A guarantee only works if people see it. The desktop/mobile split is direct evidence that the mechanism is real but suppressed on mobile by low visibility. This is the single biggest iteration opportunity: make the guarantee impossible to miss on mobile.
The Business Economics of Unclear Guarantees
Every unclear guarantee has a cost. Users who do not understand the risk-reduction policy make one of two decisions: they either proceed with lingering anxiety (which increases downstream drop-off) or they do not proceed at all.
Both outcomes cost the business money. The first increases support contacts and buyer's remorse. The second is lost revenue.
The guarantee is not a marketing message. It is an economic instrument that reduces the risk premium users mentally assign to the enrollment decision.
The Decision Framework: When to Ship a Flat Primary
Shipping a test with a flat primary and positive downstream is defensible when five conditions are met:
- The primary metric's flat result is directionally consistent (not meaningfully negative)
- Downstream metrics move in a coherent positive direction across multiple signals
- Behavioral analytics support the hypothesized mechanism
- The device segmentation explains the magnitude rather than contradicting the direction