The Invisible Crisis in a Flat Result
Most A/B test results that look flat are not flat. They are masking a struggle.
A major energy retailer ran an experiment on the first step of their online enrollment flow — the address entry page, where customers enter the location they want powered. The test was a backend change to address lookup logic: from a fuzzy matching algorithm that returned results in an unpredictable order, to a strict algorithm that returned results sorted numerically. On the surface, a clean technical upgrade.
The primary metric — users reaching the next page — moved less than one percentage point in either direction. Not statistically significant. Enrollment confirmations moved similarly. By the standard reporting taxonomy, this is a null result.
But buried in the secondary metrics was a number that should have stopped the team cold.
Manual address entry — the fallback option for when the lookup fails — increased by roughly 20% in the variant. Not statistically significant by the slimmest margin, just days short of the threshold. But directionally, consistently, unmistakably: something broke.
That number is not a footnote. It is the headline.
What Users Are Actually Doing When They Type an Address
When a customer arrives at an address entry page, they are not performing a database query. They are doing something far more intuitive: they are telling a system where they live, and they expect the system to recognize them.
We learn from our earliest interactions with institutions that systems are supposed to know us. When a system fails to find us, or presents results that feel random and irrelevant, the psychological response is not neutral. It is a violation of expectation.
Daniel Kahneman's two-system framework is useful here. System 1 — fast, intuitive, pattern-matching — handles the address entry task on autopilot. The user types their street, expects the right address to appear in the dropdown, and selects it.
When the dropdown does not show the right address — when the results are in an unexpected order, when apartments and unit numbers are missing, when the limited slots are occupied by irrelevant entries — System 1 cannot complete the task. The user is forced to engage System 2: slow, deliberate, effortful thinking.
That conscious decision, in this test, was increasingly: give up on the lookup and type it manually.
Why Strict Matching Made Things Worse
The team improved the algorithm's technical accuracy — from fuzzy matching with randomized results to strict matching with ordered results — and the user experience got worse.
To an engineer, strict matching means less noise, cleaner results. To a user, strict matching means fewer results when the input is imperfect. And address inputs are almost always imperfect. Users abbreviate street names. They omit directional prefixes. They are in apartments with unit numbers they may or may not include.
The fuzzy algorithm was not better because it was more sophisticated. It was better because it was more forgiving. When the strict algorithm failed to find an address in its limited result set, the list was useless.
This is a trust calibration failure. The system was technically more correct and perceptibly less useful.
The Economics of the Manual Entry Signal
When fewer than one in ten users are using manual entry in the control, that is already a measurable leakage point. Increase that pool by roughly 20% and the effects compound across several cost centers.
The first cost is abandonment. Manual address entry takes longer and requires more cognitive effort. Any additional friction in an enrollment flow increases the probability that users drop out.
The second cost is error rate. Users who manually enter addresses make more mistakes — misspelled streets, missing unit numbers, incorrect zip codes. Every error introduced at the address step has a long tail of operational cost.
The third cost is support volume. When users cannot complete address entry through the standard flow, a predictable fraction call or chat. Support contacts initiated during enrollment have a high cost per contact.
The Platform Divergence That Confirms the Story
On desktop, the strict algorithm showed modestly positive results — small, not significant, but directionally encouraging. On mobile, both metrics moved negative.
Desktop users have more screen space, a full keyboard, and more patience. When the list is ordered numerically, that order is legible at desktop resolutions. Mobile users see a compressed dropdown where the ordering advantage disappears and the strict matching limitation becomes more painful.
The architecture of the lookup problem is device-specific.
The Path Forward
Both algorithms have a meaningful limitation: they surface too few results, handle non-standard inputs poorly, and treat the address entry step as a lookup problem when it is actually a recognition problem.
First, the result set needs to expand. A handful of results is not enough for multi-unit buildings or streets with similar names. Second, the matching logic needs to tolerate imperfect input. Third, apartment and unit number entry needs a dedicated UI pattern. Finally, manual entry should not be a failure mode — it should be a seamlessly integrated alternative.
The lookup is not a technical problem. It is a trust interface.
The Lesson for Experimentation Programs
When a secondary metric moves sharply in the wrong direction while primary metrics appear flat, the secondary metric is probably telling you the truth.
The primary metric looked flat because the system absorbed the degradation. Some users who could not find their address gave up, depressing the metric slightly. Some users fell back to manual entry and completed the flow anyway, compensating slightly. The net result looked like nothing happened.
Something happened. Understanding what it was is worth more than the headline number.