The Setup: A Simple CTA, A Conservative Goal
An energy provider wanted to know whether adding a phone call CTA near the homepage zipcode entry would drive meaningful call volume. The test was designed conservatively — a non-inferiority test where the primary metric was not call volume but digital plan page views. The real question: does adding a phone option cannibalize the digital funnel?
The test ran for nearly two months across tens of thousands of sessions. Plan page views dipped by less than one percentage point — well within the pre-specified threshold. The phone CTA generated hundreds of calls and meaningful completed sales, overwhelmingly from mobile users. The test shipped.
And then someone looked at the session replays.