Mobile and desktop are different products. CTA tests that ignore device segmentation make ship/revert decisions on aggregates that hide opposite-direction effects on each device class.
When I review a cross-device test, I require the device analysis and decision rule to be specified before results are visible. That protects a useful diagnostic from becoming post-hoc metric shopping.
TL;DR
- The same CTA change can produce opposite-direction results on mobile vs desktop. Aggregates blend the two, often producing a noisy "directional" result that doesn't actually exist on either device.
- Device effects can differ when viewport, input, or interaction mechanics differ. That mechanism does not support a universal device-level win-rate benchmark.
- Some change types have plausible device-specific mechanisms: sticky positioning, hero size, modal-mediated routing, and form-field changes. Pre-specify device analysis when that mechanism matters.
- The decision matrix below maps eight aggregate × segmented combinations to the right ship/revert action — including the cases where the aggregate is misleading.
Why aggregates lie
Aggregate results combine device classes proportional to their traffic share. If mobile is 70% of traffic, the aggregate weights mobile more heavily — even when the mobile-specific result is wrong for desktop users.
The figures below are synthetic arithmetic examples, not experiment results.
| Mobile result | Desktop result | Aggregate (70% mobile, 30% desktop) | What aggregate tells you |
|---|---|---|---|
| +5% | +5% | +5% | Both devices win |
| +5% | -3% | +2.6% | "Directional positive" — but desktop is hurt |
| -2% | +8% | +1% | "Directional positive" — but mobile is hurt |
| -10% | +20% | +1% | "Directional positive" — but mobile regression is severe |
The last row is the warning sign. Aggregate ~+1% reads as a noisy directional win. Segmented by device, mobile is regressing 10% while desktop gains 20%. The right call is "ship on desktop, revert on mobile" — invisible to the aggregate.
Pattern observation: aggregate results can mask heterogeneity
A weighted aggregate can conceal a meaningful difference between device experiences. That is a reason to pre-specify device analysis, not evidence that segmentation itself causes more wins.
When device asymmetry is most likely
Some CTA changes are device-agnostic; others have device-specific mechanisms. Anticipate which is which before launching the test.
| CTA change type | Asymmetry likelihood | Mechanism |
|---|---|---|
| Sticky positioning | High | Mobile viewport real estate is tighter; sticky impact differs |
| Hero size / above-fold restructure | High | Mobile viewport changes what's "above the fold"; desktop has more horizontal space |
| Modal-mediated routing | High | Mobile modal UX is more disruptive than desktop |
| Form field reduction | High | Mobile typing friction higher; desktop users tolerate more fields |
| CTA copy change | Low | Copy semantics travel across devices |
| Visual hierarchy / color | Low | Same visual logic on both |
| Button placement (within-section) | Medium | Depends on whether the section's layout differs by device |
For high-asymmetry change types, segment by device before reading the aggregate.
The asymmetry signatures
Three patterns recur across device-segmented CTA tests:
| Pattern | Mobile | Desktop | Action |
|---|---|---|---|
| Mobile-friendly only | Positive | Flat or slight negative | Ship on mobile only |
| Desktop-friendly only | Flat or slight negative | Positive | Ship on desktop only |
| Mobile-hostile | Strong negative | Positive | Revert on mobile (even if aggregate says ship) |
| Desktop-hostile | Positive | Strong negative | Revert on desktop |
| Universal | Same direction | Same direction | Ship/revert sitewide |
The first four signatures may support device-conditional shipping if the segment analysis was pre-specified, adequately powered, and operationally feasible. The implementation cost and risk vary by product.
Decision matrix: device-segmented shipping
| Aggregate | Mobile segment | Desktop segment | Decision |
|---|---|---|---|
| Positive | Positive | Positive | Ship sitewide |
| Positive | Positive | Flat | Ship on mobile; hold on desktop |
| Positive | Flat | Positive | Ship on desktop; hold on mobile |
| Positive | Negative | Positive | Ship on desktop, revert on mobile — aggregate is misleading |
| Flat | Positive | Negative | Ship on mobile, revert on desktop |
| Flat | Negative | Positive | Ship on desktop, revert on mobile |
| Negative | Negative | Negative | Revert sitewide |
| Negative | Positive | Negative | Ship on mobile, revert on desktop — aggregate hides mobile win |
The matrix is a planning aid, not an automatic decision rule. Segment uncertainty, multiplicity, guardrails, and the precommitted ship policy still determine what the evidence supports.
Illustrative composite: a hierarchy iteration with device asymmetry
This composite shows how a positive aggregate can coexist with different device estimates. It is not an attributable client readout or general benchmark.
| Funnel metric | All devices | Desktop | Mobile |
|---|---|---|---|
| Page-entry rate | Low-single-digit positive | Mid-single-digit gain | Roughly flat |
| Mid-funnel completion | Mid-single-digit positive | High-single-digit gain | Mid-single-digit gain |
| Downstream conversion | Low-double-digit positive | Stronger positive | Low-single-digit positive |
The desktop segment carried most of the lift. Mobile was directionally positive on mid-funnel but negative on the upstream metric — a signature of a layout change that worked better on the desktop viewport. Decision: ship the change but plan a mobile-specific iteration to recover the upstream metric on mobile.
One plausible mobile cause is leading with an input before the value proposition, while the wider desktop viewport shows both at once. That remains a hypothesis for a follow-up test, not an isolated causal finding.
Pre-test instrumentation requirements
For high-asymmetry change types, the test needs to be set up to read by device from day one:
| Requirement | Why |
|---|---|
| Device class as a primary segmentation dimension | Standard segment, not custom-cut at analysis time |
| Per-device sample size targets | Mobile and desktop power separately; total may be powered while segments are not |
| Per-device MDE acceptance | Often need larger MDE on the smaller segment |
| Pre-committed device-conditional shipping plan | Decide before launch whether asymmetric results would ship on one device only |
Without these, a test producing strong asymmetry will be hard to interpret and harder to ship correctly.
When to NOT segment by device
A few contexts where aggregate reading is sufficient:
| Context | Why aggregate is OK |
|---|---|
| Truly device-agnostic change (copy, color, semantic) | Mechanism doesn't differ by viewport |
| Single-device test (mobile-only or desktop-only) | One segment, no asymmetry possible |
| Test with sample size only powered for the aggregate | Segment-level reads will be noise |
For other CTA tests, decide before launch whether device heterogeneity is important enough to power and report.
Bottom line
Mobile and desktop experiences can produce different effects from the same change. A single aggregate may hide that heterogeneity, but the mechanism does not establish a universal device-level win-rate advantage.
Pre-specify device analysis when the mechanism differs, size the segments, and define the conditional rollout policy before results arrive. Treat post-hoc device cuts as exploratory rather than automatic ship/revert instructions.
FAQ
Should every A/B test report mobile and desktop separately?
Use a pre-specified device view when the mechanism or rollout decision can differ by device. Avoid turning every post-hoc segment into a new success criterion.
What if the aggregate and device estimates point in different directions?
Check allocation, sample size, effect intervals, and interaction evidence. Then apply the decision rule written before results were available.
Does device asymmetry justify a split rollout?
Only when the segmented evidence and operational controls support that action. Microsoft Research discusses segment and metric interpretation pitfalls, while NIST provides general experimental-design guidance. Contact me if you need a pre-specified device policy for a priority funnel.