Every experimentation platform demo shows you the same thing: a clean setup wizard, a tidy results dashboard, a statistical engine that "just works." What no demo shows you is the part that actually determines whether the migration succeeds — how the new tool silently redefines the numbers you thought you understood.
TL;DR
- The real cost of switching tools isn't the migration effort — it's the loss of comparability. A "conversion rate" in the old platform and a "conversion rate" in the new one are almost never computed the same way, so your historical baseline quietly becomes meaningless.
- Metric definitions are the landmine. How a tool counts a session, deduplicates a user, attributes a conversion, or handles a bounce differs across platforms. The dashboards look identical; the numbers underneath are not the same numbers.
- The sales demo never surfaces the limits you'll hit in month two — sample-ratio issues under real traffic, SDK load-order race conditions, segment definitions that don't map, API rate limits on your reporting pipeline.
- The move that de-risks the whole thing is boring and non-negotiable: run A/A tests on the new platform before you trust a single result, and re-derive your core metric definitions from scratch rather than assuming they carry over.
| What you think transfers | What actually transfers |
|---|---|
| Your historical conversion baseline | A number computed under different rules — not comparable |
| "Conversion rate" as a shared definition | A label that maps to different math per tool |
| Clean traffic splitting | A splitter you haven't validated under your load |
| Your segment library | Segment logic that may not have an equivalent |
The migration doesn't fail loudly. It fails as a slow erosion of trust in the numbers, discovered months later when two tools disagree and nobody can say which one is right.
The demo is a controlled environment; your traffic is not
Every platform sells on the happy path. The setup wizard works because the demo account has clean data. The stats engine looks trustworthy because you're not stress-testing it. The problem is that the failure modes of an experimentation tool only appear under your real traffic, your real page-load conditions, and your real analytics pipeline — none of which exist in the sales environment.
This is a specific instance of a general principle senior practitioners internalize: you cannot evaluate a measurement instrument by reading its brochure. You evaluate it by running known inputs through it and checking whether the outputs are what they should be. A tool that reports beautiful dashboards on data you can't independently verify is not trustworthy — it's just confident. And confidence is not the same property as accuracy.
The same instinct that makes event tracking architecture the thing that makes or breaks your data quality applies here: the platform is only as good as the definitions and instrumentation feeding it, and switching platforms resets both.
The metric-definition trap nobody warns you about
Here is the failure that catches even experienced teams. You migrate, you run a few tests, and the results look plausible. Then someone in a stakeholder review pulls the "conversion rate" from the new tool and compares it to last quarter's number from the old tool, and they don't match. Now you're in a meeting trying to explain why conversion "dropped," when nothing changed except the ruler.
The reason is that there is no universal definition of the metrics every tool claims to measure. Consider what varies across platforms for something as basic as conversion rate:
- Sessionization. How long a gap before a new session starts? 30 minutes is common but not universal, and it changes the denominator.
- User deduplication. Does a user who converts on two devices count once or twice? Cross-device identity resolution differs wildly.
- Attribution window. Does a conversion count if it happens 20 minutes after exposure? A day? A week? The window is a setting, and the default is rarely the same across tools.
- Bounce and engagement definitions. Platforms increasingly compute "engaged sessions" with proprietary rules that don't translate.
None of these are exotic. They're the plumbing of every dashboard, and they interact with the deeper reality that attribution models are already broken before you switch anything — a migration just adds a second layer of definitional drift on top. And because each tool bakes its own choices into a metric with the same name, you get the illusion of comparability with none of the substance. Kohavi and colleagues have documented at length how much the definition of the Overall Evaluation Criterion — the metric you actually optimize — determines whether an experimentation program produces trustworthy decisions (Kohavi & Longbotham, *Online Controlled Experiments and A/B Tests*). Switch tools without re-deriving those definitions and you've changed your OEC without meaning to.
There's a useful old heuristic here — Twyman's law: any figure that looks interesting or different is usually wrong. When your conversion rate jumps the quarter you switch platforms, Twyman's law says the first hypothesis isn't "the business changed," it's "the measurement changed." After a tool migration, that hypothesis is almost always correct.
The limitations the sales cycle hides
Beyond metric definitions, every platform has operational limits that never come up in procurement because the vendor doesn't volunteer them and you don't know to ask. The recurring ones:
Traffic-splitting integrity under load. The tool splits cleanly in the demo. Under your real concurrency, with your caching layer and your CDN, you may get a sample ratio mismatch — the split isn't actually 50/50, which invalidates the test. This is not hypothetical; it's one of the most common data-quality failures in live programs, and it's exactly why experimentation governance treats SRM as a first-class check.
Flicker and load-order races. Client-side tools that swap content after page load create a visible flash of the control before the variant renders. That flicker is itself a treatment — it changes user behavior — and its severity depends on your specific page weight and script order, which no demo replicates.
Segment logic that doesn't map. Your carefully built segment library in the old tool may have no clean equivalent in the new one. "Returning mobile users in the checkout funnel" might be one click in one platform and an unsupported combination in another.
Reporting API rate limits. If you pipe results into a warehouse or a BI layer, the new tool's API quotas can throttle your reporting pipeline in ways you discover only when a dashboard silently stops updating.
The de-risking move: A/A test before you trust anything
The single practice that separates a clean migration from a painful one is calibration before use. Before running a single real experiment on a new platform, run A/A tests — identical experiences split between "control" and "variant" — and confirm the tool reports no significant difference and the traffic splits as claimed.
When I stand up a new experimentation platform, the first thing I run is a batch of A/A tests — identical experiences split between "control" and "variant." On one platform, we ran two weeks of A/A tests and the tool surfaced a sample ratio mismatch immediately: the splitter was not delivering the ratio it promised under real traffic. If we'd skipped that step and gone straight to real experiments, every result afterward would have carried the same hidden defect, and we'd have shipped decisions built on a broken instrument. A/A testing is the most tedious work in experimentation and also the highest-return — it's the difference between trusting your results because you validated the tool and trusting them because the dashboard looked convincing.
For a migration specifically, the calibration checklist is:
- Re-derive every core metric definition from the new tool's documentation — don't assume "conversion rate" means what it meant before. Write down the sessionization, dedup, and attribution rules explicitly.
- Run A/A tests for at least a full business cycle to catch SRM, flicker, and tracking gaps under real conditions.
- Run one experiment in parallel on both tools during the transition, and reconcile the difference before you decommission the old one. If they disagree, understand why before you trust the new number.
- Rebuild your baseline in the new tool's terms rather than carrying the old baseline forward. Historical comparability across a tool switch is mostly an illusion; accept that and re-baseline honestly.
Do this and a tool migration becomes a controlled re-instrumentation. Skip it and you're running a program whose foundation quietly shifted, discoverable only when the numbers embarrass you in a meeting — and a program that can't trust its numbers loses the experiment velocity it was trying to protect, because every result gets re-litigated.
FAQ
Can't I just map the old metrics to the new ones and keep my history?
You can attempt a mapping, but be honest about its limits. If the two tools sessionize differently or attribute conversions over different windows, no mapping fully reconciles them — you're approximating. For directional trend continuity, a documented mapping is fine. For anything where the exact number matters, treat the switch as a re-baseline and stop pretending the pre-migration history is directly comparable.
How long should parallel-running both tools last?
Long enough to cover a full business cycle including your normal weekly and seasonal variation, and long enough to run at least one real experiment on both. The goal isn't a fixed duration; it's enough overlap that when the two tools disagree, you can diagnose the cause rather than guess. Decommissioning the old tool before you've reconciled a disagreement is how you lose the ability to ever explain the gap.
Isn't A/A testing a waste of traffic I could spend on real tests?
It feels like a waste right up until it catches a splitter defect that would have invalidated a quarter of real experiments. The traffic spent on A/A validation is insurance against every downstream result being untrustworthy. On a new or newly migrated platform, it's not optional — it's the precondition for believing anything the tool tells you afterward.
Does this mean switching tools is never worth it?
No — sometimes the new platform genuinely fits your needs better, and staying on a tool that doesn't serve your program has its own compounding cost. The point isn't to avoid migrating; it's to migrate with your eyes open to the comparability loss and the calibration work, so the switch is a deliberate re-instrumentation rather than a silent reset of your measurement foundation.
Bottom line
Switching experimentation platforms is not a data-portability problem; it's a measurement-continuity problem. The dashboards transfer, the metric definitions underneath them don't, and the operational limits that will actually constrain you never appear in the demo. The teams that migrate cleanly treat the new tool as an uncalibrated instrument: they re-derive every metric definition, run A/A tests before trusting a single result, run both tools in parallel long enough to reconcile disagreements, and re-baseline honestly instead of pretending the old history still means something. Do that and the switch is controlled. Skip it and you'll spend the next two quarters explaining why a number changed when nothing about the business did.
Getting metric definitions and calibration right is exactly the kind of process discipline I built GrowthLayer to standardize. If you want more on the operational reality of running experimentation programs — the parts vendors don't put in the brochure — subscribe to Lean Experiments.