Atticus Li has led analytics and experimentation work across organizations with different identity, session, conversion, and revenue implementations. This guide explains why familiar metric labels must be verified in the actual data before they are used for a decision.
Early in a new organization, I ask: "Before I run a single experiment, I need to understand your data dictionary. Not only the documentation—the implementation. How do you define a user? What counts as a session? When you say 'conversion,' what event fires and under what conditions?"
The answers are often implementation-specific. The gap between what people think the data says and what it actually says is where experiments go wrong.
There's Always Going to Be Weirdness
Most mature data dictionaries accumulate quirks as analytics implementations evolve. Requirements change, features are added, edge cases appear, and temporary workarounds can become permanent. That makes documentation and validation more important than assuming a familiar metric name has a standard implementation.
The result is a data dictionary that's technically correct but practically misleading unless you understand the context behind every definition.
In one analytics implementation, an anonymous “user” was a browser identifier. The same person on a phone and laptop could appear twice, while cookie resets or privacy settings could create another identifier.
In another implementation, authenticated identity could connect some activity across devices while anonymous traffic remained tied to separate identifiers. The exact behavior depended on the configured identity and stitching rules—not the analytics vendor name alone.
The label—“unique users”—looked the same, but the counting rules differed. Applying one implementation’s mental model to another can change the reported conversion rate even when customer behavior has not changed, because the denominator is different.
The Definitions That Trip Up Experiments
Let me walk through the specific definitions that cause the most problems in experimentation.
Sessions. GA4 defines a session as a group of user interactions within a given time frame. A session expires after 30 minutes of inactivity (by default, configurable). Adobe has a similar concept but calls it a "visit" and the timeout rules can differ based on implementation. Some companies override the default timeout. Some have custom session logic for single-page applications. If your experiment measures "conversion rate per session," you need to know exactly what triggers a new session in your specific implementation.
Pageviews. Sounds simple, right? A pageview fires when a page loads. Except in single-page applications, where navigation doesn't trigger a traditional page load. Some implementations fire virtual pageviews on route changes. Some don't. Some fire them on hash changes. Some only fire on full URL changes. I've seen SPAs where navigating from /products to /products/detail counted as one pageview in some implementations and two in others. If your experiment is on a specific page and you're measuring pageviews, you need to verify that the pageview fires when you think it does.
Conversions. This is the big one. "Conversion" means whatever someone decided it means during the implementation. At one company, a conversion fires when the user clicks the submit button. At another, it fires when the server confirms the submission was successful. At a third, it fires when the payment is processed. These are materially different events that happen at different points in the funnel, with different failure rates between them.
I've seen experiments where the "conversion rate" was inflated because the conversion event fired on button click, not on successful submission. Users who clicked submit but got a validation error were counted as conversions. The experiment looked like a winner because the variant had fewer validation errors — not because it actually drove more successful submissions.
Revenue. You'd think revenue would be unambiguous. It's not. Does revenue include tax? Shipping? Does it net out discounts and coupons? Does it account for returns and refunds? Is it recognized at the point of purchase or at the point of fulfillment? Every company answers these questions differently, and the answer changes what "revenue per user" actually means in your experiment.
What Happens When You Don't Check
Here is an illustrative composite. A team runs an experiment on “add to cart rate,” records a high-single-digit improvement, and ships. Later, finance sees no corresponding revenue movement. The team discovers that the event fired on a JavaScript state change rather than the successful API call. The interface changed the state without completing the cart action, so the apparent lift was a measurement artifact.
In a second composite, an experimentation team measures “unique users who completed enrollment” and sees a mid-single-digit movement. Its identity logic counts browser sessions independently, so cross-device users can appear more than once. The apparent difference changes after the identity and completion definitions are corrected.
These examples show why complex analytics implementations create more opportunities for definitional mismatches to corrupt a result.
The Pre-Experiment Data Audit
Before I run any experiment in a new environment, I do a data audit. Here's what that looks like.
Step one: read the implementation, not the documentation. Documentation is often outdated or aspirational. I look at the actual tracking code. What events fire? Under what conditions? What parameters are attached? Where are the edge cases?
Step two: query the raw data. I pull raw event data for the metrics I plan to use in experiments and manually verify that they match my understanding. If I think "conversion" means "successful purchase," I check whether the conversion event fires before or after payment confirmation. I check whether it fires on retries. I check whether it fires for $0 transactions.
Step three: reconcile with business data. I compare the analytics numbers to the source-of-truth business systems. If analytics says 10,000 conversions last month and the CRM says 9,200, I need to understand where the 800-conversion gap lives before I trust any experiment that uses that metric.
Step four: document the quirks. Every implementation has them. I create a living document that catalogs every known quirk in the data: events that double-fire under certain conditions, metrics that exclude certain user segments, definitions that differ from common understanding. This document becomes required reading for anyone who runs experiments.
The Cross-Platform Translation Problem
If you've worked at multiple companies — or even multiple teams within the same company — you know that metrics don't translate cleanly. A "conversion rate" at one company is not comparable to a "conversion rate" at another, even if they're in the same industry. The definitions are different. The funnel shapes are different. The tracking implementations are different.
This matters for experimentation because a lot of our intuition is built on prior experience. You might think "a 5% conversion rate lift is typical for a good test" based on your experience at a previous company. But if that company defined conversion differently, your calibration is off.
I recalibrate my intuition every time I join a new organization or start working with a new analytics platform. The first month is all about understanding the data before I try to move it.
Build the Habit
Every metric in your experimentation program should have a verified definition that the entire team understands. Not the textbook definition. The actual definition in your specific implementation.
This isn't glamorous work. Nobody gets promoted for auditing event tracking. But it's the foundation that everything else rests on. An experiment built on misunderstood metrics isn't an experiment — it's guesswork with a statistical veneer.
Understand the dictionary before you try to write with it.
_Once your metrics are solid, make sure your experiments are properly powered to detect real differences. GrowthLayer's A/B test calculator keeps planning and post-test analysis in one workflow._
FAQ
Which metric definitions should be audited first?
Start with experiment exposure, eligible users, sessions, the primary conversion, cancellations, revenue, and the identity rules that connect them.
Why can two tools report different user counts?
They may use different identifiers, session boundaries, consent states, deduplication, time zones, and late-event handling. Google's GA4 user-metric guidance documents one set of definitions.
What makes a data dictionary trustworthy?
Pair the written definition with the production event, source system, owner, exclusions, known gaps, and a reconciliation check. Adobe's documentation for unique visitors illustrates why familiar metric names still need platform-specific definitions.