Randomization Unit

What Is a Randomization Unit?

The randomization unit is the entity — user, session, pageview, device, or account — that is randomly assigned to a test variant. It is a foundational design choice that determines what counts as an independent observation, shapes your sample size calculation, and influences the validity of your statistical tests. Choose wrong and every downstream analysis is compromised, no matter how careful the rest of the experiment is.

Also Known As

• Marketing teams call it assignment unit, bucket unit, or test unit.
• Growth teams say randomization unit, assignment level, or bucket level.
• Product teams use randomization unit or assignment granularity.
• Engineering teams refer to bucket unit, assignment key, or stickiness level.
• Statisticians strictly call it the unit of randomization or experimental unit.

How It Works

You're testing a new checkout flow. You have three reasonable units: user (use their user ID), session (randomize per session), or device (randomize per browser cookie). You pick user-level so returning customers see a consistent experience. Across the test, user 12345 always sees variant B regardless of device or session. User 67890 always sees control. Your sample size calculation uses unique users (50,000 per arm), not pageviews (which might be 200,000 per arm). Your statistical test treats each user as one independent observation.

Best Practices

• Default to user-level randomization for logged-in experiences.
• Use device-level (cookie) for logged-out experiences.
• Choose session-level only for truly session-independent experiences (like ad creative tests).
• Match your analysis unit to your randomization unit — if you randomize at user level, analyze at user level.
• Document the randomization key (user ID, cookie, device ID) and how it's derived.

Common Mistakes

• Randomizing at user level but analyzing pageview counts, inflating effective sample size.
• Using session randomization for logged-in workflows, creating inconsistent experiences across sessions.
• Not accounting for cookie churn on Safari (ITP) and ignoring users who bounce between variants.

Industry Context

• SaaS/B2B: Account-level randomization often matters — users within an account share context.
• Ecommerce/DTC: Device-level is common for logged-out browsing; user-level for logged-in checkout.
• Lead gen: Session-level is common since users rarely return pre-conversion.

The Behavioral Science Connection

User-level randomization preserves consistency — a core principle of behavioral design. If a user sees variant B on Monday and variant A on Wednesday, the inconsistency itself biases their experience. Cognitive dissonance shifts their behavior in unpredictable ways. User-level randomization ensures you're measuring the variant, not the inconsistency.

Key Takeaway

Default to user-level randomization, match analysis unit to randomization unit, and document the assignment key — everything else in the test depends on getting this right.