Feature flags: rollout control vs. true experimentation
A feature flag is a switch in code that determines user visibility. The core mistake is using one switch for beta access, canary releases, A/B testing, and rollbacks—these needs overlap but require different decision-making approaches.
Match your use case to the appropriate strategy:
- Gradual rollout — Learning goal: stability validation. Key risk: outages, latency issues. Best practice: slow ramp with error monitoring and kill switch.
- A/B testing — Learning goal: conversion improvement. Key risk: biased samples, false positives. Best practice: random assignment, preset metrics, adequate sample size.
- Holdout — Learning goal: long-term impact. Key risk: short-term gains masking losses. Best practice: maintain a control group for extended periods.
The fundamental principle: rollouts protect reliability while experiments protect decision making.