Reinforcement Learning for Experimentation
Using RL algorithms — multi-armed bandits, contextual bandits, and full RL — to adaptively allocate traffic across variants and maximize cumulative reward.
What Is Reinforcement Learning for Experimentation?
RL reframes experimentation as sequential decision making under uncertainty. Instead of a fixed-horizon A/B test that commits equal traffic to all variants, RL methods allocate more traffic to variants that appear to be winning while preserving exploration to avoid premature convergence. The spectrum ranges from simple multi-armed bandits (Thompson sampling, UCB) to contextual bandits (per-user feature-aware assignment) to full RL with state, action sequences, and long-horizon rewards.
Also Known As
- Data science: MAB, contextual bandit, Thompson sampling, UCB, policy gradient
- Growth: adaptive testing, traffic reallocation
- Marketing: dynamic optimization
- Engineering: sequential allocation, bandit orchestration
How It Works
You have six CTA copy variants. A Thompson-sampling policy maintains a posterior for each, samples from those posteriors, serves the variant with the highest draw, and updates after each outcome. Under suitable reward, horizon, and stationarity assumptions, adaptive allocation can reduce cumulative regret relative to uniform traffic. It does not universally outperform a fixed A/B design for causal estimation.
For contextual bandits, extend Thompson sampling to linear or tree-based models conditioning on user features. For full RL, model state-action trajectories — appropriate for multi-step onboarding or email sequences where today's action affects tomorrow's reward.
Best Practices
- Use bandits when cost of exploring losers is high and exploitation value is real-time (CTA, subject line, hero copy).
- Prefer A/B tests when you need an unbiased effect estimate — bandits optimize regret, not estimation clarity.
- Maintain exploration floors to prevent premature convergence, especially with non-stationarity.
- Log propensities to enable counterfactual evaluation later.
- Match algorithm to horizon. MAB for simple stationary choice; contextual bandits for heterogeneity; full RL for sequential action — most real problems are contextual bandits, not full RL.
Common Mistakes
- Using bandits when you actually want an effect estimate. Bandit data has correlated assignment probabilities; naive ATE estimation is biased.
- Ignoring non-stationarity. A bandit that has converged on a winner will be slow to adapt when user behavior shifts; use discounted or sliding-window variants.
- Deploying full RL where a bandit would do. Full RL is hard, data-hungry, and rarely justified by business complexity.
Industry Context
Bandits can fit high-frequency, repeatable decisions such as in-app messaging, merchandising, or creative allocation when rewards arrive quickly and the environment is stable enough. Fixed A/B designs are often clearer when the goal is an unbiased causal estimate. Neither design dominates universally.
The Behavioral Science Connection
RL formalizes the explore-exploit tradeoff. Thompson sampling and related methods make the policy explicit, but the result still depends on reward definition, horizon, priors, logging, and non-stationarity.
Key Takeaway
RL and bandits may fit problems where exploration cost is real and exploitation payoff is continuous. A fixed randomized design is often better for a clean causal estimate and a one-time ship decision. Choose from the decision objective rather than a superiority claim.