Traditional A/B testing has an uncomfortable feature: while you wait for statistical significance, you're deliberately showing half your audience a potentially inferior experience. If Variant B is clearly better after day two, you still run the test for two more weeks because the protocol demands it. The revenue you lose by showing the inferior variant during this period is called "regret" — and bandit algorithms exist to minimize it.
But bandits aren't a free upgrade over A/B tests. They solve a different problem, make different tradeoffs, and fail in predictable ways when misapplied. Understanding when each approach is appropriate requires understanding the fundamental tension they navigate: exploration versus exploitation.
The Multi-Armed Bandit Problem
The name comes from a thought experiment about a gambler facing a row of slot machines ("one-armed bandits"). Each machine has an unknown payout rate. The gambler has a fixed number of pulls. The question: how should they allocate their pulls across machines to maximize total winnings?
If the gambler pulls each machine equally (pure exploration), they learn the payout rates but waste many pulls on bad machines. If they pull only the first machine that pays out (pure exploitation), they might miss better machines. The optimal strategy balances both: explore enough to identify the best machine, then exploit it.
This maps directly to online experimentation. Your "machines" are page variants. Your "pulls" are visitor sessions. And your "payouts" are conversions. A bandit algorithm dynamically adjusts how much traffic each variant receives based on accumulated performance data.
How Bandit Algorithms Work
While specific implementations vary, the core mechanism is consistent across bandit algorithms:
1. Start with equal allocation. Initially, traffic is split evenly across all variants, just like an A/B test. This provides the initial data needed to estimate each variant's performance.
2. Update estimates. As conversion data accumulates, the algorithm updates its estimate of each variant's true conversion rate. Better-performing variants receive higher estimated values.
3. Shift traffic. The algorithm gradually sends more traffic to higher-performing variants and less to lower-performing ones. The degree of shift depends on how confident the algorithm is in the performance estimates.
4. Maintain exploration. Even as it favors the apparent winner, the algorithm continues sending some traffic to other variants. This ensures it can detect if a previously underperforming variant improves (perhaps due to changing user behavior) or if the early performance estimates were noisy.
The most common implementations include:
Epsilon-greedy. The simplest approach. Send (1-epsilon) of traffic to the current best variant and distribute epsilon equally among all others. With epsilon = 0.1, the apparent winner gets 90% of traffic and each alternative gets an equal share of the remaining 10%. Simple to implement, but the exploration rate is fixed regardless of how confident you are.
Thompson Sampling. Uses Bayesian probability to model uncertainty about each variant's true conversion rate. Variants with high uncertainty get explored more because they might be better than they appear. As data accumulates and uncertainty shrinks, exploitation naturally increases. This is considered one of the most theoretically elegant and practically effective approaches.
Upper Confidence Bound (UCB). Selects the variant with the highest upper confidence bound on its estimated conversion rate. This naturally favors both high-performing variants (exploitation) and uncertain variants (exploration), since uncertain variants have wider confidence intervals and therefore higher upper bounds.
Exploration vs. Exploitation: The Core Tradeoff
The exploration-exploitation tradeoff is one of the most fundamental concepts in decision theory, extending far beyond web experimentation. It appears in venture capital (invest in proven businesses vs. explore startups), hiring (promote known performers vs. give opportunities to unknowns), and product development (iterate on what works vs. try new approaches).
In experimentation, the tradeoff manifests as:
Exploration — showing variants to gather information about their true performance. Every visitor who sees an underperforming variant is a cost. But without this cost, you can't learn whether another variant might be better.
Exploitation — showing the current best-performing variant to maximize immediate returns. Pure exploitation means you never discover if a different variant would have been better over time.
Traditional A/B tests are pure exploration followed by pure exploitation. You explore (split traffic equally) until you reach a predetermined sample size, then exploit (ship the winner and send 100% of traffic to it). Bandit algorithms blend exploration and exploitation continuously, which reduces regret during the testing period but comes with its own costs.
Minimizing Regret
Regret, in the bandit framework, is the difference between the reward you actually received and the reward you would have received if you'd always shown the best variant. It's the cumulative cost of learning.
In an A/B test, regret accumulates linearly during the test because you're sending 50% of traffic to the worse variant the entire time. In a well-tuned bandit, regret accumulates logarithmically — the algorithm quickly learns which variant is better and shifts traffic accordingly, so less and less traffic goes to the inferior variant over time.
This regret reduction is the primary economic argument for bandits. If your test runs for four weeks and the winning variant converts 10% better, a bandit might shift 80% of traffic to the winner by end of week one — capturing most of the upside during weeks two through four that an A/B test would have missed.
However, regret minimization comes at a cost: statistical certainty. Because traffic allocation is unequal and changing, the statistical analysis becomes more complex. You can't simply compare conversion rates between groups the way you can with a fixed-allocation A/B test. The unequal allocation introduces bias in naive comparisons.
When Bandit Algorithms Work Best
Bandits shine in specific contexts where the traditional A/B testing model is suboptimal:
Short-lived content. Headline testing on a news site, promotional banners during a flash sale, or email subject lines for a one-time campaign. These decisions have a short time horizon — there's no future exploitation period to justify the exploration cost of a full A/B test. A bandit can identify and shift to the better option within hours.
High-cost exploration. When showing an inferior variant has immediate, significant costs — like in e-commerce where every lost conversion is lost revenue — bandits reduce the total cost of experimentation by minimizing the traffic exposed to underperforming variants.
Many-variant scenarios. When testing 10 or 20 headline variations, an A/B/n test would require enormous traffic to give each variant a fair evaluation. A bandit can quickly identify the top 2-3 performers and concentrate traffic there, effectively screening many options without the full traffic cost of a traditional test.
Continuous optimization. When you don't need a one-time decision ("which version do we ship?") but rather ongoing optimization ("which headline should we show right now?"), bandits provide a natural framework for continuously adapting to changing user preferences.
When Bandit Algorithms Fail
Bandits are poorly suited for several common experimentation scenarios:
When you need statistical proof. If stakeholders need a definitive answer about whether Variant B is better than the control — with quantified confidence and effect size — an A/B test provides cleaner evidence. Bandit algorithms optimize for outcomes, not knowledge. They can tell you which variant performed best, but the statistical interpretation of "how much better" is muddied by the dynamic allocation.
When the effect is small. Bandits struggle to distinguish between variants with similar performance. A 1% difference between variants might cause a bandit to oscillate between them without ever converging on a clear winner. A/B tests, with their fixed allocation and predetermined sample size, are better designed to detect small but meaningful differences.
When external factors change. Bandits assume that each variant's performance is stationary — it doesn't change over time. But web traffic isn't stationary. A variant that performs well on weekdays might underperform on weekends. A bandit trained on weekday data would over-allocate traffic to that variant on weekends, making suboptimal decisions. This is called the "non-stationary bandit" problem, and while solutions exist, they add significant complexity.
When you need to learn why. A/B tests, combined with good hypothesis documentation, build organizational knowledge about what works and why. Bandits optimize outcomes without generating insight. If your goal is to build a learning organization, the analytical discipline of A/B testing is more valuable than the regret minimization of bandits.
Bandits vs. A/B Tests: The Key Differences
The choice between bandits and A/B tests comes down to what you're optimizing for:
A/B tests optimize for knowledge. They produce clean statistical evidence about which variant is better and by how much. The cost is the regret accumulated during the fixed-allocation testing period.
Bandits optimize for outcomes. They minimize the total regret over the testing period by dynamically favoring better-performing variants. The cost is reduced statistical clarity about the true difference between variants.
In economic terms, an A/B test invests in information (exploration) upfront with the expectation that this information will compound through better decisions over the long exploitation period. A bandit reduces the upfront investment but provides less information for future decision-making.
If the decision you're making has a long future impact (a permanent page redesign), the A/B test's information investment pays dividends for years. If the decision has a short lifespan (today's homepage headline), the bandit's regret minimization is the better economic choice.
Practical Considerations
Don't use bandits as a cure for peeking. Some teams adopt bandit algorithms because they can't resist checking A/B test results early. This is treating a symptom, not a cause. The discipline to follow a predetermined test protocol is essential regardless of methodology. Bandits don't eliminate the need for statistical rigor — they change how it's applied.
Watch for the delay problem. Bandit algorithms assume they see conversions immediately. In reality, many conversions happen hours or days after the initial visit. A bandit that allocates traffic based on immediate conversions will be biased toward variants that convert quickly, even if a slower-converting variant produces higher total conversions over time.
Segment effects get hidden. Because bandits concentrate traffic on the overall winner, they can mask important segment-level differences. Variant A might be best overall but worse for mobile users. A standard A/B test with equal allocation makes segment analysis straightforward. With a bandit, the mobile segment of Variant B might have insufficient data for reliable analysis.
The Bottom Line
Bandit algorithms are a powerful tool for a specific class of problems: short-lived decisions, many variants, and situations where the cost of showing inferior variants is high. They minimize regret during the testing period by dynamically allocating traffic toward better-performing variants.
But for most experimentation programs, A/B tests remain the better default. They provide cleaner statistical evidence, support segment analysis, generate organizational learning, and work reliably with delayed conversions. The regret they accumulate during the testing period is the price of knowledge — and for decisions with long-term impact, that knowledge is worth the investment.
The most sophisticated teams use both. A/B tests for strategic decisions that shape the product for months or years. Bandits for tactical optimizations that play out over days or weeks. Understanding the tradeoff between exploration and exploitation — and choosing the right tool for each decision — is what separates mature experimentation programs from those that simply run tests.