Start with revenue per session and bet sizing, not conversion rate vibes
Revenue per session (RPS) is total revenue divided by total sessions. It bakes conversion rate and order value into a single number, making it more reliable than conversion rate alone.
Why this matters: a "Free shipping" message might boost conversions but attract lower-intent buyers, dragging down average order value. Conversion rate goes up, but RPS exposes the real story.
Before you commit traffic to a test, anchor on three baselines:
- Sitewide RPS
- Page or funnel-step RPS
- Segment RPS (new vs. returning, paid vs. organic, geo, device)
These baselines help you see when something other than your variant is moving the numbers. For example, if paid spend shifts mid-test, RPS can move even if your variant did nothing—classic attribution noise.
Also watch for local wins that lose globally. A checkout improvement might lift RPS on that step but increase refunds or support costs downstream. RPS is powerful, but it still needs context.
The bet sizing math I actually use (and why it works)
Think of each experiment as a bet. You size that bet in dollars using expected incremental revenue, then cap it by downside risk.
The core steps:
- Pick exposure (sessions exposed)
- Estimate expected RPS change (ΔRPS)
- Compute expected value in dollars
- Apply a confidence factor (0 to 1)
- Cap by downside risk using Kelly-style thinking
The working formula:
Bet = sessions_exposed × expected ΔRPS × confidence factor, capped by worst-case downside.
Example scenarios
1. Low-risk copy tweak
- Sessions exposed: 80,000
- Expected ΔRPS: $0.05
- Confidence: 0.7
Expected value (before downside cap):
- 80,000 × $0.05 × 0.7 = $2,800
2. Medium checkout friction removal
- Sessions exposed: 120,000
- Expected ΔRPS: $0.12
- Confidence: 0.5
Expected value:
- 120,000 × $0.12 × 0.5 = $7,200
3. High-risk paywall redesign
- Sessions exposed: 200,000
- Expected ΔRPS: $0.20
- Confidence: 0.25
Expected value:
- 200,000 × $0.20 × 0.25 = $10,000
Higher confidence usually comes from:
- Prior test history in similar areas
- Strong behavioral rationale
- Clean instrumentation
- Easy reversibility or kill switches
You then cap the bet based on worst-case downside: how much RPS you could realistically lose, for how long, before you detect and stop it. This is where Kelly-style thinking keeps you from over-betting on fragile ideas.
Where RPS bet sizing breaks, and how to handle it
RPS is not always the right primary metric. Distrust RPS when:
- Revenue is delayed (trials, sales-assisted deals, invoices)
- Refunds and chargebacks are meaningful
- The experiment causes equity denial (attracting bargain hunters who hurt long-term value)
- Seasonality creates large week-to-week swings
In these cases, lean on secondary metrics that better match your economics:
- Contribution margin per session
- Qualified pipeline per session
- Activated users per session
Use RPS as a supporting view, not the single source of truth.
Behavioral science and the confidence factor
Your confidence factor should be grounded in mechanisms, not vibes.
Mechanisms that usually justify higher confidence:
- Reducing hidden costs (loss aversion)
- Making default paths safer (default bias)
- Removing steps and uncertainty (friction reduction)
These have strong empirical backing and clear causal stories.
On the other hand, vague directions like "make it more modern" deserve lower confidence, even if the design looks great. Aesthetic appeal without a behavioral mechanism is a weak bet.
AI's role in this framework
AI is useful as an accelerator, not a decision-maker.
Good uses:
- Clustering session replays to spot patterns
- Surfacing themes in support tickets
- Forecasting RPS variance and seasonality
But AI should inform your confidence factor and downside estimates, not approve risky changes on its own.
Short actionable takeaway
Use this as your minimum bar before running a test:
Bet = sessions_exposed × expected ΔRPS × confidence factor, capped by worst-case downside.
If you can’t fill in these numbers without hand-waving, the test is not ready.
Conclusion
RPS gives you a common language for pricing risk in plain dollars. Bet sizing then connects that language to traffic allocation.
- Anchor on RPS baselines before you ship
- Size bets with expected ΔRPS × sessions × confidence
- Cap every bet with a downside you can defend
Your goal is not to chase conversion-rate highs. It’s to place disciplined, repeatable bets where traffic allocation matches expected value—not internal excitement.