Monte Carlo Simulation

What Is Monte Carlo Simulation?

Monte Carlo simulation is the brute-force tool of quantitative analysis: when you can't solve a problem analytically, you simulate it thousands of times and count. For experimentation, Monte Carlo is how you validate power calculations, test decision rules, estimate false positive rates under peeking, and model downstream business impact with uncertainty propagation.

Also Known As

• Data science: MC simulation, stochastic simulation
• Growth: "what-if" simulation, scenario analysis
• Marketing: probabilistic forecasting
• Engineering: randomized trials for decision validation

How It Works

You propose a sequential testing rule: "ship if P(B>A) > 0.97 at any weekly check, up to 8 weeks." What is the false positive rate under this rule? Monte Carlo it: simulate 10,000 A/A experiments with your traffic pattern, apply the rule each time, count how often it falsely ships. If 6% of simulated A/A tests trigger a ship, your actual alpha is 0.06, not 0.05. Adjust the threshold upward until simulated false positives hit your target.

Similarly for power: simulate 10,000 A/B experiments with the true effect size you care about, apply your decision rule, count how often it correctly ships.

Best Practices

• Simulate your actual decision rule, not just a textbook test. Real rules include peeking, stopping criteria, and business overrides.
• Use realistic traffic patterns — daily/weekly seasonality affects sequential tests materially.
• Run enough replications (10,000+) to estimate tail probabilities reliably.
• Validate analytical power calcs against MC — they often disagree in real-world regimes.
• Propagate uncertainty end-to-end — from conversion posterior through LTV model to annual revenue simulation.

Common Mistakes

• Simulating under one DGP and deploying under another. If your sim assumes stationarity and reality has trend, your calibrated alpha is wrong.
• Too few replications. 1,000 reps to estimate a 5% rate has margin of error ~1.5pp — your "calibrated" threshold is noise.
• Ignoring correlation structures. Users aren't independent; clustering and correlated outcomes break naive MC.

Industry Context

In SaaS/B2B, Monte Carlo lets you validate test designs under realistic low-traffic conditions where analytic approximations are unreliable. In ecommerce, MC is standard for revenue uncertainty propagation, seasonal pricing scenarios, and media mix what-ifs. In lead gen, MC models pipeline conversion funnels end-to-end, showing how upstream test uncertainty translates to downstream deal-closed uncertainty.

The Behavioral Science Connection

Monte Carlo replaces intuitions about uncertainty with empirical counts. Humans are terrible at compounding probabilities — a 10% chance of each of 5 things going wrong feels like a 10% total risk, not the actual ~41%. MC forces the counting and produces answers that survive intuition checks.

Key Takeaway

When a statistical question gets complicated enough that pencil-and-paper analysis breaks, Monte Carlo simulation is the honest path forward. It is more work than citing a formula, but it is the only way to validate rules that your team actually uses.