Bootstrap Method
A resampling technique that approximates a sampling distribution by repeatedly drawing observed units with replacement while preserving the design structure.
What Is the Bootstrap Method?
The nonparametric bootstrap approximates a sampling distribution by resampling observed units with replacement and recalculating a statistic. It assumes the empirical sample represents the target population and must preserve clustering, time dependence, or other design structure.
Also Known As
- Data science teams: bootstrap, resampling, nonparametric bootstrap
- Growth teams: empirical confidence intervals
- Marketing teams: "resampling to get a range"
- Engineering teams: BCa bootstrap, percentile bootstrap
How It Works
Imagine an A/B test measuring skewed revenue per visitor. Resample users within each variant, compute the difference in means, and repeat enough times for the desired numerical precision. The resulting bootstrap distribution can support an interval, but it remains sensitive to sample representativeness, outliers, estimator choice, and interval method.
Best Practices
- Do consider bootstrap for statistics whose sampling distributions are difficult to derive, then validate the chosen interval method.
- Do choose the number of resamples from the desired numerical precision.
- Do apply BCa (bias-corrected and accelerated) bootstrap when data is skewed.
- Do not use bootstrap mechanically on very small samples; the empirical distribution may poorly represent the target population.
- Do not treat bootstrap as a fix for fundamentally biased samples; it only captures sampling uncertainty.
Common Mistakes
- Using too few resamples (< 1,000), producing noisy intervals.
- Bootstrapping dependent data (time series, clustered users) without blocking.
- Reporting bootstrap intervals without mentioning they were produced by resampling.
Industry Context
- SaaS/B2B: Bootstrap handles MRR-per-account metrics where parametric assumptions fail.
- Ecommerce/DTC: Revenue-per-visitor is the canonical bootstrap use case.
- Lead gen/services: Pipeline-weighted metrics often require bootstrap for honest intervals.
The Behavioral Science Connection
Bootstrap is intuitive because it mirrors the way humans think about uncertainty: "what if I ran this again?" Kahneman notes people are better at reasoning with concrete simulation than with abstract probability; bootstrap is concrete simulation turned into a statistical method.
Key Takeaway
Bootstrap is the Swiss Army knife for uncertainty estimation when your data does not fit neat parametric molds.