Build vs Buy (Experimentation Platform)

What Is Build vs Buy for Experimentation?

Build vs buy is the classic platform decision applied to experimentation: do you build an in-house A/B testing and feature flag system, or buy commercial software / adopt open source? Most teams should buy. A minority with real scale and specialized needs — Airbnb, Uber, Netflix, Meta — have historically built, and shared their architectures publicly.

Also Known As

• In-house vs commercial experimentation platform
• Rolling your own A/B testing
• Platform engineering for experimentation

How It Works

A team evaluates: how many tests per year? What's the headcount to build and maintain? What's the opportunity cost? A commercial platform might cost $X per year plus 0.5 FTE to run. An in-house build might cost 2-4 FTE indefinitely plus ongoing feature gaps. For most orgs below a certain scale, the math doesn't work for building.

Best Practices

• Default to buying unless you have strong evidence of an unmet specialized need.
• If you build, treat it like a product: dedicated team, roadmap, on-call rotation. Half-built experimentation systems are worse than nothing because teams lose trust in results.
• If you buy, don't customize heavily — you'll recreate the build problem on top of a vendor contract.
• Revisit the decision every 18-24 months; both cost curves and vendor capabilities shift.

Common Mistakes

• Underestimating the maintenance cost of an in-house system (SDKs, bucketing, stats engine, UI, alerting — it's real software).
• Assuming you'll save money by building — you rarely do until well into enterprise scale.
• Building a thin wrapper over an open-source library and calling it "our platform" — you've taken on the maintenance burden without the differentiation.

Industry Context

Mega-scale consumer tech (FAANG, rideshare, streaming) have historically built. Mid-market and enterprise B2B overwhelmingly buy. Open-source options like GrowthBook have shifted the math by offering self-hosted flexibility without full build cost, which is a middle path.

The Behavioral Science Connection

Engineers suffer from the "we can build it" bias — overestimating internal capability because the prototype works. Experimentation platforms are especially deceptive: the prototype is easy; the reliability, statistics, and UX are hard. Objective evaluation requires counteracting the bias with hard numbers.

Key Takeaway

For almost every team, buy. Build only if you have demonstrable specialized needs and the engineering depth to sustain a platform team indefinitely.