Which AI Model Should You Actually Use to Build Software? (Hint: It's not the cheapest one)

Which AI Model Should You Actually Use to Build Software? (Hint: It's not the cheapest one)

Meta description: Token price and benchmark scores are the wrong scoreboard for choosing an AI coding model. The metric that matters is cost per accepted production change — and a cheap model that needs three tries is more expensive than an expensive one that lands the patch on the first. Here's the framework, the evidence, and the stack I actually run.

About two years ago I was going back and forth with a senior iOS developer about token burn rate. We were both watching the meter — input tokens, output tokens, dollars per session — and trying to optimize it the way you'd optimize a cloud bill. Pick the cheaper model, trim the context, keep the burn down.

We were wrong, and it took a few dozen projects to see exactly how wrong.

Token burn rate is a proxy. It feels like the KPI because it's the number the dashboard shows you. But the thing we actually cared about was never the tokens — it was whether those tokens produced a working feature, and whether that feature produced revenue or usage for the people we were building for. Once you write the real objective down, the token-cost obsession collapses. Optimizing the cheapest path to a broken pull request is not optimizing anything.

This is the piece I wish someone had handed me at the start of that conversation. It's about the metric that actually governs the economics of building software with AI, the benchmark theater you should ignore, which model to reach for inside a single vendor's lineup, and the specific tool stack I run today. There's real data behind all of it, including a few results that should change how you shop.

The scoreboard everyone stares at is broken

Two numbers get quoted when people argue about AI coding models: price per million tokens and benchmark score. Both are close to useless as buying guides, and the second one is now actively misleading.

Start with benchmarks, because the story there is remarkable. In February 2026, OpenAI — the company that created the most-cited coding benchmark — published a post explaining why it no longer evaluates frontier models on SWE-bench Verified. Their own audit found a large share of the benchmark's test cases were flawed in ways that reward shortcuts, and that frontier models showed signs of contamination — having effectively seen the answers during training. When the people who popularized a benchmark tell you to stop trusting it, that's not a footnote. That's the scoreboard catching fire.

Here's the gap in one line: Anthropic's Claude Opus 4.5 scores around 80% on SWE-bench Verified but only ~23% on the harder, cleaner SWE-bench Pro. Same model, same week — a 3x swing depending on which test you believe. If you picked a model off a Verified leaderboard, you bought a number that doesn't survive contact with fresh problems.

And even the honest benchmarks deliver a humbling verdict on the state of the art. SWE-Bench Mobile, which evaluates agents on real tasks pulled from a production iOS codebase — Swift and Objective-C, Figma designs, actual test suites — found that the best of 22 agent-model configurations solved just 12% of tasks. My iOS-developer friend, it turns out, had good instincts.

But the most important finding in that paper isn't the 12%. It's this:

The same model showed up to a 6× performance gap across different agents. Agent design mattered as much as raw model capability.

Sit with that. The harness around the model — how it plans, retries, reads files, runs tests — can swing results sixfold with the model held constant. Which means "which model is best?" is the wrong question to lead with. The right question is "which model, inside which workflow, at what total cost, lands accepted changes?"

The metric that actually matters

Here is the whole argument in one equation. Write it on a sticky note:

Cost per accepted production change = subscription + extra credits + retries + your steering time + bug cleanup + security review.

Every term after "subscription" is the part the token-price comparison hides. A cheap model that produces three broken attempts before you give up and rewrite it yourself is more expensive than a costly model that lands one clean patch — even though the cheap model's per-token price is a fraction of the expensive one's. The retries cost tokens. The steering costs your afternoon. The bug cleanup costs a future debugging session. The missed security issue costs something you can't price until it's too late.

And there's a metric above cost, the one that reframed everything for me and my iOS friend: does the change ship value? The full ladder looks like this:

1. Tokens — what the meter shows. A cost, not an outcome.
2. Accepted changes — code that survives review and stays merged. The first real unit of output.
3. Revenue or usage — the change moves a number a user or customer cares about. The only unit that matters.

Optimizing rung 1 while ignoring rungs 2 and 3 is the classic proxy trap. It's the software equivalent of a growth team celebrating traffic while conversions flatline. The cheapest tokens in the world are worthless if they don't climb the ladder.

Why cheap models quietly cost more

The false economy has a mechanism, and there's now data on it. A team manually analyzed more than 3,800 publicly reported bugs across the open-source repositories of Claude Code, Codex, and Gemini CLI (Engineering Pitfalls in AI Coding Tools). Two findings matter for buying decisions:

• Over 67% of the bugs were functionality bugs — the tool did the wrong thing, not just the ugly thing.
• ~37% traced back to API, integration, or configuration errors — the largest single root-cause category, concentrated in the layer where the agent orchestrates tools and executes commands.

Translate that into the equation above. A weaker model is more likely to fumble exactly that orchestration-and-integration layer — the wiring between your code, your test runner, your deploy config. Each fumble is a retry. Each retry is steering time. The savings you booked at the token meter get clawed back, with interest, at the "your afternoon" line item.

This is why I stopped chasing the cheapest model for production work. The false economy isn't a rounding error — it's the dominant cost, and it lands on the most expensive resource in the whole system: your attention.

Which model within a vendor's lineup

Most "which model" advice stops at the vendor door — Claude vs. GPT vs. Gemini. But the higher-leverage decision is usually inside one lineup. Anthropic ships Opus, Sonnet, and Haiku (each in versioned generations); OpenAI and Google have their own frontier/mid/small tiers. Using them interchangeably is how you burn money in both directions — overpaying for trivial work, underpowering the work that matters.

The rule I run:

• Frontier model (e.g., Opus-class) for anything where a mistake compounds: architecture decisions, auth, payments, data migrations, gnarly multi-file bugs, and the final review pass. Here you are buying one-shot accuracy, and one-shot accuracy is the cheapest thing you can buy.
• Mid model (Sonnet-class) for the daily driver: feature implementation, refactors, UI flows. Strong enough to land the change, cheap enough to run all day.
• Small model (Haiku-class) for the genuinely mechanical: summarizing files, generating boilerplate tests, discovery ("which files touch checkout?"), and copy tweaks.

The organizing principle underneath all three is what I call fewest total tokens to near-one-shot. The goal isn't the cheapest per-token model. It's the model that gets you closest to correct on the first attempt, because the first attempt is where token spend, wall-clock time, and your steering attention are all minimized at once. A frontier model that near-one-shots an auth refactor consumes fewer total tokens than a cheap model that needs four rounds — before you even count your time. Downgrading to "save money" per token routinely spends more money in aggregate.

Which tool, when — the stack I actually run

If agent design swings results sixfold, then your tools — not just your models — are the product. There is no universal winner, and the best study I've seen proves it. Researchers analyzed 7,156 pull requests across five agents (Codex, Copilot, Devin, Cursor, Claude Code) and stratified acceptance rates by task type (Comparing AI Coding Agents):

• Codex was consistently strong across all nine task categories (59.6%–88.6% acceptance).
• Claude Code led on documentation (92.3%) and feature work (72.6%).
• Cursor led on fix tasks (80.4%).

No single agent won everywhere. That maps to a multi-tool workflow, not a single-model religion. Here's how I assign the seats, as of mid-2026:

One caution: Google retired the consumer Gemini CLI on June 18, 2026, pushing individual users to Antigravity CLI. The cheap-scout seat is real, but the workflow is mid-transition — another reason not to make it your core.

The "master orchestrator" is a trap

The tempting next move is to build a smart router that picks the perfect model for every task and minimizes cost per accepted PR automatically. Resist it. That project is a research spiral disguised as a productivity gain.

The routing infrastructure is real — LiteLLM supports latency-, cost-, and rule-based strategies; OpenRouter gives you one API key across providers. Those are genuinely useful inside a product you're shipping, where you control the traffic and can measure outcomes. But as your personal coding control plane, a static "Claude for code, Gemini for context, GPT for reasoning" rule doesn't hold, because good routing needs feedback from actual execution results — did the change pass tests and get merged? — which a naive router doesn't have.

The best orchestrator for a solo builder is a manual policy you can state in four lines:

1. Claude Code implements.
2. Codex independently reviews or re-solves when the stakes are high.
3. Gemini/Antigravity does cheap repo reading and alternatives.
4. API routing lives inside your SaaS product, not in your editor.

The cost-control rules that actually move the needle

None of the above matters if you let an agent wander. These are the habits that keep cost per accepted change low, and most of them are free:

• Keep one `CLAUDE.md` / `AGENTS.md` per repo, under ~150 lines. Bloated context files and a pile of MCP servers burn your usage faster and dilute the model's attention. OpenAI says as much in its own Codex usage guidance.
• Never say "review the whole app" unless that's literally the task. Scope to one flow, one bug, one component, one PR.
• Kill zombie sessions. When the context gets muddy, start fresh. A long, confused session is a token furnace.
• Make the agent run tests. If there are no tests, the agent's first job is to add minimal ones around the flow it's changing. Untested agent output is a liability, not a deliverable.
• Commit before agent work. One task per branch. If it damages the repo, git reset and move on. Cheap insurance.

This is the boring, unglamorous edge. It's also the real one. The pattern serious teams follow isn't "vibe and pray" — it's specialization plus verification: small task → agent implements → tests run → a second agent reviews → a human checks architecture and security → merge only if the app still works.

What the money should look like

Concrete, because vague budget advice is useless. This is a sane starting stack for a solo founder building real SaaS in 2026:

• [Claude Code Max 20x — $200/month.](https://support.claude.com/en/articles/11049741-what-is-the-max-plan) Your primary builder. If you're already hitting the $100 Max 5x ceiling in long agentic sessions, upgrading is rational — provided it produces shipped work. The Max tiers are 5× and 20× Pro usage per session, respectively.
• [Codex via ChatGPT Plus ($20) or Pro ($100).](https://developers.openai.com/codex/pricing) Start at Plus. Move to Pro only once Codex is producing accepted PRs or saving you Claude sessions. Both tiers let you buy extra credits after you hit the shared five-hour usage window — more elastic than a hard lockout, and easier to overspend, so watch it.
• [Cursor Pro — $20/month](https://cursor.com/pricing), only if you live in the editor. Don't buy the high tier on a hunch.
• Gemini / Antigravity — free or API-metered, for specific scouting tasks. Don't pay a premium here just because someone online says "Gemini is best for code." Test it on your repos first.

Starting range: $220–$340/month. Hard cap: $400. And one rule that keeps the whole thing honest: if your AI coding spend clears $400/month before your product has users, you're substituting tooling for execution. The exception is if that spend visibly produces deployed features, landing pages, experiments, or sales assets that would cost more than $400 in contractor time. Ship or downgrade.

The thing most people get wrong

The deepest error isn't picking the wrong model. It's believing the winner is the model that writes the prettiest first answer. It isn't. The winner is the workflow that produces the most accepted, tested, deployed changes per week — and, one rung up, the most revenue or usage per unit of effort.

Cheap tokens feel like savings. Benchmark scores feel like truth. A "master orchestrator" feels like an edge. All three are seductive because they're legible — they give you a number to optimize. But the real economics live in the messy, unlegible places: the retry you didn't count, the afternoon you spent steering, the bug that shipped, the security review you skipped. Price the whole equation, not just the first term, and the answer to "which model?" stops being "the cheapest" and becomes "the one that lands the change."

That's what my iOS friend and I were circling two years ago without the words for it. The tokens were never the point. The shipped, revenue-producing feature was.

FAQ

Is the most expensive AI coding model always the right choice?

No — the rule is frontier for high-stakes, cheap for mechanical. Use a frontier model where a mistake compounds (architecture, auth, payments, migrations, final review) and a small model where it doesn't (summaries, boilerplate tests, file discovery). The goal is fewest total tokens to a near-one-shot correct result, which usually means paying up on the hard 20% and saving on the easy 80% — not defaulting to the priciest model everywhere.

How do I actually measure "cost per accepted production change"?

Track, per feature or fix: your subscription cost allocated to that work, any extra credits burned, the number of retry loops, the time you spent steering, follow-up bug-fix sessions it caused, and review effort. You won't get it to the penny — you don't need to. Even a rough tally exposes the false economy fast: the "cheap" model that needed four attempts almost always loses once your time is priced in.

Should I use Gemini as my main coding tool because it's cheaper?

Not yet. Gemini/Antigravity is an excellent cheap scout — reading many files, summarizing architecture, generating alternatives. Make it primary only after it beats your current tool on your own repositories, and note that Google's consumer CLI is mid-transition to Antigravity as of June 2026, so the workflow itself is in flux.

Do I need a router or "master orchestrator" to pick models automatically?

For personal coding, no. Routing tools like LiteLLM and OpenRouter are real infrastructure, but they belong inside a product you're shipping, where you can measure outcomes and route on real feedback. As a personal control plane, a static rule router underperforms a simple manual policy: frontier model implements, a second model reviews high-stakes work, cheap model scouts.

Are benchmark scores useless for choosing a model?

Not useless, but a weak buying guide — and SWE-bench Verified is now actively misleading after OpenAI flagged contamination and flawed tests in early 2026. Treat benchmarks as one weak signal. The stronger signals are task-stratified acceptance data (which agent wins which task type) and, above all, your own repo: run a real task through two tools and see which one's change you actually merge.

Where to go from here

If you take one action after reading this, make it small: pick a single repo and write a 10-line CLAUDE.md telling your agent exactly how to run, test, and modify that project. That one file will do more for your cost-per-accepted-change than any model swap.

If you're a founder or growth leader trying to build with AI without lighting money on fire — on product, on experimentation, on the whole tokens-to-revenue ladder — that's the work I do. Work with me or subscribe to the newsletter for more field notes from building AI-native products in public.