The Action-Velocity Signal: Why Raw Token Counts Lie About AI Agent Costs
Ask most people running AI coding agents daily what their biggest cost driver is, and they'll point at a token counter. That's the same mistake as diagnosing a CRO program by its click-through rate: it's the number that's easiest to see, not the number that explains what's actually happening. Auditing a real multi-week AI-agent workload recently surfaced a cleaner diagnostic — and a genuinely counterintuitive finding about which raw numbers actively mislead.
Three Gauges, Three Different Stories
A usage dashboard for the session in question exposed three separate percentages against the same underlying window: a cost-consumed gauge, a token-consumed gauge, and a message-count gauge. Naively, these should move roughly together — more tokens, more messages, more cost, all correlated. They didn't. The message-count gauge was running meaningfully hotter than either the cost or token gauges.
That divergence is the actual signal. When message velocity outpaces token and cost growth, it means a large number of small, rapid actions are happening — not a smaller number of expensive ones. Sustained at several actions per minute for hours, that pattern is not manual interactive use; nothing about normal human-paced review and correction produces that cadence. It's the fingerprint of autonomous, multi-step agent loops running against real work — which is a completely different cost profile than "someone had a long conversation," even when the token totals look similar.
This matters operationally because the fix is different depending on which gauge is actually binding. If cost is the constraint, the lever is model selection. If raw tokens are the constraint, the lever is context hygiene. If message velocity is the constraint, the lever is scoping how many autonomous loops run concurrently and how aggressively they fan out into sub-tasks. Treating all three as the same problem and reaching for the same fix — usually "use a cheaper model" — solves the wrong one two times out of three.
The Raw Number That Actively Lied
A separate diagnostic tool, run against the same period, reported a total token count roughly two orders of magnitude larger than what the account-level usage dashboard showed for the identical window. That's not a rounding difference — that's the kind of discrepancy that, taken at face value, would trigger a full incident response over a cost problem that didn't actually exist at that scale.
The explanation, once traced down: the larger tool was summing every token in every cached context re-read, at full face value, across a long-running session. Modern AI infrastructure caches a session's system context and re-reads it on every turn at a steep discount — officially documented as roughly a 90% reduction on cache reads — specifically because re-processing the same unchanged content from scratch every turn would be wasteful. A tool that sums raw token counts without accounting for that discount will produce numbers that are technically true and functionally meaningless, because they measure infrastructure efficiency (the thing working correctly) as if it were cost (the thing you're worried about).
The general lesson: any aggregate metric that doesn't distinguish between "new work happened" and "the same context got re-read cheaply" will systematically overstate cost on any long-running, high-context-reuse workload — and the overstatement gets worse, not better, the more efficient your caching is. The fix isn't a better calculator. It's checking the raw metric against the platform's own account-level usage reporting before treating it as ground truth, the same way you'd check a third-party analytics tool's revenue number against the payment processor's number before presenting it to finance.
Model Selection Is a Portfolio Decision, Not a Cost-Minimization Problem
The most consequential decision in the audit wasn't a diagnostic — it was a recommendation reversal. The initial default, chosen when the stated priority was minimizing spend, was the cheaper model for essentially everything. That's the correct call under a pure cost-minimization objective. It's the wrong call once the objective shifts to maximizing first-attempt success on complex, multi-step work — because a cheaper model that needs a second corrective pass frequently costs more, net, than a stronger model that succeeds once. The second pass isn't free: it's a full additional round of context, generation, and — critically — elapsed time and attention, none of which show up in the token line item but all of which are real cost.
This is the same tradeoff structure as a CRO test-tier decision. A Tier 1 test that's expensive to run but ships a large, confident lift beats ten cheap Tier 4 tests that individually cost less but collectively produce nothing shippable. The naive optimization — minimize cost per test, or minimize cost per model call — optimizes the wrong unit. The right unit is cost per successful outcome, and a model that's more expensive per call but meaningfully more likely to succeed on the first attempt can have a lower cost per successful outcome than the cheap default, especially on work where a failed first attempt cascades into an expensive correction loop.
The practical framework: for narrow, well-scoped, low-ambiguity tasks, optimize for raw cost — the success rate gap between a cheap and expensive model is small enough that price wins. For broad-scope, multi-file, high-ambiguity work — the kind where a wrong initial approach compounds — pay for the planning-quality upgrade even though the sticker price per call is higher, because the real comparison isn't "cheap model" versus "expensive model." It's "cheap model plus a probable second pass" versus "expensive model, probably once."
What to Actually Instrument
Three changes came out of applying this: track message/action velocity as a leading indicator, not just token or dollar totals, since it's the earliest signal that autonomous loops — not interactive use — are driving spend. Treat any third-party or auxiliary usage-reporting tool as a hypothesis to check against the platform's own account-level numbers, not a source of truth on its own, specifically because cache-discount accounting is easy to get wrong and produces plausible-looking totals that are off by an order of magnitude. And stop asking "which model is cheapest" as the first question for model selection — ask "what does a failed first attempt cost here," and let that answer, not the sticker price, set the tier.
None of this requires exotic tooling. It requires treating AI agent spend with the same diagnostic rigor as any other resource-allocation decision — the same instinct that stops a CRO program from optimizing win-rate frequency instead of revenue impact per test. The metric that's easiest to see is rarely the metric that explains what's happening. This is also where the verification discipline I've written about separately pays for itself twice — the same primary-source check that catches a fabricated capability claim is exactly what catches a misleading cost figure before it drives a bad model-selection decision.
FAQ
If token totals can be misleading, what should I actually watch instead?
Watch the platform's own account-level usage/quota reporting as ground truth, and treat message or action count over a rolling window as the leading indicator for whether autonomous loops are driving spend. A rapid, sustained action cadence — several actions per minute for an extended stretch — is a much earlier and more reliable signal than a token total, which lags and can be distorted by caching accounting.
Doesn't "pay for the expensive model" just mean costs go up?
Not if you're measuring cost per successful outcome rather than cost per call. On narrow, well-scoped tasks, the cheap model's success rate is close enough to the expensive model's that price should win — that half of the framework doesn't change. The upgrade only applies to the subset of work where a wrong first attempt is expensive to unwind, which is exactly the subset where the math favors paying more upfront.
How do I tell whether my own AI-agent workflow is interactive use or autonomous-loop-driven?
Check action velocity against elapsed time. Manual, human-paced review and correction rarely sustains more than roughly one action per minute for long stretches — there's inherent reading and thinking time between actions. Anything running meaningfully faster than that, sustained over multiple hours, is almost certainly an autonomous loop, a scheduled automation, or parallel agent fan-out — worth confirming explicitly rather than assuming it's "just a long session."
Auditing AI-agent spend with the same rigor as any other resource-allocation decision is the same discipline behind every experimentation program I run. If your team is scaling autonomous AI agents and wants a real diagnostic on where the spend is actually going, schedule a consultation.