Every engineering leader I talk to right now has the same chart: agent spend climbing month over month, and no clean answer to “what did we get for it.” The reflex answer is always the same too — switch to a cheaper model. It’s the most visible lever, so it gets pulled first.
It’s also the smallest one.
Run agents in production — systems that investigate bugs, write and review code, validate changes — and the bill turns into a real line item fast. What actually moves that number isn’t clever prompts, and it isn’t model shopping. It’s controlling five things:
- Which model handles each task
- How much context gets re-sent on every request
- How many tools, files, and logs the agent loads before doing anything
- How long the agent is allowed to loop
- Whether work rides on caching and batch discounts, or pays retail
And underneath all five, one accounting decision that makes the rest possible:
The optimization target is cost per accepted engineering outcome — a passing change, an approved review, a resolved incident, a merged PR. Not cost per token. A cheaper call that produces rework isn’t a saving; it’s a deferred expense with interest.
This is the operational playbook. It’s long because it’s meant to be used, not admired. There’s a slide deck version at the bottom if you want the one you can drop into a leadership meeting.
The playbook on a page#
For the people who will read exactly one screen — this is the whole thing. Print it, pin it next to the deploy checklist, send it to whoever owns the AI budget. The rest of the article is the how.
1. One gateway in front of everything. Tag every request:
user · team · repo · workflow.
2. Budget envelopes per session. Exhaustion returns partial
work + next action — never a hard kill.
3. Three model tiers. The default is cheap; escalation
requires evidence.
4. One task per session. Every request carries
GOAL / SCOPE / LIMITS / VERIFY / STOP.
5. Thin AGENTS.md (<200 lines); skills on demand;
ignore files everywhere.
6. Tools load on demand. Logs get filtered before
they enter context.
7. Stable prompt prefix first — cache reads cost 0.1×.
8. Nobody waiting on it? Batch it at ~50% off.
9. One retry, one escalation. Subagents for isolation,
not brainstorming.
10. Track cost per accepted task weekly, with a named owner.
The waste hides in five places, and the model is only one of them#
Teams that only optimize model choice leave, in my experience, a third to half of the possible savings on the table. The silent multipliers dominate real bills:
- Context pollution. Long-lived sessions accumulate old plans, failed attempts, full logs of tests that passed, and files from tasks that ended hours ago. Every one of those tokens gets re-sent on every subsequent request. You pay for the same garbage dozens of times.
- Verification theater. Agents that keep re-running tests and re-reading files after the outcome is already clear. It looks diligent. It’s a loop with no exit condition.
- Unbounded tool surface. Every MCP server, every tool schema, loaded into every request — whether the task needs them or not. Anthropic’s own numbers: a representative multi-server setup burns ~55k tokens in tool definitions before any work begins, and on-demand tool search cuts that by more than 85%.
- Escalation without evidence. The premium model gets called because a task “looks hard,” not because a cheaper attempt failed.
- Multi-agent by default. Agent teams are a power tool, and they’re priced like one — Anthropic’s docs put team sessions at roughly 7× the tokens of a standard session in plan mode, because every teammate carries its own context.
None of these show up in a model-price comparison. All of them show up on the bill.
Put every request behind one gateway before you tune a single prompt#
You cannot optimize what you cannot attribute. If every IDE, internal agent, script, and CI job calls providers directly with its own API key, your bill is a single undifferentiated number and every cost conversation is a guess.
One gateway, in front of everything:
- Mandatory metadata on every request: user, team, repository, agent, workflow, environment, task type, model tier. Spend with missing metadata is a bug.
- Centralized keys, central model allowlists, rate limits, budget enforcement, provider fallback.
- Cost and token reporting that finance and engineering both read from.
Default recommendations, depending on where you already live: LiteLLM + Langfuse for open-source-leaning orgs, Cloudflare AI Gateway if Cloudflare is already strategic, Amazon Bedrock as the front door if you’re AWS-centric — one API across models, IAM-scoped access, cost allocation through tagging, with Langfuse or Phoenix on top — and Portkey or Helicone if you want turnkey commercial. Pick one control plane and one observability system. Running four overlapping products is its own cost problem.
Budgets are an agent contract, not a kill switch#
Every agent execution needs a budget envelope: max dollars per session, max input/output/reasoning tokens, max steps and tool calls, max retries per failure signature, max subagents, max elapsed time, max premium-model calls, max files read. Budgets exist at session, user, team, repository, CI-workflow, daily, and monthly levels.
The part most teams get wrong: what happens at exhaustion. A hard kill that throws away everything the agent learned is a process failure. On budget exhaustion the agent must return five things:
- What it completed
- What remains
- The evidence it gathered
- The recommended next action
- Whether escalation to a stronger model is actually justified
That turns a budget from a circuit breaker into a checkpoint. The work survives; the spend doesn’t run away.
Three tiers. Escalation requires evidence.#
The most expensive capable model must not be the default. Route by tier:
Economy — repository search, straightforward edits, formatting, test generation, documentation, simple refactors, classification, log summarization, producing a patch from an already-approved plan.
Standard — multi-file bug investigation, moderate debugging, unfamiliar code, API integration, code review, test-failure diagnosis, ambiguous requirements.
Premium — architecture, security-sensitive work, complex migrations, hard concurrency bugs, high-impact incidents, final review of high-risk changes, and tasks that already failed on a lower tier.
The escalation rule is strict: economy → standard → premium, one step at a time, and only with evidence — a failed verification, contradictory findings, an explicit security classification, or a documented lower-tier failure on the same task. “Looks difficult” is not evidence. Neither is a missing import, noisy test output, or an ambiguous requirement when a human is one message away.
The routing itself is becoming a managed feature — Microsoft Foundry’s Model Router and Amazon Bedrock Intelligent Prompt Routing both do tiering natively, and GitHub Copilot’s auto model selection pays you to use it: a 10% discount on premium request multipliers (on models in the 0–1× multiplier range). Evaluate them against your own tasks — vendor routers optimize for vendor-average workloads, not yours.
And measure whether escalation actually improves acceptance rate. If the premium tier isn’t producing fewer rejected outcomes than standard, you’re paying for a brand, not a capability.
Context is a budget, not a backpack#
The agent doesn’t need everything you can give it. It needs exactly what the current task requires, and nothing that a previous task left behind.
Separate planning, execution, and verification. Plan short and high-quality: files, risks, acceptance criteria. Execute with the least expensive capable model. Verify with deterministic tools — tests, linters, type checkers — not with a premium model re-reading its own work. Escalate only on failure. And keep planning itself bounded; “plan mode” gets expensive fast if the agent may inspect the whole repository before defining the task boundary.
One task per session. When the task changes, the session ends. Compact after major phases. Keep the current plan, the decisions, the relevant files, the verification state — drop everything else.
Every request gets a scope and a stopping condition. Five elements: goal, allowed scope, constraints, verification, stop. In practice:
## Goal
Fix the race that sends duplicate notifications.
## Scope
- services/notifications/notification_service.ts
- Existing notification tests
- A new regression test
## Constraints
- Do not alter the public API
- Do not refactor unrelated notification code
- Do not update dependencies
## Verification
Run: pnpm test notifications --filter duplicate-send
## Stop
Stop when the regression test and existing notification suite pass.
No cleanup changes outside the listed scope.
That last block is the one that saves money. An agent without a stopping condition stops when the context window does.
Keep the always-loaded instructions thin. Root AGENTS.md well under 200 lines: repository map, standard commands, test strategy, conventions, safety rules, definition of done, and pointers to where specialized knowledge lives. Nested AGENTS.md files for package-specific rules. Vendor files (CLAUDE.md, GEMINI.md, Copilot instructions, Cursor rules) stay thin adapters generated from the canonical source.
Specialized procedures go in on-demand skills. Database migrations, incident diagnosis, PR review, dependency upgrades, release prep — each a SKILL.md loaded only when relevant, kept under ~500 lines with heavy details in conditionally-loaded references. The entire engineering handbook riding along on every request is one of the most expensive habits an org can have.
And exclude what the agent should never read: node_modules, vendor trees, build outputs, generated clients, coverage reports, minified files, huge fixtures, exported datasets. Ignore files are the cheapest optimization in this entire playbook.
Runtime discipline: load less, cache more, batch the rest#
Tools on demand. Allowlists by workflow, deferred loading with tool search, short precise descriptions, role-specific bundles. The ~55k-token / >85% numbers above are the argument. Prefer deterministic tools — rg, LSP, AST search, targeted test selection — over asking a model to reason through raw data, and filter logs to errors, failing test names, and stack traces before they ever enter the context.
Engineer prompts for cache hits. Prompt caching is a prefix match: stable material first (system instructions, org policy, repository conventions, stable tool definitions), variable task content last. A timestamp or request ID near the top of the prompt silently invalidates everything after it. The economics are worth engineering for — on Claude’s pricing, cache reads cost 0.1× base input, 5-minute cache writes 1.25×, 1-hour writes 2×. Track hit ratio and treat a collapsing cache rate as an incident.
Batch everything that isn’t interactive. Anthropic, OpenAI, and Gemini all document ~50% discounts for asynchronous work with a 24-hour window. Nightly evaluations, documentation refreshes, embeddings, backlog labeling, migration inventories, offline PR analysis — none of it needs an interactive endpoint. The policy is one sentence: if a human is waiting on the result, normal API; otherwise, queue it.
Cap the loops. One automatic retry per failure signature. One model escalation. One fallback provider. A ceiling on edit cycles and parallel subagents. Termination when no new evidence is being produced, and a human in the loop after repeated deterministic failure. Multi-agent stays an explicit high-cost mode — use subagents for context isolation (digesting a huge log without polluting the main session), not as default brainstorming.
Exact-match caching and deduplication work well for repeated development prompts, evaluation runs, and identical CI analysis. Semantic caching — treating “similar” prompts as identical — is where I draw a hard line: never for code generation, security decisions, or production remediation. Similar questions legitimately have different answers, and a cache that papers over that difference is a bug generator with a cost-savings sticker on it.
You can’t downgrade what you can’t measure#
Before any model downgrade, build a representative benchmark: 30–50 tasks covering your actual work — small bug fix, multi-file fix, test generation, refactor, review, security finding, migration, docs, incident investigation. For each attempt, record success, tests passed, human acceptance, rework required, tokens by category, total cost, tier, steps, tool calls.
The primary metric:
Cost per accepted task =
total AI cost for the workflow
÷ outputs accepted without significant rework
Track the cost of false economy right next to it — the cheaper path that increased rework or burned senior-engineer review time didn’t save anything. The weekly dashboard: cost by team/repo/workflow/tier, cost per accepted task and per merged AI-assisted PR, median and P95 session cost, cache hit ratio, premium escalation rate, retry rate, batch share, human rejection rate, spend with missing metadata. Alert on premium use without an escalation reason, sessions far above normal cost, all MCP servers enabled at once, and — the one that matters most — rising spend with a falling accepted-task rate.
Then write it down as policy with a named owner: approved models, default tier, escalation conditions, budgets, retry rules, batch-eligible workloads, caching conventions, evaluation gates, an exception process, and a monthly review where someone is accountable for the cost-per-accepted-task trend. Cost blow-ups go into the same postmortem process as reliability incidents, because that’s what they are.
What to do in the first week#
Not a quarter-long program. The first week, in order:
- Require user, team, repository, and workflow tags on every request
- Produce a baseline spend report
- Define the three tiers and change the default away from premium
- Add the one-escalation rule and session step/retry limits
- Create the root
AGENTS.md; move specialized instructions into 2–3 skills - Add ignore files; disable unused MCP servers
- Identify batch-eligible workflows
- Add cost and quality fields to evaluation results
The following two weeks: deploy the gateway and observability, implement central routing, normalize prompt prefixes for caching, move nightly jobs to batch, build the evaluation set, start premium-exception reporting, audit unused seat subscriptions, publish the policy.
And the anti-patterns to hunt down, because every org has most of them: the frontier model as default, the handbook in every prompt, every MCP server in every session, one conversation carrying five unrelated tasks, full test suites after every minor edit, complete success logs returned to the model, unlimited retries, multi-agent as a habit, models compared on token price alone, and interactive work shoved into batch to make a dashboard look better.
The savings are boring, and that’s the point#
Nothing in this playbook is clever. It’s attribution, budgets, routing, hygiene, and a metric that measures what the business actually buys. That’s why it works — and why most teams skip it in favor of another round of prompt tweaking.
The organizations that get agent economics right won’t be the ones that found a secret cheaper model. They’ll be the ones that made the cheap path the default, made the expensive path explicit, and demanded evidence that the expensive path produces a better engineering outcome. I’ve written before that AI makes code cheap to produce, not cheap to own — this is the operational answer to that gap, and it’s the same discipline that makes a fleet of agents governable at all.
Start with the control plane and the measurement. Everything else in this playbook only becomes high-leverage after attribution and budgets exist.
Slide deck: the 14-slide hands-on version of this playbook — the budget envelope, the routing ladder, the request template, the week-one checklist — ready for your next leadership or platform meeting → download the PDF.
I lead AI transformation for a global SaaS platform. This playbook is a distillation from production, not a thought experiment — and it’s still evolving. If you’ve found a silent cost multiplier it misses, or pulled one of these levers and got a different result, I want to hear about it. Find me on X, LinkedIn, or Telegram.