An agent that runs all day has a different cost personality than a chat window you open twice.
In chat, you feel the meter. In production, the meter lives in a vendor dashboard you forget to open until finance pings you with the energy of a smoke detector.
There is burned money on bloated system prompts, retrieving entire novels into context, and using a frontier model to classify “is this about parking.” All educational. Prefer learning cheaper.
Separate static from dynamic
Most agent prompts have a big static block: persona, rules, tool descriptions, safety rails. Then a small dynamic block: user message, retrieved docs, session state.
Providers that support prompt caching love that shape. Put the stable blob first. Keep it stable. Do not reshuffle it every request because you like alphabetical vibes.
If your “static” prompt changes hourly, you do not have a prompt. You have a runaway science project. Batch edits. Version them.
Retrieval should be picky
The model does not need your whole knowledge base. It needs the two chunks that answer this question.
I budget tokens like I budget ads. Top-k that is sane. Chunk sizes that are readable. Hard caps. If retrieval returns junk, fix retrieval. Do not compensate by stuffing more junk in.
Also: cite or tag what was retrieved in logs. When costs spike, you want to see whether context bloat was the villain.
Route models by job
Not every step needs the expensive brain.
- cheap/fast model: classification, routing, “is this crisis language,” simple FAQ match
- mid model: normal conversation
- expensive model: hard reasoning, messy tool plans, final judgment calls you refuse to automate fully
This is not theoretical. A classifier that runs on every turn can cost less than using your best model as a Swiss Army knife for “hi.”
Cache at more than one layer
Besides provider prompt caching:
- cache FAQ answers for identical public questions with a short TTL
- cache embeddings and retrieval results when the corpus is stable
- cache compiled configs for agent versions
Be careful caching personalized or PHI-adjacent content. Public FAQ yes. Patient-specific anything no.
Meter what matters
Track:
- cost per conversation
- cost per successful handoff or resolved ticket
- tokens in vs out by agent version
- tool call fan-out (agents that loop tools can multiply spend without multiplying value)
If you only track monthly total, you will always be surprised. If you track cost per outcome, you can decide whether a prompt change was genius or a money leak with better manners.
Guardrails against runaway loops
Set max turns. Set max tool calls. Set timeouts. Circuit-break when an agent retries the same failing tool like a raccoon with a jar.
Runaways are a reliability issue and a cost issue. Fix both with the same limits.
Cheap wins before heroic architecture
- delete dead examples from the system prompt
- stop pasting entire JSON schemas in prose if the API already enforces schema
- prefer summaries over raw transcript history beyond the last few turns
- don’t log seven copies of the same prompt into seven systems
Then do caching and routing. Heroes skip straight to architecture and leave the prompt looking like a storage unit.
Version prompts like code
Every production prompt gets a version id in logs. When cost jumps, you diff versions instead of guessing. Roll back is a config change, not an archaeology dig.
Also alert on cost per outcome thresholds the same way I alert on error rates. Money leaks deserve the same dignity as 500s.
Tell finance a story they can audit
When someone asks why the AI line item moved, you should be able to point at version ids, cache hit rates, and cost per outcome. Mystery invoices erode trust fast. Clear metering makes it easier to keep funding the stuff that works.
Bottom line
Always-on agents need always-on cost hygiene. Cache the static prompt. Retrieve less. Route models by job. Cap loops. Measure cost per outcome.
You can have a helpful agent without sponsoring your vendor’s next yacht. Mostly you need discipline, not a new religion.
