AI features do not stay “done.” Prompts drift. Vendors change defaults. Your FAQ gets stale. A well-meaning teammate edits a knowledge doc and suddenly the voice agent is inventing a Saturday clinic that does not exist.
I used to find this out from a screenshot in Slack with the energy of a fire drill. Now I find a lot of it in a weekly review that takes under an hour when the systems are healthy.
The trick is making the review boring enough that you actually do it.
Same time, same checklist
I pick a slot and protect it like a standup you cannot skip. Same agenda every week. No “strategy offsite” vibes. If it needs a slide deck, it is already too heavy.
My list fits on one page.
1. Listen or read a sample
Pull a handful of real conversations or ticket outcomes. Not only the failures. A few random wins too, or you will bias toward doom.
Ask:
- Did it follow the handoff rules?
- Did it invent anything?
- Did it sound like us?
- Did the note in the CRM make a human’s life easier?
Ten minutes of reality beats an hour of dashboard worship.
2. Check the refusal and escalation rates
If escalations spike, something changed: traffic mix, prompt, knowledge base, or a broken tool.
If escalations fall to near zero, do not celebrate yet. You might have accidentally trained the agent to wing it.
Aim for stable rates with reasons I can name.
3. Tool errors and schema failures
Agents that cannot call tools become chatty liars. Schema failures mean your structured outputs are fighting the model or the model is fighting your brief.
Look at counts, not vibes. Top failing tool. Top validation error. One owner for the fix.
4. Cost and latency
A prompt change can quietly 3x spend. A new “helpful” context pack can add seconds to every call.
I glance at weekly cost, cost per successful handoff, and p95 latency for the customer-facing paths. If numbers jump, I bisect recent prompt and retrieval changes before I invent a mythology.
5. Knowledge drift
Any fact that can go stale gets a heartbeat check: hours, locations, promo language, insurance high-level statements, hiring links, whatever your agent is allowed to say.
If a human updated a page on the website but not the knowledge base, that is a process bug. Fix the process, not just the paragraph.
6. Eval suite on the scary paths
Keep a small regression set. Crisis language. “Give me medical advice.” Jailbreak-ish prompts. The top ten happy paths.
Run it when prompts change. Run it weekly anyway. Green is not glory. Green is permission to go back to work.
7. Decide one improvement
Only one. Maybe two if they are tiny.
Weekly reviews die when they become a backlog grooming session for 47 ideas. Ship one fix. Note the rest. Move on.
Who should be in the room
Someone who owns the product behavior. Someone who hears customer pain. Someone who can change the prompt or tools without filing a quest through six teams.
If legal or compliance needs a monthly deeper dive, fine. Do not force them into every weekly unless something material changed. Respect everyone’s calendar.
What to write down
A short log:
- date
- sample notes (three bullets)
- metrics that moved
- change we will ship
- owner
That log becomes institutional memory. When a new person joins, they can see the last two months of reality instead of a myth about “the AI.”
Bottom line
Ops for agents is ops. You would not run ads without a weekly look. You should not run customer-facing models on vibes and hope either.
Keep the review short. Touch real transcripts. Watch escalations, tools, cost, and stale facts. Ship one improvement.
Boring systems compound. Dramatic systems burn weekends.
