If your agent is part of how customers reach you, it is production infrastructure. Treat it like a cute experiment and it will fail like one: quietly, at the worst hour, with no page and no postmortem.
I learned to stop asking “is the model up?” and start asking “can a real user complete the job right now?”
Those are different questions.
Agents fail in more ways than websites
A website is down or it is not. An agent can be:
- reachable but dumb because retrieval is empty
- fast in chat and dead on the phone provider
- fine on happy path and broken on tool calls
- returning 200s while every conversation ends in “please hold”
Classic uptime checks are necessary and not sufficient.
What to monitor
Transport health: SIP, WebRTC, chat websocket, webhook endpoints. Can a session start?
Model and tool health: latency, error rate, timeout rate on the LLM and on each tool (CRM, calendar, KB search).
Task success: did we capture required fields, create the ticket, book the slot, escalate cleanly?
Queue and dependency depth: if the CRM API is crawling, the agent will sound haunted even when the model is fine.
Cost spikes: sudden token burn often means loops, retries, or a prompt regression.
Aim for a dashboard that a sleepy human can read at 1am without a decoder ring.
Status pages for people who are not engineers
Internal status beats tribal Slack knowledge.
Keep a simple page or channel topic with:
- green / yellow / red for voice, chat, retrieval, and write-tools
- last deploy time and prompt version
- link to runbook
- current incident one-liner if any
External status pages are optional depending on the product. Internal honesty is not optional. Your support team should not learn about an outage from a customer.
Synthetic checks that act like users
Ping endpoints are not enough. Schedule synthetic conversations:
- start a voice or chat session in staging or a prod canary
- ask for hours or a safe FAQ
- force a tool call (search KB, create a test ticket)
- assert the response shape and that the tool side effect happened
- tear down the test data
Run that every few minutes. Alert on consecutive failures, not a single blip.
Yes, it costs a little money. Missed after-hours calls cost more.
Alerting without waking everyone for vibes
Page on:
- session start failure rate above threshold
- tool error rate above threshold for N minutes
- synthetic task success below threshold
- provider incident confirmed
Do not page on:
- one slow completion
- a single user hanging up
- token usage wobble inside normal bounds
If everything pages, nothing pages. Humans will mute you.
Runbooks beat heroes
When voice dies, Avoid a scavenger hunt.
Runbook bullets:
- Check provider status
- Check API keys and quota
- Flip to backup model or failover prompt if configured
- Disable write tools if CRM is the blast radius
- Route inbound to human overflow number
- Post the status note
Failover is a product feature. “Text Kevin” is not a failover plan.
Deploys are incidents waiting to dress casually
Prompt changes can outage you without touching servers.
Version prompts, require a diff, run the eval suite, and ship behind a flag when the agent handles money or regulated conversations. If success metrics fall after release, roll back the prompt like you would roll back bad code.
Same discipline. Different file type.
The standard you hold myself to
If an agent takes customer traffic, it gets:
- metrics
- synthetics
- alerts
- a status surface
- a rollback path
No exceptions for “it is just an MVP.” MVPs that answer phones are production.
Bottom line
Uptime for agents is not about worshipping 99.99. It is about noticing broken conversations before your morning team inherits a pile of screenshots and bad reviews. Build the boring monitoring now. Your future Saturday night will thank you.
