I can spin up a demo agent in an afternoon that makes a conference room clap. Warm voice. Clever answers. A booking flow that works when the Wi-Fi is good and the user cooperates.
Then you give it a real phone number and real traffic, and it folds like a lawn chair.
The gap between demo and production is not “more prompt adjectives.” It is engineering, ops, and judgment.
What demos optimize for
Demos optimize for:
- happy path
- short time to wow
- a scripted ask
- forgiveness from people who already want it to work
That is fine. Demos are sales tools and design tools. Just do not confuse applause with readiness.
What production requires
Production agents need:
Identity and access that will not embarrass you. Least privilege tools. No shared god keys in the prompt. Secrets in a vault or env, never in the model context if you can avoid it.
Deterministic rails around fuzzy language. Schemas for what you extract. Allowlists for what tools can do. Hard stops for crisis, legal, and medical boundaries.
Observability. Transcripts, tool call logs, latency, cost, and a way to replay a bad session without shrugging.
Handoffs. A human path that works when the agent is unsure, not only when the agent is broken.
Eval suites. A set of conversations you run before every prompt deploy.
On-call ownership. Someone gets paged. Someone can roll back.
If those are missing, you have a kiosk demo with a phone line taped to it.
Conversations are adversarial without meaning to be
Real users:
- mumble
- change the topic mid-sentence
- lie a little to get what they want
- ask for things you cannot do
- test whether you are a robot
- call from a truck with wind noise
Your demo speaker had a headset and a checklist. Production callers do not.
Design for repair. Confirm critical fields. Repeat phone numbers back. Ask one question at a time on voice. Make “I do not know, let me get you a person” a first-class outcome, not a failure state you hide.
Tool failures are normal
In demos, the CRM always responds. In production, the CRM times out while a human is mid-sentence.
Production behavior:
- retry with backoff where safe
- degrade gracefully (“I can take a message and our team will confirm”)
- never pretend a write succeeded
- surface the failure in logs with the session id
An agent that hallucinated a confirmation number is worse than an agent that apologized and escalated.
The prompt is not the product
You see teams polish system prompts for weeks while:
- the KB is stale
- there is no staging environment
- nobody listens to call recordings
- success is measured as “minutes talked”
The product is the loop: retrieve, reason, act, verify, learn. The prompt is one component. Treat it with respect. Do not worship it.
A promotion checklist you use
Before an agent gets real traffic, you want receipts:
- Written scope and out-of-scope list
- Crisis and escalation paths tested
- Tool permissions reviewed
- Synthetic monitors live
- Eval set passing at an agreed threshold
- Human review queue staffed for the first two weeks
- Rollback switch for prompt and model
- Success metrics tied to business outcomes, not vibes
If item 6 is “we will watch it when we can,” you are not ready.
How to use demos without lying
Label them. “Demo environment. Sample data. Not connected to production calendars.”
Invite a skeptic to break it. Record the breakage. Put those cases into the eval set. That is how demos fund production instead of setting traps for it.
The short version
A demo agent is a movie trailer. A production agent is a daily driver with seatbelts, logs, and a spare tire.
Build the trailer if you need buy-in. Budget time for the seatbelts. The organizations that win are not the ones with the slickest launch video. They are the ones whose agent still behaves on a rainy Tuesday when the CRM hiccups and the caller is stressed.
Bottom line
That is the bar. Everything else is theater with an API key.
