Shipping a voice agent that sounds good in a conference room is a weekend. Shipping one that can sit on a real admissions line, in a real regulated industry, with real people in a bad night, is a different sport.
There is spent a lot of hours on that second sport. Healthcare is the room Know best, but the lessons travel: finance, education, anything where a confident sentence can become a liability. The work is less “pick a voice” and more “design a system that is allowed to exist.”
If you want the short version: never give clinical or legal advice, record like you mean it, escalate faster than you talk, and assume every call will be listened to by someone who was not in the building when you wrote the prompt.
The first product is the do-not-say list
Teams want to start with the greeting. Start with the forbidden list.
The agent does not diagnose. It does not say you are covered. It does not say a bed is waiting unless a live system says so. It does not argue someone out of a crisis. It does not improvise medication, detox timelines, or “you’ll be fine.”
That list feels negative. It is the product. Everything else is a conversation around a fence.
Write the fence in plain language in the system prompt, then I enforce it with routing. If the utterance looks like a medical question, you don’t prompt harder. You change the path: collect the callback, offer a human, stop being clever.
A good voice is useless if the content can wander. Cute is not a control.
Crisis is not a ticket type
If someone is in immediate danger, your architecture is a stopwatch, not a CRM workflow.
Build the crisis path before marketing gets the number. Detect the language, stop the intake script, and connect to a human path that is actually staffed. If you cannot staff it, you should not take the call with an agent that might miss it.
It is better to the agent be blunt and fast than warm and lost. “Aim for you to talk to a person right now” is a better line than a reflective paragraph.
Test this with the ugly phrases, not the polite ones. If your detection only works on textbook wording, it does not work.
Consent, recording, and the boring legal furniture
Regulated industries have a soundtrack: disclosure. Who is speaking. That the call may be recorded. What happens to the data. How to reach a human.
Say it early. Say it clearly. Don’t hide it in a fast mumble so you can get to the demo.
Then make the backend match the mouth. If you said you record, you store with access control and retention you can explain. If you transcribe, treat the transcript as sensitive, because it is. If a vendor is in the loop, you should be able to name them without sweating.
Do not pretend to be a lawyer on the internet. Do refuse to ship a voice agent that cannot answer “where does the audio go” in one paragraph.
Humans in the loop are not a phase
There is a fantasy that you turn the agent on and the staff go home. That’s how you get an incident.
Day one, a human reviews a real sample of calls. Not as a prosecutor. As a coach. Where did it interrupt. Where did it guess. Where did it fail to escalate. Where did it sound like a robot therapist.
You keep that review. Volume can make it sampling instead of every call. It does not go away. In regulated rooms, “the model usually does fine” is not a control.
The handoff has to land in the system the team already uses. If the morning starts by hunting transcripts in a vendor portal, you didn’t ship operations. You shipped a new hiding place.
Voice is a product constraint, not a skin
People talk over the agent. They have bad connections. They give a phone number like it’s a poem. They are scared, angry, or both.
So keep questions short. I confirm the number by reading it back in groups. Do not stack four asks in one sentence. I allow “do not know” as a first-class answer.
Latency matters more than the perfect voice clone. A pretty voice that pauses too long feels drunk. A plain voice that keeps up feels competent. Competent is the brand.
When we got this wrong, it was almost never the LLM’s personality. It was turn-taking, barge-in, and a prompt that wanted a speech.
What to would not do again
you would not let a location manager paste new policy into the prompt without review. you would not let the agent quote prices from a PDF last updated during a different presidency. you would not measure success as “minutes talked” instead of “kept appointments and clean escalations.”
you would start smaller: after-hours capture and callback, not a full digital admissions director. Earn the right to do more by not lighting a dumpster on the first weekend.
Bottom line
Real voice AI in regulated industries is mostly respect. Respect for the caller, the staff, and the rules that exist because someone already got hurt. Get those right and the voice can be almost boring. Boring and on-call is a very good outcome.
