Everybody wants “safe AI.” Almost nobody wants to define what safe means in a sentence an engineer can implement.
So we get two failure modes. Soft prompts that say “please be careful” while the model happily calls delete_customer. Or a compliance cage so tight the agent can’t answer “what are your hours?” without a lawyer present.
Stay in the boring middle. Guardrails that constrain actions, not vibes. Allowlists. Schemas. Tool limits. Stuff you can test.
Prompts are not guardrails
A system prompt is a suggestion with confidence issues. Useful for tone. Useless as your only safety story.
If the model can take an action in the real world, the real world needs a gate that does not care how eloquent the model sounded. Code. Policies. Hard checks after the model speaks and before anything executes.
Still write careful prompts. Just do not pretend they are seatbelts.
Allowlists beat blacklists
Blacklists are a forever job. There is always a new weird verb.
Allowlists are rude and effective. The agent may call these five tools. It may read these tables. It may send email only to addresses in this domain. Everything else is a hard no, logged, and ideally visible to a human later.
For customer-facing agents, keep the tool list short on purpose:
- lookup hours / location / FAQ from the knowledge base
- create a lead or ticket with a fixed schema
- escalate to a human queue
- schedule a callback window (not a clinical decision)
If someone asks for a sixth tool, they need a sixth reason that survives a review. “It would be cool” is not a reason.
Schemas are how you stop creative writing in databases
Free-form model output into your CRM is how you get notes that say “patient seems vibey” and phone numbers formatted like modern art.
Force structured output. JSON schema. Typed fields. Enums for status. Required callback number. Max length on free text. If the model can’t fill the schema, it retries or escalates. It does not get to invent a new column called feelings.
This also makes evals possible. You can score whether required fields landed. You cannot score “was the vibe professional” without turning your week into a book club.
Tool limits: rate, blast radius, and confirmation
Even allowed tools need seatbelts.
Rate limits. An agent that can send 500 SMS in a loop is not “autonomous.” It is a pager duty story waiting to happen.
Blast radius. Prefer tools that create drafts or pending records over tools that mutate production irrevocably. A “propose appointment” action a human confirms beats “book anything the model hallucinated.”
Confirmation for irreversible stuff. Refunds, deletes, messages that leave the building. If a human should care, a human should click.
Yes, that slows the demo. Good. Demos that skip this are how companies learn expensive lessons in public.
Fail closed on the scary paths, fail open on the boring ones
Crisis language, self-harm signals, threats, anything that smells like a medical question outside your approved FAQ: escalate. Do not improvise. Do not “be helpful.”
Hours, parking, “do you take this insurance at a high level,” “can someone call me back”: keep the conversation moving. Over-blocking the boring path trains users to hate your product and call the main line anyway.
Usefulness dies when every answer is “let me get a human,” and liability grows when no answer ever gets a human. Pick your lane per intent.
Log the refusals
If you don’t log when the guardrail fires, you can’t tell if it’s protecting you or just annoying customers.
Take a weekly look at:
- tool calls blocked
- schema validation failures
- escalations by reason
- places the agent said “I can’t help with that” and the user hung up mad
Then you tune. Sometimes the allowlist is too tight. Sometimes the FAQ is missing an entry. Sometimes a prompt needs one clearer sentence. Guardrails without feedback rot into superstition.
Bottom line
Guardrails that work feel unsexy. They look like allowlists, schemas, and limits in code. They do not look like a paragraph of corporate poetry in the system prompt.
Build the agent so it can do a few jobs well. Block everything else. Measure the blocks. Keep a human door for the moments that matter. That is how you ship something useful without waking up to a screenshot of your agent inventing a policy that never existed.
