Changing a prompt because it “felt better” is how you ship regressions with confidence.
The demo case got prettier. The crisis case got quieter. Nobody noticed until a real caller did.
Evals are how you stop treating prompt edits like horoscopes.
What an eval is in operator language
An eval is a fixed set of examples, a way to grade the agent, and a rule for comparing version A to version B.
That is it. You do not need a research lab badge. You need discipline.
If you cannot rerun the same cases after a change, you are not iterating. You are gambling.
Start with cases from reality
Build the set from:
- real transcripts with names removed
- the weird edge cases staff complain about
- known failure screenshots
- a few synthetic extremes (empty input, rage typos, mixed language)
About thirty to fifty solid cases beats two hundred fluffy ones. Quality over cardboard.
Label each case with what good looks like. Not vibes. Checks.
Examples:
- must capture phone
- must escalate
- must not give clinical advice
- must cite the hours card, not invent Sunday hours
- must keep reply under N sentences for voice
Grade with rubrics, not applause
Avoid “rate this 1-10 how helpful.” Models and humans both get mushy.
Prefer binary or tight ordinal checks tied to outcomes. You can still have a judge model score some dimensions, but the judge needs the same rubric every time. And you should spot-check the judge.
For tool-using agents, assert on tool calls too: did it call the calendar? Did it write the CRM fields? Did it call something it should never call?
Compare versions like a grown-up
When you edit a prompt, run the suite on old and new.
Look at:
- pass rate overall
- pass rate on critical safety cases (must not drop)
- regressions list (cases that flipped from pass to fail)
- cost and latency if they matter in production
If safety regresses, the new prompt loses even if the witty FAQ score went up. Do not negotiate with that.
Online signals still matter
Offline evals catch a lot. Production catches the rest.
Watch handoff quality, human thumbs in the review queue, escalation rate, and “agent invented policy” tickets. When production shows a new failure class, it becomes a case in the suite within a day. That is how the suite stays alive.
An eval set that never grows becomes a museum.
Prompt changes are code changes
Version prompts. Note why a change happened. Keep the previous version one rollback away.
“We updated the prompt” with no artifact is how teams gaslight themselves. Put it in git. Put eval results next to the PR. Make it boring and auditable.
Common traps
Teaching to the test. If your suite is tiny and you tweak until it passes, you overfit. Add fresh cases regularly.
Only happy paths. Your agent already works on happy paths. Evals earn their keep on mud.
Metric theater. A dashboard full of averages with no safety slice will bless bad tradeoffs.
One giant prompt rewrite. Prefer small edits. Large rewrites need larger suites and more human review time. Respect the blast radius.
A lightweight setup that fits a small team
- Spreadsheet or JSON of cases with expected checks
- Script that runs the agent and records outputs
- Auto-grade what you can, human-grade the rest weekly
- Gate production prompt deploys on “no critical regressions”
You can get fancier later with CI and fancy tooling. The habit matters more than the logo on the harness.
How to talk about “better”
Better means: more of the important cases pass, none of the sacred cases fail, and the agent stays inside its lane.
Better does not mean: longer answers, more personality, or a stakeholder saying “I like this vibe” after one cherry-picked chat.
Vibe is allowed in marketing copy. Vibe is not a release criterion for an agent that touches customers.
Bottom line
You know a prompt is better when the same hard cases say so twice.
Evals turn AI work from taste arguments into system work. Systems can improve on purpose. Taste arguments just get louder.
If you ship agents without a suite, you are not moving fast. You are flying without instruments and calling the cloud cover a strategy.
