I shipped prompt changes for months on vibes.
Sometimes the new prompt felt sharper. Sometimes it quietly broke a case we had fixed two weeks earlier. We would discover the regression from a customer-shaped bruise instead of from a test.
The eval harness I wish I built earlier is not fancy. It is a folder of golden prompts, expected traits, and a scored diff before anything hits production.
What “golden” means here
A golden prompt is a real example you care about, frozen as a fixture.
Not a toy. Not “write a poem about APIs.” A messy real request. A borderline escalation. A structured extraction that must not invent fields. A coding brief that must stay inside scope.
Each golden item includes:
- input
- context fixtures (files, snippets, metadata)
- must-have traits
- must-not traits
- a note on why this case exists
That last field stops people from deleting hard cases because they are annoying.
Scored diffs, not vibes
When I change a system prompt or a tool schema, you run the suite and compare:
- did must-haves still pass?
- did must-nots newly fail?
- did latency and cost move a lot?
- did JSON validity drop?
Do not need a neural judge for everything. Exact checks, regex checks, schema validation, and a small rubric for open-ended outputs get you most of the value.
If you want a model grader later, fine. Start with boring assertions first. Boring assertions ship.
The harness shape that worked
A simple layout:
“` evals/ cases/ intake_escalation.json extract_fields.json coding_small_diff.json run.py baselines/ 2026-10-01.json “`
run.py executes each case against the candidate config, writes results, and exits non-zero if score drops below a threshold.
That non-zero exit is the whole culture change. Prompt PRs become like code PRs. They can fail CI.
Why waited too long
I told myself we were “still exploring.” Exploring without a net is how you explore the same cliff twice.
Also underestimated how fast prompt tweaks accumulate. One helpful sentence at a time becomes a different product. Without goldens, you cannot see the drift.
A minimum viable suite
If you build only ten cases, make them these categories:
- Happy path core job
- Missing information
- Hostile or junk input
- Escalation / handoff
- Schema integrity
- Refusal on disallowed advice
- Long context distraction
- Multilingual or messy formatting if you see it in prod
- Regression from a past incident
- Cost-sensitive verbose failure (model rambling)
Ten cases will already embarrass most “quick prompt edits.”
Human review still matters
Evals catch mechanical regressions. Humans still judge taste.
Use the harness to stop obvious breakage, then sample a few outputs for voice and judgment. The harness is a seatbelt. It is not the driver.
Checklist to build yours this week
- Capture five real production-like prompts
- Write must-have and must-not traits for each
- Script a runner that prints a table
- Save a baseline score
- Require the runner on prompt changes
Story
We once “improved” tone and accidentally softened a crisis escalation phrase. A human eventually caught it. An eval case would have caught it in thirty seconds. That sentence still makes my stomach drop.
Never again on vibes alone.
Closing
The eval harness I wish I built earlier is simple golden prompts and scored diffs before shipping prompt changes.
Build it while the system is small. Small systems become big systems that are afraid to change. A harness makes change safer, which means you can keep improving instead of freezing a lucky prompt like a museum piece.
Keep cases fresh without thrash
Add a case when production surprises you. Retire a case only when the product behavior intentionally changes and the old expectation is wrong.
A stale suite that blocks good changes is almost as bad as no suite. Schedule a monthly twenty-minute review. Small garden care beats a yearly rewrite.
Also store failing outputs as artifacts. Future you will want the receipt, not a vague memory that “it got weird.”
Closing
Build the boring harness. Let it fail your clever prompt. Then ship the version that still passes the goldens. That loop is how AI ops grows up.
Who owns the suite
Bottom line
Give the eval suite an owner, even if the owner is you one hour a month. Orphan suites rot. Owned suites improve. If a prompt change fails CI, the author should update either the prompt or the case with a written reason. Silent overrides are how you recreate vibes with extra YAML.
