
Applied AI · agent testing
Voice Agent Crash Testing
An AI caller that phones every one of our healthcare voice agents, runs them through every path a real call can take, and grades the result before a patient ever hears it.
The problem
A voice agent for a treatment center that loops, skips a question, or fumbles a crisis call fails a real person on a bad day. Clicking through test calls by hand does not scale across a fleet of 100+ agents, and an AI grader that makes things up is worse than no grader at all.
What I built
- Writes a test spec for each agent from its real configuration, the client profile, and its knowledge base. A strict validator throws out any rubric line that cannot point back to a real source, which kills most false failures before a call is placed.
- Places the calls with an AI caller over text, voice, or a real phone line, with crisis scenarios scripted word for word so every run is repeatable.
- Scores in layers: plain rule checks first, then two AI judges grade independently and must cite the exact turn, with a third breaking ties. Failing a safety check caps the score no matter how good the rest of the call was.
- Calibrated against transcripts I labeled by hand, so the judges are measured against a person, not against each other.
- 600+Scored test runs
- 753Hand-labeled rubric checksAcross 60 transcripts
- 3Ways to callText, voice, real phone














