Most likely your eval never used audio. In Pipecat, user.modality and judge.modality both default to text, an expectation with no within_ms gets a 60 second budget, and the judge sees tool calls but not their results — while a Twilio Media Streams leg is mulaw at 8,000 Hz. Nine blind spots, one test.
Why my voice agent passes every eval but still gets it wrong on real calls
A green suite is not a claim about your agent. It is a claim about the artefacts your harness produced, and by default those artefacts are text. The sentence that brings people here — “my voice agent passes every eval but still gets it wrong on real calls” — is almost always a fidelity gap rather than a coverage gap: the scenario was tested, through a pipeline that removed the thing that broke. Everything below is read out of Pipecat, LiveKit Agents and Vapi, and out of the Twilio Media Streams specification for the carrier leg: the RTVI text path your tester used, the DTMF frames it injected, and the G.711 narrowband audio your callers were actually on.
Call the underlying rule the marking-your-own-homework rule: if the artefact your evaluator reads was produced by the same pipeline you are testing, the evaluation cannot detect a fault in that pipeline. A judge reading llm_response cannot find a text-to-speech fault, because the text-to-speech never ran. A judge reading a transcript your own aggregator assembled cannot find a fault in that aggregator. It is the evidence standard we apply to vendor claims in why there are no trustworthy AI SDR benchmarks: a number is evidence only if something outside the system under test produced it.
This page diagnoses a suite that is already green. If you have not built one yet, the four-layer ladder and the run-count arithmetic live in how to test an AI voice agent before go-live; that ground is ceded here and not repeated.
Nine eval blind spots, ranked by how often they are switched on by default
Ordered by frequency, not severity — and the honest frequency proxy is how many suites have the blind spot active without anyone choosing it. Every mechanism below is quoted from framework source or vendor documentation read on 16 September 2026, with the file or issue named so you can check it.
| # | Blind spot | What the evaluator saw | What the caller got | Mechanism, at source |
|---|---|---|---|---|
| 1 | The judge never heard the bot | The raw LLM text | A mangled surname, a postcode read as a year, a clipped ending | Pipecat src/pipecat/evals/script.py: judge.modality “text (the default) skips TTS and judges the LLM text (llm_response), which is faster and silent” |
| 2 | The tester typed; speech recognition never ran | A perfect input string | Whatever the recogniser heard, accent and all | Same file: audio “streams synthesized user audio to the bot (exercising its STT for real); text (the default) sends RTVI send-text“ |
| 3 | Nobody talked over the bot | A polite alternating conversation | A caller who cuts in at 500 ms and derails the turn | Same file: “A turn is sent once the bot has finished speaking, like a caller who waits for the end of the sentence”. Barge-in has to be scheduled explicitly with a send_after: block, the documented example being {event: llm_started, delay_ms: 500} |
| 4 | Dead air is invisible below the backstop | A conversation that eventually continued | Silence long enough to hang up on | Pipecat simulation.py: max_silence_s defaults to 30 s and ends the whole run as silence rather than failing a turn, so a shorter lull is never measured |
| 5 | The latency assertion is effectively off | A pass | A pause no human waits through | Pipecat script.py: within_ms is a “latency budget from the most recent anchor (optional; defaults to 60s when omitted)”. In simulation.py, text-mode and audio-mode latency measures are explicitly “not comparable” |
| 6 | The judge graded the tool call, not the tool result | A function name and its arguments | A confident, fluent, wrong readback of backend data | Pipecat simulation.py: “The judge sees the bot’s tool calls (name and arguments), not their results.” Added after issue #5664, closed as completed on 8 September 2026 |
| 7 | The transcript the eval read is not what was said | A clean, complete transcript | Holding lines and late text missing or attached to the wrong utterance | pipecat #5639 (open as at 16 September 2026): lines spoken with append_to_context=False never reach the turn events transcripts are built from. livekit/agents #7227, same class, closed 14 September 2026 by merged PR #7264 |
| 8 | Keypresses and transfers leave the tested pipeline | A synthetic frame injected straight into the agent | Carrier-negotiated tones, and a leg that your harness cannot follow | Pipecat script.py: each DTMF character is sent as one InputDTMFFrame “regardless of the scenario’s user/judge modality” — the audio path and the carrier are both bypassed |
| 9 | Test audio was never telephone audio | Clean synthesised speech at the eval sample rate | 8 kHz narrowband, packet loss, jitter | Twilio Media Streams: the encoding “is always audio/x-mulaw“ and the sample rate “is always 8000“, one channel |
Rows 1 to 5 are defaults: nobody chose them. Row 6 is deliberate, because a simulation judges the conversation from the caller side and a caller cannot see your backend either. Rows 7 to 9 depend on how you deploy. Vapi states the residual limit for its own harness in the same terms: synthetic callers “do not fully represent real callers and their environments”, and cannot reliably reproduce “background noise, degraded audio, a range of accents, silences, and interruptions” (docs.vapi.ai/test/voice-testing, read 16 September 2026).
The One-Variable Replay: the test that tells you which blind spot is yours
Take one real call that failed. Do not add scenarios yet. Replay that one conversation through your own harness, changing exactly one fidelity variable per run, and stop at the first run that fails. That run names your cause. Changing two variables at once turns a diagnosis back into a guess.
| Run | The one change | If it fails here, your cause is | Then do this |
|---|---|---|---|
| A | Replay the failed call as a scripted scenario, text in, text judged | Coverage, not fidelity: the scenario was never in the suite | Keep the scenario and stop |
| B | judge.modality: audio, so the bot speaks and the harness transcribes its real audio |
Blind spot 1: pronunciation, pacing or a truncated utterance | Assert on response, not llm_response |
| C | user.modality: audio, so persona turns pass through your recogniser |
Blind spot 2: recognition error on names, numbers or accent | Fix at the recogniser, not the prompt |
| D | Replace the synthesised caller with the real recording via audio:, at 8 kHz mulaw |
Blind spot 9: narrowband audio, noise or codec artefacts | Test on the carrier leg you will ship on |
| E | Add one barge-in turn: send_after: {event: llm_started, delay_ms: 500} |
Blind spot 3: turn-taking under interruption | See why your AI voice agent interrupts callers for the endpointing side of this |
| F | Rewrite the success criterion to name the backend value the tool returned | Blind spot 6: the judge graded intent to look something up | Write the expected value in and keep mocks deterministic |
Run F is the one people skip, and the one Pipecat documented. The issue author wrote success: criteria expecting the judge to notice a reply that contradicted a tool result — a wrong next-available time, a cancelled appointment read back with the wrong date — and it could not. If the replay passes A to E and fails F, your agent is calling the right function and then saying something else, which extra scenarios will never surface.
If every run passes and real calls still fail, you are in blind spot 7: the artefact itself is wrong. Stop trusting the transcript and read the call from traces and outcome events instead, the approach set out in AI agent observability: traces, logs and outcomes.
How much of my suite actually ran audio?
Before rewriting anything, measure the suite you have. Three counts, from the scenario directory:
grep -rl "modality: audio" scenarios/ | wc -l # audio-path scenarios
grep -rl "send_after:" scenarios/ | wc -l # barge-in scenarios
grep -rlE "audio:" scenarios/ | wc -l # real-recording scenariosWorked, with inputs you substitute for your own. Say a 40 scenario suite returns 3, 0 and 0. Audio coverage is 3 divided by 40, or 7.5%. Barge-in coverage is 0%. Real-caller-audio coverage is 0%. The suite therefore reports a pass rate for the text path on 92.5% of its scenarios, and says nothing at all about turn-taking under interruption.
Now weight it by traffic, which is what predicts customer impact. If the three audio scenarios cover 12% of call volume, 88% of your traffic carries text-only assurance: at 4,000 calls a month that is 3,520 calls covered only by a test that never made a sound.
Two caveats. The greps count declared intent, not executed runs, so check the run output for the mode each scenario actually reported. And a scenario that runs once is an anecdote: Pipecat defaults runs to 1, and the docstring says why — “a persona does not say the same thing twice, so one run is an anecdote and three are a check.”
What each fix costs to run, and when to hand it over
Every fidelity fix above is available to you and none requires a vendor. Raising modality to audio is a two-line change per scenario file; adding a barge-in turn is one send_after block. The cost is not the edit, it is the running: audio-mode runs consume speech spend on every execution, wall clock is bounded by your concurrency limit rather than by CPU, real recordings must be curated and re-cut whenever the flow changes, and dead air thresholds move with every model swap — the mechanisms behind those are in why your AI voice agent goes silent on tool calls.
The threshold below is a decision rule, not a measurement — it is built from what the suite must cover, and you should substitute your own numbers.
| Your situation | Do it yourself | Hand it over |
|---|---|---|
| Under about 5 scenarios, one language, no backend lookups | Yes — a scripted suite plus one supervised live call per release | No |
| About 5 to 40 scenarios, one country, tool calls reading customer data | Yes, but budget an engineer on scenario files plus recurring speech spend | Only if nobody owns the suite |
| Over about 40 scenarios, or more than one language or country | Maintenance becomes the job, not a task inside it | Yes |
| Regulated data that cannot leave your infrastructure | Only if your harness runs inside the boundary too | Only to a platform that deploys inside it |
That last row decides most enterprise evaluations, because a harness that ships call audio to a third party inherits the same data boundary as the agent. Zian AI runs live phone, SMS, email and WhatsApp agents in 30+ languages and supports private model deployment on customer infrastructure, which is the configuration in which the audio leg you test and the audio leg you ship are the same leg. Whichever way you go, ask your suite what this page asks: what produced the artefact the evaluator read?
Frequently asked questions
Why does my voice agent pass every eval but fail on real calls?
Because the eval most likely ran in text mode. In Pipecat both the user modality and the judge modality default to text, so the speech recogniser never ran and the judge read raw model output rather than a transcription of synthesised audio. The scenario was covered; the failing layer was removed before it was graded.
Does a passing eval mean my agent handles interruptions?
No. Pipecat sends each turn only once the bot has finished speaking, which is a caller who waits politely for the end of the sentence. Barge-in has to be scheduled deliberately with a send_after block relative to an earlier event. If no scenario contains one, nothing in the suite ever talked over the agent.
Can an LLM judge see what my tools returned?
Not in a Pipecat simulation. The documentation states that the judge sees the tool calls, name and arguments, but not their results, and this was written into the simulation module after issue 5664, closed as completed on 8 September 2026. The fix is to write the expected backend value into the success criterion and keep mocks deterministic.
Why does my suite report good latency when callers complain about delays?
Two reasons. An expectation written without an explicit within_ms value gets a 60 second budget by default, which almost nothing fails. And a latency figure measured in text mode counts time to first token, while an audio-mode figure counts time to the first spoken sentence. The Pipecat documentation says plainly that the two are not comparable.
Is telephone audio different from the audio in my eval?
Yes, and materially. Twilio Media Streams documents that the encoding is always audio slash x-mulaw at a sample rate of always 8000 hertz, in one channel. A harness that synthesises clean speech at a higher rate is not exercising the narrowband path your callers are on.
My transcript looks complete. Can I trust it?
Only if something outside the agent produced it. Two dated examples of the opposite: pipecat issue 5639, open as at 16 September 2026, covers spoken lines that never reach the events transcripts are built from, and livekit agents issue 7227 covered late transcripts attaching to the wrong audio burst until it was closed on 14 September 2026 by merged pull request 7264.