Answer in brief: Test it in four layers: turn-level assertions in pytest or Vitest, chat simulations, voice simulations over the full STT-LLM-TTS path, then a supervised live pilot. Each layer is blind to the faults the next one catches. To be 95% confident of catching a fault that fires on 1 call in 20, run that scenario 59 times.
How do I test my AI voice agent before it takes real calls?
Run the Pre-Flight Ladder: four layers, each of which catches a class of failure that no earlier layer can structurally see. The layers are not interchangeable. Both vendors quoted below are candid about the limits of their own harness — Vapi gives it a section heading, Know what automated tests cover, and LiveKit points you at four partner services for the full audio path — but each documents its own tooling, and neither page tells you how many runs it takes before a passing scenario means anything. That number is the second half of this page.
The ladder below is built from what LiveKit and Vapi publish about their own tooling, read on 10 September 2026, plus the telephony layer neither harness reaches.
| Layer | What it is | What it catches | What it cannot catch | What a run costs |
|---|---|---|---|---|
| 1. Turn assertions | Unit tests over a single agent turn, run in LiveKit’s pytest/Vitest test framework or Vapi Evals | Wrong tool called, wrong arguments, missing required question, a refusal that should have fired, a handoff to the wrong agent | Anything downstream of text: mishearing, mispronunciation, barge-in timing, real latency, carrier behaviour. Also any code path that calls get_job_context() |
Seconds. Runs locally and in CI. No call minutes, no speech spend |
| 2. Chat simulations | A synthetic caller with a goal, holding a complete conversation in text | Multi-turn flow, memory across turns, misuse resistance, recovery when the caller changes their mind, whether the outcome was actually reached | Every speech-path failure. A perfect transcript-mode pass says nothing about what the STT would have heard | Minutes. Tester tokens plus judge tokens. Cheapest layer that tests an outcome rather than a turn |
| 3. Voice simulations | The same scenarios rerun over audio, or through a dedicated audio-path tester | Transcription errors, pronunciation of names and addresses, pacing, turn-taking and interruption handling, end-to-end latency | Real caller environments. Vapi states plainly that “synthetic callers do not fully represent real callers and their environments” — background noise, degraded audio, a range of accents, silences | Real call minutes plus speech spend, and wall-clock time bounded by your parallelism |
| 4. Supervised live pilot | Controlled real calls on the production number, with a human listening and a kill switch | Phone-number routing, carrier behaviour, voicemail detection, caller ID presentation, what real people actually say in the first eight seconds | Rare faults. A pilot is volume-limited, and it is not reproducible, so it can never serve as your regression gate | Staff time on live calls, and reputational exposure on every call you do not intercept |
The rule that falls out of the table: a test layer only proves the things it actually executed. Layer 1 executed a string. Layer 3 executed a waveform. They are not evidence about each other.
Why a green test suite still ships a broken agent
The single most common pre-launch failure is an agent that has passed hundreds of tests and has never once been through speech-to-text.
This is not a criticism of the harnesses; it is stated in their own documentation. LiveKit’s testing page says: “The test framework runs in text mode, using an LLM through LiveKit Inference or a model plugin.” On simulations it says: “Simulations communicate with your agent using text by default, and can also run over audio to exercise the full STT-LLM-TTS pipeline.” And its recommendation is explicit about the trade: “Text mode is the most cost-effective and deterministic way to test agent behavior, so use it while you iterate and reserve audio runs for turn-taking and speech-specific issues.” All three sentences are on docs.livekit.io/agents/start/testing, read 10 September 2026.
Two details on that same page are worth reading before you plan a suite around it. First, the page is not consistent about simulation mode. Its Text-first testing section says simulations “can also run over audio to exercise the full STT-LLM-TTS pipeline,” and its testing-options table lists agent simulations as running “over text or audio.” The third-party tools section further down opens with a flat statement to the contrary: “First-party simulations run in text mode. To test the full audio pipeline or monitor deployed agents in production, consider these third-party services” — and then names Bluejay, Cekura, Coval and Hamming. All three passages were on that one page when we read it on 10 September 2026, and we are not going to resolve the discrepancy on LiveKit’s behalf: check your own run output for the mode a given simulation actually used, rather than inferring it from any single sentence. Second, “Agent simulations are in beta and currently support Python agents only,” and they “run on LiveKit Cloud, in parallel up to your project’s concurrency limit,” which is what sets your wall-clock time in the calculation below.
Vapi splits the same problem differently and publishes its own comparison: an Eval answers “At this point in the conversation, did the agent make the right next decision?” against a fixed context, while a Simulation answers “By the end of the conversation, did the agent reach the right outcome?” with a tester that adapts. Its coverage caveat is the sentence to put on the wall: “Evals test text-based decisions. Voice Simulations exercise speech recognition, voice output, and conversation timing, but synthetic callers do not fully represent real callers and their environments.” Read on docs.vapi.ai/test/voice-testing, 10 September 2026.
So the ladder carries one hard gate. Call it the Audio Floor rule: no scenario reaches production having never been run through speech. If a scenario is worth testing at all, at least one of its runs goes over the full STT-LLM-TTS path, on the number and carrier that will carry it in production. Text-mode iteration is correct and cheap; text-mode sign-off is not sign-off. If your callers are Australian, the audio layer is where that shows up first, and the method is worked end to end in our post on testing Australian accent accuracy and word error rate.
How many test runs do I need before I trust a scenario?
Neither vendor page above answers this with a number — Vapi says to use multiple iterations for critical scenarios and stops there, and LiveKit gives no run count anywhere on the page we read on 10 September 2026. The question does have an answer. A voice agent is non-deterministic: the same input can pass once and fail the next time. One clean run is not evidence.
Treat each run as independent, and assume the fault you are hunting fires with probability p on any given run. The chance that n clean runs all missed it is (1 − p)n. Set that equal to your acceptable miss rate and solve:
n = ln(1 − C) ÷ ln(1 − p), where C is the confidence you want.
Worked, for a fault that hits 1 call in 20 at 95% confidence: ln(0.05) = −2.9957; ln(0.95) = −0.05129; n = 58.4, so 59 runs. That is the Rule of 59, and it is the number to quote when someone proposes signing off a scenario on three good calls.
| Fault rate you cannot afford to miss | Runs for 90% confidence | Runs for 95% confidence | Runs for 99% confidence |
|---|---|---|---|
| 1 in 5 (20%) | 11 | 14 | 21 |
| 1 in 10 (10%) | 22 | 29 | 44 |
| 1 in 20 (5%) | 45 | 59 | 90 |
| 1 in 50 (2%) | 114 | 149 | 228 |
| 1 in 100 (1%) | 230 | 299 | 459 |
Read it the other way and it is more sobering: a scenario that has passed 20 times in a row is still consistent with a fault rate of about 14% at 95% confidence, because 1 − 0.051/20 = 0.139. Twenty clean calls feels like proof. It is not.
Vapi says the same thing in words on its testing page: “Use multiple iterations for critical scenarios because one pass does not prove consistent behavior.” The table just tells you how many.
Now cost it out. Take five critical scenarios at 59 voice runs each: 295 runs. Assume a three-minute scenario — substitute your own average handle time — and that is 885 call-minutes of speech per release. LiveKit runs simulations “in parallel up to your project’s concurrency limit”; at a concurrency of 10 that is roughly 89 minutes of wall clock. At 40 scenarios the same arithmetic gives 2,360 runs and 7,080 call-minutes, which is no longer something you fit between a merge and a deploy.
Two honest caveats on the model. It assumes runs are independent and that p is stable; a fault triggered by one specific carrier, one accent or one time of day is not random, and no number of synthetic runs will surface it if your generator never produces that input. Sampling frequency does not fix a coverage gap.
What do I do on day one, and what does the suite look like at go-live?
Day one, before you write a single test: open your last 20 real or demo transcripts and list every point at which the conversation went wrong. If you have no calls yet, write down the five sentences a caller says most often. That list is your scenario backlog, and it beats any invented test plan, because it is drawn from behaviour rather than imagination. Vapi puts the discipline well: “When you find a real failure, add it to your test set so the same issue gets caught next time.”
Then, in order:
- Write the refusals and the required questions first (layer 1). Consent and disclosure lines, identity verification before any account action, the tool that must never fire without a confirmed booking date. These are the assertions that protect you legally, and they are the cheapest tests you will ever write. Our guide to human-in-the-loop controls for AI sales agents covers which actions should never be fully autonomous in the first place.
- Assert tool arguments, not just tool names. Most production incidents are a correctly chosen tool called with a mangled argument — a date parsed to the wrong year, a phone number missing its country code.
- Add an LLM-judge assertion for tone and intent where an exact string match would be brittle. LiveKit exposes this as
.judge(llm, intent="...")on the expected event. - Promote the top scenarios to chat simulations (layer 2). Give the synthetic caller a goal and let the path vary. If your test only passes when the agent asks questions in one particular order, it is testing your script, not your outcome.
- Run the audio layer (layer 3) on the accents, names and addresses your callers actually use. This is where latency budgets get verified too; the thresholds are in our post on sub-second response latency for voice AI.
- Check the suite into the repository and wire it to CI. LiveKit notes that “A checked-in scenarios.yaml is reproducible, so you can run the same scenarios on every change” — reproducibility is the whole point, and a suite that lives on one engineer’s laptop is not a gate. Note the documented gap:
get_job_context()“is unavailable in test environments and raises a RuntimeError when called,” so any path that calls it needs mocking or it is simply untested. - Book the supervised pilot (layer 4) before you announce a date. Real calls, real number, human listening, kill switch armed. This is also where you discover telephony problems no harness reaches; Vapi is explicit that you should “Place controlled test calls when you need to verify phone-number routing, carrier behavior, voicemail, or other telephony setup.” Registration and verification lead times are the usual reason this step slips — we mapped them in our AI voice agent US go-live timeline.
The finish state. You are ready when: every critical decision has a checked-in layer-1 assertion; every revenue-carrying outcome has a layer-2 simulation; your top five scenarios have layer-3 audio runs at the count the Rule of 59 gives you; the suite runs automatically on every prompt, model, voice or transcriber change; and a supervised pilot has produced at least one failure you fixed. If nothing failed in the pilot, your pilot was too small.
After go-live the ladder does not retire — it inverts, and production becomes your richest source of new scenarios. What to instrument for that is a separate job, covered in our three layers of AI agent observability.
What does running this actually cost, and when should I hand it over?
The method above is complete and genuinely usable — a competent engineer can build all four layers. What changes the answer is not difficulty, it is the size of the matrix and how often it has to be rerun.
| Signal | Run it yourself | Hand the testing layer to a platform |
|---|---|---|
| Scenario count | Under about 10 critical scenarios | 40 or more — that is 2,360 voice runs per release at the Rule of 59 |
| Change frequency | Prompt changes weekly or less | Prompts change automatically, so the suite must run on a schedule, not on commit |
| Languages | One | Every language multiplies the whole matrix; a 30-language agent has 30 audio paths, not one |
| Regulated content | No consent, disclosure or payment handling | Any scenario where a wrong turn is a compliance event, not a bad call |
| Audio-path coverage | Manual calls plus a handful of recordings | You need repeatable audio runs at volume and your framework defaults to text |
| Who maintains it | The engineer who wrote the agent | Nobody, currently — if that is the honest answer six weeks after launch, the suite has already stopped being a gate |
That second row is the one teams miss. Continuous optimisation and regression testing pull against each other: Zian’s autonomous sales agents include PrecisionPitch AI™, which continuously split-tests scripts and approaches against real success outcomes, and SmartReach AI™, which varies message, channel and timing by country and profile. When the agent’s own behaviour is being optimised without a human editing a prompt, “run the suite on every commit” stops being sufficient, because there is no commit. The suite has to run on a schedule and compare against a baseline. Zian also runs 30+ languages and can be deployed as a private model on a customer’s own infrastructure, both of which multiply the audio matrix rather than the text one.
Demand for a supported answer here is documented rather than assumed. In livekit/agents issue 6707, opened 5 August 2026 and still open, a builder asks the maintainers for the supported pattern for “post-call diagnosis and repeatable regression testing”, scoring turn handling, transcript integrity, tool reliability and response latency. On the Pipecat side the tooling is moving under teams as they build on it: PR 5414, “feat(cli): accept scenario directories in eval run”, merged 25 August 2026, and PR 5520, “Report eval progress through BaseObject events”, merged 2 September 2026. If you are building the harness yourself, budget for the harness itself changing.
Where each figure on this page comes from
Who owns each figure, so nothing here can be traced back to us that is not ours. Every external claim was opened at the owner’s own URL on the date shown; nothing is taken from a summariser or a comparison blog.
| Figure or claim used above | Owner (organisation) | Source URL | Date checked |
|---|---|---|---|
Test framework runs in text mode; simulations text by default with optional audio; simulations in beta and Python-only; parallel up to project concurrency limit; checked-in scenarios.yaml is reproducible; get_job_context() raises RuntimeError in tests; partner services Bluejay, Cekura, Coval, Hamming |
LiveKit | docs.livekit.io/agents/start/testing | 10 September 2026 |
| Evals versus Simulations core questions and comparison table; one successful test call is not enough; synthetic callers do not fully represent real callers and their environments; use multiple iterations for critical scenarios; place controlled test calls for telephony verification | Vapi | docs.vapi.ai/test/voice-testing | 10 September 2026 |
| Open request for a supported reliability-scoring and regression-testing pattern, filed 5 August 2026 | livekit/agents contributor (public issue tracker) | github.com/livekit/agents/issues/6707 | 10 September 2026 |
| Eval CLI accepts scenario directories (merged 25 August 2026); eval progress reported through BaseObject events (merged 2 September 2026) | pipecat-ai maintainers (public repository) | pull/5414 and pull/5520 | 10 September 2026 |
| The Pre-Flight Ladder, the Audio Floor rule, the Rule of 59, the run-count table, the 14% figure and the 295-run and 2,360-run worked examples | Zian AI — original to this page | Derived here from n = ln(1 − C) ÷ ln(1 − p); no external source, reproduce it yourself | 10 September 2026 |
| Zian capabilities named above: SmartReach AI™, PrecisionPitch AI™, 30+ languages, private model deployment | Zian AI | zian.ai/features-integrations | 10 September 2026 |
Frequently asked questions
More questions of this kind are answered on our AI sales agent FAQ hub.
Can I just make a few test calls myself before go-live?
Not as sign-off. Vapi states on its Testing voice agents page, read on 10 September 2026, that because the agent can respond differently to the same caller, one successful test call is not enough, and that a small change to any part of the stack can improve one conversation and break another. Manual calls remain valuable for the things no harness reproduces: background noise, degraded audio, accents and interruptions. Use them as evidence, not as a gate. See the Vapi testing documentation.
Do text based tests catch speech problems?
No. LiveKit documents that its test framework runs in text mode, and that simulations communicate with the agent using text by default and can also run over audio to exercise the full STT-LLM-TTS pipeline. A test that never produced a waveform cannot tell you what the transcriber would have heard. That is why the ladder treats voice runs as a separate layer with its own gate.
How many times should I run a scenario before go-live?
Use n = ln(1 minus C) divided by ln(1 minus p). For a fault that fires on 1 call in 20, at 95% confidence, that is 59 clean runs. Twenty consecutive passes is consistent with a fault rate as high as about 14%. The model assumes runs are independent and that the fault rate is stable, so it does not help with a fault that only appears on one carrier or one accent.
Can I run voice agent tests in CI?
Yes, and that is the point of checking scenarios into the repository. LiveKit documents that a checked-in scenarios.yaml file is reproducible so the same scenarios run on every change, and that agent simulations are in beta and currently support Python agents only, running on LiveKit Cloud in parallel up to your project concurrency limit. One documented gap to plan around: get_job_context() is unavailable in test environments and raises a RuntimeError when called, so mock it or avoid those code paths.
Does any of this apply if I am not on LiveKit or Vapi?
Yes. The four layers are a property of the stack, not of a vendor: any agent that turns speech into text, text into a decision and a decision back into speech has the same four failure surfaces. Only the tooling names change. If your platform publishes no test harness at all, layers 1 and 2 can be built against its API and layer 3 through a dedicated audio-path tester.
What should I test on the morning of go-live?
Three things, in this order: place a live call to the production number from a phone on a different carrier and confirm routing and caller ID; run the refusal and disclosure assertions once more against the exact prompt version being deployed; and confirm the kill switch and the escalation path work with a human actually on the other end. Everything else was tested earlier. These three change at deploy time.
Zian AI runs autonomous phone, SMS, email and WhatsApp agents with SmartReach AI™ and PrecisionPitch AI™, in 30+ languages, with API and CRM integrations and private model deployment. Zian is in partnership-application beta — there is no free trial and no self-serve signup. Apply For Partnership.