The API usually gets the blame. Often it is not the API. In livekit/agents issue 5826, open since 24 May 2026, a builder measured 590–1000 ms of dead air per tool turn inside the framework’s streaming layer, on models that speak before they call a tool. Measure three clocks first.
This page is about one specific silence: the gap a caller hears when your voice agent decides to look something up. The general anatomy of a turn — endpointing, ASR, time-to-first-token, TTS, telephony transport, and the perception research on how long a human tolerates a pause — is covered in our post on voice AI latency and sub-second responses and is not restated here. What to say during a lookup, and how to pre-fetch most of what agents fetch live, is covered in what a mid-call knowledge lookup actually costs. This page answers the question those two do not: why is the silence there, and which layer is producing it?
Why does my AI voice agent go silent for a few seconds when it looks something up?
Because a tool turn is not one wait, it is a chain of them, and the chain contains a step most teams never instrument. The model has to decide to call the tool, serialise the arguments, hand them to the framework, wait for your HTTP round trip, read the result, and only then generate speech that a TTS engine can start streaming. Every one of those steps can be the dominant term, and the two that most often are — argument serialisation and framework buffering — produce no log line in your API and no metric on your LLM provider’s dashboard.
That is why “the CRM is slow” is such a durable misdiagnosis. A HubSpot or Salesforce lookup that returns in 180 ms server-side can still sit inside two and a half seconds of caller-audible silence, and nothing in the CRM’s own timing tells you so. The silence a caller hears and the duration your API reports are different measurements, and on tool turns they routinely disagree by a factor of five or more.
The three-clock test: which clock is running while the caller hears nothing
This is the diagnostic we would run first, and it costs one afternoon of logging rather than a re-architecture. Three clocks run during a tool turn. Log all three per turn, with the same time source, then compare them.
| Clock | Where you read it | What it actually measures | If this is the biggest term |
|---|---|---|---|
| The tool’s clock | Your own API server: request received → response written | Only the work your service did. Excludes DNS, TLS setup, queueing and the network both ways. | Optimise the endpoint, add a warm connection pool, or pre-fetch the field before the phone rings. |
| The model’s clock | Your LLM provider’s timing: request sent → first token, and first token → last token of the tool-call arguments | Time-to-first-token plus the time the model spends writing the JSON arguments — a term that grows with argument size and with long tool schemas. | Shorten the prompt, cut unused tools out of the schema, or route the tool-selection turn to a smaller model. |
| The stream’s clock | Your agent framework: last text delta received → first TTS audio frame emitted | The framework’s own buffering between the model and the speech engine. Invisible to both vendors. | You have a stack problem, not a latency problem. livekit/agents 5826 is the documented instance. |
The decision rule: if your tool’s own server-side duration accounts for less than half the silence the caller heard, stop optimising the API — the rest of the gap is in your stack. We call it the three-clock test because two clocks are not enough: a team that logs only the caller-audible gap and the API duration knows a gap exists and has no idea where it lives.
What this costs, honestly: per-turn structured logging on three surfaces, one shared time source so the numbers are comparable, and roughly a week of production calls before a p95 means anything. It is a real engineering task, not a configuration toggle, and it is the only way to know which of the fixes below is the one you need.
The measured case where the silence was not the API at all
The number worth knowing here comes from a named person, on a named version, with a method attached. In livekit/agents issue 5826, GitHub user bnovik0v reported on 24 May 2026 that inference.llm.LLMStream._parse_choice in livekit-agents v1.5.7 (and then-current main) silently drops every streaming tool-call delta until finish_reason arrives. A streaming TTS engine holding the model’s spoken preamble therefore has nothing to flush on, and waits out the entire argument-serialisation window.
The measurement, verbatim from the issue: “~590–1000 ms of dead air before the agent speaks the preamble on every tool turn. Measured end-to-end against live Cartesia.” The reporter’s environment was a Qwen mixture-of-experts model served on sglang through an OpenAI-compatible endpoint, with Cartesia streaming TTS, exercised through production code paths rather than a synthetic benchmark. A sample timeline in the issue shows the flush at 1,941 ms, the first audio frame 182 ms later at 2,123 ms, and the LLM stream not ending until 2,563 ms — audio starting roughly 440 ms before the stream that was previously gating it.
Scope this honestly, because it does not hit everyone. The report is specific to models that emit a text preamble before a tool call in the same assistant turn — Hermes and Qwen chat templates, common on vLLM and sglang and on Hermes-style fine-tunes. If your model never speaks before it calls a tool, this particular defect is not your silence. The transferable finding is the class, not the instance: a buffering step between the model and the speech engine can hold audio that was ready to play, and neither vendor’s dashboard will show it.
Status as we read it on 10 September 2026: the issue is open, last active 23 July 2026. A community pull request, livekit/agents 6387, was opened on 11 July 2026 proposing a marker chunk at the first tool delta; it is also still open and unmerged. Treat both as reported and unresolved rather than as settled behaviour, and verify against the version you actually run.
One thing to note about all three reports on this page: they are public because these are open frameworks with public issue trackers. A closed voice platform can carry precisely the same class of defect with nothing you can read. The absence of an issue tracker is not the absence of the bug.
How long may a tool call take before the caller must hear something?
We have not found this budget published anywhere, so here is ours as a decision table. The bands are anchored to the published perception thresholds set out in our latency post — where the peer-reviewed sources live — and the mitigations are our rule, not a vendor benchmark. Use p95 of the caller-audible gap, not the median: a ten-minute call contains dozens of turns, so your worst one-in-twenty happens several times per call.
| Tool round-trip, p95 (caller-audible) | What the caller experiences | Required mitigation | Why that and not more |
|---|---|---|---|
| Under 400 ms | Nothing unusual. The lookup disappears into an ordinary turn gap. | Nothing. Do not add a filler. | A filler phrase in front of a 300 ms tool makes the turn longer than the silence it was meant to cover. |
| 400 ms – 1.2 s | A pause. Read as thinking, not as a fault. | One short pre-synthesised preamble sentence, played before the request is issued and interruptible. | One sentence covers the gap. A progress update here interrupts a caller who was never worried. |
| 1.2 s – 4 s | Read as a dropped call. Callers start talking, and collide with the reply when it lands. | Preamble plus at least one progress update, and the tool moved to a background or async tool so the agent can keep the conversation going. | Both frameworks ship this: LiveKit’s tool documentation (read 10 September 2026) describes async tools that “run long-running tools in the background so the agent can keep talking”, and Pipecat exposes the same shape via cancel_on_interruption=False. |
| Over 4 s | Hang-ups and talk-over, whatever you play over the top. | Do not do it on the call. Take the work off-call, end the turn with a commitment, and deliver the answer by callback or SMS. | Filler changes perceived latency, not real latency, and it works once. A second filler is an admission. |
| Not known | Unknown, which in practice means unbounded. | Instrument first. You have a measurement problem, and possibly the timeout defect below. | You cannot pick a mitigation for a distribution you have never seen. |
The four-second cut-off is the row people argue with and the one that saves calls: above four seconds at p95, the honest answer is that the work does not belong on the call.
Work out your own tool-turn budget
Here is the arithmetic end to end, with example inputs so you can substitute your own. Only the streaming-buffer row is a published measurement (livekit/agents 5826, cited above), and it only applies if your model speaks a preamble before it calls a tool — if it does not, set that row to zero. Every other figure below is an illustrative placeholder for a number you must measure on your own stack.
| Term | Example input | Where the number comes from |
|---|---|---|
| End-of-turn detection and ASR finalisation | 300 ms | Yours to measure. Endpointing config dominates. |
| LLM time-to-first-token, plus writing the tool arguments | 400 ms | Yours to measure. Grows with prompt and tool-schema size. |
| Streaming-buffer tax on tool turns — only on models that speak before calling a tool | 590–1000 ms (zero if your model does not) | Measured and published: livekit/agents 5826, v1.5.7, 24 May 2026, scoped to Hermes and Qwen style preamble-then-tool turns. |
| Tool HTTP round trip at p95 (your CRM, calendar or pricing service) | 700 ms | Yours to measure, at the caller’s end, not server-side. |
| TTS time-to-first-byte | 150 ms | Yours to measure. Vendor-published figures rarely cover your voice and language. |
| One-way telephony transport | 100 ms | Yours to measure, per region. Media routing across continents stacks onto every turn. |
| Caller-audible silence | 2,240–2,650 ms | Sum of the above. |
| Same budget with the buffering term removed | 1,650 ms | Sum, less the 5826 term. |
Two things fall out of that arithmetic, and both are the point of building it. First, where it applies the framework term is 26–38% of the total silence in this example, which puts it above every other voice-stack component here — endpointing, the model, TTS and transport alike. Second, at the top of its measured range it is larger than the CRM call it is hiding behind — 1,000 ms against 700 ms — so a team optimising the CRM in that scenario is working on the second-biggest term. Run your own numbers before you pick a target. Against the table above, this example agent lands in the 1.2–4 s band, so it needs a preamble, a progress update and a background tool, not a faster endpoint.
The tool timeout that quietly turns itself off
A documented failure worth reading before you trust your own deadlines. Pipecat’s function-calling documentation describes a per-tool timeout_secs and states plainly: “A call that runs past its deadline is cancelled: the handler is thrown an asyncio.CancelledError so it can clean up, the call settles as cancelled, and inference runs so the bot can say it didn’t complete.” The same paragraph narrows what that covers: “The deadline covers the handler’s own execution — work it spawns into a task of its own isn’t cancelled with it.” The same page documents intermediate results for async functions: “They do not close the function call; the call remains in progress until the final result is sent.” (Pipecat function-calling documentation, read 10 September 2026.)
In pipecat-ai/pipecat issue 5481, opened 28 August 2026 by GitHub user mannyb223 and still open on 10 September 2026, those two documented behaviours are reported to collide. The report is that in LLMService._run_function_call the shared result callback cancels the timeout task on every delivery, while only the final one settles the call — so a handler that reports “working on it” a millisecond in permanently disarms its own deadline. The reporter checked this on pipecat 1.8.0 and on current main and published a deterministic repro: with a timeout_secs of 1.0 and one intermediate update, the timeout never fires and the call settles only when the handler returns at about 4.0 s; the control run, identical but without the intermediate update, broadcasts the cancel frame at about 1.0 s as documented.
The pattern to recognise: the progress update you added to reassure the caller is the same call that removed the ceiling on how long they can be left waiting. It is a reported issue rather than a confirmed vendor position, so treat it as a hypothesis to test on your own version — and the test is cheap. Register a handler with a one-second deadline that sleeps for four seconds, run it once with an intermediate update and once without, and check whether the cancel actually fires in both. If your timeout only fires in the control run, every band in the threshold table above is theoretical for you, because nothing is enforcing an upper bound at all.
Silence long enough that the agent stops noticing the caller
There is a third-order failure that only appears on genuinely long tool runs, and it is the reason “just make the tool async” is not a complete answer. In livekit/agents issue 6883, opened 17 August 2026 by GitHub user 0xNuru against livekit-agents 1.6.10 and closed as completed on 21 August 2026, the reporter described a user waiting quietly through a 30–90 second async tool run. The session flipped that user to an “away” state after the idle window, and because the away timer only re-armed while the user was listening, the agent finishing its work and speaking the result did not restart it. Any idle etiquette built on the state-change event — check in once, then say goodbye and close the room — was silently disabled for the rest of the session, and the room stayed open on nobody.
Unlike the other two, this one has been fixed upstream, which changes what you should do about it. LiveKit merged pull request 6937 on 21 August 2026, closing both this issue and the related 6904; the pull request states: “The away timer no longer arms while a tool runs, and the last tool to land restarts the window in full.” The user never enters the away state during a tool call at all. If you are pinned to a build from before that merge, the behaviour above is still yours. Either way the point survives the fix.
A long tool run does not just cost you the caller’s patience; it can also cost you the machinery that was supposed to notice the caller had gone. When you move a slow tool to the background, test what happens to your idle, away and end-of-call logic while it runs — and check how your platform reports the call afterwards, because a session that hangs open shows up in your minutes before it shows up in your transcripts. Related reading: why AI voice agent calls drop and what the ended-reason codes mean, and what to do when an AI voice agent transfer to a human fails.
Where Zian fits
Zian AI is an autonomous AI sales-agents platform running live phone, SMS, email and WhatsApp outreach, with research and knowledge-base lookups and API and CRM integrations including HubSpot, Salesforce, HighLevel and Zapier — which means tool turns, and therefore this problem, are part of the design surface rather than an edge case. Zian does not publish a tool-turn latency figure, or any latency figure, and we would rather you ran the three-clock test against us than read a number with no method attached. For teams whose lookups hit systems that cannot leave their own network, Zian supports private model deployment on customer infrastructure; the features and integrations overview sets out what connects to what, and our AI sales agents FAQ hub collects the rest of these questions in one place.
Zian AI has been running outbound acquisition since 2017 and is in partnership-application beta. There is no free trial and no self-serve signup — a limited number of teams are taken on by application. Apply For Partnership.
FAQ
Why does my AI voice agent go silent when it looks something up?
Because a tool turn chains several waits, and at least two of them are invisible to the API you are calling. The model decides to call the tool, writes the arguments, your framework passes them on, your service responds, the model reads the result, then speech synthesis starts. In livekit/agents issue 5826 a builder measured 590 to 1000 ms of that silence arising inside the framework itself, on models that speak a preamble before calling a tool. If your model never speaks before it calls a tool, that particular defect is not your silence.
Is the dead air my API or my voice framework?
Log three clocks on the same time source: your API server duration, your model provider timing from request to first token, and your framework timing from the last text delta to the first audio frame. If your API duration accounts for less than half the silence the caller heard, the remainder is in your stack and optimising the endpoint will not recover it.
How long can a tool call take before the caller notices?
Use p95 of the caller-audible gap, not the median. Under 400 ms needs nothing. Between 400 ms and 1.2 seconds, play one short pre-synthesised sentence before the request goes out. Between 1.2 and 4 seconds, add a progress update and move the tool to the background. Above 4 seconds, take the work off the call entirely and deliver the answer afterwards.
Does a progress update keep my tool timeout running?
Test it rather than assuming. Pipecat documents that a call running past its deadline is cancelled, and separately that an intermediate result does not close the call. Issue 5481 on the pipecat-ai/pipecat tracker, opened 28 August 2026 and open on 10 September 2026, reports those two behaviours colliding: on pipecat 1.8.0 and current main, a one second deadline never fires once an intermediate update is delivered, and the call settles only when the handler returns at about four seconds.
Should I just play a filler sound during tool calls?
Only in the right band, and only as pre-synthesised audio that the caller can talk over. Routing a filler through text to speech means paying time to first byte on the very thing meant to cover a delay. Filler changes perceived latency, not real latency, and it works once: if the lookup still has not returned, say so and hand the call to a person.
Does Zian publish a tool-turn latency figure?
No. Zian does not publish a headline latency number of any kind, because a single figure without a stated measurement window and method is a claim about nothing. Apply the same test to Zian that this page asks you to apply to every platform on your shortlist: a recorded call with timestamps, the three clocks logged separately, and p95 taken across a week of real calls rather than a demo.