Voice Agent Latency: What Vendor Numbers Measure

Quick answer

Five vendors, four different stop-points – and most of their clocks start exactly where your stopwatch starts, when the caller goes quiet. So the gap is usually at the far end, not the near one: Deepgram publishes 100-500 ms for the end-of-turn decision alone, and none of the five publishes a turn-level figure that runs to the audio your caller actually hears.

A figure can be accurate and still be nothing like what you measured. The market is full of millisecond numbers and short of definitions, so this page supplies the definitions for the five whose pages publish one: Vapi, Retell AI, Deepgram Flux, LiveKit Agents and Pipecat. Where each clock starts, where it stops, and which legs of a real phone call sit outside it. Every figure below was read on the owner’s own page on 20 September 2026.

Where does a vendor’s latency clock start and stop?

A latency figure is two timestamps, and the start one is the half nobody prints. Four of the five start their turn-level clock in the same place, the caller’s speech ending, then stop in four different places; one starts much later, at its own transcriber. Ranking numbers that stop in different places is not a comparison, it is a unit error.

Stop-point	Clock starts	Clock stops when	Published figure or metric (owner’s own page)	Outside that number
1. Turn decided	The caller’s speech ends	The end-of-turn event fires	Deepgram Flux: ~260ms end-of-turn detection, and 100-500 ms in Deepgram’s own streaming latency guide	LLM, TTS, network back, playout
2. First LLM token	The model request is sent	The model emits its first token	Vapi’s Model component, Time to first token; LiveKit llm_node_ttft; Deepgram ttt_text_latency	Everything before and after the model
3. First TTS byte	Varies: the transcriber (Vapi) or the caller’s speech ending (LiveKit, Deepgram Agent API)	The voice model returns its first audio byte	Vapi’s displayed total, simply the sum of the three component medians; LiveKit’s documented sum end_of_utterance_delay + llm.ttft + tts.ttfb; Deepgram total_latency, End-to-end: user utterance end to first audio byte	Leading silence, network back, playout
4. Agent starts speaking	The caller’s speech ends	The first frame is handed to the transport	Retell AI, latency as low as 600ms, measured from when the user stops speaking to when the AI agent begins responding; LiveKit e2e_latency; Pipecat UserBotLatencyObserver	Return carrier leg, playout at the handset
5. Caller hears a word	The caller’s speech ends	Audio is audible to the person on the call	None of the five publishes a turn-level figure that stops here. LiveKit’s playback_latency is near-zero unless a remote avatar worker is in the chain and reports playback via the lk.playback_started RPC	Nothing. This is your stopwatch

The distance between stop-point 1 and stop-point 5 is most of the pipeline: a number that stops at stop-point 1 has not called the language model yet. The LiveKit half of row 5 – what playback_latency sees, and the sibling metrics that empty a panel rather than erroring – is documented in full on the post-transfer turn that is slow and invisible at the same time, so we take it as read here.

What each platform says its own number covers

Vapi publishes two figures, on two properties, and neither page mentions the other. The vapi.ai home page carries <500ms against the label average latency; its documentation FAQ answers a differently scoped question differently: Yes, Vapi is designed to achieve low latency, typically around 800 milliseconds for end-to-end voice processing. What the methodology page does say is what the dashboard total covers, and that The total excludes endpointing and transport time. Its latency page adds the caveat a marketing line cannot carry: Compare figures only when their scope and conditions match.

Retell AI scopes its 600 ms in the sentence that gives it (quoted in the table above), and the e2e field in its Get Call API carries two documented exclusions: Only turns where the agent speaks are counted — turns where the agent just runs tool calls without saying anything are excluded, and Does not account for network trip time from the Retell server to the user’s frontend. Both are disclosures rather than defects, and both make the number narrower than the call.

Deepgram publishes one figure per boundary and labels each. Its streaming latency guide separates transcript latency, How far behind the transcription is from the audio being sent, from end-of-turn latency, The time from when a user stops speaking to when an EOT event is received. This is the critical metric for voice agent applications, where it directly determines how quickly your agent can begin responding. The guidance table on that page gives network transit 20-200 ms, transcription 150-300 ms, total transcript latency 200-500 ms and Flux end-of-turn detection 100-500 ms. A third page, the Agent API latency report, carries the widest boundary as a per-turn field, total_latency, beside a stage-level tts_latency: three boundaries, three pages, each labelled.

LiveKit and Pipecat publish definitions rather than headlines. The first term of LiveKit’s documented sum is where the endpointing wait lives: end_of_utterance_delay is Time (in seconds) from the end of speech (as detected by VAD) to the point when the user’s turn is considered complete. This includes any transcription_delay. So a LiveKit-shaped sum contains the end-of-turn decision and a Vapi-shaped sum does not – two totals of three stages that are not the same three stages. Pipecat separates the byte from the sound: Time To First Audio is the time from a TTS request to the first audible audio sample. This includes the time to first byte plus any leading silence padding added by the service, reported as a leading_silence field, so logging only TTFB makes that padding free in your data and audible on the call. Which TTS figures exclude what, model by model, is traced in what each published TTS latency number leaves out.

Why is my measured latency higher than the vendor’s number?

Because the two clocks differ at one end or at both, and which end depends on where the vendor’s clock starts. Write it as a ledger: one row per leg of the turn, one column per published scope, ticked where that scope contains the leg. We call it the stop-point ledger, and it is the Four-Column Rule we published for TTS figures – metric, transport, concurrency, network position – applied one layer up, to the turn rather than the model.

Leg of the turn	Published size (owner’s own page)	In Flux end-of-turn (~260 ms)?	In Vapi’s displayed total?	In a speech-end figure (Retell 600 ms, LiveKit e2e_latency, Pipecat observer, Deepgram total_latency)?	In your stopwatch?
1. Handset to your media server (carrier or SIP)	Not published on these pages. Retell publishes a client-to-API threshold: A round-trip time consistently above 300ms can cause latency spikes	No	No	No	Yes
2. Chunk accumulation before the first send	Deepgram: buffers should be between 20 and 100 milliseconds of audio; 80 ms strongly recommended for Flux	Yes, by definition	No	Yes	Yes
3. End-of-turn decision, including the STT round trip	Deepgram Flux 100-500 ms published; 447.7-472.0 ms mean measured in issue 7204’s own trials	This is the figure	Partly: the total excludes endpointing by name, but a transcriber median prices the STT round trip inside this leg	Yes: LiveKit’s sum opens with end_of_utterance_delay	Yes
4. LLM time to first token	Your own value	No	Yes	Yes	Yes
5. TTS first byte	Your own value	No	Yes	Yes	Yes
6. First byte to first audible sample	Pipecat reports it as ttfa minus ttfb; no range on these pages	No	No	No	Yes
7. Emission to playout, plus the return leg	Not published; Retell excludes it by name, LiveKit measures it only via an avatar RPC	No	No	No	Yes

One trap in the arithmetic, and Deepgram’s own pages close it: legs 2 and 3 are not additive. The guidance table defines the 100-500 ms figure as the time from speech end to the EndOfTurn event, so a chunk still filling sits inside that window rather than in front of it, and the same page says its sibling figure, total transcript latency, includes transcription processing, network overhead, and buffer sizes. The proof is in issue 7204: the only variable its author changed was chunk size, and the measured end-of-turn interval moved from 472.0 ms to 447.7 ms. If changing leg 2 moves measured leg 3, adding the two double-counts. One nuance, because it is this page’s subject in miniature: Deepgram’s published definition starts at speech end, while the measurement recipe in the same guide starts its clock from when that audio was sent over the WebSocket – one chunk later.

So there are two reconciliations, and your start-point picks which.

• Against a figure that starts at the transcriber – Vapi’s displayed total is the worked case – your extra legs are 1, 2, 6, 7 and the endpointing part of 3. Not the whole of leg 3: Vapi excludes endpointing by name but includes a transcriber median, and Deepgram’s own guidance table puts transcription latency at 150-300 ms inside the same 100-500 ms end-of-turn window. Counting that whole window as your gap double-counts the transcriber.
• Against a figure that starts when the caller goes quiet – Retell’s 600 ms, LiveKit’s e2e_latency, Pipecat’s observer, Deepgram’s total_latency – legs 2 and 3 are already inside it, and your remaining gap is legs 6 and 7 only: leading silence and playout, the two legs none of the five publishes a range for. That is how a vendor figure and your stopwatch can both be right about the same turn.

How much of that gap you can afford, and the human turn-taking research behind it, is the subject of the sub-second voice AI latency budget, which also carries ITU-T G.114’s one-way limits for leg 1.

The three legs my dashboard never priced

Chunk duration is a latency term most teams never see. The size of the audio messages your client sends sets a floor on how soon the far end can decide anything, and Deepgram states its preference in the Flux quickstart: Chunk Size: 80ms audio chunks strongly recommended for optimal model performance and latency. Two LiveKit plugins shipped a different value. Issue 7204, opened 10 September 2026 and still open as at 20 September 2026, reports that in the tested Deepgram plugin version, 1.6.1, SpeechStreamv2._run sets samples_per_channel to sample_rate // 20, which is 50 ms, and asks for the value to be configurable. It carries the author’s own trials – 24 direct WebSocket runs from a LiveKit Cloud worker in us-east – reporting mean end-of-turn latency of 472.0 ms at 50 ms chunks against 447.7 ms at 80 ms, a difference of 24.4 ms or about 5.2%. That is the issue author’s first-party figure, not a Deepgram or LiveKit claim, and it points the opposite way to the intuition that more buffering must be slower. The ElevenLabs equivalent, issue 7182, was opened 9 September 2026 against a hardcoded 50 ms in the realtime STT send loop and closed as completed on 16 September 2026 when pull request 7183 merged. What the decision inside that window is doing – confidence scores rather than clocks – is compared in STT turn detection across Flux, Scribe and Speechmatics.

Emission is not playout. Pipecat issue 5592, opened 2 September 2026, reported that on the WebSocket transports write_audio_frame returned early without pacing whenever the serializer produced no payload for a frame – the normal path for telephony serializers, whose stream resampler accumulates audio across calls. The consequence as filed: the output queue and the TTSStoppedFrame behind it drain about 3.5x faster than playout, and BotStoppedSpeakingFrame fires early. On an 11-second utterance on a live call that frame fired after 3.2 seconds. Pull request 5593 merged on 11 September 2026 and shipped in Pipecat v1.10.0 about ten hours later, whose release notes put it in this page’s terms: frames are now paced and pushed downstream like any other, so the output queue drains at playback speed again. If a framework can lose playout for two releases, a metric named after the agent starting to speak measures emission and only emission.

Some turns are not in the number at all. Retell documents that its e2e excludes turns where the agent only runs tool calls; LiveKit had a wider version of the same problem after an in-session handoff, fixed from livekit-agents 1.8.1 and covered on its own page. Mid-call lookups are their own budget line, in retrieval latency during mid-call knowledge lookups.

How do I compare my number with a vendor’s on one call?

The method, in order. It needs a recording, a spreadsheet and no deploy.

1. Take your stopwatch number from audio, not a dashboard. One recorded call, one turn: the caller’s speech ending to the first audible agent word – stop-point 5, the only number your caller experienced.
2. Ask the vendor for two timestamps, not a figure. What starts the clock and what stops it. The wider version of that question, with the rest of the claims worth checking, is our checklist for auditing an AI vendor’s technical claims.
3. Fill in the ledger, then pick your reconciliation, which differs depending on whether their clock starts at the transcript or at the caller going quiet.
4. Measure legs 2 and 3 together, never separately, because one is inside the other. Chunk duration is in your client configuration; the end-of-turn wait is in your turn-detection settings, whose published defaults across six platforms are tabulated in turn-detection defaults and why the agent cuts callers off.
5. Re-run it after every dependency bump, because two of the legs above changed behaviour inside a fortnight in September 2026, in shipped releases, with no model changing.
6. Report a percentile over calls, not over turns, with the denominator: how many turns carried a value at all.

What it costs, honestly: about half a day to reconcile one call properly, an owner for the dashboard, and a repeat run on every upgrade. Most teams do it once, find the answer is legs 1 and 3 rather than the model they were about to swap, and never repeat it. Defensible, as long as it is a choice.

Zian AI builds autonomous AI sales agents for live phone, SMS, email and WhatsApp outreach in 30 or more languages, and supports private model deployment on customer infrastructure – the lever that moves leg 1 when your callers are nowhere near a US region. Zian has been running outbound acquisition since 2017, and publishes no headline latency figure deliberately: a figure without its two timestamps is what this page exists to decode. More questions of this shape sit on our AI sales agent FAQ hub.

Where every figure on this page comes from

Figure or quotation	Who published it	Link (their own page)	Date read
<500ms average latency; sub-500ms latency	Vapi	vapi.ai	20 Sep 2026
Typically around 800 milliseconds for end-to-end voice processing	Vapi	docs.vapi.ai/faq	20 Sep 2026
Displayed total is a sum of three component medians; excludes endpointing and transport	Vapi	docs.vapi.ai metrics methodology	20 Sep 2026
Compare figures only when scope and conditions match	Vapi	docs.vapi.ai understanding latency	20 Sep 2026
Latency as low as 600ms, user stops speaking to agent begins responding	Retell AI	docs.retellai.com estimated latency	20 Sep 2026
e2e excludes tool-only turns and server-to-frontend network trip time	Retell AI	docs.retellai.com actual latency	20 Sep 2026
Round-trip time consistently above 300ms can cause latency spikes (client to api.retellai.com)	Retell AI	docs.retellai.com troubleshoot latency	20 Sep 2026
~260ms end-of-turn detection; 80 ms chunks strongly recommended	Deepgram	Deepgram Flux quickstart	20 Sep 2026
Network transit 20-200 ms; transcription 150-300 ms; total transcript 200-500 ms; Flux end-of-turn 100-500 ms; buffers 20-100 ms; buffer sizes inside total transcript latency	Deepgram	Deepgram measuring STT latency	20 Sep 2026
total_latency: user utterance end to first audio byte; tts_latency: first text token to first audio byte	Deepgram	Deepgram Agent API latency report	20 Sep 2026
e2e_latency, end_of_utterance_delay, playback_latency and the documented stage sum	LiveKit	LiveKit data hooks reference	20 Sep 2026
TTFA equals TTFB plus leading silence padding; UserBotLatencyObserver	Pipecat	Pipecat metrics reference	20 Sep 2026
50 ms chunking in plugin 1.6.1; 472.0 ms vs 447.7 ms mean end-of-turn latency (author’s own trials)	livekit/agents issue 7204, open	github.com/livekit/agents/issues/7204	20 Sep 2026
ElevenLabs realtime STT chunk duration made configurable; closed 16 Sep 2026	livekit/agents issue 7182	github.com/livekit/agents/issues/7182	20 Sep 2026
Output queue and TTSStoppedFrame drain about 3.5x faster than playout; 3.2 s on an 11 s utterance	pipecat-ai/pipecat issue 5592, closed	github.com/pipecat-ai/pipecat/issues/5592	20 Sep 2026
Frames are now paced so the output queue drains at playback speed again	Pipecat v1.10.0 release notes, 12 Sep 2026	Pipecat v1.10.0	20 Sep 2026

Frequently asked questions

What does a vendor voice latency number actually measure?

One of four things: the end-of-turn decision, the first LLM token, the first generated TTS byte, or the moment the agent hands its first audio frame to the transport. Deepgram Flux publishes ~260 ms for end-of-turn detection alone, a clock that stops before the language model is called. Ask which two timestamps bound the figure.

Why is my measured latency higher than the number on the vendor site?

Because the two clocks differ at one end or at both. If the vendor clock starts at its own transcriber, as Vapi documents for its displayed total, your number also contains the end-of-turn decision, which Deepgram publishes at 100 to 500 ms for Flux. If it starts when the caller goes quiet, as Retell and LiveKit document, the difference is the leading silence plus the playout leg at the handset, and neither is published.

Does end-to-end latency include the time the caller waits to hear audio?

Not on the pages read for this article. LiveKit defines e2e_latency as the time from when the user stopped speaking to when the agent began responding, and keeps playback as a separate field its own data hooks reference calls near zero unless a remote avatar worker reports playback over an RPC. Emission and playout are different clocks.

How much latency does audio chunk size add?

Deepgram recommends 80 ms chunks for Flux and says streaming buffers should sit between 20 and 100 ms of audio. Do not add that to a published end-of-turn figure: Deepgram defines that range from the moment speech ends, so the buffering is already inside it. In the 24 trials published in livekit/agents issue 7204, moving from 50 ms to 80 ms chunks changed mean end-of-turn latency from 472.0 ms to 447.7 ms, about 5.2% faster rather than slower.

Can I compare latency figures from two vendors directly?

Only if both state the same start and stop points, and one vendor says so itself: Vapi advises comparing figures only when their scope and conditions match. The figures read here stop at four different points, so ranking them as published is a unit error rather than a comparison.