A mid-call knowledge lookup runs inside the silence the caller is already hearing. ElevenLabs documents around 250 ms of added latency for RAG in its agents platform, with no methodology attached; Retell AI exposes a knowledge_base latency percentile on calls that use its knowledge base; Pinecone’s latency guidance publishes no millisecond figure at all. All checked 2 September 2026.
This post is about one step: the lookup. The rest of the turn — endpointing, ASR, time-to-first-token, TTS, telephony transport — is covered in the companion piece on voice AI latency and sub-second responses and is not restated here. The point is that retrieval gets no budget of its own. It runs inside the same gap, competing with everything else in it.
What happens inside the gap when an agent “looks it up”
“The agent checks your knowledge base live” describes a sequence, not an event. ElevenLabs documents its own four steps for retrieval-augmented generation in agents: query processing, embedding generation, retrieval and response generation. On the first, its documentation describes the caller’s question being analysed and reformulated before anything is searched (ElevenLabs Agents RAG documentation, checked 2 September 2026).
Read that first step carefully. Reformulating a query is usually itself a model call, so you pay time-to-first-token twice in one turn — once to write the query, once to answer with the results. A stitched stack then adds an embedding call, a vector search, often a reranking pass, sometimes a document fetch, and finally the answering model has to read the retrieved text before it can start speaking.
Function calls into your own systems behave the same way, and worse. LiveKit’s engineering guidance is blunt: “If your agent calls function tools, this happens before the reply is generated and can add large, variable latency. Since agents can invoke multiple function tools per turn and tools are executed sequentially, a single turn can greatly increase perceived latency.” (LiveKit, “Understand and Improve Voice Agent Latency”, 13 April 2026, checked 2 September 2026.)
“Real-time knowledge base” is a design claim, not a feature
Every voice platform can call an API mid-turn. That is not the claim worth buying. The claim worth buying is that the lookup finishes inside a gap the caller does not read as hesitation. Four things have to be true at once, and none are visible on a feature grid:
- The index sits in the same cloud region as the inference. Pinecone’s own advice is explicit: “To decrease latency, access Pinecone/deploy your application from a cloud environment instead, ideally from the same cloud and region as your index.” (Pinecone, “Decrease latency”, checked 2 September 2026 — that page publishes no millisecond figures at all.)
- Connections are already open. A cold TLS handshake spends a round trip you have not budgeted for. RFC 8446 notes of TLS 1.3: “A zero round-trip time (0-RTT) mode was added, saving a round trip at connection setup for some application data, at the cost of certain security properties.” (IETF RFC 8446, August 2018, checked 2 September 2026.)
- There is a hard timeout shorter than the conversational budget, and a defined behaviour when it fires.
- Retrieval latency is measured per call, separately from model latency, and is available to you — not just to the vendor.
Ask which of those a vendor has implemented. A “retrieval latency” number that measures only the vector query, with the reformulation call and the reranker outside the measurement window, is a number about the wrong thing. How to structure the source documents so a lookup returns one clean chunk is covered in chunk-aware writing.
Most “live” lookups could have been fetched before the call
The pre-fetchable set is far larger than the marketing implies. On an outbound call you know who you are dialling before the phone rings. On an inbound call you usually know the number before the second ring. That is enough time to have already loaded: the account record, order or claim status, plan and renewal date, last support ticket, open invoice, calendar availability, the current price book, and consent or Do Not Call status.
Platforms publish the mechanism for exactly this. Retell AI documents dynamic variables that inject per-call data into the agent’s prompt and opening message, with order ID and appointment date among its own worked examples (Retell AI dynamic variables documentation, checked 2 September 2026). Nothing is retrieved mid-turn because nothing needs to be.
The test is simple: if you can enumerate the field in advance, pre-fetch it. What genuinely cannot be pre-fetched is anything keyed on something the caller says that you could not have predicted — a serial number they read off a device, a competitor’s quote they mention, a policy question about a clause nobody anticipated, a date they propose. That residue is real, and it is small. Design for it rather than routing every fact through it.
Caching, warm connections and starting early
Three techniques do most of the work, and each has a published cost.
Prompt caching attacks the read-the-context step rather than the search step. Anthropic publishes a default five-minute cache lifetime with a one-hour option, cache reads priced at 0.1x base input tokens for most models, and minimum cacheable prefix lengths that vary by model (Anthropic prompt caching documentation, checked 2 September 2026). Neither Anthropic nor OpenAI publishes a percentage latency reduction — both publish cost multipliers. Read the refresh rule before planning around the five-minute default: Anthropic states the cache is refreshed at no additional cost each time the cached content is used, so an active call keeps its own prefix warm. The exposure is the quiet gap — a long hold, a transfer, a caller who stops talking — after which the next turn pays a cache miss.
Speculative execution starts the work before the caller has finished. Deepgram’s Flux model emits an eager end-of-turn event ahead of the confirmed one so the stack can begin generating; its documentation frames this as trimming the last 100–200 ms of end-to-end latency at the cost of 50–70% more model calls, and specifically names complex RAG or function calling as the case where it is worth it (Deepgram, eager end of turn, checked 2 September 2026). LiveKit ships the same idea on by default: “Speculatively begins LLM requests before end-of-turn is detected to reduce response latency. Increases LLM token usage because speculative responses may be discarded.” (LiveKit AgentSession documentation, checked 2 September 2026.)
The trade is worth stating plainly: speculation converts latency into spend. Speculate on retrieval as well as generation and you also multiply reads against your CRM, which is where rate limits live.
Published figures for each step of a mid-call lookup
| Step | What it does | Published figure (source, checked 2 September 2026) | Who controls it |
|---|---|---|---|
| Query reformulation | Rewrites what the caller said into a searchable query | Not published — ElevenLabs documents the step, not its duration | Platform |
| Embedding generation | Turns the query into a vector | Not published — ElevenLabs names the embedding model, no timing | Platform |
| Vector search | Finds the closest chunks | Not published — Pinecone’s latency page gives architecture advice, no numbers | You (region, namespace, filters) |
| Reranking | Reorders candidates for relevance | Not published — Cohere’s Rerank overview states no latency figure | You (whether to use it at all) |
| Whole RAG step, packaged | All of the above inside one platform | “RAG adds on slight latency to the response time of your agent, around 250ms” — ElevenLabs | Vendor |
| Function call to your own system | CRM, order status, calendar | Not published — LiveKit describes it as “large, variable latency” | You (your API is the bottleneck) |
| Reading the retrieved text | Model processes context before speaking | Not published as latency — Anthropic and OpenAI publish cache pricing only | Shared |
| Measurement | Reporting what retrieval actually cost | Retell AI exposes knowledge_base p50/p90/p95/p99 per call, populated only when the agent uses its knowledge base (docs) |
Vendor exposes, you monitor |
| Telephony transport | Carries the audio regardless | Twilio tags a call when Twilio-internal RTP time exceeds 150 ms, or when Voice SDK round-trip time breaches 400 ms in 3 of the last 5 samples (Voice Insights FAQ) | Carrier and routing |
| Zian AI | Retrieval latency | Not published | — |
Seven of the ten rows read “not published”. That is the honest state of the market on 2 September 2026, and it is why a demonstrated call beats a quoted figure.
When the lookup overruns: three bad options
Sooner or later a lookup misses its window. The agent has three options, and they are not equally bad.
Worst: answer without the data. A model that never received the record but has been told it is a helpful assistant will produce a plausible balance, delivery date or policy clause. On a sales call that is a fabricated commitment made in your name, and you will not learn about it until someone acts on it. This alone justifies a hard timeout that passes an explicit “retrieval failed” instruction into the prompt rather than falling through silently.
Second worst: dead air. Silence is information to a caller, and once it runs past a normal conversational pause it reads as a dropped call or a machine. They start talking, and their speech collides with the agent’s reply when it finally arrives.
Least bad: a short, true, interruptible filler. LiveKit’s guidance recommends playing a thinking sound during tool execution and telling the caller before making the call, so they are not left without feedback. Two details decide whether it helps. The filler must be pre-synthesised audio, not generated text — routing it through TTS means paying time-to-first-byte on the very thing meant to cover a delay. And it must be barge-in safe: if the caller speaks over it, the agent stops.
Filler changes perceived latency, not real latency, and it works once. If the lookup still fails, say so and route the call to a person.
p50 looks fine; p99 loses the call
Retrieval is the step most likely to have a well-behaved median and an ugly tail. The median query hits a warm index and a cached prefix. The tail hits a cold namespace, a cache that expired mid-call, a reranker under load, a retry after a timeout, or your own CRM’s rate limiter.
Averages hide all of it. Retell AI publishes per-call latency percentiles including a dedicated retrieval field, defined in its documentation as “Knowledge base retrieval latency from triggering retrieval to receiving all relevant context.” That field, at p99, across a week of production calls, is the number worth asking for. A ten-minute call contains dozens of turns; if even three trigger retrieval, a multi-second p99 is not a rare event across your call volume — it is a daily one. Instrumenting for it is the discipline described in AI agent observability.
What to ask a vendor to demonstrate
- A recorded call with timestamps, measured at the caller’s ear, in which the agent retrieves something it could not have known in advance. Invent the fact on the call — read out a reference number they have never seen.
- The per-call retrieval latency field, exported, at p95 and p99, over at least a week of real calls rather than a demo.
- The timeout value, and a recording of what the agent says when it fires.
- Which steps the quoted figure covers: reformulation, embedding, search, rerank, document fetch, and the model reading the result.
- Where the index runs relative to the model and to your callers.
Where Zian fits
Zian AI runs autonomous sales agents across phone, SMS, email and WhatsApp, and has been running outbound acquisition since 2017. Zian does not publish a retrieval latency figure, or any latency figure. Apply the test above to us and to everyone else on your shortlist: a recorded call, with timestamps, retrieving something the agent could not have known in advance. A number without a methodology is a number about nothing.
Zian AI is in waitlist beta and is accepting a limited number of partner applications. Apply For Partnership.
FAQ
How much latency does a mid-call knowledge lookup add?
The only widely published packaged figure comes from ElevenLabs, whose agents documentation states: “RAG adds on slight latency to the response time of your agent, around 250ms” (ElevenLabs Agents RAG documentation, checked 2 September 2026). Pinecone’s own latency guidance and Cohere’s Rerank overview carry no millisecond figures for search or reranking, so a stitched stack has no published end-to-end number — you have to measure it.
Can the agent look things up before the call instead?
Usually, yes, and for far more than vendors imply. Account records, order status, renewal dates, calendar availability, price books and consent status are all knowable before the phone rings. Platforms publish the mechanism — Retell AI’s dynamic variables are supplied at call creation and interpolated into the prompt. Reserve live retrieval for facts keyed on something the caller says that you could not have predicted.
What should happen when a lookup is too slow?
Set a hard timeout shorter than your conversational budget and define the fallback explicitly. A short, true, pre-synthesised and interruptible filler is the least bad outcome; dead air is worse; answering without the retrieved data is worst, because the model will produce a plausible figure and commit you to it. Never let a timeout fall through silently to generation.
Why does p99 matter more than the average for retrieval?
Because retrieval fails in bursts, not uniformly. Cold indexes, expired caches, reranker load and your own API rate limits all land in the tail. Retell AI’s documentation exposes a per-call retrieval metric defined as “Knowledge base retrieval latency from triggering retrieval to receiving all relevant context” alongside p50, p90, p95 and p99 (Retell AI latency documentation, checked 2 September 2026). Ask for the p99 across production calls, not the median from a demo.
Does starting the lookup early actually help?
It can, at a cost the vendors publish. Deepgram’s Flux emits an eager end-of-turn signal so work can start before the caller has certainly finished, which its documentation frames as trimming the last 100–200 ms of end-to-end latency in exchange for 50–70% more model calls. LiveKit enables speculative generation by default and states plainly that it increases token usage because speculative responses may be discarded. Speculation converts latency into spend.
Does Zian publish a retrieval latency number?
No — not published. Zian AI does not publish latency figures of any kind. Ask us, and every other vendor, for a recorded call with timestamps in which the agent retrieves something it could not have known in advance, plus exported p95 and p99 retrieval latency over a week of real calls.