How to Build Failover Into an AI Voice Agent

Quick answer: Build failover in four layers, in this order: SIP trunk, transcriber, voice, then model. Twilio moves to your next origination URI after 4 seconds of SIP silence. LiveKit Agents v1.8.0 ships a 10.0-second attempt timeout for STT and 5.0 seconds for the LLM. You are finished when every layer has a named second provider and a written, caller-audible symptom.

This is a build sequence, not a theory page. Each step below is a configuration change on a stack you already run, and each one ends with a test you can execute before the outage rather than during it. The four layers are the SIP trunk that carries the call, the speech-to-text service that hears the caller, the text-to-speech service that speaks back, and the large language model that decides what to say. They fail differently, they sound different to the caller, and — the part almost nobody plans for — two of them fail loudly and two of them fail silently.

One framing to carry through: failover is not a switch you turn on. It is a decision about what the caller hears while the switch is being thrown. Get that decision wrong and a working fallback still loses the call.

Which layer do I actually need to protect first? The four-layer failover ladder

We call this the four-layer failover ladder. Build it bottom-up: a perfect model fallback is worthless if the SIP INVITE never arrives. Every symptom and default in this table is taken from the named vendor’s own documentation, read on 10 September 2026.

Layer	What my caller hears if it fails unprotected	Redundancy mechanism	Documented default if I configure nothing	Build it when
1. Telephony trunk (SIP origination and termination)	Ringing that never connects, or a fast busy. The agent never runs at all, so no application-level logging fires.	Multiple origination SIP URIs with priority and weight, a second edge location, and a Disaster Recovery URL pointing at TwiML.	Twilio fails over after 4 seconds of no SIP response, but does not fail over on ten specific SIP codes including 404, 486 and any 6xx.	Before the first production call. It is the only layer where nothing you do inside the agent can rescue the call.
2. Transcriber (STT)	Dead air. The caller speaks, nothing happens, and the call usually ends without either party knowing why.	A second transcription provider behind the first, in your agent process or server-side at the platform.	Vapi: the call ends with an error. LiveKit: no fallback exists unless you construct the adapter.	Before the first production call. It is a configuration change, not an architecture change.
3. Voice (TTS)	Either a sentence that stops mid-word, or the same sentence starting over in a different voice. Which one depends on where the fallback runs.	Two or three voice providers with similar characteristics, ordered.	Vapi: the call ends with an error. LiveKit Inference restarts the request from the beginning; the in-process adapter skips fallback once audio has reached the speaker.	Before you build a persona around one cloned voice. Voice cloning ties you to a single vendor unless the clone exists in two places.
4. Model (LLM)	A long silence, then either nothing or an answer that restarts with a different personality.	An ordered model list. Note that server-side fallback at LiveKit covers STT and TTS only, so this layer runs in your process.	LiveKit: mid-stream fallback is off by default and the adapter raises the error instead. Vapi: automatic, and the schema exposes a manual list on OpenAI models only.	Last. It is the only layer where the backup changes what the agent says, not just how it sounds.

The ladder rule: protect the silent layers first. A caller forgives a changed voice; a caller does not forgive twenty seconds of nothing.

Step 1: give the SIP trunk somewhere to fail to

Start at the carrier, because this layer fails before your code exists. Twilio’s Elastic SIP Trunking documentation lists a Disaster Recovery URL in its feature set and describes it plainly: “In the case of a disaster preventing your calls from being delivered to your origination SIP URI above, you can configure a Disaster Recovery URL pointing to an application built on Twilio’s powerful scripting tool called TwiML.” The same page notes the commercial consequence, which is the honest bit teams miss: “When calls are redirected to your disaster recovery URL, normal Twilio Voice rates apply” (Twilio, Elastic SIP Trunking, read 10 September 2026).

Three concrete actions, in order:

1. Configure more than one origination SIP URI, with priority and weight. Twilio accepts up to ten. Priority runs 0 to 65535 where the lowest number wins, and weight distributes load among equal-priority records. Twilio states that “If all three servers with priority 10 are unavailable, the record with the next lowest priority value will be chosen.”
2. Do not point termination at a single IP. Twilio publishes 3 to 4 A records per edge location, “distributed across multiple Availability Zones”, and recommends the retry rule it uses itself: “A common strategy, which we deploy internally and what we have instructed our carriers to do towards us as well, is that if there is no response to an INVITE, go to the next IP after 4 seconds.” It goes further and recommends a cross-region plan: “if there is a complete Ashburn outage, it is recommended that you failover to another edge location (e.g. If connecting to ashburn, failover to umatilla)”. The published edges include ashburn, umatilla, dublin, frankfurt, singapore, tokyo, sao-paulo and sydney.
3. Read the non-failover code list before you celebrate. This is the trap. Twilio states: “If any of the following SIP status codes are returned (“2xx”, “400”, “404”, “405”, “410”, “416”, “482”, “484”, “486”, “6xx”), Twilio will not fail over to the next origination SIP URI. If there is no SIP response from a given server, Twilio will fail over after 4 seconds.” A session border controller that answers 486 Busy Here under load is answering. Your redundancy will sit idle while every call is rejected by the one box that is overloaded.

Test for step 1: null-route your primary origination target and place a call. The call must connect through the secondary within roughly 4 seconds. Then repeat with the primary returning 486 instead of timing out, and confirm you can distinguish the two outcomes in your carrier logs — our guide to AI voice agent call-ended reason codes maps those fields across platforms.

Step 2: put a second transcriber behind the first

The transcriber is the layer that fails silently, which makes it the one to fix next. Vapi documents the unconfigured default in one sentence: “Without any fallback plan configured, your call will end with an error if your chosen transcription provider fails” (Vapi, transcriber fallback configuration, read 10 September 2026). Vapi offers two routes: a manual list under fallbackPlan.transcribers tried in order, and an automatic mode via transcriber.fallbackPlan.autoFallback.enabled, described as the safety net used when manual fallbacks are exhausted or absent.

On LiveKit the equivalent is stt.FallbackAdapter, and its behaviour is more aggressive by design. LiveKit documents the trigger condition for both of its adapters as “any error from the primary provider, including connection failures, timeouts, HTTP errors (4xx, 5xx), and mid-stream disconnects”, and states that the STT layer has “no partial output guard. The adapter switches to the next provider on any error” (LiveKit, Fallback strategies, read 10 September 2026). Once a provider fails, LiveKit marks it unhealthy and stops sending requests to it, then keeps “periodically probing the failed provider in the background and restoring it once it responds successfully”.

One carve-out that will bite an accent-sensitive deployment: Vapi warns that native endpointing does not survive the switch. Its documentation states that if your primary transcriber supports native endpointing, “that capability won’t carry over to the fallback provider. Vapi will automatically switch to heuristic-based endpointing to continue detecting turn boundaries.” You do not lose turn detection, but it changes character at the moment your provider changes, which matters most on the accents that were already marginal. That interacts directly with the sub-second latency budget a voice agent has to hit.

Test for step 2: revoke the API key of the primary transcriber mid-call and confirm the caller keeps talking to a working agent. A revoked key returns an HTTP error, which is the fast path. Testing a stall is a separate test, and step 5 explains why it is the one that hurts.

Step 3: decide what my caller hears when the voice provider fails

This is the step teams skip, and it is the most interesting finding on this page: the same TTS provider failure produces two opposite caller experiences depending only on where you placed the fallback.

If fallback runs server-side in LiveKit Inference, LiveKit states: “If the primary provider fails partway through streaming a response, the service switches to the next model and restarts the request from the beginning.” The caller hears the sentence start over, in a different voice.

If fallback runs in your own agent process, LiveKit states the opposite guard: “TTS: if audio has already been pushed to the speaker, the adapter does not switch to a backup provider mid-utterance. Fallback is skipped and the partial audio plays through.” The caller hears the sentence stop, mid-word. The v1.8.0 source agrees with the documentation: when the emitter reports pushed audio, the adapter logs a warning about audio already synthesised and returns without falling back (livekit/agents, tts/fallback_adapter.py, read 10 September 2026).

Neither is wrong. A restarted sentence is better for an agent reading a confirmation number and worse for an agent delivering a long disclosure. Pick deliberately, write the choice down, and tell your quality reviewers which one is expected so they do not log it as a defect.

Vapi describes its own switch honestly in the same terms: “Users may notice a brief pause and a change in voice characteristics when switching to a fallback voice”, and it recommends “2-3 fallbacks from different providers for optimal reliability”. Without a plan, the documented default is again terminal: “Without a fallback plan configured, your call will end with an error in the event that your chosen voice provider fails” (Vapi, voice fallback configuration, read 10 September 2026).

If your brand rests on a cloned voice, this layer is a lock-in question as much as a reliability one. LiveKit notes that for custom voices on its Inference service, “TTS fallback across providers is automatic. Each cloned voice is cloned to more than one provider, so LiveKit Inference automatically falls back to another provider if the primary one is unavailable.” If your clone lives at exactly one vendor, you have no voice fallback at all, only a different voice.

Step 4: add model fallback last, and know what it changes

Model fallback goes last because it is the only layer where the backup changes the content of the call rather than its delivery. Two vendor-specific facts decide how you build it.

First, LiveKit’s server-side option does not cover this layer. Its Inference Fallback Adapter “Supports STT and TTS only”; the Agent Fallback Adapter, which runs in your agent process, “Supports STT, TTS, and LLM”. So LLM redundancy on LiveKit is code you own. And mid-stream it is conservative by default: “LLM: if text or tool calls have already been streamed to the user, the adapter raises the error rather than restarting the response with a different model. Set retry_on_chunk_sent=True on llm.FallbackAdapter to override this and allow mid-stream fallback.”

Second, we counted the model layer in Vapi’s own machine-readable schema rather than trusting a summary of it. Method: we fetched Vapi’s published OpenAPI document at api.vapi.ai/api-json on 10 September 2026 and enumerated every schema whose name ends in “Model”. Result: 28 model schemas, of which exactly one — OpenAIModel — carries a fallbackModels property. AnthropicModel, GoogleModel, GroqModel, XaiModel, DeepSeekModel, OpenRouterModel, CustomLLMModel and the rest do not expose that array in that document. The same document contains 20 provider-specific fallback voice schemas and 13 provider-specific fallback transcriber schemas (a fourteenth name, FallbackTranscriberPlan, is the wrapper object rather than a provider), so the asymmetry is specific to the model layer, not a gap in the spec.

Read Vapi’s own description of the field before drawing a conclusion, because it points the other way: “These are the fallback models that will be used if the primary model fails. This shouldn’t be specified unless you have a specific reason to do so. Vapi will automatically find the fastest fallbacks that make sense.” The honest reading is that Vapi treats model fallback as a managed behaviour and exposes manual control on one provider family. The practical reading is that if you need a named, ordered, auditable model chain on a non-OpenAI model, you should confirm the mechanism with Vapi directly rather than assume the array exists. That is the same discipline we describe in auditing a vendor’s technical claims before you build on them.

Model fallback also overlaps a slower failure that is not an outage at all: the model you pinned being retired on a schedule. That has its own playbook in what happens to an AI agent when its base model retires, and the two plans should name the same backup model so you are not maintaining two lists.

What do the default timeouts actually cost my caller?

Here is the arithmetic almost nobody does. The numbers below are the constructor defaults in livekit-agents package version 1.8.0, read from source on 10 September 2026. The retry interval of 0.1 seconds for a first retry comes from APIConnectOptions._interval_for_retry, documented in the same package as “The first retry is immediate, and then uses specified retry_interval”.

Layer	Attempt timeout	Internal retries per provider	Retry interval	Worst-case dwell on a stalled primary before the backup is tried
LLM (llm.FallbackAdapter)	attempt_timeout 5.0 s	max_retry_per_llm 0	retry_interval 0.5 s (unused at 0 retries)	5.0 s
STT (stt.FallbackAdapter)	attempt_timeout 10.0 s	max_retry_per_stt 1	retry_interval 5 s (first retry is 0.1 s)	20.1 s (10.0 + 0.1 + 10.0)
TTS (tts.FallbackAdapter)	no attempt_timeout parameter; inherits the connect timeout of 10.0 s	max_retry_per_tts 2	2.0 s inherited	32.1 s (10.0 + 0.1 + 10.0 + 2.0 + 10.0)

Two caveats, because this is derived arithmetic on published defaults and not a figure any vendor states. It applies only to a provider that stalls — one that returns an error fails over immediately, which is the common case. And the TTS figure applies only before any audio has reached the speaker; after that, the guard in step 3 takes over.

The dead-air rule: your failover budget is the silence your caller will tolerate, not the timeout your framework happens to ship with. On stock settings the silent layer gets four times the patience of the loud one. If you change one number after reading this page, change attempt_timeout on the transcriber.

The counter-pressure is real and it is documented. LiveKit issue 6916, “FallbackAdapter: allow a separate attempt timeout for fallback LLMs” (open, filed 19 August 2026) argues that one shared value cannot serve both providers, because a fallback “typically runs cold — no warmed prompt cache, no session affinity — so its time-to-first-token is meaningfully higher than the primary’s”, and that a tight shared timeout “regularly cuts off a healthy fallback mid-attempt”. Tune the primary down, and verify your backup can still answer inside the window you left it. A related pull request, 7185 (open, 9 September 2026), proposes exposing system-default TTS fallback controls and notices, which suggests this surface is still moving.

What does done look like? The finish state and its acceptance test

You are finished when all five of these are true. Anything less is a partial build, and partial failover is worse than none because it produces confidence without coverage.

1. Every layer names a second provider, written down, with the vendor and model or voice identifier, not “a backup”.
2. Every layer has a written caller-audible symptom — dead air, dropped call, restarted sentence, truncated sentence, changed voice — so your reviewers know what is expected behaviour rather than a defect.
3. The trunk test passes twice: once against a timeout, once against a 486, with a different documented outcome each time.
4. Failover emits an event you can alert on. LiveKit emits an error event on AgentSession when a fallback triggers, and the Agent Fallback Adapter emits an availability-changed event (stt_availability_changed, llm_availability_changed or tts_availability_changed) when a provider is restored. If nothing pages you, you will discover the outage from your call quality scores a week later.
5. Your attempt timeouts are your numbers, not the defaults, and your backup has been measured cold, from a fresh process with no warmed cache, inside the window you set.

The honest cost of running this: four vendor relationships instead of two, four sets of credentials and rate limits to monitor, a second cloned voice to commission, and a recurring test you have to actually execute because the failure you are protecting against is by definition rare. That is the real price, and for a low-volume internal line it may not be worth paying. For a revenue line where a silent call is a lost customer, it is a day of work.

Where every figure on this page comes from

Figure or statement	Owner (organisation)	Source URL	Date checked
Failover to the next origination URI after 4 seconds; the ten SIP codes that suppress failover; Disaster Recovery URL; 3 to 4 IPs per edge; ashburn to umatilla advice	Twilio	twilio.com/docs/sip-trunking	10 Sep 2026
Trigger conditions; unhealthy marking and background probe; Inference restarts the request from the beginning; per-layer partial-output guards; adapter coverage table	LiveKit	docs.livekit.io/agents/logic/fallback-strategies	10 Sep 2026
Constructor defaults: attempt_timeout 5.0 and 10.0, max_retry_per_llm 0, max_retry_per_stt 1, max_retry_per_tts 2, first-retry interval 0.1 s	LiveKit, livekit-agents v1.8.0 source	stt/fallback_adapter.py, llm/fallback_adapter.py, tts/fallback_adapter.py	10 Sep 2026
Worst-case dwell figures of 5.0 s, 20.1 s and 32.1 s	Zian AI, arithmetic on the LiveKit defaults above (not a vendor claim)	Shown in full on this page	10 Sep 2026
Call ends with an error when no voice or transcriber fallback plan is set; brief pause and changed voice characteristics; endpointing does not carry over; 2-3 fallbacks recommended	Vapi	voice fallback plan, transcriber fallback plan	10 Sep 2026
1 of 28 model schemas exposes fallbackModels; 20 fallback voice schemas; 13 fallback transcriber schemas	Zian AI, count over Vapi’s published OpenAPI document	api.vapi.ai/api-json	10 Sep 2026
Separate fallback attempt timeout proposal; TTS system fallback controls	LiveKit contributors (open issue and open PR)	issue 6916, PR 7185	10 Sep 2026

Frequently asked questions

What happens to my AI voice agent if the transcriber goes down mid-call?

Without a fallback, the call ends. Vapi documents that with no fallback plan configured, a call ends with an error if the chosen transcription provider fails (Vapi transcriber fallback configuration, read 10 September 2026). With a fallback configured, LiveKit switches on any error from the primary, including timeouts and mid-stream disconnects, and the caller hears a gap rather than a disconnect.

Does my caller hear the sentence again when a voice provider fails?

It depends on where the fallback runs, and this is the single most overlooked design choice in the build. Server-side fallback in LiveKit Inference switches to the next model and restarts the request from the beginning, so the caller hears the sentence start over. The in-process adapter does the opposite: once audio has reached the speaker it skips fallback and lets the partial audio play through, so the caller hears the sentence stop.

Do I need model fallback as well as voice and transcriber fallback?

Yes, but build it last, because it is the only layer where the backup changes what the agent says rather than how it sounds. Note that LiveKit server-side fallback covers speech-to-text and text-to-speech only, so model redundancy runs in your own agent process, and mid-stream model fallback is disabled by default until you set retry_on_chunk_sent to true.

Will Twilio fail over if my session border controller returns a busy signal?

No, and this is the most expensive misunderstanding on this list. Twilio documents ten SIP status codes, including 400, 404, 486 and any 6xx, on which it will not fail over to the next origination SIP URI, and it fails over after 4 seconds only when there is no SIP response at all (Twilio Elastic SIP Trunking documentation, read 10 September 2026). An overloaded box that answers with 486 is answering, so your second URI never gets the call.

How do I test failover without waiting for a real outage?

Run two tests, not one. Revoke the primary credential to produce an immediate error, which is the fast path every vendor documents. Then simulate a stall by black-holing traffic to the primary, because a stall is governed by the attempt timeout rather than by an error, and on stock LiveKit defaults a stalled transcriber can hold a caller in silence for roughly 20 seconds before the backup ever receives audio.

How many fallback providers should I configure per layer?

Two is the working minimum and three is where the returns flatten. Vapi recommends 2-3 fallbacks from different providers for optimal reliability, and the different providers part matters more than the count, because two voices from the same vendor share an outage. Order them by similarity to the primary so the switch is least noticeable to the caller.