Quick answer: If you are asking “why is my AI voice agent not following its instructions?”, check delivery before wording. In 10 of the 23 public reports we classified on 26 September 2026, the instruction was not in force when the agent broke it: a mid-call change never arrived as an instruction (6), the session config was rejected or beaten by the first turn (3), or a summariser deleted it (1).
Why is my AI voice agent not following its instructions?
Two answers tie, and they need opposite fixes. Either the model never had the instruction in force, or, as far as anyone established, it had it and chose something else. As single causes they are level at 6 reports each; grouped, delivery failures (causes 1, 3 and 6) make up 10 of 23. One is a plumbing bug you can prove and close; the other is a rate you can only lower.
We set the order by counting. On 26 September 2026 we read 23 public reports in which a developer said a voice agent did not do what its instructions said: 12 threads on the OpenAI developer community forum and 11 issues on the LiveKit Agents and Pipecat GitHub trackers, covering the OpenAI Realtime API (gpt-realtime, gpt-realtime-1.5, gpt-realtime-mini), Gemini Live, xAI Grok and Gemma 4. Each cause is as established on the thread, by the reporter, a maintainer or a reproduction; where no pipeline fault was established, the report went to cause 2. If an API refused the instructions, or cut them to fit a limit, the rules affected were not in force, so that report counts as delivery (cause 3) even when the lesson is about prompt size. Two excluded items are explained below.
Two limits: framework trackers attract framework bugs, so delivery failures are probably over-represented, and 23 is small. Read the ranking as an order to test in, not a rate.
| Cause, as established on the thread | Reports (of 23) | Distinguishing test | First fix |
|---|---|---|---|
| 1. A mid-call instruction change did not reach the model as an instruction | 6 | Log the instructions actually sent with the next request after the change | Use the framework’s instruction-update call, not a system message injected into history |
| 2. Delivered and not followed, or no pipeline fault established | 6 | Proof of delivery, then a failure rate over repeated identical calls | Move the rule out of the prompt into code or a parameter |
| 3. The API rejected the session config, or the first turn ran before it was in force | 3 | First response.created against first session.updated; any error after session.update |
Hold caller audio until the update is acknowledged; treat a rejected update as fatal |
| 4. The model changed underneath the prompt | 3 | Replay the same scripted call on the previous dated snapshot | Pin a dated snapshot; retest before its shutdown date |
| 5. The API accepted the prompt, but it was too long or complex for the model | 2 | A canary rule at the tail of the prompt, on your longest real call | Cut to always-needed rules; move reference data behind tools |
| 6. Context management deleted the instructions | 1 | Does the failure start at a fixed message count? | Keep instructions in the protected instruction slot, not in history |
| 7. The rule asked the prompt to do a parameter’s job | 1 | Does a documented setting control the same behaviour? | Set the parameter, then verify it |
| 8. Rules in the prompt conflicted with each other | 1 | Remove rules one at a time until the behaviour returns | Simplify; resolve the conflict explicitly |
Two commonly suspected causes scored zero: no report attributed a failure to temperature, and none to a speech-to-text error mistaken for disobedience. That means they are not where public reports cluster, not that they never happen. The quotable line: in 10 of 23 public reports of a voice agent ignoring its instructions, the instruction was not in force when the agent broke it.
The 23 reports, one row each, as at 26 September 2026
Issue states and linked pull requests were read from each GitHub issue page, and forum threads the same day. An issue closed as “completed” can mean fixed, working as designed, or a warning added; the last column says which.
| # | Report | What the developer saw | Cause as established on the thread | State as at 26 Sep 2026 |
|---|---|---|---|---|
| 1 | livekit/agents #4242 | Instructions updated inside a tool; the reply to that tool result used the old ones | Cause 1: the reply was generated from a context copy taken before the update | Closed 18 Mar 2026; issue page links merged PR #5072 |
| 2 | livekit/agents #1917 | update_instructions left behaviour unchanged (1.0 RC8) |
Cause 1: chat context not updated | Closed 8 Apr 2025; maintainer comment links PR #1921, merged 8 Apr 2025 |
| 3 | livekit/agents #5496 | On gemini-3.1-flash-live-preview, instruction updates logged “will be ignored” and were a no-op | Cause 1: provider restriction, guarded by the plugin | Closed 16 Sep 2026 after a maintainer said updates now work on Gemini 3.1 Live; the PR the page links (#5754, a docs note) was open |
| 4 | livekit/agents #4875 | System message added to chat context in an OpenAI realtime session was ignored | Cause 1: the plugin stripped system messages from context updates | Closed 18 Feb 2026; page links merged PR #4942 (merged 26 Feb 2026) |
| 5 | livekit/agents #4497 | System messages injected mid-call into Gemini realtime silently disappeared | Cause 1: Gemini realtime turns carry only user and model roles | Closed 16 Jan 2026; page links merged PR #4513, which adds a warning rather than changing behaviour |
| 6 | livekit/agents #7020 | Gemma 4 answered its own question “as the caller” after a mid-call instruction | Cause 1: instruction delivered as a trailing system turn the model’s template does not expect | Closed 9 Sep 2026; maintainer points to PR #6591, merged 11 Sep 2026 |
| 7 | OpenAI forum 1023389 | Per-response instructions ignored; the model repeated the reply that had been deleted | Cause 2: no cause established on the thread | Two posts; last 27 Oct 2025, “I have the same issue” |
| 8 | OpenAI forum 1066408 | session.updated confirmed a new prompt; behaviour did not change |
Cause 2: delivery confirmed by the server event, compliance did not follow | Seven posts; last 31 Aug 2025, unresolved |
| 9 | OpenAI forum 1280308 | “Always say no”: first reply complied about 80% of the time, then drifted | Cause 2 | No replies |
| 10 | OpenAI forum 1362990 | Tool arguments outside a declared enum despite strict: true |
Cause 2 | Three posts; unresolved |
| 11 | OpenAI forum 1366289 | Agent switched language on caller names (an Italian surname, Italian; “Amir”, Hebrew) despite a rule against it | Cause 2; worked around with a forced starting language | Eight posts; last 24 Nov 2025 |
| 12 | OpenAI forum 1366626 | When the caller spoke before the greeting, the agent sometimes replied in the wrong language | Cause 2: no pipeline fault established. The reporter used the SIP API, used the WebSocket only to monitor events and send the greeting, and fixed the delay with a warm cloud function; nobody on the thread showed the instructions were missing | Resolved by the reporter, 17 Nov 2025 |
| 13 | pipecat #5354 | First reply ignored the system instruction, intermittently, with no error | Cause 3: server VAD started the first response before session.update took effect |
Open as at 26 Sep 2026; linked PR #5355 also open |
| 14 | livekit/agents #1183 | After an API error, the voice reverted to default and “all the safeguards from the system prompt are ignored” | Cause 3: session.update rejected (a tool description of 1198 characters against a 1024 maximum), session continued on defaults |
Open as at 26 Sep 2026 (the reporter closed it as working on 21 Jan 2025, then reopened it the same day) |
| 15 | OpenAI forum 1378932 | Rules at the end of the prompt lost on calls with returning customers | Cause 3: instructions over the API’s 16,384-token limit drew an error, and the reporter describes the prompt as silently truncated, losing rules from the end; either way those rules were not in force (counted as delivery; see cause 5 for the size lesson) | Closed by OpenAI staff 16 Sep 2026 as a feature request |
| 16 | OpenAI forum 1379147 | Per-response “repeat this phrase” instruction ignored on gpt-realtime-1.5, honoured on gpt-realtime | Cause 4 | Five posts; reporter’s workaround dated 25 Apr 2026 |
| 17 | OpenAI forum 1380643 | Replacement model invented clinic departments absent from the supplied data | Cause 4: the validated dated snapshot was scheduled for shutdown, and its listed replacement (the undated alias) pointed at a newer snapshot | Open thread; last substantive reply 2 Jun 2026 |
| 18 | OpenAI forum 1375106 | Text output with tools on gpt-realtime-1.5 ignored language instructions, leaked tool JSON | Cause 4: model-side defect | OpenAI staff reported a fix pushed to production, 30 Mar 2026 |
| 19 | OpenAI forum 1310832 | Nonsense replies once the system prompt passed a size threshold (fine at 12k tokens, failing at 18k) | Cause 5, reproduced by a second forum member on one model slug | 14 posts, Jul 2025 |
| 20 | livekit/agents #4305 | Grok realtime agent behaved generically, ignoring its prompt | Cause 5, in the reporter’s words: the model “not being able to handle long complex prompts with tools”; the maintainer could not reproduce a delivery bug | Closed by the reporter, 23 Dec 2025 |
| 21 | pipecat #5595 | After the 21st message the agent had no instructions and replied with a bare greeting | Cause 6: prompt sent as a developer-role message was summarised away | Closed 22 Sep 2026, no linked PR; maintainer: “working as expected” |
| 22 | OpenAI forum 971100 | “Speak faster” in the instructions had little effect | Cause 7: a later reply pointed to the session speed parameter |
17 posts; last 18 Dec 2025 |
| 23 | pipecat #4345 | Gemma 4 announced tool calls and never made them | Cause 8: the reporter traced it to their own formatting rules and fixed it by simplifying them | Closed 22 Apr 2026 |
Two exclusions. Pipecat PR #4291 claimed the Responses API ignores updated instructions when previous_response_id is attached; its author closed it unmerged on 14 April 2026: “The premise of this PR no longer reproduces.” And livekit/agents #4843 was filed by a code-analysis bot with no agent behaviour observed, so it fails our inclusion rule.
Cause 1 (6 of 23, tied first): my instruction change never reached the model as an instruction
Symptom. The original prompt works, but anything changed mid-call (a language switch, an escalation rule, a handoff) is silently ignored.
Cause. All six are one shape: the change was made in the developer’s code and never became an instruction in the request the model answered. In livekit/agents #4242 a tool result’s reply came from a context copy taken before the tool changed the instructions. In #4875 and #4497, system messages added to history were stripped before reaching the realtime session; maintainers said instructions go through the instruction-update call. In #7020 the instruction arrived as a trailing system turn Gemma did not expect, and the model replied as if it were the caller.
Distinguishing test. Log the exact instruction text attached to the first request after the change, at the last point before it leaves your process, or read the provider’s echo of it (on the OpenAI Realtime API, the session.updated event). If the new text is not there, stop tuning the wording.
Fix. Change instructions through the framework’s instruction slot, not a system message in history, and check the provider’s limits: on Gemini 3.1 Live, livekit/agents #5496 records updates ignored outright until a maintainer reported them working on 16 September 2026. For the very next reply, LiveKit maintainers point to per-reply instructions (generate_reply); forum 1379147 found those ignored on gpt-realtime-1.5, so verify on your model.
Cause 2 (6 of 23, tied first): no delivery fault found, and the agent still broke the rule
Symptom. The instruction is provably in force, and the agent still breaks the rule some of the time: a “say only no” agent that complies on the first reply about 80% of the time and then drifts (forum 1280308); tool arguments outside a declared enum despite strict: true (1362990); a language rule broken on foreign-sounding names (1366289). The bucket also holds reports with no pipeline fault established, such as forum 1366626.
Cause. The vendor says this is possible. OpenAI’s Realtime reference states that “the instructions are not guaranteed to be followed by the model”. Replying on forum 1379147 in April 2026, OpenAI’s support account was blunter: “The newer realtime model treats instructions more like guidance than strict rules.”
Distinguishing test. A diagnosis of exclusion: prove delivery first (causes 1, 3 and 6), then run the same scripted call many times and count failures. One bad call tells you nothing.
Fix. What worked in the sample moved the rule out of the model’s discretion: the language thread shipped a “force starting language” option; a reply on the enum thread routes tool calls through a second text model that extracts the arguments. OpenAI’s advice in 1379147 was “short, explicit rules”. If a rule has a price or a booking attached, validate it in code: a prompt is a request, a validator is a guarantee.
Cause 3 (3 of 23): my agent ignores its prompt on the first turn, or after an error
Symptom. The first reply of some calls ignores the prompt while later turns are fine, or the agent loses its rules after an API error.
Cause. On the OpenAI Realtime API your instructions arrive after the session exists. The Realtime reference says the server’s default instructions “will be used if this field is not set”, that session.updated is returned “unless there is an error”, and that in Server VAD mode “the server will create Responses automatically”. So a caller who speaks first can get a reply generated before your instructions land. Pipecat #5354 reproduces that against OpenAI’s backend: told to answer only “acknowledged”, the first turn answered “Paris is the capital of France.” It is open as at 26 September 2026, as is its linked fix, PR #5355. In livekit/agents #1183 (open: the reporter closed it as working on 21 January 2025, then reopened it the same day), a rejected session.update (a tool description over the 1024-character limit) left the session on defaults; that report is against livekit-agents 0.11.3, and we have not tested current releases. The third is forum 1378932, where instructions over the API’s 16,384-token limit drew an error and, as the reporter describes it, the rules at the end of the prompt were silently lost.
Distinguishing test. The event log settles it. We wrote the checker below for logs that record every event in both directions as one JSON line each ({"ts": ..., "dir": "sent"|"recv", "event": {...}}). It flags a first response.created before any session.updated, an error event tied to a session.update, instructions in force that differ from what you sent, per-response overrides, and an undated model alias.
import json, re, sys
def check(path, expected_model_pinned=True):
events = [json.loads(l) for l in open(path) if l.strip()]
first_updated = first_response = None
sent_updates = {} # event_id -> instructions sent
last_sent_instr = None
findings = []
for i, e in enumerate(events):
ev, d = e["event"], e["dir"]
t = ev.get("type")
if d == "sent" and t == "session.update":
sent_updates[ev.get("event_id")] = ev["session"].get("instructions")
if "instructions" in ev["session"]:
last_sent_instr = ev["session"]["instructions"]
elif d == "recv" and t == "session.updated":
if first_updated is None:
first_updated = i
got = ev["session"].get("instructions")
if last_sent_instr is not None and got != last_sent_instr:
findings.append(f"MISMATCH at line {i+1}: sent {len(last_sent_instr)} chars, "
f"in force {len(got or '')} chars")
model = ev["session"].get("model", "")
if expected_model_pinned and not re.search(r"\d{4}-\d{2}-\d{2}$", model):
findings.append(f"UNPINNED model alias '{model}' at line {i+1}")
elif d == "recv" and t == "response.created" and first_response is None:
first_response = i
elif d == "recv" and t == "error":
err = ev["error"]
if (err.get("event_id") is not None and err.get("event_id") in sent_updates) or (err.get("param") or "").startswith("session."):
findings.append(f"REJECTED session.update at line {i+1}: {err.get('code')} "
f"on {err.get('param')} -- this update was not applied")
elif d == "sent" and t == "response.create" and ev.get("response", {}).get("instructions"):
findings.append(f"OVERRIDE at line {i+1}: per-response instructions replace session "
f"instructions for this response only")
if first_response is not None and (first_updated is None or first_response < first_updated):
findings.append(f"EARLY first response.created at line {first_response+1}, before any "
f"session.updated -- your instructions were not confirmed in force for it")
return findings or ["OK: instructions confirmed in force before the first response"]
if __name__ == "__main__":
for p in sys.argv[1:]:
print(p); [print(" " + f) for f in check(p)]
We ran it on 26 September 2026 against seven fixture logs built from the event shapes in OpenAI’s reference, and each gave the expected flag: healthy (OK), the #5354 race (EARLY), the #1183 rejection (REJECTED and EARLY), shortened instructions on an undated model (MISMATCH and UNPINNED), an override (OVERRIDE), an unrelated error with a null event_id (OK; a reviewer caught an earlier version calling it REJECTED), and a session configured at creation (EARLY, a false positive). It has not been run on a live session log. Two caveats: whitespace normalised in the echo would trip MISMATCH, and if instructions are set when the session is created (an ephemeral client secret, or the SIP accept call), no session.updated ever arrives, EARLY fires on every call, and the instructions to check are the ones in session.created. For any other setting, see our guide to proving a voice agent setting is actually applied.
Fix. Do not send caller audio, and do not trigger a greeting, until session.updated has arrived; treat any error tied to a session.update as fatal for that call rather than logging it and carrying on.
Cause 4 (3 of 23): the model changed underneath my prompt
Symptom. Nothing in your code changed, and an instruction that used to work stopped working.
Cause. A prompt is tuned to one model. In forum 1379147 a per-response instruction worked on gpt-realtime and was ignored on gpt-realtime-1.5. In forum 1380643 a Romanian-language team on the dated gpt-realtime-mini-2025-10-06 snapshot faced its shutdown, and its listed replacement, the undated gpt-realtime-mini alias, in a clinic test told a patient about a neurology department absent from its data. Pinning buys time, not permanence: OpenAI’s deprecations page, read 26 September 2026, says access to that snapshot “was shut down on July 23, 2026”.
Distinguishing test. Read the model string the provider echoes back in the session event, not the one in your config file, then replay the same scripted call on the previous dated snapshot while you still can.
Fix. Pin dated snapshots and calendar every shutdown date as a retest deadline. It is the discipline behind continuous script split-testing such as Zian’s PrecisionPitch AI™: one call is an anecdote, a rate across versions is evidence.
Cause 5 (2 of 23): my prompt is too long for the model to hold
Symptom. Replies degrade, or rules are ignored, once the prompt passes a certain size: often only on the longer calls, or the ones where you inject more context.
Cause. This covers prompts the API accepted and the model then handled badly. In forum 1310832 replies turned to nonsense above a size threshold (fine at 12k tokens, failing at 18k); in livekit/agents #4305 the reporter concluded the model could not handle “long complex prompts with tools”. Hard limits are counted under cause 3, but the lesson is the same. In forum 1378932 a restaurant ordering agent on gpt-realtime-1.5 was refused with “Instructions cannot be longer than 16384 tokens, you have provided 17199 tokens” only on calls with returning customers; it “took us weeks to diagnose”. OpenAI’s support reply describes a 32k context window with about 4k reserved for output, shared by instructions, history, tools and audio. The thread also reports about one call in ten with tool errors, including duplicate items when a caller interrupted mid-tool-call, a turn-taking problem covered in our guide to stopping tool calls on an unfinished caller turn.
Distinguishing test. Measure instructions plus tool definitions on your longest real call, not your test call, and put a harmless canary rule at the very end of the prompt (see the canary test below).
Fix. Keep only “always needed” rules in the prompt and move less critical logic into tools, which is what OpenAI’s support reply recommended; reference data such as a full menu is the obvious first candidate. If caller memory is what pushes you over, our guide to voice agent memory between calls covers injecting memory at session start without bloating the prompt.
Cause 6 (1 of 23): my agent forgets its instructions after about 20 messages
Symptom. Fine for the first few minutes, then no prompt at all, at roughly the same point in every long call.
Cause. A context manager removed the instructions with the history. In pipecat #5595 the prompt was a developer-role message; Pipecat’s summariser protects only a leading system message, fires by default at 20 unsummarised messages, and the agent “continued after the 21st message with no instructions at all”. A maintainer replied the same day, 2 September 2026, that this is “working as expected. Long-lived instructions should be provided via the system_instruction“; the issue closed on 22 September 2026 with no code change linked. (Not counted: a code-analysis bot found the same loss in LiveKit’s truncation code, livekit/agents #4843, fixed February 2026.)
Distinguishing test. If the failure starts at a message count rather than a clock time, it is this. Our page on voice agent context loss mid-call separates this from a silently replaced realtime session, which fails by clock time instead.
Fix. Put long-lived instructions in the slot your framework protects (in Pipecat, system_instruction), and never in a history message a summariser is allowed to touch.
Two single reports: a rule the prompt cannot enforce, and rules that fight
A parameter’s job. In forum 971100 developers tried to make the voice speak faster through the prompt until a reply pointed to the session speed parameter, which OpenAI’s reference documents as 0.25 to 1.5, default 1.0, “a post-processing adjustment to the audio after it is generated”. Later posts report it having less effect than expected, and on 17 December 2025 OpenAI staff said they would look into it. If a documented setting controls the behaviour, use it, then verify it.
Conflicting rules. In pipecat #4345 a Gemma 4 agent kept saying it would call a tool and never did. The reporter concluded “the problem was my prompt”, suspected a strict formatting rule, and fixed it by simplifying the rules. The test is ablation: remove rules one at a time until the behaviour comes back.
If you are writing the prompt from scratch rather than diagnosing one, our guide to writing a system prompt for a voice AI agent covers the setup side.
The head-and-tail canary test: one test call that sorts the causes
You need a test number and two lines of prompt: a trigger-gated rule at the very top (“if the caller says canary head, reply only with marigold“) and one at the very bottom (“if the caller says canary tail, reply only with juniper“). No real caller says those phrases. Say both on turn one, and again after turn 25.
| What you hear | Most likely cause | Next step |
|---|---|---|
| Neither canary on turn one | Session setup failed (cause 3) or instructions not delivered (cause 1) | Run the event-log checker |
| Head canary works, tail canary does not | Prompt too long for the model (cause 5), or cut at an API limit (cause 3) | Measure instructions plus tools on your longest call |
| Both work on turn one, both fail after turn 25 | Context management (cause 6) | Find the message count where they stop |
| Both work on the previous snapshot, fail on the new one | Model change (cause 4) | Pin the snapshot; retest the prompt |
| Both canaries always work, the real rule still breaks | Delivered, not followed (cause 2) or conflict (cause 8) | Measure a failure rate; ablate rules; move the rule into code |
This is a procedure we designed, not a measurement, and a canary can itself fail for cause 2 reasons, so repeat it before trusting a negative.
What this costs to run yourself
All of this can be done in-house, and for one agent on one stack it should be. The checks are cheap: one log parse per call, one canary call per deploy. The cost is keeping the ground truth current: frameworks change how instructions travel, aliases move, snapshots get shutdown dates, and trackers change state. pipecat #5595 was opened on 2 September 2026, answered “working as expected” by a maintainer the same day, and closed on 22 September 2026. Someone has to re-read the issues on your stack at every upgrade.
If you would rather hand the realtime layer over, Zian AI builds autonomous phone, SMS, email and WhatsApp sales agents in 30+ languages, including private model deployment on your own infrastructure, and has run outbound acquisition since 2017. It is in partnership-application beta.
Where every figure on this page comes from
| Figure | Who published it | Link | Date read |
|---|---|---|---|
| 23 reports classified; 10 with the instruction not in force; per-cause counts 6, 6, 3, 3, 2, 1, 1, 1; zero for temperature and speech-to-text; 2 items excluded | Zian AI, our own classification of the 23 linked threads and issues | This page, table above | 26 Sep 2026 |
| First reply complied “80% of the time” | Reporter, OpenAI developer community | Thread 1280308 | 26 Sep 2026 |
| Tool description 1198 characters against a 1024 maximum | Reporter’s logged OpenAI error, livekit/agents tracker | Issue #1183 | 26 Sep 2026 |
| Instruction limit 16,384 tokens; 17,199 provided; about one call in ten with errors | Reporter, quoting the OpenAI API error | Thread 1378932 | 26 Sep 2026 |
| 32k context window, about 4k reserved for output (gpt-realtime-1.5) | OpenAI support account, in the same thread | Thread 1378932 | 26 Sep 2026 |
| About 17.8k tokens succeeding, about 31.3k failing with HTTP 504 | A developer’s local estimate, same thread | Thread 1378932 | 26 Sep 2026 |
| Fine at 12k tokens, failing at 18k | Reporter, OpenAI developer community | Thread 1310832 | 26 Sep 2026 |
| Summariser default of 20 messages; failure after the 21st message | Reporter, Pipecat tracker | Issue #5595 | 26 Sep 2026 |
| gpt-realtime-mini-2025-10-06 shut down 23 July 2026 | OpenAI | Deprecations | 26 Sep 2026 |
| Speed setting 0.25 to 1.5, default 1.0 | OpenAI | Realtime client events reference | 26 Sep 2026 |
| Issue states, close dates and linked PR numbers and merge dates | LiveKit and Pipecat maintainers, on each issue page | Linked in the 23-row table | 26 Sep 2026 |
Frequently asked questions
Why does my voice agent ignore its system prompt on the first turn only?
Usually because the first reply was generated before your instructions were in force. In pipecat issue 5354, open as at 26 September 2026, server side turn detection answered before session.update took effect. OpenAI documents that session.updated shows the full effective configuration and that default instructions apply until you set your own (OpenAI Realtime client events reference, read 26 September 2026). If the first response.created in your log arrives before the first session.updated, that is your cause.
Does session.updated mean the model will follow my new instructions?
No. It proves the instructions were stored and are in force, which rules out a delivery failure. It does not prove compliance: the same OpenAI reference says the instructions are not guaranteed to be followed by the model. In OpenAI community thread 1066408 a developer received session.updated with the new prompt and the agent still behaved as before. Once delivery is proven, measure a failure rate over repeated calls.
Why did my agent stop following instructions after a model upgrade?
Because a different model is reading the same prompt. Three of the 23 reports we classified were model changes, including a per response instruction that worked on gpt-realtime and was ignored on gpt-realtime-1.5. Pin a dated snapshot, keep the previous one available for an A/B replay, and treat the shutdown date on the OpenAI deprecations page as your retest deadline.
How long can the instructions for a realtime voice agent be?
Shorter than the context window suggests. One production team hit the error Instructions cannot be longer than 16384 tokens, you have provided 17199 tokens, and only on calls where returning customer details were added. Another developer, by local estimate, saw about 17,800 tokens of instructions plus tools succeed and about 31,300 fail with HTTP 504 on gpt-realtime-1.5 and gpt-realtime-2. Count both together, on your longest real call.
Why does my agent forget its instructions after about 20 messages?
Check your context summariser. In Pipecat it triggers by default at 20 unsummarised messages and protects only a leading system message, so a prompt sent as a developer role message was summarised away after the 21st message. A maintainer answered on 2 September 2026 that this was working as expected, because long lived instructions belong in system_instruction, and the issue was closed on 22 September 2026.
Can temperature make my voice agent ignore its instructions?
None of the 23 reports we classified attributed an instruction failure to temperature, so start elsewhere. If you suspect it, first prove the value reaches the model at all, then compare failure rates over repeated identical calls at each setting.
Want sales agents whose instructions are proven in force rather than assumed? Apply For Partnership.