To make a voice agent remember a caller between calls, write the outcome of each call to your CRM, resolve the inbound caller to a contact record before the greeting, and inject 3 to 8 retrieved facts into the session. That is layer 3 of the four memories below, and it is a retrieval problem, not a model problem.
The four memories of a voice agent
Most confused conversations about voice agent memory are two people using one word for four different mechanisms. Separating them is the whole job, so give the separation a name: the four memories of a voice agent. The layers map onto real framework boundaries — a LiveKit Agents activity, a Pipecat context aggregator — and the defects cited below sit on exactly those boundaries.
| Layer | What it actually is | Lifetime | Where it lives | What breaks it |
|---|---|---|---|---|
| 1. Turn context | Messages inside the model context window | One call | Framework context object | Summarisation silently dropping messages |
| 2. Call state | Name given, intent, consent captured, what the agent promised | One call, must survive a handoff | Session store, keyed by call ID | A handoff creating a fresh activity that inherits nothing |
| 3. Caller recall | What you know about this human from earlier calls | Weeks to years, under a retention rule | Retrieved at session start from layer 4 | Wrong identity resolution, so you recall the wrong person |
| 4. System of record | Contact, deal, consent log, disposition history | Until a retention rule destroys it | Your CRM | Nothing, and that is the point |
The decision rule: layer 3 is a retrieval problem against layer 4, not a model memory problem. Treating it as model memory is the expensive mistake. It sends teams shopping for a vector store and a longer context window when the facts they want are already in a HubSpot or Salesforce row that is inside their retention policy and visible to the humans who talk to that caller. The vector store becomes a second, worse, unaudited copy of the CRM.
How do I make my voice agent remember a caller between calls?
On day one, do nothing to the model. Open your CRM and confirm there is one field on the contact record where a call outcome can be written and read back. If there is not, that is the entire first task.
- Pick the identity key and write down its failure modes. This is the skipped step that produces the embarrassing outcomes.
- Define the memory record as a short set of structured fields on the contact, not a free-text blob and not a transcript.
- Write at end of call. Have the agent emit a structured disposition when the call ends and write it to the contact through your existing integration. The trade-offs between native, Zapier and direct API writes are set out in our guide to AI agent CRM integration patterns.
- Read at session start. Resolve identity, fetch the record, and render it into the initial prompt before the agent speaks. It sits on the critical path to your greeting, so it carries the same budget as any other mid-call knowledge lookup latency.
- Re-assert across handoffs. Do not assume layer 2 survives a transfer between agents. Pass it explicitly.
- Attach a retention rule to the memory record on the day you create it, not later.
You are finished when a caller who rang last week rings again and the agent opens by acknowledging the commitment made on the previous call, without the caller repeating their name, and without the agent volunteering anything before it has confirmed who it is talking to.
Step 1: resolve the caller before you recall anything
Keying memory on the calling number alone is wrong in four specific ways. The Office of the Australian Information Commissioner lists a phone number directly among its examples of personal information, so the key itself is regulated material before you have stored a single fact against it.
| Situation | What a naive lookup does | What it should do |
|---|---|---|
| Shared household or office number | Recalls whoever rang last and greets the wrong human | Treat a single number match as provisional. Let the caller supply their own name, then match |
| Caller ID withheld | No key, so it falls back to a generic script and loses everything | Handle as a first-time caller by design, and offer a lookup on something the caller volunteers |
| Number reassigned to a new subscriber | Recalls a stranger and discloses the previous subscriber details to them | Age out the key and confirm from scratch after a long gap. The US FCC Reassigned Numbers Database exists because carriers report permanently disconnected numbers that later go to somebody else |
| Same person, different number | No match, so recall is lost although the record exists | Resolve on the contact record. Store every observed number against one contact and key memory on the contact ID |
| Several contacts share the number | Picks the most recently updated and is quietly wrong half the time | Treat ambiguity as unresolved, and unresolved means no recall |
Two rules carry this section. First, normalise every number to E.164 before matching, or the same Australian mobile will exist in your database three times. Second, and this one matters legally as well as operationally: the agent confirms identity before it discloses anything. The caller supplies the identifier and the agent verifies it. An agent that opens with a recalled fact has already disclosed that fact to whoever picked up the phone.
Do not solve the shared-number problem with a voice print. The OAIC lists a voice print among its examples of personal information, and under section 6(1) of the Privacy Act 1988, biometric information that is to be used for automated biometric verification or identification, and biometric templates, are sensitive information — a point set out in the OAIC key concepts guidance. Sensitive information generally has a higher level of privacy protection, and it carries a consent requirement a convenience feature will not clear.
Step 2: persist decisions, not dialogue
The second named rule here is the one that most changes what people build: persist decisions, not dialogue. Store what the call concluded, not what was said. The conclusion is smaller, more useful at retrieval, far less sensitive, and unlike a transcript it is actually complete.
That last point is not theoretical. In Pipecat 1.8.1, issue #5639 (open as at 14 September 2026, raised 5 September 2026) documents that a line spoken with TTSSpeakFrame(append_to_context=False) never reaches on_assistant_turn_stopped, and the saved-transcript pattern in the documentation is built from exactly those turn events. The issue is explicit that a holding line queued and then cut off by the caller is indistinguishable from one that was heard. Persist the transcript as your memory and you are persisting an artefact that can both omit what the bot said and record as spoken something the caller never heard.
| Memory field | Where it belongs | Retain for | What it costs you |
|---|---|---|---|
| Identity and every observed number | CRM contact record | Life of the relationship | Nothing new |
| Consent and do-not-call state, with timestamp and wording used | CRM, a dedicated field, never free text | As long as the law you rely on requires the evidence | Field design time. The row you cannot afford to get wrong |
| Commitments made by the agent, such as a promised callback | CRM task, so a human sees it too | Until discharged, then archive | Almost nothing, and the highest-value row here |
| Disposition and next action, as an enum | CRM | Standard sales data policy | Schema discipline up front |
| Stated preferences, such as a preferred time or number | CRM | Review annually | Prompt tokens |
| Do-not-repeat markers for topics already covered | Memory record | Months, not years | Prompt tokens, growing if you never prune |
| Not: the raw transcript | Keep as a record if you must, governed separately | Shortest defensible period | Lossy, large, more sensitive than the outcome |
| Not: audio, card numbers, or sensitive information volunteered in passing | Never the memory layer. Each has its own regime | Not applicable | Compliance scope you did not plan for |
Consent state is the one memory field where being wrong is an enforcement question rather than an awkward moment. Store the state, the timestamp and the exact wording that produced it, which is why the wording and records side of consent language for AI calls is worth settling first.
Step 3: inject memory at session start without bloating the prompt
Retrieve, render and pin. Render the memory record as a compact structured block rather than prose, and cap it at a fixed budget. Three to eight facts is enough for a recall greeting. The failure mode at the other end is an agent that opens by reciting a dossier, which is unsettling and a disclosure to whoever answered.
There is a trap specific to injecting anything at session start. Pipecat issue #5595 (open as at 14 September 2026, raised 2 September 2026, against 1.8.1) reports that the context summarisation utility protects the initial prompt only when the first message carries the system role. Sent with the developer role instead, it falls inside the summarised range, and separately its input_text parts are counted as roughly zero tokens. The reporter states the net effect plainly: on the default trigger of 20 unsummarised messages, the whole initial prompt disappears from the context. The memory you carefully injected can be summarised away mid-call. Test what your framework compactor actually protects.
The same scepticism applies at a handoff. LiveKit Agents issue #7157 (raised 7 September 2026, closed as completed by merged pull request #7167, verified 14 September 2026) describes a handoff creating a new AgentActivity whose first generation began with none of the metrics held by the activity that handled the preceding user turn. That bug is fixed; the boundary it exposed is not. A handoff is a real state boundary, and whatever you want on the far side has to be passed deliberately — the same discipline as AI-to-human handoff context transfer.
When memory stops being a feature and starts being a personal information store
The moment you keep what a caller told you so a future call can use it, you hold personal information about an identified individual, and the questions stop being engineering questions.
Three things follow. Purpose: under Australian Privacy Principle 6, information collected for one purpose cannot be used for another unless the individual consented, or the individual would reasonably expect the use and the new purpose is related to the original one (directly related, if the information is sensitive). A caller who gave a delivery address to resolve a complaint may not reasonably expect it to open a sales call three months later. Destruction: APP 11.2 requires an entity that no longer needs personal information for any permitted purpose, where no Australian law and no court or tribunal order requires it to be retained, to take such steps as are reasonable in the circumstances to destroy it or ensure it is de-identified. Reach: your retention policy has to cover the memory store. A deletion routine that clears the CRM contact and the recordings but leaves a memory record keyed on the same number has destroyed nothing. Run the test — delete a contact, ring in from that number, and see what the agent knows.
Where the data may live is a separate question, and for regulated buyers it is usually the binding one. Zian AI supports private AI deployment on customer infrastructure, which keeps the model and the memory store inside a boundary the customer controls, alongside CRM integrations for HubSpot, Salesforce, HighLevel and Zapier so the system of record stays the system of record.
Build it yourself, or do not
| Your situation | Do this |
|---|---|
| Under roughly 50 repeat callers a month, one queue, one CRM | Build it. A lookup, six fields and a write |
| Several numbers, several agents, handoffs between them | Build it, but budget for identity resolution and handoff state as the real work |
| Regulated vertical, or data that cannot leave the country | The memory store is a compliance artefact. The deployment boundary decides this, not features |
| You cannot say what happens to the memory record when a contact is deleted | Stop and fix the retention path first. Recall on an undeletable store is a liability, not a feature |
The honest cost of the do-it-yourself path is not the code. It is schema discipline, the identity-resolution edge cases above, a retention job nobody wants to own, and tracking framework changes like the two open issues cited here. None of it is hard. All of it is permanent.
Frequently asked questions
Is remembering what a caller said personal information under Australian law?
Yes, in almost every realistic case. The Office of the Australian Information Commissioner lists a phone number among its examples of personal information, so a memory record keyed on a number and containing what the person said is personal information about an identified individual. That brings the Australian Privacy Principles into scope for the memory store itself, not only for your CRM.
Can I just use the phone number as the memory key?
No, not on its own. A household number is shared, a caller can withhold caller ID, a disconnected number can be reassigned to a different subscriber, and the same person rings from a mobile one day and an office line the next. Key the memory on a resolved contact record, store every observed number against that contact, and treat any ambiguous match as unresolved.
Should I store the full call transcript as the memory?
No. Store the decisions the call reached, not the dialogue. Pipecat issue 5639, open as at 14 September 2026, documents that text spoken without appending to context never reaches the assistant turn events the documented transcript pattern is built from, so the saved transcript can be incomplete. A transcript is also larger and more sensitive than the outcome you want to recall.
Does caller memory survive a handoff to another agent mid call?
Only if you pass it. LiveKit Agents issue 7157, raised 7 September 2026 and closed as completed, describes a handoff creating a new agent activity whose first generation started with none of the user turn metrics held by the activity that handled the previous user turn. That specific bug is fixed, but the boundary is real. Assert what you need on the other side of a handoff rather than assuming it carries.
How long should I keep caller memory?
Set a period per category rather than one global number, and make sure the memory store is inside it. Australian Privacy Principle 11.2 requires an entity to take such steps as are reasonable in the circumstances to destroy or de-identify personal information it no longer needs for any permitted purpose, where no Australian law requires it to be kept. The text of the principles is published by the OAIC.
What if the data cannot leave Australia?
Then the deployment boundary decides the design before any memory feature does. Private model deployment on customer infrastructure keeps both the model and the memory store inside a boundary you control, and the system of record stays in your own CRM. Settle residency first, because retrofitting it onto a live memory store means migrating personal information rather than changing a setting.
Talk to us about agents that remember the right things
Zian AI builds autonomous AI sales agents that work against your system of record rather than around it, with private model deployment on customer infrastructure where the data boundary matters. Zian AI is in partnership-application beta.