Quick answer: three levers, three different jobs
Prompt and context engineering changes how an agent behaves, retrieval changes what it knows, and fine-tuning changes the form and consistency of what it says. Most teams reaching for fine-tuning actually needed retrieval. For a sales agent that has to quote live pricing and availability on a call, knowledge has to arrive at call time, not be baked into weights.
That resolves most of the argument, but not what each option costs in latency, in evaluation effort, or on the day your provider ships a new model version. Those costs are below, with the constraints that only appear on a live phone line.
The three levers, defined by what they actually change
Prompt and context engineering: behaviour
Everything in the model’s context window on a given turn is behaviour control: system instructions, the tools you expose, examples, retrieved records, the conversation so far. Microsoft’s Azure OpenAI documentation on customising a large language model says the process “utilizes in-context learning (zero shot and few shot) and, with iteration, improves accuracy and relevancy in responses”.
It is the fastest lever and the only fully portable one. A prompt is text, so it moves to a new model version or provider with no retraining. Google’s tuning documentation for Gemini is direct about the ordering: “We recommend starting with prompting to find the optimal prompt. Then, move on to fine-tuning (if required) to further boost performances or fix recurrent errors.”
It is not unlimited. Anthropic’s prompt engineering overview notes that “Not every success criteria or failing eval is best solved by prompt engineering.” Long instruction stacks cost tokens on every turn, and on a voice call they cost time as well.
Retrieval: knowledge
Retrieval-augmented generation has a precise origin. Patrick Lewis and co-authors introduced the term in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, submitted 22 May 2020), describing RAG as “models which combine pre-trained parametric and non-parametric memory for language generation”. Their abstract names the sales-agent problem: for large pre-trained language models, “providing provenance for their decisions and updating their world knowledge remain open research problems”.
Parametric memory sits in the weights. Non-parametric memory is an index you can edit. If your pricing changes on a Tuesday, you want it in the second kind. Microsoft’s customisation guidance puts it plainly: “RAG is helpful when you might want questions answered that are recent (for example, before the cutoff date of when the model version was last trained).”
Fine-tuning: form and consistency
Fine-tuning adjusts the model’s weights on your examples. The same Microsoft customisation guidance describes when it earns its keep: “Good cases for fine-tuning include steering the model to output content in a specific and customized style, tone, or format, or tasks where the information needed to steer the model is too long or complex to fit into the prompt window.”
Note what is absent from that list: teaching the model your product catalogue. Fine-tuning on a hundred pricing examples gives you a model that has learned the shape of a pricing answer and will produce confident, well-formatted, out-of-date numbers.
Where fine-tuning genuinely pays is efficiency. Google lists “Lower inference latency and cost due to shorter prompts” as a benefit of tuning, and OpenAI’s model optimisation track agrees: “While sometimes the same outcomes could be achieved by simply prompting the model, fine-tuning allows to achieve the same results with a shorter prompt, and maybe even a smaller model.”
Comparison: what each lever costs you
| Dimension | Prompt / context engineering | Retrieval (RAG) | Fine-tuning |
|---|---|---|---|
| What it changes | Behaviour, tone, framing, tool choice | Knowledge available this turn | Form, consistency, task accuracy |
| Time to first result | Minutes | Days to weeks (ingest, chunk, index) | Weeks (dataset, training, evaluation) |
| Where cost sits | Tokens every turn | Index storage plus a query per turn | Training upfront, then hosting |
| Per-turn latency | Grows with instruction length | Adds a round trip before generation | Can reduce it via a shorter prompt |
| Fixing a wrong answer | Edit the instruction, redeploy now | Correct the document, re-index | Collect examples, retrain, re-evaluate |
| Staleness | Only if you forget to update it | As fresh as your index refresh | Frozen at the training set |
| Base model changes | Portable; re-test and adjust | Portable; index is model-independent | Pinned; retraining required |
| Provenance | None inherent | Citable to the retrieved passage | None inherent |
| Evaluation burden | Low to moderate | Two systems: retrieval and generation | High; needs a baseline |
The version-change problem nobody budgets for
A fine-tune is not a standalone asset. It is a derivative of a specific base model snapshot and it lives and dies with that snapshot. Microsoft’s Foundry fine-tuning considerations page states the consequence directly: “Fine-tuning may need to be repeated whenever the data is updated, or when an updated base model is released.” In 2026 that stopped being hypothetical.
OpenAI’s supervised fine-tuning guide now carries the notice: “OpenAI is winding down the fine-tuning platform. The platform is no longer accessible to new users, but existing users of the fine-tuning platform will be able to create training jobs for the coming months.” Its deprecations page gives the schedule in three steps: from 7 May 2026, “Creating fine-tuning jobs or training is not available to organizations that have not previously run fine-tuning”; from 2 July 2026, “Creating fine-tuning jobs is no longer available to organizations that have not run inference on a fine-tuned model in the past 60 days”; and on 6 January 2027, “Active existing customers will no longer be able to create new fine-tuning jobs on this date.” Existing work is not switched off: the same page states that “Inference on fine-tuned models will continue to be available until the base models are deprecated.”
Fine-tuning is not going away. It remains available through Microsoft Foundry, Google’s Gemini platform and open-weight models you host yourself. But the lesson generalises: a fine-tune carries a maintenance obligation tied to somebody else’s release calendar, and a prompt and a retrieval index do not. Ask any vendor directly, alongside the rest of an enterprise AI sales agent readiness checklist: what happens to our customisation when you upgrade the model?
Evaluation is the part teams skip
Every provider gives the same order of operations and almost nobody follows it. OpenAI’s supervised fine-tuning guide is unambiguous: “Only invest in fine-tuning after setting up evals. You need a reliable way to determine whether your fine-tuned model is performing better than a base model.” Microsoft names the failure mode: “Fine-tuning with bad data makes the base model worse, but without a baseline, it’s hard to detect regressions.” A fine-tune trained on your best transcripts learns the habits of your best rep and any compliance shortcuts in them.
Retrieval has a subtler problem: you are grading two systems at once. A wrong answer can be a retrieval failure (the passage was never fetched) or a generation failure (it was fetched and ignored). If your logging cannot separate those, you cannot fix either. That is the argument behind AI agent observability.
What changes when the agent is on a live phone call
Generic RAG-versus-fine-tuning writing assumes a chat box where a two-second pause is invisible. Voice is a different problem.
The retrieval round trip lands inside the turn
On a call, every lookup happens in the silence between the prospect finishing a sentence and the agent starting one. OpenAI’s Realtime API documentation acknowledges the trade when describing manual response control, calling it useful “for moderation or input validation or RAG patterns, where you’re comfortable trading a bit more latency in the interaction for control over inputs”.
The answer is not to remove retrieval but to hide it. OpenAI’s realtime prompting guide recommends speaking and looking up at once: “Some use cases could benefit from the Realtime model providing an audio response at the same time as calling a tool. This leads to a better user experience, masking latency.” It also warns that spoken preambles “Used poorly, they become filler and increase perceived latency.”
Why the milliseconds matter, and what the human turn-taking baseline is, we covered in voice AI latency and sub-second responses. That post is the budget; this one is how you spend it. Retrieval is affordable running concurrently with speech under a hard timeout, and unaffordable as a blocking step in series.
Knowledge has to be true at the moment of the call
A sales agent’s most consequential facts change hourly: what is in stock, which slot is open, whether the lead spoke to someone yesterday, what the current rate is. None of that can live in weights. It is a live lookup against a system of record, which makes it an integration problem, not a training problem.
It also has to be labelled. OpenAI’s realtime prompting guidance recommends a structured context block with explicit status and retrieval-time fields on each fact, cautioning: “Do not rely on the model to infer source priority from a raw transcript or large context dump. Use structure.” The agent should know that availability fetched this turn outranks availability it mentioned eleven minutes ago.
The highest-value adaptation is not in the weights
For a sales agent, the change that moves the number is almost never a weight change. It is the record of which opener, which objection response and which callback timing actually converted, fed back into what the agent says next. That is measurement and orchestration: it sits above the model and survives every model upgrade. We have written before about continuous split-testing of sales scripts and the sceptical question of whether AI sales agents actually learn. Given that most vendor learning claims describe orchestration rather than training, these three levers are where a change can actually be made.
The hybrid reality
Production systems do not choose. Microsoft’s framing is that “These three techniques are not mutually exclusive but are complementary methods that in combination can be applicable to a specific use case.”
In a realistic voice stack, prompt and context engineering defines the persona, disclosure, compliance guardrails and escalation rules. Retrieval supplies product facts, account history and the live calendar. Fine-tuning, if used at all, goes to a narrow sub-task such as intent classification, or shrinking a long instruction stack so a smaller model can carry it. Microsoft lists that combination as legitimate, noting fine-tuning “trains the model to effectively use retrieved data while filtering out irrelevant information”.
The sequence matters more than the choice. Prompt first, because it is reversible in minutes. Retrieval second, because it fixes knowledge permanently and portably. Fine-tune last, only for a measured failure the first two could not solve.
Where Zian sits
Zian AI runs autonomous sales agents across phone, SMS, email and WhatsApp. SmartReach AI orchestrates message, channel and timing by country, industry and profile, with intelligent follow-up pacing. PrecisionPitch AI runs continuous split-testing of scripts and approaches, optimised for real success outcomes. Research and web and knowledge-base lookups are a listed platform capability. That is a capability statement, not a claim to run a retrieval pipeline over your corpus, and we do not publish our model-adaptation architecture. The learning engine tracks around 420,000 data points, across more than 10,000 leads a day. Private model deployment on customer infrastructure is available, covered in our note on private AI deployment.
Zian is in waitlist beta: no public pricing, no free trial, no self-serve signup.
How we sourced this
Every technical claim here comes from a primary source we opened on 28 August 2026: the original RAG paper on arXiv for the definition, and the providers’ own documentation for guidance and lifecycle facts. Specifically, Microsoft Learn’s page on customising a large language model and its Foundry fine-tuning considerations page; Google’s tuning introduction for Gemini; Anthropic’s prompt engineering overview; and OpenAI’s supervised fine-tuning guide, model optimisation track, deprecations page and Realtime API guides. Quotation marks indicate verbatim text from the named source.
What we could not verify and did not claim: comparative benchmark figures for fine-tuning versus retrieval (published numbers are task-specific and do not transfer to sales calls), any cost figure for training or hosting a fine-tuned model, and any latency figure for a retrieval round trip, which depends on your index, region and network path. We did not check other providers’ roadmaps, so the wind-down above is OpenAI-specific.
Frequently asked questions
What is the difference between RAG and fine-tuning?
Fine-tuning changes the model’s weights so it produces outputs in a particular form. Retrieval-augmented generation leaves the weights alone and fetches documents at query time. The original paper by Lewis and co-authors, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, 2020), defines RAG models as ones that “combine pre-trained parametric and non-parametric memory for language generation”. To change a fact in a fine-tuned model you retrain; in a retrieval system you edit a document.
Should I fine-tune a model so my AI sales agent knows my products and pricing?
Almost certainly not. Product details and pricing change, and weights do not update themselves. Put that information in a retrieval index or a live integration with your system of record. Microsoft’s guidance is that RAG suits answers grounded in private data, or in information more recent than the model’s training cut-off. Fine-tuning governs how the agent speaks, not what it knows.
Does fine-tuning make an AI voice agent faster?
Indirectly. Google lists lower inference latency and cost from shorter prompts as a benefit of tuning, because tuning lets you strip few-shot examples out of the prompt. If your system prompt has grown to thousands of tokens of examples, folding those into a tuned model is a real latency win. It does nothing for the latency of a knowledge lookup, which is a separate round trip.
What happens to my fine-tuned model when the provider releases a new base model?
Your fine-tune is pinned to the base model snapshot it was trained on. Microsoft states that fine-tuning “may need to be repeated whenever the data is updated, or when an updated base model is released”. OpenAI’s deprecations page states that “Inference on fine-tuned models will continue to be available until the base models are deprecated”. Budget periodic retraining as a standing cost.
Can an AI sales agent look up live calendar availability or stock during a call?
Yes, through tool calls to your systems, and that is the correct design. The constraint is timing: the lookup must fit inside the pause between turns. OpenAI’s Realtime API documentation describes issuing an audio response at the same time as calling a tool, specifically to mask that delay. Design for concurrent speech and lookup with a hard timeout, not a blocking wait.
How much training data do I need before fine-tuning is worth trying?
OpenAI’s supervised fine-tuning guide recommends starting with 50 well-crafted demonstrations and evaluating the results, noting that if 50 examples have no impact you should rethink the task or the prompt before adding data. Google suggests roughly 100 or more. The binding constraint is rarely volume; it is having an evaluation set good enough to prove the fine-tune helped.
Does Zian fine-tune a custom model for each customer?
Zian does not publish its model-adaptation architecture, so we will not characterise it here. What is public: SmartReach AI orchestrates message, channel and timing, PrecisionPitch AI continuously split-tests scripts and approaches, the platform performs research and knowledge-base lookups, and private model deployment on customer infrastructure is available. Zian is in waitlist beta with no public pricing.
Deciding this for your own team
Run the test in order. If the agent is wrong about facts, that is retrieval. If it is right but inconsistent, verbose or badly formatted, that is a candidate for fine-tuning once you have a baseline. If it is technically correct and still not converting, none of the three is your answer, because that is a script and orchestration problem. To work through which applies to your calling, Apply For Partnership.