Quick answer: No, not on volume alone. An AI sales agent only improves when every call is joined to a real outcome and to the script variant that produced it. To tell a 2% booking rate from 2.5% at 95% confidence and 80% power you need about 13,791 leads per arm, which is roughly 276 booked meetings in the control arm. Without outcome labels, 13,791 calls teach the agent nothing.
Will my AI sales agent get better the more calls it makes?
Only if something turns those calls into labelled outcomes and acts on them.
In one sentence: an AI sales agent gets better when a change in what it says or does produces a measurably higher rate of a real outcome, and that change is kept while the losing version is retired. Improvement is a comparison between two rates. A comparison needs outcomes on both sides, and it needs enough of them for the difference not to be chance.
The boundary matters as much as the definition. None of the following is the agent getting better, although each is often described that way:
- A bigger transcript archive. Ten thousand recordings in storage are data, not learning. Nothing changes until someone or something reads them against outcomes.
- A foundation model update. When OpenAI, Anthropic or Google ship a new model version, the agent may sound better. That improvement came from the lab, on the lab’s schedule, and would have arrived with zero calls. Our explainer on whether AI sales agents actually learn over time walks through the three layers where a learning claim can live.
- Memory of a single lead. Remembering that a prospect asked for a Thursday callback is state, not improvement.
- Fluency. A reply that reads more naturally is not necessarily a reply that books more meetings.
The quotable version: call volume without an outcome label is a larger transcript archive, not a smarter agent.
The Outcome-Label Test: three questions that decide whether volume can teach
We call this the Outcome-Label Test. Put these three questions to your own logs, not to a sales deck.
- Can every call be joined to its outcome? For each conversation, can you look up whether it produced a booked meeting, whether that meeting showed, whether it closed, and whether the lead opted out? If the booking lives in the calendar, the show lives in HubSpot or Salesforce and the call lives in the voice platform, with no shared key, the answer is no.
- Does every call carry the variant it was delivered under? A booked meeting is only evidence if you know which opener, which objection response and which call-to-action produced it. An outcome with no variant attached tells you the campaign worked; it cannot tell you what to change.
- Is there a pre-committed rule that changes what the agent says next? A dashboard that shows variant B ahead is not a loop. A loop has a decision rule fixed before the test (a sample size, or a proper sequential method), and when the rule fires, the winner becomes the new control and the loser is retired.
The shortest form of the test: if your agent’s logs cannot tell you which calls produced a booked meeting that showed, more volume cannot teach it anything.
Question one is usually the one that fails, because the outcome that matters is recorded in a different system, days or weeks after the call. Our guide to AI agent observability and outcome attribution covers the trace and log structure that makes the join possible.
How many calls does it take to tell 2% from 2.5%?
This is the worked calculation. It uses the standard large-sample formula for comparing two independent proportions with equal group sizes, as set out by Hae-Young Kim in Statistical notes for clinical researchers: Sample size calculation 2 (Restorative Dentistry & Endodontics, 2016):
n per arm = (zα/2 + zβ)² × [p1(1 − p1) + p2(1 − p2)] ÷ (p1 − p2)²
Where p1 is the control rate, p2 is the rate you want to be able to detect, zα/2 = 1.96 for a two-sided test at 95% confidence, and zβ = 0.84 for 80% power. Those are the two z-values the paper uses.
Step 1: check the formula against the source. The paper’s own example compares 20% with 30% and computes 290.08, which it rounds up to approximately 291 subjects per group. We re-ran it: (1.96 + 0.84)² × (0.2 × 0.8 + 0.3 × 0.7) ÷ 0.1² = 7.84 × 0.37 ÷ 0.01 = 290.08. It matches, so the inputs below are being fed to the same formula the paper prints.
Step 2: 2% against 2.5%. The variance term is 0.02 × 0.98 + 0.025 × 0.975 = 0.0196 + 0.024375 = 0.043975. The squared difference is 0.005² = 0.000025. So n = 7.84 × 0.043975 ÷ 0.000025 = 13,790.56, which rounds up to 13,791 leads per arm, or 27,582 across both arms.
Step 3: count the outcomes, not the calls. At 2%, the control arm of 13,791 leads is expected to produce about 276 booked meetings (13,791 × 0.02 = 275.8). The variant arm at 2.5% produces about 345. The test is decided by those roughly 620 meetings; the other 26,960 conversations are the price of finding them.
Step 4: 2% against 3%. The same formula gives 7.84 × (0.0196 + 0.0291) ÷ 0.0001 = 3,818.08, so 3,819 leads per arm, with about 76 booked meetings expected in the control arm.
Step 5: cross-checks. We ran the calculation in Python rather than trusting hand arithmetic. Using unrounded z-values (1.959964 and 0.841621) gives 13,807 per arm instead of 13,791, a difference of 0.1%. A Monte Carlo run of 20,000 simulated tests at 13,791 leads per arm, true rates 2% and 2.5%, detected the difference 80.3% of the time, which is the 80% power the formula promises. And Evan Miller’s rule of thumb in How Not To Run an A/B Test, n = 16σ²/δ² with σ² = p(1 − p), gives 16 × 0.0196 ÷ 0.000025 = 12,544 per arm: the same order of magnitude, from a cruder shortcut.
This is the snippet we ran. It prints 290.08, 13791 and 3819:
from math import ceil
def n_per_arm(p1, p2, za=1.96, zb=0.84):
return (za + zb)**2 * (p1*(1-p1) + p2*(1-p2)) / (p1 - p2)**2
print(round(n_per_arm(0.20, 0.30), 2)) # source check: 290.08
print(ceil(n_per_arm(0.02, 0.025))) # 13791
print(ceil(n_per_arm(0.02, 0.03))) # 3819Substitute your own control rate and the smallest lift you care about: the result is the volume your agent must process per variant before one script change can be trusted.
Why a 25% lift needs about 280 outcomes per arm, whatever the base rate
Rearrange the same formula and something useful falls out. When the base rate is low, as it is for every sales outcome, the (1 − p) terms are close to 1 and the number of outcomes needed in the control arm depends almost entirely on the size of the relative lift, not on the base rate:
outcomes per arm ≈ (zα/2 + zβ)² × (1 + r) ÷ (r − 1)², where r is the ratio of the variant rate to the control rate.
For a 25% relative lift, r = 1.25: 7.84 × 2.25 ÷ 0.0625 = 282.2. That is the source of the rule of thumb on this page. The approximation slightly overstates the need because it drops the (1 − p) terms; the exact figure at a 2% base rate is 275.8.
| Relative lift you want to detect | Ratio r | Outcomes per arm (approximation) | Outcomes per arm (exact, 2% base) | Leads per arm at 2% base |
|---|---|---|---|---|
| 10% (2.0% to 2.2%) | 1.10 | 1,646 | 1,612 | 80,588 |
| 25% (2.0% to 2.5%) | 1.25 | 282 | 276 | 13,791 |
| 50% (2.0% to 3.0%) | 1.50 | 78 | 76 | 3,819 |
| 100% (2.0% to 4.0%) | 2.00 | 24 | 23 | 1,137 |
| 300% (2.0% to 8.0%) | 4.00 | 4.4 | 4.1 | 203 |
Two things follow. First, an AI sales agent learns at the rate it collects outcomes, not the rate it makes calls. A campaign making 1,000 calls a day at a 0.3% close rate collects closed-deal labels about as fast as one making 150 calls a day at 2%. Second, big lifts are cheap to confirm and small ones are expensive. A move from 2% to 8% shows up in about 203 leads per arm. A move from 2% to 2.2% needs more than 80,000 per arm. If a large gain is built step by step, every step has to be confirmed on its own, and each small step costs the expensive rows of this table.
How fast each feedback signal arrives at 10,000 leads a day
Different learning loops listen to different outcomes, and those outcomes differ in how often they occur and how long they take to arrive. This threshold table applies the formula above to each signal, for a 25% relative lift. All rates and lags in it are illustrative inputs, not measurements; replace them with your own. The 10,000 leads a day column is a round pooled volume chosen to show scale, not a measurement of any Zian test; under the comparable-volume condition below, only the comparable share of a pooled volume can feed any one test. The 200 a day column is closer to what one account on its own produces.
| Feedback signal | Illustrative rate per lead | Illustrative lag after the call | Events per week at 10,000 leads/day | Leads per arm for a 25% lift | Time to verdict at 10,000 leads/day | Time to verdict at 200 leads/day | Use it as the optimisation target when… |
|---|---|---|---|---|---|---|---|
| Positive reply (a proxy) | 5.0% | Minutes to hours | 3,500 | 5,324 | About 1 day | About 53 days | Only for early screening of variants; replies can be won by lines that never book |
| Booked meeting | 2.0% | Same day to 2 days | 1,400 | 13,791 | About 3 to 5 days | About 138 to 140 days | You can join each booking to its call and variant; the default target for setter scripts |
| Meeting that showed | 1.4% (2.0% × 70% show) | 1 to 14 days after booking | 980 | 19,839 | About 5 to 20 days | About 200 to 215 days | Pooled volume is high enough; otherwise measure it as a check on the booking winner |
| Closed deal | 0.28% (1.4% × 20% close) | 30 to 90 days | 196 | 100,479 | About 50 to 110 days | About 2.8 to 3.0 years | Almost never as a per-script target; use it to validate the booking and show targets quarterly |
| Opt-out (a guardrail) | 1.0% | Immediate | 700 | 27,903 | About 6 days | About 279 days | Always monitored, never optimised; a variant that raises opt-outs is stopped regardless of bookings |
How the columns were derived: events per week = 10,000 × 7 × rate. Leads per arm come from the formula with p2 = 1.25 × p1. Time to verdict = (2 × leads per arm ÷ daily leads) + lag; for the show row the lag includes up to 2 days to book plus 1 to 14 days to the meeting. For the closed-deal row at 200 a day, that is 200,958 ÷ 200 = 1,004.8 days of lead flow, plus 30 to 90 days of sales cycle, or about 1,035 to 1,095 days.
The table makes the counter-argument to “it gets smarter the more you use it” concrete. At 200 leads a day, an agent optimising for booked meetings has room for two or three tests a year sized to detect a 25% lift; optimising for closed deals, it cannot finish one in two years. That is not a limitation of any particular model. It is the arithmetic of rare outcomes.
What a 500-call “winner” is worth
The failure mode that looks most like learning is a small test declared early. Suppose two script variants both truly convert at 2%, and each gets 500 leads. We computed the exact binomial probability that one arm nonetheless shows at least 50% more conversions than the other. It is 40.2%. A Monte Carlo run of 4,000 simulated pairs gave the same 40.2%. At 1,000 leads per arm, the chance is still 21.8%.
An agent that promotes such winners automatically is not learning; it is drifting, and it will report every drift as progress.
Peeking makes this worse. In Evan Miller’s worked example (a 50% conversion rate, a significance test after every observation, stopping at the first significant result or after 150 observations), a test the operator believes has a 5% false-positive rate actually has one of 26.1%. He calls it a worst case, but not an unheard-of one. The protection is the same in both cases: fix the sample size before the test starts, from the formula above, and do not act before it is reached. Our guide to split-testing sales scripts with AI covers the operational side, including which lines must be locked out of any test.
What is the smallest improvement one account can detect?
Turn the question round. If a single account puts 200 leads a day through its agent for four weeks and splits them evenly between two variants, each arm gets 2,800 leads. Solving the formula for the smallest detectable p2 at a 2% control rate gives 3.19%, a 59% relative lift. Anything smaller than that is invisible at that volume, even though it may be real.
At 10,000 leads a day for one week, split evenly, each arm gets 35,000 leads and the smallest detectable lift from 2% falls to 2.31%, or about 15%. That gap is the case for pooled learning, and its risk.
When does more volume actually make an AI sales agent better?
The answer changes from no to yes when four testable conditions hold together.
- The Outcome-Label Test passes. Every call is joined to its outcome and its variant, and a pre-committed rule acts on the result.
- The outcome count reaches the threshold before the market moves. A test that needs 140 days of lead flow ends in a different market from the one it started in: seasonality, competitors’ offers and your own pricing have shifted.
- The volume being pooled is comparable. Pooling leads across campaigns shortens the path to a verdict, but only if the campaigns are alike enough that a pattern in one applies to another. Pool a dental campaign with a freight campaign and the pooled winner may win in neither. Stratify by campaign, or pool only within one vertical.
- The optimisation target sits as far down the funnel as the volume allows. Use the threshold table: optimise on bookings, confirm with shows, audit against closed deals and opt-outs.
This is where the design choices behind Zian AI sit. Zian’s team has been running outbound acquisition since 2017, at one point operating campaigns for roughly a hundred businesses in the same vertical simultaneously. That is the comparable-volume condition in practice: a pattern that one account would need months of lead flow to confirm can reach the outcome threshold much sooner when many similar campaigns contribute to it. Zian’s learning engine tracks around 420,000 data points across more than 10,000 leads a day. PrecisionPitch AI™ runs continuous split-testing of scripts and approaches optimised for real success outcomes, and SmartReach AI™ orchestrates message, channel and timing by country, industry and profile, with intelligent follow-up pacing. Zian has taken accounts from roughly 2% conversion to around 8%, and in some cases beaten a client’s existing system by five times.
Pooling has a real governance side as well as a statistical one: what is shared across accounts, and what stays inside yours. Our comparison of single-tenant versus cross-account agent learning sets out the questions to put to any vendor in writing, us included.
If you run the loop yourself, the honest economics are these. You need a shared key between your voice platform, calendar and CRM; a variant ID on every call; a pre-registered sample size per test; and someone to write the next challenger when a test closes. At 200 leads a day, the table above leaves room for two or three booking-rate tests a year, each sized to detect a 25% lift, and not every test will produce a winner. That is a legitimate programme and many teams should run it. The question is only whether the outcome count your own volume produces is enough for the pace you need.
What vendors mean when they say their agent learns
Three examples, quoted from each vendor’s own page:
- Spara’s guide to AI SDRs says LLM-native AI SDRs “train on your company’s data—sales playbooks, past conversations, product documentation—and get smarter over time”, and that the agent “improves conversion rates as it handles more conversations” (spara.com).
- Qualified’s explainer says “The more an AI SDR interacts with leads, the better it becomes at understanding their needs and responding to questions” (qualified.com).
- 11x’s homepage describes its digital workers as “Always learning”, able to “Learn and adapt continuously, expanding knowledge and maximizing performance with every interaction” (11x.ai).
None of these claims is contradicted by this page. Each can be true, and the way to find out is the same for all of them, including Zian: ask which outcome the system learns from, how each call is joined to that outcome, how many outcomes a change needs before it is promoted, and whether you can see the log of what was tested and what won. A vendor running a real loop can answer all four in a sentence each.
Frequently asked questions
Why does the outcome count matter more than the call count?
Because a script test is decided by its outcomes, not its conversations. To confirm a lift from 2% to 2.5% at 95% confidence and 80% power you need about 13,791 leads per arm, but the verdict rests on the roughly 276 booked meetings in the control arm. Calls that are not joined to an outcome and to the variant that produced it add transcripts, not improvement.
What sample size does a 2% booking rate need for a script test?
It depends on your base rate and the smallest lift you care about. Using the two-proportion formula in Hae-Young Kim’s 2016 statistical note, at 95% confidence and 80% power, 2% against 2.5% needs 13,791 leads per arm and 2% against 3% needs 3,819. As a rule of thumb, count about 280 outcomes per arm for a 25% relative lift.
Why can’t my AI agent learn from closed deals?
It can, but slowly. Closed deals are rare and late. At an illustrative 0.28% close rate per lead, a 25% lift needs about 100,479 leads per arm, plus a 30 to 90 day sales cycle. At 200 leads a day that is roughly three years. Optimise on bookings and use closed deals to audit.
Is a 50% lift after 500 calls real?
Often not. If two variants both truly convert at 2% and each gets 500 leads, one will show at least 50% more conversions than the other about 40% of the time by chance alone. Fix the sample size before the test starts and do not stop early.
Does pooling calls across many accounts make an agent learn faster?
Yes, if the pooled campaigns are comparable. Pooling shortens the time to reach the outcome count a test needs. Pooling unlike campaigns can produce a winner that wins in none of them, so stratify by campaign or pool within one vertical, and check what data is shared across accounts.
What is the Outcome-Label Test?
A three-question check on your own logs. Can every call be joined to its outcome? Does every call carry its variant? Is there a pre-committed rule that changes what the agent says next? If any answer is no, more call volume cannot teach the agent anything.
Where every figure on this page comes from
| Figure | Who published it | Link | Date read |
|---|---|---|---|
| Two-proportion sample-size formula; z = 1.96 (two-sided α = 0.05) and z = 0.84 (80% power); worked example of 290.08 per group for 20% vs 30% | Hae-Young Kim, Restorative Dentistry & Endodontics 2016;41(2):154–156 | rde.ac | 2026-09-24 |
| Rule of thumb n = 16σ²/δ²; 26.1% false-positive rate in his worked example (50% base rate, test after every observation, cap of 150 observations) | Evan Miller, How Not To Run an A/B Test (18 April 2010) | evanmiller.org | 2026-09-24 |
| 13,791 and 3,819 leads per arm; 276 outcomes; 282 outcomes approximation; 12,544 cross-check; 80.3% simulated power; 40.2% and 21.8% false-winner rates; 3.19% and 2.31% minimum detectable rates; all threshold-table outputs | This page’s own calculation from the two sources above, run in Python on 2026-09-24 | Formula and code shown on this page | 2026-09-24 |
| Illustrative rates (5% reply, 2% booked, 70% show, 20% close, 1% opt-out), lags, and the 200 leads a day account volume | Illustrative assumptions chosen for this page, not measurements | Not applicable | 2026-09-24 |
| Spara statement on AI SDRs getting smarter over time | Spara | spara.com/blog/ai-sdr | 2026-09-24 |
| Qualified statement on AI SDRs improving with more interactions | Qualified | qualified.com/plus/articles/ai-sdr | 2026-09-24 |
| 11x statement “Always learning” | 11x | 11x.ai | 2026-09-24 |
| Outbound acquisition since 2017; roughly a hundred businesses in the same vertical; around 420,000 data points; more than 10,000 leads a day; roughly 2% to around 8%; five times a client’s existing system | Zian AI, first-party figures released by the company | First-party; no external URL | 2026-09-24 |
Want the loop run on outcomes rather than call counts? Zian AI is in partnership-application beta. Apply For Partnership. Before you do, run the threshold table on your own booking, show and opt-out rates.