Do not ask the vendor — test it. Place one call with a canary phone number and a revoked API key, then grep every log sink for both strings. Then run four searches against the platform public tracker. Of the four closed issues I read in full on 16 September 2026, one was closed by a merged code fix.
Security review usually answers this with a questionnaire, and a questionnaire returns a promise. The version people actually type is how do I check whether a voice platform writes our API keys or caller data into its logs, and it has a verifiable answer, because a speech stack leaks in three places that leave public evidence. This page is the independent-verification counterpart to our voice-AI vendor security questionnaire: that page asks, this one checks the answers without asking.
What leaks in a voice stack, and where it lands
MITRE catalogues the class as CWE-532, Insertion of Sensitive Information into Log File, described in one sentence: the product writes sensitive information to a log file. In a speech pipeline it happens through three doors, each needing its own test.
Door one: the credential inside an exception. Voice plugins open authenticated WebSockets to speech and model providers. When the provider refuses the upgrade — a rotated, expired or rate-limited key — the exception object can carry the request headers, and the auth header rides with it into whatever prints that exception. A key rotation is enough; nothing has to be attacking you.
Door two: the caller identifier on every log line. Telephony stacks name a room, session or job after the inbound call, and the phone number is the obvious unique value. That identifier reaches metrics labels, trace spans, recording filenames, webhook bodies and support tooling, each with its own retention and access list. Transcript redaction never touches it, because it is not in the transcript.
Door three: the transcript downstream of redaction. Redaction happens at one point, in one component. Anything that copied the text earlier — an error tracker, an analytics pipeline, a prompt cache — kept the unredacted version. Spoken card numbers are the sharpest case, handled in whether an AI voice agent should take card payments.
The tracker audit: four queries that answer the question without asking
Most serious voice platforms are built on open-source projects with public issue trackers: LiveKit Agents, LiveKit SIP, Pipecat and their plugin ecosystems. A public tracker is a security artefact no marketing team curates. Call this the tracker audit: four title-scoped searches against every repository in the vendor stack. Title scoping matters, because unscoped boolean queries return hundreds of loose body matches, which is how this check usually gets abandoned.
| # | Query (substitute the repository) | The failure class it finds |
|---|---|---|
| 1 | repo:OWNER/REPO is:issue "api key" in:title |
Secret in log: credentials reaching stdout, exception reprs or crash handlers |
| 2 | repo:OWNER/REPO is:issue PII in:title |
PII in identifier: caller data encoded into names, labels and filenames |
| 3 | repo:OWNER/REPO is:issue GHSA in:title (repeat with CVE) |
Dependency advisory: transitive findings the vendor inherits |
| 4 | repo:OWNER/REPO is:issue is:open redact in:title sort:created-ascthen repeat with redaction in:title |
Unfixed-open count: the requests sitting longest without resolution. GitHub issue search does not stem, so redact does not match redaction and one spelling alone misses issues |
Only query 4 carries information as a count; the other three are for reading, not scoring. The output is a list of dated issue numbers, which is a very different artefact to bring to a vendor call than a completed questionnaire.
What the four queries returned on 16 September 2026
Run against livekit/agents, livekit/sip and pipecat-ai/pipecat through the GitHub search API on 16 September 2026. Counts are per repository, because an aggregate hides where a finding lives:
- Query 1 returned 15 across the three — agents 10, pipecat 5, sip 0 — none open.
- Query 2 returned 4 — agents 3, sip 1, pipecat 0 — and one of them is open.
- Query 3 returned 1 on
GHSA(pipecat), plus 3 more on theCVErepeat (sip 1, pipecat 2), all closed. - Query 4 returned 0 on
redactand 1 onredaction. That difference is the point, below.
The queries take under five minutes. Then the slow part, which is the actual work: opening each result and reading how it closed.
- Closed by a merged fix. livekit/agents #7031, opened 29 August 2026, reported that a rejected WebSocket upgrade escaped the error handlers at six credentialed connect sites, carrying the auth header into an unretrieved task exception. PR #7032 merged 30 August 2026 and closed it — roughly 27 hours from report to merge. A follow-up, PR #7152 merged 7 September 2026, fixed the regression test, which errored before reaching its assertion and so could not distinguish a working redaction from a removed one. Both re-checked 16 September 2026.
- Closed because the fix is coming from elsewhere. livekit/sip #821 (opened 30 August 2026) asked for field-level redaction of one log line; livekit/sip #726 (opened 15 June 2026) asked for room names omitting the caller number. A maintainer closed both — #821 on 3 September 2026, #726 on 20 July 2026 — explaining in each case that webhook-based dispatch rules, not a redaction switch, are the intended solution. Only the #726 comment offers an interim step, randomised room naming. What exists now is documentation, not code, and it is explicit: the LiveKit dispatch rule reference warns that an individual dispatch rule names each room after the caller phone number, that room names are recorded in logs and traces throughout LiveKit, and that they are not removed by PII redaction. Read 16 September 2026.
- Closed as a no-op, finding still live upstream. pipecat-ai/pipecat #5627, opened and closed 3 September 2026, tracked the NLTK advisory GHSA-8mgp-746c-j5xp (CVE-2026-81726, high severity, published by NLTK on 12 August 2026 and added to the GitHub Advisory Database on 2 September 2026). As at 16 September 2026 it still lists an affected range of nltk 3.10.3 and below with no patched version identified, the latest NLTK release is 3.10.3 from 13 August 2026, and the Pipecat pin on
mainis unchanged. The maintainer reply is the model answer: aware of it, the existing range already permits the patched build, the floor gets raised when one exists. - Open, old, and the one my first pass missed. livekit/agents #6050, an enhancement request titled STT-agnostic PII redaction for transcripts before they hit logs/storage, was opened 10 June 2026 and was still open, last updated 23 July 2026, when checked on 16 September 2026 — 98 days. It asks for a provider-agnostic redaction hook rather than reporting a defect, and work is visibly moving around it: community pull request #6342, open since 7 July 2026, calls itself Phase 1 of a redaction contract unifying #6050 and #6204, and LiveKit merged #6356 on 20 August 2026 to tag content-bearing trace and log keys for redaction. This is the finding the threshold table below ranks highest.
Why it was missed. Query 4 returned zero on redact across all three repositories and I reported that zero. GitHub issue search does not stem: redact does not match redaction. Run it on redaction and #6050 appears. Row 5 of the table below says a zero usually means the query is wrong rather than the platform clean, and I published the zero before applying my own row.
Of the four closed issues read in full above — #7031, #821, #726 and #5627 — GitHub records all four as closed-completed and one was closed by a merged code fix. That is a fact about those four threads, not a score for any platform. The state was accurate; it does not encode the outcome.
Closed is not the same as fixed: the threshold table
The value of a tracker finding is set by how it closed, and each state changes a different thing in your evaluation.
| What you find | What it actually means | What to change | What to ask the vendor |
|---|---|---|---|
| Closed by a merged PR that names the issue | Strongest signal available. The class was found, fixed and dated in public | Nothing. Treat as evidence the process works | Which release carries it, and does a regression test for it run in CI |
| Closed by a maintainer saying it will be solved another way | Not fixed. There may be a documented interim mitigation | Find the docs page and check it applies to your configuration | Which mitigation is applied on our tenancy, and when the permanent fix lands |
| Closed as a no-op against an upstream advisory | A live transitive finding, and not the vendor own code | Record it as inherited risk, with the advisory ID | The current version pin, and who watches the advisory for a patched release |
| Open, no maintainer reply, older than 90 days | The real signal in the set. Volume of open issues is noise; age with silence is not | Assume the behaviour is current and design around it | The workaround, in writing, and whether it is on the roadmap |
| Zero results across all four queries | Usually the query is wrong, not the platform clean | Widen to the plugin repositories and the docs site before concluding anything | Where security issues are tracked, if not here |
One more threshold. Issue #7031 opened and closed inside 27 hours, so a quarterly audit would have missed it entirely: report, fix and merge all fell between two reviews. Run this monthly as a saved search, not once as a procurement ritual, and date every count you record.
The canary call: the test that works without a public repository
The tracker audit only works on open-source-backed platforms. The canary call works on any of them, and tests your deployment rather than the upstream project. Run it on day one, before the commercial conversation.
- Mint and record a canary credential. Issue a fresh API key for one non-production provider and store the literal value. Do not grep for guessed key prefixes; you hold the actual string, so match on it exactly.
- Choose a canary caller identity. A number in E.164 form that exists in no other system of yours, plus a spoken token: an invented surname and a nine-digit number. Note the exact minute you will call.
- Force door one. Revoke the canary key, then trigger the component that uses it. A revoked key produces the rejected-handshake path #7031 describes, which is precisely when a credential can reach a log.
- Place one real call from the canary number, speaking the token twice: once early, once after the point where redaction is supposed to engage.
- Enumerate every sink before you search. Platform log viewer, exported logs, trace backend, metrics labels, recording filenames and object keys, transcript store, webhook payloads, error tracker, analytics warehouse, vendor support portal. That list of ten is our own working expectation rather than a measured figure, and sinks you did not list are sinks you did not test.
- Search each sink for both strings, then for the number pattern rather than your number, because door two hides in fields you were not looking at.
- Finish state: a one-page sink map. One row per sink, three columns filled: did the canary appear, what is the retention, who can read it. That artefact is the deliverable, and you re-run it after every configuration change.
The two commands that matter on exported logs:
grep -rF "$(cat canary-key.txt)" ./logs/ # door one: the literal credential
grep -rEo '\+[1-9][0-9]{7,14}' ./logs/ | sort -u # door two: any E.164 number, anywhereThe second is the one that surprises people. It does not look for your canary number; it lists every phone number in the export. A room name, job ID or recording key built from a caller number shows up there even when the transcript is spotless.
What a checkable redaction claim looks like
Two published examples, read 16 September 2026, show the precision that makes a claim testable rather than reassuring. LiveKit Cloud documents PII redaction as recognising 41 categories across 10 groups with 36 on by default, names the five that are off (date, language, organization, political affiliation and religion), and states that it runs at recording upload after the session rather than live, is available only for LiveKit Cloud projects, and is best effort. Deepgram documents a redact parameter with the groups pci, pii, phi, numbers and aggressive_numbers, or selection from more than 50 individual entity types; entity redaction is English only, and on the Flux streaming endpoint number redaction is the only kind supported, any other value being rejected with an HTTP 400 at connection time.
Every sentence there is falsifiable in one test call. A vendor sentence that names the boundary of its own feature beats an attestation.
The honest economics, and these are our own estimates rather than measured figures: the queries cost minutes, the canary call costs one call and an hour, and the sink map costs roughly half a day per platform for someone who can read a stack trace and has admin access to every log destination. Cheap once. The cadence is not — three or four platforms re-audited monthly, because trackers move in 27-hour windows, is a recurring day a month plus an escalation path when a finding needs a vendor answer. Below about two platforms under evaluation, do it by hand and keep the sink maps in a folder. Above that it is a standing process, and the same arithmetic governs auditing an AI vendor technical claims and how to test an AI voice agent before go-live.
Running the same audit on Zian
An audit method that exempts its author is advertising, so apply it to us. Zian AI is a closed-source platform in partnership-application beta, so queries 1 to 4 return nothing for us — an unavailable result, not a clean one. That is true of most commercial voice vendors, which is why the canary call is the part of this page that generalises. Zian holds no SOC 2, ISO/IEC 27001 or HIPAA certification, so there is no attestation to substitute for the test either.
What is on the capability list is private model deployment on customer infrastructure, and it changes this audit more than anything else here: when the deployment is yours, the sinks are yours, the retention is yours to set, and step 5 stops being a request and becomes a query you run. Put the canary call to us, ask for the sink map, and score us on the same table as everyone else. The rest of the security-review furniture sits in our enterprise readiness checklist for AI sales agents.
Frequently asked questions
How do I check whether a voice platform writes our API keys into its logs?
Revoke a canary key, trigger the component that uses it, and search every log sink for the literal key value. A rejected authentication handshake is the condition under which a credential most often reaches a log, because the exception object can carry the request headers. MITRE catalogues the class as CWE-532, Insertion of Sensitive Information into Log File, whose description is simply that the product writes sensitive information to a log file.
Does a closed issue on a vendor tracker mean the problem is fixed?
No, and this is the most common mistake in the method. Of four credential and PII logging issues checked across LiveKit and Pipecat repositories on 16 September 2026, all four carried a closed-completed state and only one had been closed by a merged code fix. Two were closed by a maintainer explaining the problem would be solved another way, and one as a no-op against an upstream advisory that is still unpatched. Open the thread and read the closing comment.
What should I grep my own call logs for?
Two things. The literal value of a credential you issued, because you hold the string and do not need to guess a prefix. And every E.164 phone number in the export, not just the one you tested with, because caller data most often escapes through identifiers such as room names, job IDs and recording filenames rather than through transcript text.
Is a room name really personal information?
It can be, and at least one platform documents that plainly. The LiveKit dispatch rule reference, read 16 September 2026, warns that an individual dispatch rule names each room after the caller phone number, that room names are recorded in logs and traces throughout LiveKit, and that they are not removed by PII redaction. Treat identifiers as in scope for your retention and access rules.
Can I run this audit on a closed-source platform?
Partly. The four tracker queries need a public repository, so they return nothing for a closed-source vendor, which is absence of evidence rather than evidence of safety. The canary call, the sink enumeration and the sink map work regardless, because they test the deployment in front of you rather than the upstream project.
Apply For Partnership
If you want to run the canary call and the sink map against us, that is the conversation we want in beta. Apply For Partnership and bring your sink map template with you.