“Our AI gave medical advice”: the first 24 hours - Zian AI

“Our AI gave medical advice”: the first 24 hours

Preserve the audio and the transcript before you touch the agent configuration. An ordinary Australian private practice has no 24-hour legal clock here: where a mandatory notification to Ahpra is engaged at all, the National Law sets no date, only as soon as practicable, and the one published 24-hour candour deadline, the Victorian one, binds hospitals and day procedure centres rather than clinics.

What actually happened on that call

A booking agent is a general-purpose language model with a booking instruction layered on top. That instruction is a preference expressed to the model, not a filter in front of it, and three mechanisms account for most of these calls.

One: the model answers from its own training when retrieval finds nothing. Your knowledge base holds hours, fees and practitioners. The caller asks whether the swelling can wait until Tuesday. Nothing matches, and a model that is fluent by construction produces a fluent sentence anyway.

Two: your own website is the source. It is common to load the practice website into the knowledge base wholesale, and those pages carry clinical prose — the “conditions we treat” section, written by a clinician for a different purpose. The agent did not invent the advice. It read it back.

Three: the model leaves the script entirely. LiveKit Agents issue 5662, filed 6 May 2026 and still open when we checked it on 18 September 2026, reports Gemini models in SIP telephony sessions speaking their entire system instruction block to the caller and emitting pseudo-code instead of calling a tool. In the reported scenarios the system prompt ran to about 7,500 characters, and the reporter writes that the model “breaks out” of the tool-calling mode and treats the system instructions as the next sequence to predict. That issue is not a report of an agent giving clinical advice, and we are not citing it as one: it is maintainer-tracked evidence that a configured agent can leave its instructions, which is the mechanism rather than the content.

Where the line sits is settled ground and not this page: the privacy and medical-device boundary for AI phone agents in Australian dental and medical practices covers the Privacy Act and TGA analysis, and the triage limits for AI receptionists at Australian vet clinics works it from the veterinary statutes. This page is about the call that already happened.

Is this as urgent as it feels?

Usually less urgent than it feels at 11pm. Four things would make it genuinely urgent, and none is “the agent said something clinical”.

  • Someone acted on it and came to harm. A medico-legal matter from the first minute; ring your indemnity insurer and a clinician, not a software vendor.
  • A registered practitioner at your practice is implicated. Not the software — a person.
  • Personal information went to the wrong human being. A different obligation, on a different clock.
  • The caller has already complained in writing to a body. The timetable is then theirs, not yours.

If none of those is true, you have a configuration defect and a call to make. Both matter. Neither is a deadline.

Be precise about the deadline that does exist, because it is the one people half-remember. Victoria legislated a statutory duty of candour from 30 November 2022 under s 128ZC of the Health Services Act 1988 (Vic). Requirement 1 of the Victorian Duty of Candour Guidelines, a legislative instrument made by the Minister for Health under s 128ZF, gives a health service entity no longer than 24 hours to apologise and provide initial information after a serious adverse patient safety event is identified; Requirement 3 gives ten business days to hold the meeting. That is a real 24-hour clock, and it runs only on the entities the Act names. In the authorised consolidated text as at 4 September 2026, s 3(1) defines a health service entity as a public health service, a public hospital, a multi purpose service, a denominational hospital, a private hospital, a day procedure centre, an ambulance service, a licensed non-emergency patient transport service, Youth Mental Health and Wellbeing Victoria, the Victorian Institute of Forensic Mental Health, or a prescribed entity that provides health services. Youth Mental Health and Wellbeing Victoria was added on 30 October 2024 and is not in the 2022 Guidelines list. A suburban general practice, dental surgery or physio clinic is on neither.

Ahpra sets no number at all. Section 141(2) of the Health Practitioner Regulation National Law requires a practitioner who forms the relevant reasonable belief to notify the National Agency “as soon as practicable”, and the Ahpra guidelines on making a mandatory notification say a notification to Ahpra should be made as soon as practicable.

What to do in the next 24 hours: freeze four artefacts

The instinct now is to open the dashboard and fix the prompt. Do not, and do not assume the platform has kept a record of what the agent was told. Some have: Vapi documents assistant versioning that keeps every published version so you can review what changed and when. On a platform without it, the prompt as it stood during the call is gone the moment someone saves. Freeze before you fix: four artefacts, captured out of the platform and into storage you control, before any configuration change. It costs nothing. The general version of this, seven artefacts and the published default retention window for each platform, is already on our page on what to do when an AI agent tells a customer something untrue; what follows is the health practice cut of it, and the regulatory half below is what that page does not cover.

Artefact Where it usually lives What destroys it
The call audio The platform call record, or your own storage bucket A retention setting, a zero-retention mode, or deleting the call
The transcript The same record, sometimes only in the end-of-call webhook payload Privacy modes. Under Vapi Zero Data Retention no transcript is kept at all
The configuration as it was System prompt, knowledge base documents, model identifier, tool definitions Your own repair. Copy it to a dated file first
The call metadata Telephony provider logs: timestamp, call identifier, caller number, duration Provider log retention windows. Confirm yours, do not assume

The second row catches careful practices hardest: the more privacy-hardened the configuration, the less likely any recording exists. Vapi documents that it records calls and stores logs and transcriptions by default; its HIPAA mode moves storage and, in the vendor description, “limits access to certain features, such as reviewing call logs or transcriptions”; Zero Data Retention removes the content entirely. Check your mode before assuming there is anything to preserve. What you may lawfully do with an Australian recording is covered in the call recording laws that apply to AI agents in Australia.

Then, still inside the first day, and still free:

  1. Disable the intent, do not rewrite it. Route every clinical-sounding turn to a human or your after-hours number tonight. A rewrite needs testing you do not have time for; a hard route needs none.
  2. Read the transcript yourself. What the agent said is almost never what the caller reported, in either direction.
  3. Have a clinician contact the caller. Not the practice manager, not the vendor, not an email. Say what the agent said, say it was not clinical advice, say what the correct next step is. Ordinary open disclosure resolves most of these.
  4. Write a contemporaneous file note. Date, time, who listened, what was said, what was done. Ten minutes, and it is what your insurer asks for.

Who owns this, and whether a notification obligation is engaged

Three of the eight rows below engage no notification obligation at all, and the only flat yes belongs to Victorian hospitals and day procedure centres rather than to a clinic. That is the honest shape of it, and why a threshold is more useful than a warning.

What happened Notification obligation engaged? Who owns it, and on what timing
The agent read back published clinic policy and the caller found it alarming No Nobody. Log it, fix the wording
The agent offered an opinion about one caller and nobody acted on it No. Section 140 of the National Law defines notifiable conduct in relation to a registered health practitioner, and software is not one Your practice. Contact the caller, document it
A registered practitioner configured or supervised the line in a way that may be a significant departure from accepted professional standards, placing the public at risk of harm Possibly, under s 140(d), via s 141 (practitioners) or s 142 (employers) Ahpra. Section 141(2): as soon as practicable, no fixed number of days
Someone acted on the answer and suffered a complication Your policy, not a statute Your medical indemnity insurer. Avant states members must notify if a patient suffers a complication from treatment or if an incident may lead to a claim or complaint
Personal information was accessed without authorisation, disclosed to the wrong person, or lost Possibly. Needs unauthorised access, disclosure or loss, plus a likely risk of serious harm OAIC. All reasonable steps to complete the assessment within 30 calendar days of becoming aware of the grounds for suspicion
You are a Victorian private hospital or day procedure centre and a patient sustained moderate harm, severe harm or prolonged psychological harm Yes, the statutory duty of candour The patient, under the Victorian Duty of Candour Guidelines that Safer Care Victoria publishes. Apology and initial information within 24 hours; the meeting within ten business days
The caller lodges a complaint No self-notification obligation. The route is theirs The state health complaints entity. In NSW the Health Care Complaints Commission takes complaints about health organisations as well as practitioners and aims to assess complaints within 60 days
Your vendor markets the agent as assessing symptoms or urgency A medical-device question about the intended purpose stated by the manufacturer The TGA, and your vendor. Analysis on the dental and medical practices page above

One thing is yours regardless of the row. Ahpra and the National Boards put it this way on their artificial intelligence resource, reviewed 22 August 2024: “Regardless of what technology is used in providing healthcare the practitioner remains responsible for delivering safe and quality care”, and “Practitioners must apply human judgment to any output of AI.” No configuration moves that.

The next 7 days

The single call is rarely the problem. The question for the week is how many other callers got the same behaviour and never mentioned it.

  1. Pull the whole window, not the one call. Export every call since the configuration last changed and filter transcripts for the shape of the failure, not clinical vocabulary. Callers say “is this normal”, “should I be worried”, “can it wait”. Keyword lists built from clinical words miss almost all of it.
  2. Count and state the scope. Calls in the window, calls where the agent formed a view, calls where the caller could have acted on it. Three numbers decide whether you contact one person or twelve, and they are the first thing an insurer asks for.
  3. Tell your insurer in writing, even if nothing came of it. The Avant position above is that notification by itself has no adverse impact on claims history. Ring your own insurer and ask; do not take a web page for it.
  4. Decide whether a practitioner is implicated. If the answer is even arguably yes, that belongs to a medico-legal adviser or the relevant National Board, not to your IT provider and not to us.
  5. Write down what you changed and when. The dated record turns an incident into a defensible process.

The fix that stops it happening again

A better system prompt is not the fix, for the reason in the first section. Four changes, in order of how much they buy you.

  1. Deny by intent, at the routing layer. Classify the turn before the model answers it, and route is-this-an-emergency, can-it-wait, what-should-I-do and is-this-medication-safe straight to a human. A classifier that fires outside the model cannot be talked out of firing.
  2. Check the output, not just the input. A second pass over what the agent is about to say catches the cases where the caller never asked a clinical question but the agent volunteered one. Patterns are in the guide to writing guardrails for autonomous AI agents.
  3. Take clinical prose out of the knowledge base. Treatment pages belong on your website, not in the retrieval index of a booking agent. A half-hour job that removes a whole class of failure.
  4. Keep a regression set of ten real calls and replay it after every change. Prompt edits, model bumps and new knowledge-base documents all reopen intents you closed last month.

The honest economics: steps 1 to 3 are about a day of work once. Step 4 is the one that lapses — nothing triggers it, nobody notices it was skipped, and the cost is another recorded call in which your practice said something only a registered practitioner may say. Whether that is worth outsourcing is your arithmetic, and the architecture that makes it repeatable is in the compliance architecture for AI sales agents in regulated industries.

Where Zian sits

Zian AI builds autonomous phone, SMS, email and WhatsApp agents for outbound work, with SmartReach AI™ orchestrating message, channel and timing and PrecisionPitch AI™ split-testing approaches. It is an outbound sales platform, not a medical receptionist product, and we will not pretend otherwise to be useful to you tonight. If the problem is an inbound clinic line, the four fixes above are the work, and the people to involve are your clinicians, your indemnity insurer and your software provider. If the work is outbound — recalls, reactivation, follow-up — the same boundary applies word for word. Zian has run outbound acquisition since 2017 and is in partnership-application beta. Apply For Partnership.

Frequently asked questions

Does an AI receptionist giving medical advice trigger a mandatory notification to Ahpra?

Not by itself. Section 140 of the Health Practitioner Regulation National Law defines notifiable conduct in relation to a registered health practitioner: practising the profession while intoxicated by alcohol or drugs, sexual misconduct in connection with the practice of the profession, placing the public at risk of substantial harm because of an impairment, and placing the public at risk of harm by practising the profession in a way that constitutes a significant departure from accepted professional standards. Software is not a registered practitioner. What determines the answer is whether the conduct of a person at your practice falls inside one of those four, and section 141(2) then requires notification as soon as practicable rather than within a fixed number of days.

Is this a notifiable data breach?

Usually not. The scheme is triggered by unauthorised access to, unauthorised disclosure of, or loss of personal information, plus a likely risk of serious harm. An agent that said something clinical to the caller it was already speaking to has disclosed nothing to anyone else. If a recording went somewhere it should not have, that is a different question, and an entity must take all reasonable steps to complete the assessment within 30 calendar days. The OAIC guidance on the notifiable data breaches scheme sets out the test.

How long do we have to tell the caller?

There is no general statutory deadline on an ordinary Australian private practice. The only published clock of that kind is Victorian, it gives no longer than 24 hours to apologise and provide initial information after a serious adverse patient safety event is identified, and it applies to a defined list of health service entities that includes private hospitals and day procedure centres but not ordinary clinics. Elsewhere, contacting the caller promptly is professional practice rather than a legal deadline. Do it the same day anyway.

Should we delete the recording?

No. Preserve it and decide later: the recording is the only reliable account of what the agent said, and your indemnity adviser or a complaints body will ask for it. Some platform settings mean it never existed. The Vapi documentation on Zero Data Retention states that with the mode active it retains no recordings, transcripts, messages, summaries or structured outputs, and that call content not sent to your own systems cannot be recovered.

Related Blogs

Related from Zian AI