AI Hallucinations in Healthcare: Real Cases, Costs and the Safeguards That Work

An AI hallucination in healthcare is output that reads as fact but isn't supported by the record or the evidence: a symptom the patient never mentioned in a visit note, a plan step no clinician chose, a confident answer to a clinical question that's wrong. It costs more here than in most industries because the error can reach a chart, a patient or a regulator before anyone checks it.

A better model alone doesn't stop it. What stops it is a check on every output by an evaluator that didn't write it, a fix or a human review for anything that fails, and a record of both. This guide covers where healthcare AI hallucinates, real cases, what the errors cost and the safeguards that work.

Where AI Hallucinates in Healthcare

Hallucinations appear wherever a model writes something a person will act on. In healthcare, that's five places:

• Clinical documentation: ambient scribes and visit summaries that add, drop or change what was said.
• Patient-facing chatbots and messages: answers to health questions that sound expert and are wrong.
• Clinical decision support: suggested diagnoses, tests or orders the evidence doesn't support.
• Education and assessment: AI that teaches or grades nursing and medical skills.
• Operations: prior authorization letters, coding and billing summaries built from the chart.

The dangerous hallucinations are the plausible ones. An obviously wrong sentence gets caught. A dose, a negation or a date that looks right gets copied forward. The seven types of AI hallucination each need their own check.

Real Cases

Transcription that invents text

In October 2024 the Associated Press reported that OpenAI's Whisper speech-to-text model sometimes invents text, including, according to researchers it interviewed, "imagined medical treatments" (Associated Press, October 26, 2024). One study it cited found 187 hallucinations in more than 13,000 clear audio snippets. OpenAI had warned against using the tool in "high-risk domains."

The same report found that over 30,000 clinicians and 40 health systems had started using a Whisper-based tool from Nabla, which said the tool had transcribed an estimated 7 million medical visits. Nabla's tool erases the original audio "for data safety reasons," so a transcript can't be checked against the recording. Nabla said providers must quickly edit and approve the notes.

The lesson: if the source is deleted, no one can catch the error later.

A vendor's accuracy claim, challenged by a state

In September 2024 the Texas Attorney General settled with Pieces Technologies, a Dallas company whose generative AI summarized patients' condition and treatment for staff at at least four major Texas hospitals. Pieces had advertised a "severe hallucination rate" of "<1 per 100,000." The Attorney General's investigation found those metrics "likely inaccurate" and said they "may have deceived hospitals" (Texas Attorney General, September 18, 2024).

The settlement included no payment. For five years, whenever Pieces advertises an accuracy metric, it must say what the metric means and how it's calculated, unless an independent auditor backs the claim (WilmerHale summary of the settlement). Pieces "vigorously denies any wrongdoing" and pointed out that no industry-wide way to classify hallucinations in inpatient clinical summaries exists today (HIPAA Journal).

The lesson: an error rate means nothing until you know how it was defined and measured.

Patient safety experts now rank AI as the top hazard

ECRI, an independent patient safety organization, put AI-enabled health technologies first on its 2025 list of health technology hazards. It warned that AI systems "can produce false or misleading results, or 'hallucinations'" (ECRI, December 4, 2024).

For 2026, ECRI ranked misuse of AI chatbots in healthcare first. In one of its tests, a chatbot said it was appropriate to place an electrosurgical return electrode over a patient's shoulder blade, advice that would put the patient at risk of burns (ECRI, January 21, 2026).

How Often Clinical AI Gets It Wrong

Measured carefully, the rate is low, and the errors that remain land where they matter. A May 2025 study in npj Digital Medicine had clinicians check 12,999 sentences in 450 AI-written clinical notes, line by line, against the consultation transcripts (Asgari et al., npj Digital Medicine, 2025):

• 1.47% of the note sentences were hallucinations: 191 sentences.
• 44% of those were major: they could change a diagnosis or treatment if left uncorrected.
• Made-up content was the most common type, and it appeared mostly in the plan section of the note.
• 3.45% of transcript sentences were left out of the notes, and 16.7% of those omissions were major.

Two things follow. A rate near 1.5% sounds small until it's multiplied by every note a health system writes in a year. And omissions count: a note that's accurate but leaves out an allergy is still unsafe. The same study cut major errors by changing its prompts and workflow, which only works when every change is measured.

What Hallucinations Cost a Healthcare Organization

The cost isn't the wrong sentence. It's what happens after someone acts on it.

• Patient harm. A wrong dose, a missed allergy or bad advice reaches the patient when no one checks.
• Liability stays with the clinician. The Federation of State Medical Boards says the physician "is ultimately responsible for the use of AI and should be held accountable for any harms that occur" (FSMB policy, April 2024).
• Regulators read accuracy claims. The Texas case was brought under existing consumer protection law, with no new AI law needed (Manatt).
• Review time eats the savings. If clinicians must recheck every AI note by hand because they can't tell which ones are wrong, the time the AI saved is gone.
• Lost trust. One confident, wrong note teaches clinicians to ignore the tool, including when it's right.

Safeguards That Work

The safeguards that work share one idea: check every output before it's used, and keep a record of what happened to each one.

1. Check every output, not a sample

Score each output before it reaches a chart or a patient. DeepRails' guardrail metrics score how much of an answer is verifiably true (Correctness), how closely it sticks to the source it was given, such as a transcript or chart extract (Context Adherence), whether it answers every part of the request (Completeness) and whether it contains unsafe content (Comprehensive Safety) (DeepRails guardrail metrics).

2. Use a judge that didn't write the answer

A model checking its own work tends to miss its own mistakes. DeepRails' evaluation engine judges each part of an output with two different models in parallel, and the model that wrote the output never judges itself (Multimodal Partitioned Evaluation). Here's why using one AI to grade another needs care.

3. Set the bar by risk

A scheduling reminder and a medication summary don't need the same bar. In DeepRails Defend, each use case is a workflow with its own metrics and thresholds, and any score below its threshold counts as a hallucination (hallucination classification).

4. Fix it, or send it to a person

A failed output shouldn't reach anyone as it is. Defend can repair it (FixIt), write a new one (ReGen) or record the failure for your own process (Do Nothing), and it scores each new version again before it's returned (Defend overview). For clinical content, send anything that still fails to a clinician.

5. Keep the source and the record

Keep the input, the output, the scores and every attempt to fix it. Defend logs each attempt with its scores, its pass or fail and its output, so you can show later what was checked and why it passed (Defend details). The Whisper case shows the opposite: without the source, no one can check the output.

6. Govern it like any clinical tool

Ask every AI vendor what its error rate means, how it was measured and on what data. ECRI recommends AI governance committees, AI training for clinicians and regular audits of AI tools' performance (ECRI, 2026).

In Practice: Grading Nursing Skills at Scale

DeepRails built AI grading for a healthcare education institution that evaluates 90,000 nursing skill videos a year across 59 skill assessments (read the case study).

The first problem wasn't the AI. The same video could get "Meets" from one grader and "Partially Meets" from another, so DeepRails worked with the lead nursing evaluators to rebuild the rubrics into precise, checkable criteria. Results the system isn't confident about go to faculty reviewers instead of being guessed. The program set its own bar, and the AI cleared it: 92% agreement with expert graders.

Questions to Ask Before You Deploy

1. Which AI outputs reach a chart, a patient or a claim without a person reading them?
2. What is each output checked against: the transcript, the chart or a guideline?
3. Is the checker a different model from the one that wrote the output?
4. What happens to an output that fails: fixed, rewritten or sent to a clinician?
5. Can you show, months later, what was checked and why it passed?
6. How does your vendor define and measure its error rate?