Is your AI scribe making things up? An accuracy checklist for fire and EMS

A vendor shows you an AI tool that writes your crews’ narratives. The demo looks great. A clean chart appears in about ten seconds.

But one question stays in your mind: What if it makes something up?

That is the right question. Every chief and medical director asks us. We would ask it too.

A patient care report is a legal record. Hospitals, quality assurance (QA) teams, attorneys, and sometimes juries may read it. A writing tool carries more risk than an inventory tool. If it adds the wrong fact, someone could get hurt. Your department is responsible for the final chart.

We will not tell you to stop worrying. We will give you a clear way to test the tool.

What “making things up” actually means

What people mean is that the AI adds something that was not said on scene.

A general-purpose model like ChatGPT is trained to produce text that sounds right. When information is missing, it can fill the gap with something that fits the pattern. Maybe no one said the blood pressure. Maybe the audio dropped for four seconds. In a chart, the result could be a blood pressure no one took, a lung sound no one heard, or a denial of chest pain the patient never gave.

That is the risk. It is real, and it is why this checklist exists. Pasting patient details into a consumer chatbot also creates risks beyond HIPAA. We cover that topic in our guide to writing EMS narratives.

A tool built for EMS can be designed to leave a gap blank. Your job is to make sure it does.

The checklist: six questions to ask any AI documentation vendor

Ask us these questions. Ask every other vendor too. If a vendor will not give you a clear answer, pay attention.

1. Does it ever add facts that were not said? Do not accept “it’s very accurate.” Ask for a written yes-or-no answer: does the system create clinical details that no one said on the call? If a vendor will not put the answer in an email, do not expect the vendor to defend it in sworn testimony.

2. Does a person review and sign the chart before it is filed? No AI should write directly into an electronic patient care report (ePCR) without review. The medic who ran the call must read the draft, fix it, and sign it. Automatic filing may save time, but it also adds risk.

3. Can you see the source? The narrative should link back to a transcript of what was said on scene. Without that source, you cannot check the draft. Ask where the transcript is stored and how long it is kept.

4. What happens offline or when the audio is bad? Think about basements, rural roads, a running engine, a screaming child, or a patient who mumbles. When the system cannot hear a word, does it guess? Or does it leave a gap for the medic? Those are two very different products.

5. Who tests it? A score from the vendor’s lab is not enough. Ask for a field trial where your crews use it on your calls. Have them compare each draft with what was said. A vendor confident in the tool should agree.

6. What record will remain if someone challenges a chart in two years? A complaint or lawsuit may arrive long after everyone forgets the call. Ask whether the record shows when and where the session happened, how long it ran, and who approved the chart. This record is called an audit trail.

How vitalvoice answers each one

Here are our answers in the same order. Hold us to them.

The draft comes only from what was said on scene. The product is built around that rule, and the medic’s review is the check on it. We will put it in writing for your compliance officer.

A medic reviews, edits, and signs every draft. Ambient Scribe creates a draft. Nothing reaches the ePCR until the person who ran the call reads and signs it. That step is required.

Every narrative has a source transcript. The draft comes from what was said on scene. A medic can check the chart against that source. Encounter content, including the transcript, stays encrypted on the crew’s device for 90 days by default, or on the department’s wipe schedule. Our servers process it and discard it: no transcript, narrative or audio is stored there. A Business Associate Agreement (BAA) is available.

It works offline, and it is built to leave gaps blank. vitalvoice runs without cell signal. That matters in basements and on county roads. When audio is unclear, the draft is designed to leave a gap instead of making a likely guess. A blank for the medic to fill is safer than a number no one said.

Your crews test it. Every department starts with a 30-day field trial. Your medics use it on real calls and compare the drafts with what they remember saying. CityEMT’s crews did that, and 95% would recommend it (CityEMT). Crews rate the app 5.0 on the App Store.

Every session keeps its time and GPS location. We keep this operational metadata, which we treat as ePHI, along with a hash-chained audit log. If someone reviews a chart two years later, there is a record of when and where the session happened.

The comparison that actually matters

When you judge an AI scribe, you may compare it with a perfect scribe. That person would never mishear, leave something out, or add an opinion. No software can pass that test.

But that is not the real choice. Today, a medic may write from memory at hour 47 of a 48-hour shift. The call may have happened three calls earlier. A family member may interpret during a stroke check. A phone interpreter may leave no record to review later.

Those methods can also produce mistakes. The errors may be hard to find because there is no source to check. A tool with a transcript, review step, and timestamp makes gaps easier to see. That matters in a legal record. If you are also making your narratives more consistent, read our guide to EMS narrative formats.

Hold AI to a high bar. Hold your current process to the same bar.

---

Quick answers

Can an AI scribe invent vital signs in an EMS narrative? A general-purpose model can add text that seems to fit a gap. A tool built for EMS should not. vitalvoice drafts only from what was said on scene. The medic reviews and signs the draft before anything reaches the ePCR.

How do I verify an AI-generated narrative is accurate? Check the draft against the source transcript. Make sure a person reviews and signs it. If a vendor cannot show you the transcript, you cannot check where the details came from.

Does an AI scribe work without cell signal? vitalvoice does. It runs offline, and when audio is unclear it is built to leave a gap for the medic instead of guessing.

Run this checklist against us

Chiefs and medical directors: Book a 30-minute demo and bring all six questions. We will answer each one. Then start the field trial so your own crews can compare the drafts with what they said on the call.

Crews: Download vitalvoice from the App Store. Each mode includes free sessions. Use one on your next call, then read the draft and check every detail before you sign.