At Ambience, our clinical speech recognition pipeline processes audio incrementally, building a transcript as new audio arrives.

In one patient encounter, our pipeline transcribed “Brilinta,” a medication, as “Burlington,” a city. The words sound similar, but the substitution lost a clinically important detail before the note was ever written.

Our challenge was to reduce errors like that while maintaining low-latency transcription. That meant working with incomplete audio: transcribing the available signal even though subsequent audio might change its interpretation. We explored a series of changes to our streaming pipeline, examining which errors each approach corrected and which it left behind. This post walks through those experiments and the design decisions that made a difference.

Read full paper here.

Where we started

Our starting point was already a step beyond raw ASR. We started with a single, domain-adapted, streaming speech-recognition model. This model is fine-tuned on clinical audio. Off the shelf ASR models are strong, but to perform well across many different clinical settings, specialties, and regions, an in-house fine-tuned model is necessary. Incoming audio is first segmented by a voice activity detection (VAD) step: a lightweight model that listens for when someone is actually speaking and splits the stream into natural chunks. Each chunk is transcribed as it arrives.

This pipeline was already a significant improvement over off the shelf open source ASR models or even proprietary ones.

First move: chart-aware context

A fine-tuned model transcribes what it hears. It has no idea who is in the room.

“Burlington” is a perfectly reasonable guess at that audio. It is only a bad guess if you know the patient is on an antiplatelet.

The recognized text goes to a reconciliation LLM that uses context from the patient’s chart to improve transcription. That context helps distinguish between similar-sounding words, such as “Brilinta,” which is on the patient’s medication list, and “Burlington.”

The result: keyword error down 57%. Normalized keyword error down 62%.

This was the single largest improvement we measured, and it came from a grounded reconciliation pass rather than a bigger model. But it is still one pass over one hypothesis: whatever the single ASR model missed acoustically, no amount of chart context can recover.

Second move: a second pair of ears

Different speech-recognition models fail differently. A large fine-tuned model is strong on structure and narrative coherence, but may stumble on rare medication names. The other, a smaller model trained specifically on medical terminology is especially strong on drug names but struggles with unseen proper nouns, a consequence of training on de-identified data.

We stopped choosing between them and started running both in parallel.

Both ASR models receive the same chunks, so their outputs stay aligned and the reconciliation LLM can compare them word for word.

Each audio chunk gets transcribed by both models simultaneously. Their outputs go to the reconciliation LLM, which compares two competing transcripts, identifies disagreements, and arbitrates using the structured patient context that neither model had access to on its own. The LLM reads both outputs and makes a judgment call grounded in clinical evidence.

The result: keyword error down 24%. Normalized keyword error down 27%.

Most of the gain came from medication names and clinical terms.

We call this configuration mini-batch ensemble. It worked well, but it had a structural limit.

Once a chunk is processed, its audio is gone. If the model makes a mistake at a chunk boundary, no later step can fix it.

The problem at the boundary

Here’s a motivating example of the problem. Consider a clinical utterance with a brief pause mid-sentence:

“I can’t 〈pause〉 recommend surgery at this time”

With a short audio chunk ending at the pause, the trailing “t” in “can’t” gets swallowed. The mini-batch ensemble outputs:

I can recommend surgery at this time

Despite both sentences being grammatically and medically sound, the meaning has completely flipped. The LLM cannot make up for the information it has lost in the audio.

With a wider audio window that overlaps the pause boundary, the system hears enough context to get it right:

I can’t recommend surgery at this time

The information needed to get this right was never missing from the stream. It just wasn’t inside the chunk. Revisit that decision once more audio arrives, and the prosody and surrounding context make the negation obvious.

Third move: let the system change its mind

Instead of processing each chunk once and moving on, the rolling-window ensemble architecture keeps a sliding window of recent audio in scope. On each cycle, the system transcribes the newest audio along with the last several buffers, so that the reconciliation LLM always sees overlapping context across chunk boundaries.

The critical difference: previous output is now revisable as a function of increased audio context. If a later cycle reveals that an earlier transcription is wrong thanks to a larger audio window, the LLM can revise it.

The result: keyword error down 16%. Normalized keyword error down 17%.

Same speech-recognition models. Same reconciliation LLM. Same structured context. The only difference is whether the audio stays in scope long enough for the system to accurately transcribe it. This is not free. A wider window means more audio to transcribe and more text to reconcile on every cycle, so the median cycle takes roughly 40% longer. The slowest cycles are no worse, and the pipeline continues to produce incremental transcripts.

What we measured

Across a 226-encounter evaluation spanning 28 specialties and 15 clinicians, the gains compound. Chart context did the most, but each later stage removed a class of error the ones before it could not reach:

A note on the evaluation metric

Keyword error rate (KER) depends on an LLM tagging which words are clinically significant (medication names, dosages, proper names) in both the reference and the model’s output, then counting how many go missing. Normalized keyword error is the same measure ignoring case and punctuation. We chose it because it scores the errors that actually reach the chart. WER weights a dropped “the” the same as a dropped beta blocker. On our main comparisons, KER and WER moved in the same direction, which is some assurance the gains aren’t an artifact of the tagging step. Entity-level metrics still don’t capture severity, and LLM-as-judge scoring and clinician-satisfaction measures are where we’re taking the evaluation next.

What we learned

The biggest improvement did not come from asking the LLM to reason harder. It came from changing what evidence the system was still allowed to use. Rolling-Window Ensemble kept recent audio in scope long enough for later context to correct an earlier transcription.

That distinction reduced errors while preserving incremental transcription. The improvement was upstream, so everything downstream starts from a cleaner transcript.

The broader lesson is simple: in systems built on noisy, real-time inputs, deciding when an output becomes final is part of the model design.