Extracting Biomedical Entities from Noisy Audio Transcripts

Automatic Speech Recognition (ASR) technology is pivotal in converting spoken language into written text and finds critical applications within clinical contexts. One important use is expediting medical transcription processes and efficiently documenting doctor-patient interactions. This seamless conversion reduces the time and resources traditionally spent on manual transcription, affording healthcare professionals more time for focused patient care. Specifically, ASR can seamlessly integrate into Electronic Health Record (EHR) systems, enabling real-time dictation of diagnoses, treatment plans, and patient notes, thereby augmenting the accuracy and immediacy of clinical documentation. Hence, this technology holds substantial promise in revolutionizing healthcare documentation practices. After successful conversion from audio to text, natural language processing (NLP) tools can be applied to the transcriptions for various tasks Szymański et al. (2023). Unfortunately, transcription is not accurate, particularly in noisy environments. Moreover, when NLP models are applied to noisy data that does not match the training data distribution, large drops in performance may be observed.

This paper focuses on the biomedical NLP entity recognition (NER) task applied to noisy audio transcripts. Named entity recognition is vital for many important clinical tasks, from extracting social determinants of health mentions from clinical notes to extracting mentions of adverse drug reactions. Clinicians may not be able to capture everything stated to them by a patient (e.g., specific adverse reactions to a drug), particularly if they need to transcribe information after an interaction via rote memory. Hence, if ASR can be used to record patient-clinician interactions, then NER systems can be applied to extract clinically relevant information for later use. We explore the viability of NER systems applied to noisy transcripts to better understand their performance in real-world settings, where records may have multiple speakers and background sounds.

Szymański et al. (2023) has recently called this difference in performance the ASR-NLP gap. At a high level, there are two primary causes for the ASR-NLP gap. First, transcription errors can completely change the words mentioned. For instance, if someone mentions the word “headache” (which could be a mention of a drug side effect), but if it is recognized as “headway,” then a traditional NER system would be unable to identify it. Second, the data distribution changes. Models trained on clean, non-transcribed data may capture different patterns in the text that are not available in the transcribed text. The patterns may be as simple as differences in punctuation and capitalization, but such patterns are essential for accurate NER.

Much of the prior work on studying ASR systems, particularly in biomedical domains, has focused on either developing or evaluating ASR systems for novel patient populations Tran et al. (2023) or training and evaluating NLP systems on carefully corrected and relatively clean transcripts. For work evaluating ASR systems in the clinical domain, there have been low word error rates (WER) reported (e.g., 11% Tran et al. (2023) 24.3% Hacking et al. (2023), and 10% King et al. (2023)). However, the studies often report results on relatively clean recordings (e.g., without multiple background speakers or substantial background noise). Sometimes transcripts that are very noisy are completely removed from the evaluation data King et al. (2023), potentially resulting in overly optimistic performance. Prior works have reported WERs much worse than the reported numbers in the clinical setting Kodish-Wachs et al. (2018), with WERs in the range of 30% to 60%. Moreover, Kodish-Wachs et al. (2018) also evaluated concept extraction software on transcriptions. However, they did not compare the performance difference between clean data and noisy transcripts. The numbers are still generally reported on “clean” transcripts with minimal background noise and background speakers. Finally, they do not provide any natural next steps for improving performance. Hence, the results may be much worse when evaluating substantially noisy environments.

In this paper, we develop a new dataset so biomedical NLP researchers can directly improve and explore the biomedical ASR-NLP gap. Specifically, we introduce a dataset that extracts adverse drug reaction mentions and a dataset that extracts fruits and animals that would be mentioned as part of the Brief Test of Adult Cognition by Telephone (BTACT) exam. To the best of our knowledge, this will be the first publicly available dataset to allow for careful evaluation of the ASR-NLP gap in the biomedical domain.

In summary, based on current research gaps in the ASR-NLP gap for biomedical applications, this paper makes the following contributions:

1. (i)

   We introduce a novel dataset of nearly 2000 clean and noisy recordings for biomedical-related ASR-NER called BioASR-NER.¹¹1The dataset is available at https://zenodo.org/records/10864063.

2. (ii)

   We introduce a simple approach to improving model performance via a transcript-cleaning procedure using GPT4. We explore both zero-shot and few-shot methodologies for when ground-truth noisy and cleaned transcription pairs are limited.

3. (iii)

   Finally, we perform an informative error analysis showcasing the types of errors made by the transcription software, the type of errors GPT4 corrects, and the types of errors GPT4 cannot handle accurately.

In this work, we describe two major research lines relevant to this paper: Biomedical ASR-NLP, which includes work on Biomedical ASR technologies and NLP applied to transcriptions (clean and noisy if available), and Biomedical NER, which discusses some recent work on developing methods to extract biomedical entities from text.

This paper uses CADEC Karimi et al. (2015) and a Synthetic BTACT dataset. CADEC is a popular NER dataset for extracting adverse drug reactions from experiences written by patients, and the Synthetic BTACT dataset is a novel dataset we created that simulates questions of the Brief Test of Adult Cognition by Telephone (BTACT). For both datasets, we have research assistants read each item and record an audio file of the reading. We generate noisy audio files by merging the files of multiple speakers and background noises/sounds. A high-level overview of the data collection process is shown in Figure 1. The details of the curation and creation are described in the following subsections.

This study attempts to investigate the performance of NER systems on biomedical ASR-transcribed data. In this regard, after developing a set of baselines trained on original scripts, we transcribe the noisy audio and evaluate the NER performances on the noisy and original transcripts.

In addition, we introduce a simple framework to improve the NER system performance that requires no training on domain-specific transcribed data. We use the fourth iteration of the Generative Pre-trained Transformer (GPT4) OpenAI (2023) to post-process the ASR transcripts using its advanced capacity in contextual understanding and extensive knowledge, which covers a very broad range of biomedical concepts. During this post-processing, GPT4 is provided instructions to evaluate and refine the transcribed outputs to improve the downstream NER performance. In this regard, we study two approaches: zero-shot prompting and few-shot in-context learning.

In this section, we evaluate and contrast ASR-NER performances on biomedical noisy transcripts with those of the two proposed methods. We evaluate the models based on their performance in detecting the named entities as well as their categories. We use micro precision, recall, and f1 as the evaluation metrics in which a correct prediction happens only if the named entity and the corresponding tag category $\hat{y^{\hat{c_{i}}}_{i}}=(\hat{n_{i}},\hat{c_{i}})$ is predicted correctly and matches those of the corresponding original ground truth $y^{c_{i}}_{i}=(n_{i},c_{i})$. If a script has multiple pairs of the same named entity and category, we treat each as a separate prediction to account for repeated terms. This way, our evaluation would be very similar to CoNLL’s Sang and De Meulder (2003); however, we do not perform a Span detection or consider the BIO tags as the position of words changes in all of our noisy transcript datasets due to the existence of mistranscribed words.

Table 3 shows the performances on the CADEC dataset. As you can see, and was expected, the performance on the NER has significantly dropped (on average 62% of micro f1 scores) on the ASR noisy transcript data for different reasons, including mistranscribed words/terms, the words that are picked up from the background and those that are missed to name but a few reasons. Another reason is the covariate shift between the training and testing set. For example, the ASR output may write the full name of a drug/disease, but the original script may use abbreviations, or they may use different punctuation. PLM-based models’ performances have dropped less due to their robustness to covariate shift and their ability to handle Out-Of-Vocabulary (OOV) scenarios (in comparison with traditional pretrained embeddings such as GloVe). T5 performs consistently better on the recall and precision, which shows its superior robustness, and it would potentially be an ideal choice for fine-tuning if one decides to fine-tune the NER on the noisy transcript data.

The zero-shot prompting consistently improves the micro f1 score by an average of 0.14 across all models, which is 59% improvement, fixing the 22% of the drop (reduces the 62% drop to 40%). This shows the value and effectiveness of providing the context and GPT4’s capability of utilizing its knowledge to refine the transcripts. With the improvement of the transcripts and reduction in noise models, performances improved, and Flair pretrained embedding surpassed other models, consistent with the original scripts’ results. BioBERT and Flair improved more than others, especially T5, because GPT4 may replace some terms with new medical terms that T5 has not seen in the training set, but BioBERT and Flair PubMed embeddings are good at recognizing them.

The few-shot in-context learning additionally improves the micro f1 score by an average of 0.03 across all models. This is 79% improvement from the ASR noisy transcripts that happen consistently across all models and fixes the f1 drop by 27%. Furthermore, the difference between the performance of the models is more similar to those of the original scripts, even in comparison with the zero-shot prompting, which is because the post-transcribed scripts are more similar to the original ones. However, the BioBERT model, which is the best-performing model, has shown the most improvement by improving by 32% in terms of micro-F1. Flair and BioBERT are the best models for zero-shot and few-shot learning when used with GPT4 because of their pretraining on medical terms. We hypothesize that more specialized language models that use medical databases for pretraining can potentially improve performance.

Table 4 shows the performances on the Synthetic BTACT dataset. Similar to the CADEC dataset, we see that the performance significantly drops on the noisy transcripts, and zero- or few-shot learning will consistently improve the performance. The generalization of improvements across the two datasets suggests the effectiveness of the proposed approaches. However, the improvement on BTACT data is substantially lower than CADEC–on BTACT data, the micro-f1 drop reduces from 41.4% in the noisy ASR-NER to 35.9% using few-shot learning, i.e., 5.5% improvement. This is due to the unnatural style of the BTACT data and the lack of context to be leveraged by GPT4.

The error in ASR-NER can come from different sources, such as noise in the original script, NER errors, or ASR errors in which words may be mistranscribed or not transcribed from the background speaker by mistake. The noise from the original script includes grammatical errors, typos, or even biomedical misconceptions. Although in a general setting, we cannot address the input data noise, in an ASR setting, this causes the model to either mistranscribe or fix the issue, which causes a mismatch between the predictions and ground-truth tags. For example, (’flu like symtoms’, ’ADR’) is transcribed to (’flu-like symtoms’, ’ADR’). Another example, if the script talks about a drug that can stop multiple episodes of migraines, it should say

“It stops migraines”

rather than

“It stops migraine.”

GPT4 realizes these issues and fixes them using its biomedical knowledge.

The errors that come from the NER system cannot be remedied, but sometimes the noisy transcribed data can add terminologies or a combination of words, which results in errors in the NER model; for example, NER is prone to misidentify pronouns as ADRs, e.g. (we, ADR) or (i, ADR), especially when there are not many named-entities in the input text. Therefore, when the input is very noisy, and ASR cannot transcribe any ADR terms, the output includes many such errors. Furthermore, the covariate shift between the ASR transcript and the original script can also cause errors in the NER system. However, both types of errors are mitigated to some extent by few-shot in-context learning as the GPT4 has seen these errors and changes in the style of the transcript compared to the original script.

The error can also come from Whisper’s limited vocabulary or noisy predictions during uncertainty. For example, Whisper cannot detect many drug names, especially phonetically similar to a more common term. For example, it transcribes Arthotec as “arthritis” or “arthrotype,” but providing the context that the script is about adverse drug effects fixes the problem. However, due to the intrinsic randomness of GPT4, there are examples in which the term is not fixed in either zero- or few-shot learning–this happens for many other terms, and very commonly, one approach has the right answer, which suggests the use of ensemble approaches.

We also find unique situations where GPT4 hallucinates new contexts. For instance, The original text

“Both husband and wife on a low dosage (10 mg). We are having extreme reactions to heat.”

is transcribed by Whisper to

“There you have it everyone, some kind of low dose of temperature, we are having extreme reactions.”.

But, after applying GPT4, it hallucinates a change in temperature, e.g.,

“We both have extreme reactions to temperature changes.”

The new context has unexpected impacts on the NER models, e.g., “changes in temperature” is detected as an ADR. Future research is required to reduce these hallucinations.

This paper explores the ASR-NLP gap in the biomedical domain, particularly for noisy audio. This challenge is especially pronounced in biomedical Named Entity Recognition (NER) tasks. While advancements in ASR show promise in controlled environments, real-world noisy conditions present significant obstacles (e.g., lack of publicly available datasets). To address this, we’ve introduced the BioASR-NER dataset, offering a mix of clean and noisy biomedical recordings. Coupled with our innovative GPT4-based transcript cleaning approach, we’ve made strides toward mitigating transcription errors.

There are two avenues for future research. First, our methodology to fix transcripts is based on the transcription text. Incorporating the audio information, particularly with recent advances in transformer-based audio-representations Gong et al. (2022), could substantially improve performance. Second, other biomedical NLP-based tasks, particularly when applied to noisy transcripts, deserve attention. These tasks include tasks such as text summarization Mishra et al. (2014) and question answering Jin et al. (2022). How these models generalize when applied to audio-generated transcripts with background voices and noises is unclear. Hence, we will explore these tasks as future work.

This material is based upon work supported by the National Science Foundation (NSF) under Grant No. 2145357.

\c@NAT@ctr