A Question Answering Based Pipeline for Comprehensive Chinese EHR Information Extraction

The extraction of information from health records and clinical documents has a long history. In its early stages, rule-based and expert-based systems were commonly used methods for Information Extraction (IE) [12, 13]. Software such as cTAKES [14] and MetaMap [15], equipped with NLP analysis engines, proved to be beneficial for clinical tasks. However, the drawback of these methods was the need to create numerous rules for each specific task, which made them less efficient.

Consequently, machine learning and deep learning-based approaches have gained significant interest in research for more efficient IE methods in the clinical domain [16]. Support vector machine and conditional random field are widely used methods for detecting entities or events [17, 18]. Deep learning methods in the clinical domain have also benefited from transfer learning from the general domain. Apart from the popular BERT (Bidirectional Encoder Representations from Transformers) backbone[19], other strategies such as MT-clinical BERT that use multi-task learning for knowledge sharing among subtasks [20], OIE4KGC that employs a UMLS knowledge-graph enhanced model [21], and self-supervised and retrieval-augmented methods have shown effectiveness [22].

The applications of information extraction are diverse. For instance, phenotyping involves identifying diseases with specific features, like childhood obesity and neuropsychiatric disorders [23]. IE can also be employed in feature engineering for the classification of cancer types and stages [24, 25]. Additionally, drug-related studies can draw on extracted examples for insights [26], and the extracted information can be utilized to optimize clinical workflows or even provide clinical decision support [27, 28].

However, as far as our knowledge extends, no existing methods have utilized QA to assist in EHR IE tasks, such as NER and relation extraction. The use of QA offers the advantage of simultaneously addressing all the subtasks in a more convenient manner.

In our focus on Machine Reading Comprehension (MRC) QA, the model provides an answer based on both the question and a given context. This QA task encompasses four main types of answers: yes-no questions, multi-choice questions, extractive questions, and generative questions [10]. For the first three types, models typically use only the encoder/embedding to obtain word representations. The widely adopted three-stage architecture proposed by FastQA [29] consists of semantic understanding, interaction, and prediction. Since the emergence of BERT-like models, enhancing the semantic understanding stage has become the mainstream approach to improving model performance [30, 31, 32]. However, these three types differ in the predicting module. For instance, extractive QA predicts the probability of being start or end tokens, yes-no QA conducts binary classification [33, 34], and multi-choice QA may select the most plausible hypothesis from constructed options [35, 36]. The SQuAD 2.0 dataset introduced unanswerable questions for extractive QA [37], which is the task used to define our problem.

On the other hand, generative QA always requires an additional decoding module and is more commonly used in open-domain QA rather than machine reading comprehension. Interaction-aware embeddings are fed into a seq2seq decoder [38], transformer decoder [39], or other structures like T5 [40], GPT, etc. However, evaluating generative answers is challenging [41], and it is even more difficult to determine whether the model provides unfounded answers. As a result, generative QA is not the preferred approach for our pipeline.

The raw annotations of our EHR data only compose of general entity annotations, which specify the textual content, type and start token of the entities, and the dependencies between them. The type of an entity $A$ is denoted as $t(A)$ and each dependency pair is denoted as $(A,B)$, indicating that a dependency relation exists from $A$ to $B$. Each sample pair $(A,B)$ is categorized into the class named ”$t(A)$-$t(B)$”. There are 15 entity types and 21 relation classes in total in the annotations.

Given a specific key (the information we are interested in), the output should consist of corresponding retrieved values. After preprocessing, the input should be the concatenation of a question $Q=(q_{1},\cdots,q_{n})$ together with a context $X=(X_{1},\cdots,X_{m})$. The pipeline output should include one or several spans $A_{1}=(X_{a_{1s}},\cdots,X_{a_{1e}}),\cdots,A_{t}=(X_{a_{ts}},\cdots,X_{a_{te}})$ from the context, where $a_{is}$ and $a_{ie}$ represent the start and end tokens of the $i^{th}$ span, respectively.

The preprocessing consists of three parts which will be introduced below. We will still utilize the examples in Figure 1.

The main body of the Question Answering (QA) model we employ is the original Retro-Reader[42], one of the benchmark models for SQuAD 2.0 (Stanford Question Answering Dataset). And we will give a brief introduction to the model. Retro-Reader comprises two primary components: a sketch reading module responsible for making a coarse judgment about answerability, and an intensive reading module that generates candidate answer spans. Additionally, it has integrated two threshold-based verification modules to further enhance the model’s capability for predicting answerability. The architecture can be concluded in Figure 2.

In the sketchy reading module, the input is first encoded by a pretrained multi-layer Transformer[39, 19]. The last layer of hidden states $\bm{H}_{1}$ is used as the embedding, which is sent to a fully connected layer for a two-way classification. And the predicted logits of not having an answer $\hat{y}_{i}$ are used for loss computation, where $y_{i}\in\{0,1\}$ is the true label:

The intensive reading module uses another Transformer but starts from the same pretrained weights. We denote the embeddings by $\bm{H}_{2}=(H_{Q},H_{P})$, standing for the question and passage respectively. The module adopts cross attention to compute the question-aware representation of the texts, which set $Q=H,K=V=H_{Q}$ in the multi-head attention in [39]. The final embedding $H^{\prime}$ after being weighted by attention is applied to two linear layers with a softmax operator to compute start and end position probabilities. The loss of the span prediction is written as:

which is the sum of the log probabilities of the true start and end positions. Another binary cross entropy loss $L_{ans2}$ is introduced to render this module also able to identify unanswerable questions. So the loss of this whole module is:

For our usage, both reading modules are trained on sentence-level and paragraph-level data to deal with multi-granularity in preprocessed data. As the answerability judgment is essential in the pipeline, we slightly enlarge the weight of $L_{ans2}$. Finally, three scores: an external verification score, a has-answer score and a no-answer score are computed for rear verification:

Here $s_{1},e_{1}$ is the start and end possibility of the [CLS] token. If the difference score $score_{diff}=score_{null}-score_{has}$ and the mixture score $\beta_{1}score_{diff}+\beta_{2}score_{ext}$ are both larger than a chosen threshold $\delta$, the QA model will output the answer span. Otherwise, it will predict the null string.

To ensure consistency in preprocessing and evaluation, we establish the format of the gold standard. The answer should be a concatenation of the initially annotated general entities, with one comma placed between each entity. The presence of the comma will be taken into account when computing the EM (exact match) score but not in the computation of the F1 score. If the answer of one sentence is empty, it will not be concatenated. In the case where all the answers of the sentences are empty, we consider the question to be unanswerable.

In a real-world setting, conducting comprehensive information extraction for all the EHRs is essential. Depending on the type of EHR, different templates will be applied and queried. If the EHR contains disease, abnormality and body part information, the NER step should be performed first, and the resulting entities will be utilized in the templates to query the text. Otherwise, the template itself with several fed-in words is sufficient for the query. Once all the possible queries are listed, we split the paragraph, obtain the model outputs, and merge them to complete the task. Questions that yield an empty string as the final answer are dismissed as non-informative.

We utilized real de-identified EHR data from a single hospital as the basis for our study. The corpus comprises various EHR types, such as past medical history, personal history, family history, CT reports, and radiology reports. Our annotators were instructed to first label the general entities and then establish the dependency relationships between them. The identified entities encompass 15 types, including numbers, sizes, trends, properties of abnormalities, diseases, body parts, immune groups, and values. These general entities are further connected to form 21 distinct relation classes. For training and testing purposes, we selected 18 high-frequency classes.

Following preprocessing steps, we compiled a dataset consisting of 1718 EHR passages, resulting in a total of 14528 question-answer pairs based on dependencies. The training set consists of 11451 QA pairs, while the dev set contains 1510 pairs, and the test set contains 1567 pairs. Notably, the dataset contains 3576 impossible questions, with 350 instances each in the dev and test sets. Additionally, the dataset includes 5600 NER-like questions found in 908 passages. To handle the two types of questions we process them separately.

The model backbone is detailed in Section III. For recognition of Chinese EHR texts, we employed the ”roberta-base-chinese” and ”bert-base-chinese” pretrained embeddings. To optimize performance, the model was deployed in parallel on two GTX 2080Ti 11G GPUs. Each GPU was assigned a train and eval batch size of 2, and the max length was set to 512. The training process encompassed the entire training set for four epochs. The training was completed within approximately 40 minutes, while producing predictions for evaluation took around five minutes. We set $\alpha_{2}=0.8$ in the intensive reading module loss, and all other hyperparameters remained consistent with those utilized in [42].

The NER-like QA evaluation is tested only on questions that pertain to diseases, abnormalities and body parts, the only three entity types not categorized into general entities or events. The evaluation standard we adopt involves both entity-level Exact Match (EM) score and token-level micro F1 score. A token is considered a True Positive if both its position and label are correct, and a False Negative if either of them mismatches.

For Chinese NER, we opted to use the most commonly employed baseline model, which consists of BERT/RoBERTa (Chinese) in combination with a classifier (linear layer) for NER sequence tagging. The performance of each baseline model as well as our pipeline is summarized in Table I.

The initial performance of the QA pipeline is notably poor when applied without finetuning. This discrepancy in results can be attributed to the inherent differences between the clinical domain and the general domain. However, following the training process, our pipeline demonstrates significant improvement and achieves competitive results on the NER task.

In this section, we evaluate the other question types for information extraction collectively. For evaluation, we employ the F1 and EM metrics. The token-level F1 computation is similar to the NER task, with the distinction that QA does not involve entity class labels. Since the extraction of information within given keys cannot be directly classified into existing tasks, such as Open IE or relation extraction, we establish our baseline model as a pipeline without any training.

The performance improvement in Table II demonstrates the effectiveness of the processing pipeline, which render the model able to answer EHR-related questions by constructing such question-answer pairs from original data. We also manually checked the model’s predictions. Among the 122 samples in the test set that do not exactly match the gold standard, approximately 2/3 of the errors result from a margin mismatch of one to two tokens. These errors also include some subtle inconsistencies among the annotations. We provide some examples in Figure 3, where the ”Boundary Mismatch” line illustrates these cases. The figure caption contains the translations. The red-labeled characters representing the mismatch may be locative prepositions or function words. Both predicted spans are plausible in these cases. So intuitively, the performance seems even better than indicated and appears suitable for practical usage. Other main causes of error are finding the wrong correspondence to another left entity of the same entity type at the paragraph level, and predicting a long sequence unexpectedly while the true answer is short. The accuracy of judgment, on the other hand, turned out to be very close to 1 after finetuning as shown in Table III. This actually guarantees the validity of merging sentence-level outputs.

n this section, we aim to validate the necessity of our design, which consists of two main components in the preprocessing stage. The first component involves splitting some paragraphs into sentences, which addresses many-to-one correspondence and discontinuous answer spans. The second component is the construction of impossible questions with plausible answers, which enhances the model’s ability to capture dependencies and discriminate answerability.

Table IV and Table V provide evidence of the design’s effectiveness. Each component contributes to a gain of approximately $0.02$ to $0.05$ in the EM score and $0.05$ to $0.08$ in the F1 score for relation-like questions. However, for NER-like questions, the performance experiences a significant drop without the component. This drop can be attributed to the fact that most paragraphs contain more than five entities of the same type, making it challenging for the model to handle NER-like questions effectively. It’s worth noting that the ablation of plausible answers is not conducted for NER-like questions due to the absence of plausible answers when a specific entity type does not exist in the sentence.

A case study can help us better understand how these tricks work. In Figure 3, we present two instances to illustrate their effectiveness. In the first example, we have two abnormalities that point to the same body location but are mentioned in two separate clauses. If we didn’t perform the splitting operation during preprocessing, the model would likely predict only one of the abnormalities. We observed similar cases even when the two spans are adjacent in the original annotations, as they are initially annotated in this way.

In both examples, there are two clauses in analog structures. When the model lacks fine-tuning on impossible questions with plausible answers, there is a higher probability of mixing up the relativity between the clauses and predicting the wrong span of the same type. So both of the tricks are necessary to achieve more stable and accurate extractions.

Finally, we aim to explore whether the model can be utilized to extract information beyond the annotations, simulating a zero-shot setting. For this purpose, we manually propose 40 questions not covered in the 18 relation classes. These questions comprise 20 wh-questions and 20 yes-no questions. As our pipeline relies on extractive Question-Answering (QA), we expect the model to extract the negation word if the answer is ”no,” and extract the relevant statement if the answer is ”yes.” Therefore, if the model predicts a negation word, we consider its answer as ”no,” and as ”yes” otherwise.

The generalization results are presented in Table VI. The model performs well on yes-no questions, achieving satisfactory accuracy. However, it only performs acceptably on other types of questions. The primary reason for this discrepancy is that the model lacks an understanding of the EHR writing format during the fine-tuning process, making it challenging for it to answer questions related to overall information.

The results obtained from our pipeline demonstrate its superiority, meeting most of the requirements in our task settings. However, due to limitations in annotations and equipment, several potential improvements are not explored.

Firstly, we acknowledge the need to address a combination of yes-no questions, multiple-choice questions, and extractive questions within the context of EHR extraction. In certain situations, while extracting information from EHRs, we may encounter queries related to the patient’s diagnosis of specific diseases, and disease classification or severity, which typically require constrained options for answers. Although our pipeline showcased competence in handling yes-no questions, it would be better to apply targeted models. An additional module, such as rule-based or BERT-based classifiers, could achieve high accuracy in distinguishing among these three question classes. By incorporating such classifiers, the models described in Section II can be adapted accordingly to form again a unified framework.

Secondly, we recognize the importance of enabling few-shot or even zero-shot capabilities for our models. In real-world scenarios, medical concepts may lack available annotated data[43], or the existing data may not align precisely with practical requirements. Pretrained Language models possess the transferring function, and LLMs even have emergent ablities[44]. Prompts or other tricks may be invented to trigger such capacity. As new techniques continue to emerge, other avenues for improvement may become possible, ultimately leading to the development of a more comprehensive and powerful EHR IE system in the foreseeable future.