DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queries

Electronic Health Records (EHRs) are digitized records of patients’ medical history, which can be either in structured or unstructured form. MIMIC-III (Medical Information Mart for Intensive Care) [Johnson et al., 2016a] is an example of a large EHR database containing health-related information for over 40,000 patients who were admitted in critical care units in the period between 2001 and 2012 in the Beth Israel Deaconess Medical Center. MIMIC-III contains multi-relational tables with patient-specific data on diagnosis, medications, admission, lab tests, etc. Question answering over EHRs aid doctors in diagnosing, while helping patients obtain answers to health-related queries. The structured relational database has multiple tables which store information about the patient’s demographics, diagnoses, medications, lab tests along with their results. Conversely, the unstructured data, are notes entered by clinicians that contain a detailed description of every patient’s visit, their past medical history, their problem, symptoms and more. Thus, to benefit from both the modalities, there arises a need for a multi-modal QA dataset on EHRs.

We present DrugEHRQA, the first QA dataset which uses both the structured tables and the unstructured clinical notes of an EHR to answer questions. The answers from the clinical notes are used to support or provide evidence to the answers retrieved from the structured tables. The former gives better context to support the latter. Moreover, there can be cases where a guaranteed answer might not be available in the structured tables, due to missing data/relation. For example, if the question is ‘What medication is the patient with an admission ID of 105104 taking for Hypoxemia?’ The MIMIC-III tables have no direct relation between medicines and problems. The tables DIAGNOSES_ICD and D_ICD_DIAGNOSES of MIMIC-III can be used to verify if the patient with admission ID 105104 is suffering from Hypoxemia, and the PRESCRIPTIONS table of MIMIC-III can be used to fetch all the medicines prescribed to the patient, having an admission ID of 105104. However, the patient could have been prescribed medicines for non-Hypoxemia related conditions, which will be contained in the tables. In such cases, the unstructured clinical notes can be used to identify the medicines from this list, since the information about the medicine for Hypoxemia is directly present in the clinical notes.

One reason for the lack of any pre-existing multimodal QA dataset on EHRs is due to the tedious amount of time and effort that is required to annotate such a dataset. In this work, we introduce a novel method to automatically generate a template-based QA (DrugEHRQA) dataset (on drug-related questions) from the MIMIC-III database. DrugEHRQA contains the following: 1) natural language questions, 2) its corresponding SQL Query that can be used to retrieve answers from the multi-relational MIMIC-III tables, 3) the answers from either or both the modalities, and 4) the ‘best selected’ multi-modal answer. DrugEHRQA contains 70,381 QA pairs that have been generated using nine different template types. We also generated three paraphrases for every natural language question template, and analyzed the effects of paraphrasing on the baseline models. DrugEHRQA was benchmarked against existing models like TREQS [Wang et al., 2020b], RAT-SQL [Wang et al., 2020a], BERT QA [Devlin et al., 2019] and ClinicalBERT [Alsentzer et al., 2019] to test the validity of the DrugEHRQA dataset for the individual modalities.

We also introduce a simple baseline model for multimodal text-table QA in EHRs. Similar to ManyModelQA [Hannan et al., 2020], our model (MultimodalEHRQA) contains a modality selection network. With questions fed as an input, the modality selection network predicts the more reliable modality (i.e. text or table) for question answering. The predicted modality is used to choose the corresponding state-of-the-art model for table and text QA. The more complex task of leveraging answers from both structured and unstructured data to provide a contextualized answer is left to future work.

The main contributions of this paper are as follows:

1. 1.

   Introduce DrugEHRQA¹¹1https://github.com/jayetri/DrugEHRQA-A-Question-Answering-Dataset-on-Structured-and-Unstructured-Electronic-Health-Records, scripts to generate DrugEHRQA dataset, the first QA dataset on multi-modal EHRs, containing QA pairs from structured tables and unstructured clinical notes from MIMIC III. The dataset contains natural language questions, its corresponding SQL query for querying multi-relational tables in MIMIC-III, the retrieved answer(s) from one or both modalities, and the combined multi-modal answer.

2. 2.

   Introduce a novel technique to automatically generate a template-based dataset using existing annotations of a non-QA application, skipping the need for manual annotations specifically for QA.

3. 3.

   Develop a simple baseline model for multimodal QA on EHR, using modality selection network.

The remainder of the paper is organized into 8 sections. Section 2 discusses existing related work, Section 3 describes the DrugEHRQA dataset generation, Section 4 presents the analysis of DrugEHRQA, Section 5 discusses the baseline model for multimodal QA, Section 6 presents the experiments and results, Section 7 discusses the reproducibility and limitations of our work, Section 8 proposes the broader impact of our dataset in the EHR QA research community, and Section 9 concludes the work and discusses possible future work.

Even though there are some available multimodal QA datasets in non-clinical domains [Hannan et al., 2020, Chen et al., 2020, Talmor et al., 2021], but there are no existing multimodal QA datasets which uses structured with unstructured EHR data to answer questions. There are some existing works in the clinical genre on multi-modal understanding from text-image pairs  [Moon et al., 2021, Khare et al., 2021, Li et al., 2020] as well as clinical QA  [Singh et al., 2021] on text-image data. But as far as the authors’ knowledge, so far there is no multi-modal clinical dataset that encorporates structured and unstructured EHR data for QA. QA in EHRs has been limited to QA over knowledge bases [Wang et al., 2021], EHR tables [Wang et al., 2020b, Raghavan et al., 2021] or clinical notes [Johnson et al., 2016b, Pampari et al., 2018]. emrQA [Pampari et al., 2018] and CliniQG4QA [Yue et al., 2021] are QA datasets that utilize unstructured text of EHRs to generate QA datasets. The emrQA contains 1 million question-logical forms along with over 40,000 QA evidence pairs, extracted from clinical notes of five n2c2 challenge datasets ²²2https://www.i2b2.org/NLP/DataSets/. CliniQG4QA on the other hand, contains 1287 annotated QA pairs on 36 discharge summaries from clinical notes of MIMIC-III. CliCR [Šuster and Daelemans, 2018] is another large medical QA dataset which is constructed from clinical case reports. It is used for reading comprehension in the healthcare domain. The reports used in CliCR are proxy for electronic health records, since the clinical reports look similar to the discharge summaries of EHR even though it lacks features of EHR text.

There are QA datasets that are generated using template-based method like MIMICSQL [Wang et al., 2020b] and emrKBQA [Raghavan et al., 2021] which utilize the structured EHR tables of MIMIC-III for QA. emrKBQA contains 940,000 questions, logical forms and answers which uses the structured records of MIMIC-III. Both emrKBQA and emrQA use semi-automated methods to retrieve the answers. The question templates and logical forms are generated by physicians, followed by a slot-filling process and answers retrieved from MIMIC-III KB [Johnson et al., 2016b]. On the contrary, our dataset - DrugEHRQA uses both structured tables and clinical notes containing elaborate details of MIMIC-III to generate the QA dataset. We use an automatic novel methodology to create the dataset (described in Section 3).

The dataset has been generated using a template-based method. The dataset (DrugEHRQA) contains over 70,000 natural language questions. Each line in DrugEHRQA consists of a natural language question, its corresponding SQL query to retrieve answers from the MIMIC-III tables, the retrieved answers from MIMIC-III tables and/or answers from clinical notes of MIMIC-III, and the selected multi-modal answer. As stated earlier, generating a multi-modal dataset is time-consuming mainly because the data must be manually annotated, which is a very tedious process. To overcome this, we introduce a novel strategy to automatically generate the dataset. The dataset generation framework of DrugEHRQA is illustrated in figure 1. The dataset generation process can be explained using five steps: (1) Annotation of question templates, (2) Extraction of drug based relations from n2c2 repository, (3) Answer extraction from MIMIC-III tables, (4) Paraphrasing, (5) Selecting multi-modal answers. The following subsections explain in detail the five steps involved in automatic data generation.

The SQL queries generated in the DrugEHRQA dataset can be classified into easy, medium, hard and very hard SQL queries (Refer Table 1). The generated SQL queries were classified using the complexity determination method used in RAT-SQL [Wang et al., 2020a]. Complexities of the SQL queries are determined by factors like number of tables in the SQL query, number of conditions, presence of nesting etc. The DrugEHRQA dataset contains more complicated SQL queries (containing nested queries) than the existing text to SQL datasets like MIMICSQL [Wang et al., 2020b]

DrugEHRQA contains a total of 70,381 questions along with answers from either the multi-relational tables, or the unstructured clinical data of MIMIC-III, or from both the sources. The dataset also contains an automatically generated multi-modal answer. The DrugEHRQA dataset is diverse with respect to the source of the answers (refer to figure 3). There are questions that can be answered by individual modalities (i.e. structured or unstructured EHR). Roughly 41 % of the drug-related queries can be answered individually by the structured data and unstructured data. It also contains questions that have answers in both the modalities (18.1%, i.e. 12,738 queries). 11% of the questions have at least one overlapping answer in structured and unstructured EHR data (e.g., row 2 and 3 of table A4) whereas the remaining 7.1% of the questions have distinct (or non-overlapping) answers in the two modalities (e.g., row 4 of table A4). 15% of this section of the dataset have missing relations (or information) in the structured tables, i.e. they are derived relation queries (as explained in Section 3.5. Hence, among the queries containing answers in both the modalities, the answers from unstructured EHR data is more reliable (than structured EHR data) for 15% of the queries.

Figure 4 shows the percentage of questions with at least one-answer overlap between table and text QA for the nine templates. Some of the templates like medicine names, form-drug, and route-drug have a high percentage of overlapping answers. Confidence in the accuracy of answers increases when the answers are the same, e.g.: row 2 and row 3 of Table A4. Table A3 shows examples where multi-modal QA in EHRs can help provide additional context. We observed that the answers from the modalities were different, but the dual modalities together provide the complete answer. The answer from structured data gives the dosage in milligrams, whereas the answer retrieved from the clinical notes presents the dosage based on the number of tablets. Both of the answers are right, which can be verified from the last column, since the dosage recommended in row 1 is one 325 mg tablet, to be taken daily. In short, answers from one modality can help to provide better context to the answers retrieved from the other modality.

We now discuss all the baseline models used for performing QA tasks on DrugEHRQA. Among the questions containing answers in both the modalities (which comprises 18.1% of the dataset), 60.8% of these queries have repeated answers in the two modalities. For such questions, either modality (text QA or table QA) can be used. For questions with one or more overlapping answers, we used the non-derived relation queries for QA on structured tables, and have used unstructured data to answer the derived relation queries, as explained in Section  3.5. For questions having answers in only a single modality (table or text), we use unimodal QA and separate QA baseline models to validate our QA dataset on structured EHRs and unstructured EHRs. Two existing models - TREQS [Wang et al., 2020b] and RAT-SQL [Wang et al., 2020a] are used for the text-to-SQL tasks on DrugEHRQA which use MIMIC-III tables.

For questions having no common answers in the two modalities, we have developed a multimodal QA pipeline (MultimodalEHRQA). The pipeline consists of a modality selection network which predicts the modality (i.e. table or text) for a given question. If the network predicts ’text‘ as the modality (i.e. clinical notes), then extractive QA using BERT or ClinicalBERT is performed, predicting the span of text from the clinical EHR notes as the answer. If the network predicts ’table‘ as the modality, then text-to-SQL is performed by the TREQS model. The predicted answers from the chosen modality is compared to the golden standard multimodal answer.

Our dataset has questions with answers in a single modality (structured or unstructured EHR data), as well as questions where answers exist in both the modalities (refer to figure 3). Due to the diversity of our dataset, we have performed experiments on single-modality QA pairs, as well as for multimodal non-overlapping questions. Furthermore, we have also evaluated the overall performance of our dataset on the MultimodalEHRQA model. We define two versions of our dataset - DrugEHRQA (basic) and DrugEHRQA (extended). The difference between the two versions is that the DrugEHRQA (basic) contains all the questions of the DrugEHRQA dataset except those which have answers only in the structured EHR data and contain nested SQL queries. Note, that we can’t use BERT/ClinicalBERT for these questions, as they have answers only in the MIMIC-III tables. Also, we cannot use TREQS to validate these questions, so we limit using MultimodalEHRQA to the DrugEHRQA (basic). We use RAT-SQL to determine the accuracy of the structure of the predicted SQL queries for these questions.

The user needs to request for credentialed access for PhysioNet⁴⁴4https://physionet.org/. The user must download the MIMIC-III data, retrieve the drug relations from the ‘2018 (Track 2) Adverse Drug Event (ADE) and the Medication Extraction Challenge dataset’ from the n2c2 repository (after requesting for access to n2c2 datasets). Once this is done, the user can just replicate the steps described in the dataset generation process (Section 3) to produce the DrugEHRQA dataset. Even though the multimodal QA dataset generation process is automatic, without the need for long hours of annotation. But this procedure is limited only to the MIMIC-III database. The same steps cannot be reproduced for other EHR databases. In fact, MIMIC-IV [Johnson et al., 2020] is the latest version. But since the dataset generation process is dependent on the drug relations extracted from the challenge dataset, so our dataset generation process was limited to the MIMIC-III database. Furthermore, the diversity of the questions (or question templates) in the DrugEHRQA dataset are limited by the type of relations extracted from the challenge dataset. But despite these limitations, the DrugEHRQA dataset is an invaluable resource which will encourage further research in multimodal QA over EHRs.

The DrugEHRQA dataset helps to put a spotlight on multimodal EHRs. The data in the structured and unstructured EHR may contain duplicated information (improves confidence of the answer), they may contrast each other, and may also aid in adding context to each other. This opens up new avenues of research in multimodal QA in EHRs. DrugEHRQA can be used as a benchmark model for all QA models that uses multiple EHR tables and clinical notes for information retrieval. Since in a lot of cases, the data in structured and unstructured EHR sources helps to provide additional context to each other, another possible application of DrugEHRQA is in improving QA over structured (or unstructured), by using information or evidence from the unstructured EHR source (or structured).

To conclude, EHRs contain a large amount of up-to-date patient information in the structured databases, along with clinical notes containing elaborate details. We have introduced a novel methodology to generate a large multimodal QA dataset, containing answers from multi-relational tables and discharge summaries of a publicly available EHR database (MIMIC-III). It is the first QA dataset which contains natural language questions, SQL queries, and answers from either or both structured EHR tables and unstructured free text. Additionally, we use an automated methodology to generate the multimodal answer. Following this, human annotators verified the answers for a sampled dataset. We have also introduced a simple baseline model for multimodal QA on EHRs. In the future, we will try to work on multimodal QA models for EHRs which jointly trains the model on both table and text. Using QA over one modality to improve QA over other modalities is another promising direction of this work. The DrugEHRQA dataset introduces new horizons of research in multimodal QA over EHRs.

This work is partially support ed by US National Science Foundation grant IIS-1526753, and Arnold and Lisa Goldberg funds.