SCORE-IT: A Machine Learning-based Tool for Automatic Standardization of EEG Reports

Analysis and interpretation of EEG data is important in the diagnosis of neurological conditions such as epilepsy. Recently, machine learning-based analyses are becoming the mainstay of quantitative EEG-based decision making [1]. EEG reports are the primary source of metadata for quantitative EEG studies because they contain rich information regarding the patients’ condition at the time of EEG recording including seizures, interictal abnormalities, and background activity such as posterior dominant rhythm. However, due to a lack of standardization in the organization of information presented and terminology in EEG reports, automatic extraction of those information has proven to be a difficult task. As a result, manual information retrieval has been the primary way of generating metadata for EEG-based ML studies. Clearly, this is not scalable and warrants the development of natural language processing-based automated tools.

The standardized computer-based organized reporting of EEG (SCORE) guidelines seek to standardize reporting of EEG studies by specifying the content and terminology of characteristics that are described in an EEG report [2, 3]. As such, those guidelines provide an ideal target for automated information retrieval and validation. In this work, we present a semi-automated tool for extracting structured information from unstructured natural text EEG reports with an eye towards converting past EEG reports to the standardized format stipulated in the SCORE guidelines. Furthermore, we also leverage the classification tasks made available through various labeled subsets of the TUH dataset as a valuable proxy to evaluate the capabilities of our system.

The development of such a system poses several challenges: 1) data sparsity: from an entire patient record consisting of dozens of sentences, often only a single phrase is relevant to making a classification decision; 2) lack of labeled data: the lack of clinical data available around many of the SCORE attributes; and 3) accounting for varied clinician practices. We took a two-step approach to address those challenges: first, we leverage a previously developed named entity recognition approach using BERT Transformer models trained on the National NLP Clinical Challenges (n2c2) dataset [4]; second, hand-crafted rules are applied to these extracted entities, considering factors such as the SCORE lexicon and numerical values to identify information relevant to SCORE attributes. This hybrid approach allows us to leverage a) existing well-trained ML models for relevant entity identification, and b) domain knowledge to further classify them into SCORE attributes.

In the current work, we focus on identifying three SCORE attributes: (1) the type of seizure if present (complex partial, simple partial, absence, myoclonic, generalized tonic-clonic, other/none); (2) whether the recording contained any abnormal events (e.g., seizures); and 3) whether the patient is being evaluated for epilepsy. In addition, we designed and validated our system on the Temple University Hospital (TUH) EEG dataset [5], containing over 16,000 EEG recordings and reports. We utilized subsets of the TUH EEG corpus consisting ground truth labels for the three tasks we focused on. We achieved weighted F1 scores of 0.92 (on 171 test records), 0.82 (on 561 records), and 0.97 (on 2727 records) for the seizure classification, normal/abnormal classification, and epilepsy classification tasks, respectively.

Existing work on classification from patient medical records includes Track 1 of the 2018 National NLP Clinical Challenges (n2c2), which involves making binary classification decisions about whether a patient meets some criteria, such as alcoholism or history of myocardial infarction over past 6 months [6]. Existing strategies for this type of task include extraction of clinical entities, hand-crafted features that are used to train simple machine learning models, and creation of lexicons to feed into rules [7]. On the other hand, there are related classification tasks in the clinical domain for which there is ample labeled data available, e.g., clinical/biomedical named entity recognition (NER) [4]. Prior work on performing NER has included the use of classical Machine Learning models, such as Conditional Random Fields, as well as deep neural networks, including LSTM networks [8] and, more recently, fine-tuning transformer networks, such as BERT [9].

However, our work on automatically extracting useful metadata such as seizure type, EEG classification, and diagnostic information from EEG reports, to our knowledge, is the first attempt at developing automated information retrieval approaches for EEG reports. Considering that the majority of the (past and present) EEG reports are written in natural text format, our work offers the potential to standardize EEG reports and to generate useful metadata for subsequent ML-based analyses.

The TUH EEG Corpus (TUEG) is a collection of over 30,000 clinical EEG records collected and made available by Temple University Hospital (TUH) [5]. Corresponding patient medical reports, in plaintext format, are also made available alongside the EEG recordings. There are also subsets of the corpus, containing labeled data for several tasks. In this work, we utilize the TUH EEG Epilepsy Corpus (TUEP), TUH Abnormal EEG Corpus (TUAB), and TUH EEG Seizure Corpus (TUSZ). Table 1 describes the number of labeled samples for each of the subsets. (As the Epilepsy subset had no train and test partition, unlabeled records from outside the subset were used to develop rules.)

In this work, we present a system that, given a natural-language patient report, predicts, 1) the type of seizure that was observed in the recording, per physician impression; 2) whether the session recording was normal or abnormal according to physician impression; and 3) whether a patient has epilepsy or not.

To construct such a system, we provide a framework for classifying EEG records that divides the process into two steps: a broad parsing step, for which tasks are well-defined and considerable training data exists; and a narrow parsing/classification step built on top of the broad parsing, for which tasks are highly domain-specific and little training data is available. In this manner, we address issues of data sparsity and support while simultaneously leveraging effective data-driven, deep learning techniques that can be adapted to this specific domain.

A high-level architecture of the system is detailed in Figure 1. In the following subsections, we discuss each component.

The system was evaluated on three specific tasks that are subsets of the TUH EEG dataset, and for which gold annotations are provided:

1. 1.

   Seizure type classification: given a record, determine the type of seizure noted in the EEG record (complex partial, simple partial, absence, myoclonic, tonic-clonic, other/none).
   

2. 2.

   Epilepsy classification: given a record, determine whether the patient is being evaluated for epilepsy or not.
   

3. 3.

   Norma/abnormal classification: determine if the clinical impression in a given record is normal or abnormal.

A detailed information for each of these tasks is provided in the following subsections.

On the seizure classification task, our system performed with a weighted F1 score of 0.92 on 171 test records. On the epilepsy classification task, our system performed with a weighted F1 score of 0.82 on 561 records. On the abnormal classification task, our system performed with a weighted F1 of 0.97 on 2727 records. Notably, the system performed poorer on classes with a very small support, due to difficulty in creating generalizable hand-crafted rules from a very small dataset. More detailed results are provided in Table 2.

We proposed an automated approach for extracting clinically useful metadata from natural text EEG reports. Major challenges addressed in our work include 1) designing strategies for integration of clinical domain knowledge, 2) accounting for varied clinician practices, 3) establishing ground truth standards, and 4) robust verification of the system. Applications of this system include enabling better search and filtering of clinical reports, as well as auto-labeling unstructured datasets for research purposes.

As mentioned previously, one complication with interpreting these results is that for certain tasks, such as epilepsy classification, the labels are derived from the EEG findings and not necessarily explicitly mentioned in the patient medical record, making it difficult to fully accurately judge the performance of the system.

Although the broad and narrow parsing architecture of the system helped in the development of more general rules, the system still performed better on classes with greater support. This can be attributed to the fact that having access to a wider range of training samples allowed rules to be developed that could capture more of the variance. On the other hand, the system performed poorer on classes with few samples, as the rules developed with under 5 training samples were not able to be validated to ensure sufficient generalizability.

The current iteration of the system still performs relatively poorly in classification circumstances with low data support. Consequently, one important area of focus is reducing the amount of hand-crafted rules needed, while increasing the generalizability of the system, in order to be able to apply the system across a wider range of use-cases. To that effect, a prominent area of future focus remains using existing NLP tasks, such as semantic textual similarity [10], as a means to perform zero-shot classification in place of hand-crafting rules. Another area for future work is implementing and validating our approach on additional datasets.

We described a semi-automated ML-based system to extract standardized components from unstructured EEG reports. Our system can facilitate better indexing, searching and organization of existing EEG report collections by providing an semi-automated methodology of standardizing important data within free-text EEG reports. Furthermore, our system can be used to generate labels for large, unlabeled EEG corpora like the TUH dataset to motivate future research.

This research was supported by National Science Foundation’s Computer and Information Science and Engineering Research Initiation Initiative Award SCH-2105233. We would like to thank Dr. Susan Herman and her team from the Barrow Neurological Institute, as well as Neeraj Wagh from the University of Illinois at Urbana Champaign, for contributions to discussions on where this work can be applied.