\model: Improving Generalization of Clinical Text
De-identification Models via Data Augmentation

Clinical text in electronic health records (EHRs) often contain sensitive information. In the United States, Health Insurance Portability and Accountability Act (HIPPA)²²2http://www.hhs.gov/hipaa requires that protected health information (PHI) (e.g., name, street address, phone number) must be removed before EHRs are shared for secondary uses such as clinical research Meystre et al. (2014).

The task of identifying and removing PHI from clinical texts is referred as de-identification. Although many neural de-idenfication models such as LSTM-based Dernoncourt et al. (2017); Liu et al. (2017); Jiang et al. (2017); Khin et al. (2018) and BERT-based Alsentzer et al. (2019); Tang et al. (2019) have achieved very promising performance, identifying PHI still remains challenging in the real-world scenario: even well-trained models often fail to generalize to a new dataset. For example, we conduct cross-dataset test on i2b2 2006 and i2b2 2014 de-identification challenge datasets³³3https://portal.dbmi.hms.harvard.edu/projects/n2c2-nlp/ (i.e., train a widely-used de-identification model NeuroNER Dernoncourt et al. (2017) on one dataset and test it on the other one). The result in Figure 1 shows that model’s performance (F1-score) on the new dataset decreases up to $33\%$ compared to the original test set. The poor generalization issue on de-identification is also reported in previous studies (Stubbs et al., 2017; Yang et al., 2019; Johnson et al., 2020; Hartman et al., 2020).

To explore what factors lead to poor generalization, we sample some error examples and find that the model might focus too much on specific entities and does not really learn language patterns well. For example, in Figure 2, given a sentence “She met Washington in the Ohio Hospital”, the model tends to recognize the entity “Washington” as the “Location” instead of the “Name” if “Washington” appears as “Location” in the training many times. Such cases appear more frequently in a new testing set, thus leading to poor generalization.

To prevent the model overfitting on specific cases and encourage it to learn general language patterns, one possible way is to enlarge training data Yang et al. (2019). However, clinical texts are usually difficult to obtain, not to mention the requirement of tremendous expert effort for annotations Yue et al. (2020). To solve this, we introduce our data augmentation method \model, which consists of PHI augmentation and Context augmentation. Specifically, PHI augmentation replaces the original PHI entity in the training set with a same type named-entity sampled from external sources (such as Wikipedia). For example, in Figure 2, “Ohio Hospital” is replaced by an randomly-sampled “Hospital” entity “Alaska Health Center”. In terms of context augmentation, we randomly replace or insert some non-stop words (e.g., verb, adverb) in sentences to create new sentences as an example shown in Figure 2. The augmented data does not change the meaning of original sentences but increase the diversity of the data. It can better help the model to learn contextual patterns and prevent the model focusing on specific PHI entities. Data augmentation is widely used in many NLP tasks Xie et al. (2017); Ratner et al. (2017); Kobayashi (2018); Yu et al. (2018); Bodapati et al. (2019); Wei and Zou (2019) to improve model’s robustness and generalizability. However, to the best of our knowledge, no work explores its potential in the clinical text de-identification task.

We test two LSTM-based models: NeuroNER Dernoncourt et al. (2017), DeepAffix Yadav et al. (2018) and one BERT-based Devlin et al. (2019) model: ClinicalBERT Alsentzer et al. (2019) with our \model. Cross-dataset evaluations on i2b2 2006 dataset and i2b2 2014 dataset show that \modelcan boost the models’ generalization performance up to 8.6% in terms of F1-score. We also discuss how much augmentation we need and conduct the ablation study to explore the effect of PHI augmentation and context augmentation. To summarize, our \modelis simple yet effective and can be used together with any existing machine learning-based de-identification systems to improve their generalizability on new datasets.

To understand what factors lead to the poor generalization, we check some error examples and find that most of the PHI entities in these error examples do not appear in training set or appear as a different PHI type (e.g., Washington [Name v.s. Location]). We argue that neural models might focus on too much on specific entities (e.g., recognizing “Washington” as “Location”) but fail to learn general language patterns (e.g., “met” is not usually followed by a “Location” entity but a “Name” entity instead). Consequently, such unseen or Out-Of-Vocabulary PHI entities might be hard to be identified correctly, thus leading to lower performance. To help models better identify these unseen PHI entities, we may encourage models to learn contextual patterns or linguistic characteristics and prevent models focusing too much on specific PHI tokens.

We construct 11 candidate lists for sampling different PHI types. The lists are either obtained by scraping the online web sources (e.g., Wikipedia Lists) or by randomly generating based on pre-defined regular expressions (the number and the source of each candidate list is shown in Table 1).

For each sentence containing PHI entities in the corpus, we can apply both PHI augmentation and Context augmentation to obtain the augmented data $D_{aug}$. We can run $\alpha$ times (by setting different random seeds) to obtain different sizes of augmented data (e.g., $\alpha$ = 2 means augmenting the original dataset twice). Though with the $\alpha$ increases, we can obtain larger augmented training corpora, it may also bring much noise. We recommend a small value for $\alpha$ (See more discussions in Section 4.2). Then we merge the $D_{aug}$ with the original dataset $D$ to form the final dataset $D_{new}$ for training: $D_{new}$ = $D$ $\cup$ $\alpha$ $D_{aug}$.

In summary, \modelcan significantly increase the diversity of training data without involving more labeling efforts. The augmented data can increase data diversity and enrich contextual patterns, which could prevent the model focusing too much on specific PHI entities and encourage it to learn general language patterns.

In this paper, we explore the generalization issue on clinical text de-identification task. We propose a data augmentation method named \modelthat augments both PHI and context to boost model generalization. The augmented data can increase data diversity and enrich contextual patterns in training data, which may prevent the model overfitting on specific PHI entities and encourage it to focus more on language patterns. Experimental results demonstrate that our \modelcan help improve models’ generalizability, especially in the low-resource training case (i.e., the size of the original training set is small). We also discuss how much augmentation to use and how each augmentation method influences the performance. In the future research, we will explore more advanced data augmentation techniques for improving the de-identification models’ generalization performance.

We thank Prof. Kwong-Sak LEUNG and Sunny Lai in The Chinese University of Hong Kong as well as anonymous reviewers for their helpful comments.