BBAEG: Towards BERT-based Biomedical Adversarial Example Generation for Text Classification

Recent studies have exposed the importance of biomedical NLP in the well-being of human-beings, analyzing the critical process of medical decision-making. However, the dialogue managing tools targeted for medical conversations Zhang et al. (2020), Campillos Llanos et al. (2017), Kazi and Kahanda (2019) between patients and healthcare providers in assisting diagnosis may generate certain insignificant perturbations (spelling errors, paraphrasing), which when fed to the classifier to determine the type of diagnosis required/detecting adverse drug effects/drug recommendation, might provide unreasonable performance. Insignificant perturbations might also creep in from the casual language expressed in the tweets Zilio et al. (2020). Thus, the classifier needs to be robust towards these perturbations.

Generating adversarial examples in text is challenging compared to computer vision tasks because of (i) discrete nature of input space and (ii) preservation of semantic coherence with original text. Initial works for attacking text models relied on introducing errors at the character level or manipulating words Feng et al. (2018) to generate adversarial examples. But due to grammatical disfluency, these seem very unnatural. Some rule-based synonym replacement strategies Alzantot et al. (2018), Ren et al. (2019) have lead to more natural looking examples. Jin et al. (2019) proposed TextFooler, as a baseline to generate adversaries for text classification models. But, the adversarial examples created by TextFooler rely heavily on word-embedding based word similarity replacement technique, and not overall sentence semantics. Recently, Garg and Ramakrishnan (2020) proposed BERT-MLM-based Devlin et al. (2019) word replacements to create adversaries to better fit the overall context.

Despite these advancements, there is much less attention towards making robust predictions in critical domains like biomedical, which comes with its unique challenges. Araujo et al. (2020) has proposed two types of rule-based adversarial attacks inspired by natural spelling errors and typos made by humans and synonym replacement in the biomedical domain. Some challenges include: 1) Biomedical named entities are usually multi-word phrases such as colorectal adenoma. During token replacement, we need the entire entity to be replaced, but the MLM model (token-level replacement) fails to generate correct synonym of entity fitting in the context. So, we need a BioNER+Entity Linker Martins et al. (2019), Mondal et al. (2019) to link entity to ontology for generating correct synonyms. 2) Due to several variations of representing medical entities such as Type I Diabetes could be expressed as ’Type One Diabetes’, we explore numeric entity expansion strategies for generating adversaries. 3) Spelling variations (keyboard swap, modification). While we evaluate on two benchmark datasets, our method is general and is applicable for any biomedical classification datasets.

In this paper, we present BBAEG (Biomedical BERT-based Adversarial Example Generation)¹¹1https://github.com/Ishani-Mondal/BBAEG.git, a novel black-box attack algorithm for biomedical text classification task leveraging both the BERT-MLM model for non-named entity replacements combined with NER linked synonyms for named entities to better fit the overall context. In addition to replacing words with synonyms, we explore the mechanism of generating adversarial examples using typographical variations and numeric entity modification. Our BBAEG attack beats the existing baselines by a wide margin on both automatic and human evaluation across datasets and models. To the best of our knowledge, we are the first to introduce a novel algorithm for generating adversarial examples for biomedical text whose success attack is higher than the existing baselines like TextFooler and BAE Garg and Ramakrishnan (2020). The overall contributions of the paper include: 1) We explore several challenges of biomedical adversarial example generation. 2) We propose BBAEG, a biomedical adversarial example generation technique for text classification combining the power of several perturbation techniques. 3) We introduce 3 type of attacks for this purpose on two biomedical text classification datasets. 4) Through human evaluation, we show that BBAEG yields adversarial examples with improved naturalness.

Problem Definition: Given a set of $n$ inputs ($D,Y$) = [($D_{1}$, $y_{1}$), . . .($D_{n}$, $y_{n}$)] and a trained classifier $M$ : $D$ $\rightarrow$ $Y$, we assume the soft-label black-box setting where the attacker can only query the classifier for output probabilities on a given input, and has no access to the model parameters, gradients or training data. For an input of length $l$ consisting of words $w_{i}$, where 1 $\leq$ $i$ $\leq$ $l$, $(D_{i}=[w_{1},...,w_{l}],y)$, we want to generate an adversarial example $D_{adv}$ such that $M(D_{adv})$ $\neq$ $y$. We would like $D_{adv}$ to be grammatically correct, semantically similar to $D$ ($Sim$($D$, $D_{adv}$) $\geq$ $\alpha$), where $\alpha$ denotes the similarity threshold.

Datasets and Experimental Details: We evaluate BBAEG on two different biomedical text classification datasets: 1) Adverse Drug Event (ADE) Detection Gurulingappa et al. (2012) and 2) Twitter ADE dataset Rosenthal et al. (2017) for the task of classifying whether the sentence contains mention of ADE (binary).

We use 6 classification models as $M$: Hierarchical Attention Model Yang et al. (2016), BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), BioBERT Lee et al. (2019), Clinical-BERT Huang et al. (2019), SciBERT Beltagy et al. (2019). We fine-tune these models on the training data (of each corpus) using Adam Optimizer Kingma and Ba (2015) with learning rate of 0.00002, 10 epochs and perform adversarial attack on the test data. For the BBAEG non-NER synonym attacks, we use BERT-base-uncased MLM to predict the masked tokens. We consider top $K$=10 synonyms from the BERT-MLM predictions and set threshold $\alpha$ of 0.75 for cosine similarity between Reimers and Gurevych (2019) embeddings of the adversarial and input text, we set $p$=2 characters for rotation to introduce noise in input. For more details refer to the appendix.

Automatic Evaluation Results: We examine the success of adversarial attack using two criteria: (1) Performance Drop (Adrop): Difference between original (accuracy on original test set) and after-attack accuracy (accuracy on the perturbed test set) (2) Perturbation of input (%): Percentage of perturbed words in adversary generated. Success of attack is directly and indirectly proportional with criteria 1 and 2 respectively.

During ablation analysis, we observe that the synonym replaced perturbed samples looked more natural to the human evaluators compared to the spelling perturbed samples and number replaced entities. When considered jointly, the number replaced and synonym replaced samples seemed more natural to the annotators compared to spelling perturbed samples. This arises due to the fact that the number replaced entities when thrown to the annotators they could easily interpret the meaning correctly when given in combination with the original sample. For instance, in the examples shown in table 2, the number replaced samples (21-year old $\rightarrow$ twenty-one-year old) look more natural and easily interpretable compared to spelling perturbed samples (clozapine $\rightarrow$ clpazoine).

In this paper, we propose a new technique for generating adversarial examples combining contextual perturbations based on BERT-MLM, synonym replacement of biomedical entities, typographical errors and numeric entity expansion. We explore several classification models to demonstrate the efficacy of our method. Experiments conducted on two benchmark biomedical datasets demonstrate the strength and effectiveness of our attack. As a future work, we would like to explore more about retraining the models with the perturbed samples in order to improve model robustness.

The author would like to thank the annotators for hard work, and also the anonymous reviewers for their insightful comments and feedback.