Leveraging Expert Guided Adversarial Augmentation
For Improving Generalization in Named Entity Recognition

Deep learning models have achieved great performance on many natural language processing (NLP) problems Bahdanau et al. (2016); Devlin et al. (2019). However, many recent works have shown that these models often rely on spurious correlations which are not necessarily the causal artifacts. Thus, these models perform well on the in-distribution test set but are likely to exhibit a huge performance decline on out-of-distribution data (e.g. real world data) Tu et al. (2020); Kaushik and Lipton (2018); Poliak et al. (2018); Gururangan et al. (2018); Zhang et al. (2019); Glockner et al. (2018). Prior works have constructed adversarial examples for benchmarking the generalization ability of state-of-the-art NLP models on out-of-distribution examples Kaushik et al. (2020); Zhang et al. (2019); Glockner et al. (2018). Proposed approaches such as random word swapping Jin et al. (2020) and the appending of a sentence to the end of text Jia and Liang (2017) do not take into consideration the unique linguistic properties and variations associated with named entities. As a key problem setting involving the classification of semantic categories of entities (e.g., Organizations, Locations) Nadeau and Sekine (2007), NER is still in need of improved benchmarks of true generalization.

Previous works Bernier-Colborne and Langlais (2020); Fu et al. (2020); Stanislawek et al. (2019) have shown that words which have different entity labels in different scenarios often lead to frequently occurring errors of NER models. This can be especially problematic in specific domain applications where this challenging case is common. For example, when training an NER model for political text mining, it would be of great importance to differentiate between the categories of Clinton (Person) and the Clinton Foundation (Organization). We make use of this as the inspiration for designing expert-guided heuristic linguistic patterns for creating a high quality adversarial dataset for NER.

Leveraging such expert-guided heuristics, we propose an automated procedure for adversarial augmentation. We use this automated procedure to first generate adversarial examples from the test data. Since some of these automatically generated adversarial examples may lack quality in terms of syntax or semantics, we manually select only the examples that are of high quality for the construction of the challenging test set. The performance of state-of-the-art NER systems drops severely on this challenging test set. To alleviate this degradation, we first use the proposed heuristics to augment the training examples (without manually filtering the data for quality), which proves to be effective. We further utilize mixup Zhang et al. (2018); Chen et al. (2020) as a regularization technique to interpolate the representations of the original examples and the augmented examples, leading to a smoother decision boundary and improved generalization ability (Lee et al., 2020; Wang et al., 2021b).

Current NER models often deal with unambiguous cases where one entity often gets assigned to the same label. By inducing challenging cases using the Overlapping Categories Fu et al. (2020) that alter the entity and its label, models can then be tested to see whether they are only learning spurious correlations between the token and the label. For the construction of adversarial examples by the altering of entity types, we define three components: (i) Eligibility Check: We only augment entities that are eligible to change their entity types. (ii) Entity Token Change: By adding or deleting certain predefined tokens, we change the entity type of the original tokens to a target type. (iii) Entity Context Change: To deal with ambiguous tokens, we further add some predefined contexts that correspond to the target entity type. Note that predefined words/phrases/contexts used in different scenarios form different predefined word phrase sets, into which embed expert knowledge. During the automatic generation process, we randomly sample from the corresponding word phrase sets. Table 1 contains examples of expert-guided adversarial augmentations. The three components are defined below for their use in the transition to each target entity type (organization, person, location):

Adversarial training improves a model’s robustness to adversarial examples by directly training on adversarial examples, however, such training might hurt generalization (Raghunathan et al., 2019) or cause overfitting on adversarial features (Lee et al., 2020) (predefined word phrases in our case). To this end, we leverage mixup Zhang et al. (2018); Verma et al. (2019) to mitigate these issues and further improve generalization on the basis of adversarial training (Lee et al., 2020).

Given a pair of data points $(x,y)$ and $(x^{\prime},y^{\prime})$, where $x$ denotes a data point and $y$ denotes its label in a one-hot representation, mixup (Zhang et al., 2018) creates a new data point by the interpolation of the data and their labels as shown below with $\lambda$ being drawn from a beta distribution:

$$\hat{x}=\lambda x+(1-\lambda)x^{\prime}$$

(1)

$$\hat{y}=\lambda y+(1-\lambda)y^{\prime}$$

(2)

In this work, $(x,y)$ is a training example that is eligible for heuristic augmentation and is paired with its heuristically modified version $(x^{\prime},y^{\prime})$. Since textual data is discrete and cannot be mixed in the input space, the interpolation of the two examples is computed in the hidden space.

Following Chen et al. (2020), Let $\mathbf{h}^{m}=\{h_{1}..h_{n}\}$ be the hidden representations after the $m$-th layer where they are the concatenation of the token representations. The hidden representation for each token in the original example at the $m$-th layer $\mathbf{h}^{m}$ is linearly interpolated with $\mathbf{h}^{m\prime}$, the representation for each token in the augmented example, by a ratio $\lambda$:

$$\mathbf{\hat{h}}^{m}=\lambda\mathbf{h}^{m}+(1-\lambda)\mathbf{h}^{m\prime}$$

(3)

Then $\mathbf{\hat{h}}^{m}$ is passed to the $(m+1)$-th layer, and the labels for the final output logits are mixed at the same ratio. $m$ is randomly sampled from $\{8,9,10\}$. The mixing parameter $\lambda$ is sampled from a beta distribution: $\lambda\sim\mathit{B}(\alpha,\beta)$, where $\alpha$ and $\beta$ determine the skew of the beta distribution. In this work, we use two different beta distributions from which to sample $\lambda$. For each pair of data points, two mixed data points are generated. One data point is closer to the original examples and the other is closer to the adversarial examples. See Appendix B for more details.

This work proposed an expert-guided adversarial augmentation for NER consisting of the altering of entity types by strategic selection and modification of tokens and their contexts. Using this augmentation strategy on CoNLL 2003 and manually filtering the generated examples for quality, we constructed a high-quality challenging test set for the NER task. We show that SOTA NER systems suffer from dramatic performance drop when evaluated on our challenging set. Beyond simply using the proposed augmentation for adversarial training, we demonstrated that leveraging mixup between original examples and their augmented versions can outperform state-of-the-art baselines on in-distribution data, the challenging set, and few-shot generalization to out-of-domain data.

We would like to thank the anonymous reviewers for their helpful comments, and the members of the Georgia Tech SALT lab for their feedback.