Evaluating Neural Machine Comprehension Model
Robustness to Noisy Inputs and Adversarial Attacks

We examine how changes to the context paragraph (excluding the answer span) affect the model’s performance using the following perturbations:

• •

  Character-level. In computer security, this is known as a homograph attack. These attacks attacks have been investigated to identify phishing and spam (Fu et al., 2006b, a; Liu and Stamm, 2007) but to our knowledge have not been applied in the NLP domain. We replace 25% of characters in the context paragraph with deceptive Unicode characters¹¹1From https://www.unicode.org/Public/security/12.1.0/intentional.txt that to a human are indistinguishable from the original.

• •

  Word-level. We randomly replace 25% of the words in the context paragraph with their nearest neighbor in the GLoVe (Pennington et al., 2014) embedding space.²²2Several alternative embedding techniques could be used to find the nearest neighbors e.g., Word2Vec or FastText. We use GLoVe for consistency with previous work (Li et al., 2018).

• •

  Sentence-level. We use Improved ParaBank Rewriter (Hu et al., 2019), a machine translation approach for sentence paraphrasing, to paraphrase sentences in the context paragraph. We perform sentence tokenization, paraphrase each sentence with the paraphraser, then recombine the sentences.

For character and word perturbations, we use 25% as this is where the performance curve in Heigold et al. (2018) flattens out.³³3Belinkov and Bisk (2018) perturbed text at 100% (every word), while Heigold et al. (2018) experimented with 5–30% perturbations. Regardless of the type of perturbation, we do not perturb the context that contains the answer span, so that the answer can always be found in the context unperturbed. Because paraphrasing is per sentence, we only modify sentences that do not contain the answer span. An example of each perturbation type is shown in Tab. 2.

For each perturbation type, we experiment with perturbing the training data at differing amounts. All models are tested on fully perturbed test data.

• •

  None: clean training data

• •

  Half: perturb half the training examples

• •

  Full: perturb the entire train set

• •

  Both: append the entire perturbed data to the entire clean data⁴⁴4This has twice the amount of data as other settings so is not directly comparable, but many papers show that doing this can improve a model’s performance.

• •

  Ens: ensemble model that relies on none, half and full perturbed data; we rely on ensemble voting and only include the word in the predicted answer if any two models agree.

We experiment with several recent models and embeddings that have been state-of-the-art on machine comprehension.

• •

  BiDAF model with ELMo Seo et al. (2017); Peters et al. (2018). ELMo is a deep, contextualized, character-based word embedding method using a bidirectional language model. The Bi-Directional Attention Flow model is a hierarchical model with embeddings at multiple levels of granularity: character, word, and paragraph. We use pre-trained ELMo embeddings in the BiDAF model implemented in AllenNLP (Gardner et al., 2018).

• •

  BERT (Devlin et al., 2019). BERT is another contextualized embedding method that uses Transformers (Vaswani et al., 2017). It is trained to recover masked words in a sentence as well as on a next-sentence prediction task. The output layer of BERT is fed into a fully-connected layer for the span classification task. Pre-trained embeddings can be fine-tuned to a specific task, and we use the Huggingface PyTorch-Transformers package, specifically bert-large-cased-whole-word-masking-finetuned-squad model. We fine-tune for two epochs in each experimental settings.

We experiment on two benchmark MC datasets:

• •

  SQuAD (Rajpurkar et al., 2016). The Stanford Question Answering Dataset is a collection of over 100K crowdsourced question and answer pairs. The context containing the answer is taken from Wikipedia articles.

• •

  TriviaQA (Joshi et al., 2017). A collection of over 650K crowdsourced question and answer pairs, where the context is from web data or Wikipedia. The construction of the dataset differs from SQuAD in that question answer pairs were first constructed, then evidence was found to support the answer. We utilize the Wikipedia portion of TriviaQA, whose size is comparable to SQuAD. To match the span-based setting of SQuAD, we convert TriviaQA to the SQuAD format using the scripts in the official repo and remove answers without evidence.

Ensemble adversarial training has recently been explored (Tramèr et al., 2018) as a way to ensure robustness of ML models. For each perturbation type, we present results ensembled from the none, half, and full perturbed settings. We tokenize answers from these three models and keep all tokens that appear at least twice as the resulting answer (Tab. 3). Even when all three model answers differ (e.g. in the word perturbation case), ensembling can often reconstruct the correct answer. Nevertheless, we find that this ensembling only helps for TriviaQA, which has an overall higher error rate (bottom row of each figure in Fig. 1).

We did not observe a large increase in errors with paraphrase perturbations (Fig. 1), perhaps because paraphrasing, unlike the char and word perturbations, is not a deliberate attack on the sentence. Here we experiment with a novel strategic paraphrasing technique that targets specific words in the context and then generates paraphrases that exclude those words. We find the most important words in the context by individually modifying each word and obtaining the model’s prediction and confidence, a process similar to Li et al. (2018). Our modification consists of removing the word and examining its effect on the model prediction. The most important words are those which, when removed, lower the model confidence of a correct answer or increase confidence of an incorrect answer. The Improved ParaBank Rewriter supports constrained decoding, i.e. specifying positive and negative constraints to force the system output to include or exclude certain phrases. We specify the top five important words in the context as negative constraints to generate strategic paraphrases.⁶⁶6The number of constraints does not necessarily indicate the number of words that are changed in the context.

We experimented on 1000 instances in the SQuAD dev set as shown in Tab. 4. Our results indicate that strategic paraphrasing with negative constraints is a successful adversarial attack, lowering the F1-score from 89.96 to 84.55.

Analysis shows that many words in the question are important and thus excluded from the paraphrases. We also notice that paraphrasing can occasionally turn an incorrect prediction into a correct one. Perhaps paraphrasing makes the context easier to understand by removing distractor terms; we leave this for future investigation.

In a black-box setting, model confidence is one of the only indications of the model’s inner workings. The models we employed do not provide a single confidence value; AllenNLP gives a probability that each word in the context is the start and end span, while the BERT models only give the probability for the start and end words. We compute the model’s confidence using the normalized entropy of the distribution across the context words, where $n$ is the number of context words, and take the mean for both the start and end word: $1-\frac{H_{n}(\mathbf{s})+H_{n}(\mathbf{e})}{2}$, where $\mathbf{s}$ and $\mathbf{e}$ are probability distributions for the start and end words, respectively. Low entropy indicates certainty about the start/end location. Since the BERT models only provide probabilities for the start and end words, we approximate the entropy by assuming a flat distribution, dividing the remaining probability equally across all other words in the context.

Comparing confidence across models (Fig. 2(a)), the BERT model has lower confidence for misclassifications, which is ideal. A model should not be confident about errors. Fig. 2(b) compares confidence across perturbation type. In the none training setting, character perturbations introduce the most uncertainty compared to word or paraphrase perturbations. This is expected, since character perturbations result in unknown words. In the adversarial training settings, word perturbations lead to the highest number of high-confidence misclassifications. Thus, to convincingly mislead the model to be highly confident about errors, one should use word perturbations.

We examine if models make more errors on specific types of questions in adversarial training, i.e., some questions could just be easier that others. We first examine question type:⁷⁷7Computed as the first word of the question. Many how questions are how many or how much, rather than how in the “in what manner” sense. who, what, which, when, where, why, how, and other. The majority of SQuAD questions are what questions, while most TriviaQA questions are other questions, perhaps indicating more complex questions (Fig. 4(a)). We see that models usually choose answers appropriate for the question type; even if they are incorrect, answers to when questions will be dates or time word spans, and answers to how many questions will be numbers. Fig. 4(a) presents key findings on differences in model misclassifications between two datasets given specific question types. On the SQuAD dataset, the model finds certain question types, e.g. when and how, easiest to answer regardless of the perturbation type. Responses to these questions, which generally expect numeric answers, are not greatly affected by perturbations. For TriviaQA, in general we observe more errors across question types compared to SQuAD, i.e. more errors in what, which and who questions.

Regarding question length, SQuAD and TriviaQA have similar distributions (Fig. 4(b)). Both datasets have a mode answer length around 10 words; TriviaQA has a slightly longer tail in the distribution. We did not find question length to impact the error. Regarding context length, SQuAD and TriviaQA have vastly differing context length distributions (Fig. 4(c)), partly due to how the two datasets were constructed (see Sec. 3.4 for details). For both datasets, the error distribution mirrors the context length distribution, and we did not find any relation between model errors and context length.

Our analysis shows that the length of the model’s answer is a strong predictor of model error in the adversarial setting: the longer the answer length, the more likely it is to be incorrect. Fig. 3 plots the proportion of correct to incorrect answers. We notice a downward trend which is mostly consistent across experimental settings. For both SQuAD and TriviaQA, the models favored shorter answers, which mirrors the data distribution.

Here, we examine the effect of task complexity on model performance under adversarial training, using inter-annotator agreement as a proxy for question complexity and paragraph readability as a proxy for context complexity.

Inter-annotator agreement represents a question’s complexity: low agreement indicates that annotators did not come to a consensus on the correct answer; thus the question may be difficult to answer. We examine SQuAD, whose questions have one to six annotated answers. In Fig. 5, we present inter-annotator agreement (human confidence) plotted against model confidence over the four training perturbation amounts, looking only at the incorrect predictions. The setting is SQuAD BERT with character perturbation. We observe that the models are generally confident even when the humans are not, which is noticeable across all perturbation amounts. However, we see interesting differences in model confidence in adversarial training: models trained in the none and half settings have confidence ranging between 0 and 1 compared to the models trained in full and both setting with confidence above 0.8, indicating training with more perturbed data leads to more confident models.

To evaluate the effect of context complexity, we use the Flesch-Kincaid reading level (Kincaid et al., 1975) to measure readability. For questions the model answered incorrectly, the median readability score was slightly higher than the median score for correct responses (Tab. 5), indicating that context with higher reading level is harder for the model to understand. TriviaQA contexts have higher reading levels than SQuAD.