Few-Shot Question Answering by Pretraining Span Selection

The standard approach to question answering is to pretrain a masked language model on raw text, and then fine-tune it with a span selection layer on top Devlin et al. (2019); Joshi et al. (2020); Liu et al. (2019). While this approach is effective, and sometimes exceeds human performance, its success is based on the assumption that large quantities of annotated question answering examples are available. For instance, both SQuAD Rajpurkar et al. (2016, 2018) and Natural Questions Kwiatkowski et al. (2019) contain an order of 100,000 question and answer pairs in their training data. This assumption quickly becomes unrealistic as we venture outside the lab conditions of English Wikipedia, and attempt to crowdsource question-answer pairs in other languages or domains of expertise Tsatsaronis et al. (2015); Kembhavi et al. (2017). How do question answering models fare in the more practical case, where an in-house annotation effort can only produce a couple hundred training examples?

We investigate the task of few-shot question answering by sampling small training sets from existing question answering benchmarks. Despite the use of pretrained models, the standard approach yields poor results when fine-tuning on few examples (Figure 1). For example, RoBERTa-base fine-tuned on 128 question-answer pairs from SQuAD obtains around 40 F1. This is somewhat expected, since the pretraining objective is quite different from the fine-tuning task. While masked language modeling requires mainly local context around the masked token, question answering needs to align the question with the global context of the passage. To bridge this gap, we propose (1) a novel self-supervised method for pretraining span selection models, and (2) a question answering layer that aligns a representation of the question with the text.

We introduce Splinter (span-level pointer), a pretrained model for few-shot question answering. The challenge in defining such a self-supervised task is how to create question-answer pairs from unlabeled data. Our key observation is that one can leverage recurring spans: n-grams, such as named entities, which tend to occur multiple times in a given passage (e.g., “Roosevelt” in Figure 2). We emulate question answering by masking all but one instance of each recurring span with a special [QUESTION] token, and asking the model to select the correct span for each such token.

To select an answer span for each [QUESTION] token in parallel, we introduce a question-aware span selection (QASS) layer, which uses the [QUESTION] token’s representation to select the answer span. The QASS layer seamlessly integrates with fine-tuning on real question-answer pairs. We simply append the [QUESTION] token to the input question, and use the QASS layer to select the answer span (Figure 3). This is unlike existing models for span selection, which do not include an explicit question representation. The compatibility between pretraining and fine-tuning makes Splinter an effective few-shot learner.

Splinter exhibits surprisingly high performance given only a few training examples throughout a variety of benchmarks from the MRQA 2019 shared task Fisch et al. (2019). For example, Splinter-base achieves 72.7 F1 on SQuAD with only 128 examples, outperforming all baselines by a very wide margin. An ablation study shows that the pretraining method and the QASS layer itself (even without pretraining) both contribute to improved performance. Analysis indicates that Splinter’s representations change significantly less during fine-tuning compared to the baselines, suggesting that our pretraining is more adequate for question answering. Overall, our results highlight the importance of designing objectives and architectures in the few-shot setting, where an appropriate inductive bias can lead to dramatic performance improvements.

Extractive question answering is a common task in NLP, where the goal is to select a contiguous span $a$ from a given text $T$ that answers a question $Q$. This format was popularized by SQuAD Rajpurkar et al. (2016), and has since been adopted by several datasets in various domains Trischler et al. (2017); Kembhavi et al. (2017) and languages Lewis et al. (2020); Clark et al. (2020), with some extensions allowing for unanswerable questions Levy et al. (2017); Rajpurkar et al. (2018) or multiple answer spans Dua et al. (2019); Dasigi et al. (2019). In this work, we follow the assumptions in the recent MRQA 2019 shared task Fisch et al. (2019) and focus on questions whose answer is a single span.

The standard approach uses a pretrained encoder, such as BERT Devlin et al. (2019), and adds two parameter vectors $\mathbf{s},\mathbf{e}$ to the pretrained model in order to detect the start position $s$ and end position $e$ of the answer span $a$, respectively. The input text $T$ and question $Q$ are concatenated and fed into the encoder, producing a contextualized token representation $\mathbf{x}_{i}$ for each token in the sequence. To predict the start position of the answer span, a probability distribution is induced over the entire sequence by computing the inner product of a learned vector $\mathbf{s}$ with every token representation (the end position is computed similarly using a vector $\mathbf{e}$):

The parameters $\mathbf{s},\mathbf{e}$ are trained during fine-tuning, using the cross-entropy loss with the start and end positions of the gold answer span.

This approach assumes that each token representation $\mathbf{x}_{i}$ is contextualized with respect to the question. However, the masked language modeling objective does not necessarily encourage this form of long-range contextualization in the pretrained model, since many of the masked tokens can be resolved from local cues. Fine-tuning the attention patterns of pretrained masked language models may thus entail an extensive learning effort, difficult to achieve with only a handful of training examples. We overcome this issue by (1) pretraining directly for span selection, and (2) explicitly representing the question with a single vector, used to detect the answer in the input text.

We formulate a new task for pretraining question answering from unlabeled text: recurring span selection. We replace spans that appear multiple times in the given text with a special [QUESTION] token, except for one occurrence, which acts as the “answer” span for each (masked) cloze-style “question”. The prediction layer is a modification of the standard span selection layer, which replaces the static start and end parameter vectors, $\mathbf{s}$ and $\mathbf{e}$, with dynamically-computed boundary detectors based on the contextualized representation of each [QUESTION] token. We reuse this architecture when fine-tuning on question-answer pairs by adding a [QUESTION] token at the end of the actual question, thus aligning the pretraining objective with the fine-tuning task. We refer to our pretrained model as Splinter.

To evaluate how pretrained models work when only a small amount of labeled data is available for fine-tuning, we simulate various low-data scenarios by sampling subsets of training examples from larger datasets. We use a subset of the MRQA 2019 shared task Fisch et al. (2019), which contains extractive question answering datasets in a unified format, where the answer is a single span in the given text passage.

Split I of the MRQA shared task contains 6 large question answering datasets: SQuAD Rajpurkar et al. (2016), NewsQA Trischler et al. (2017), TriviaQA Joshi et al. (2017), SearchQA Dunn et al. (2017), HotpotQA Yang et al. (2018), and Natural Questions Kwiatkowski et al. (2019). For each dataset, we sample smaller training datasets from the original training set with sizes changing on a logarithmic scale, from 16 to 1,024 examples. To reduce variance, for each training set size, we sample 5 training sets using different random seeds and report average performance across training sets. We also experiment with fine-tuning the models on the full training sets. Since Split I of the MRQA shared task does not contain test sets, we evaluate using the official development sets as our test sets.

We also select two datasets from Split II of the MRQA shared task that were annotated by domain experts: BioASQ Tsatsaronis et al. (2015) and TextbookQA Kembhavi et al. (2017). Each of these datasets only has a development set that is publicly available in MRQA, containing about 1,500 examples. For each dataset, we sample 400 examples for evaluation (test set), and follow the same protocol we used for large datasets to sample training sets of 16 to 1,024 examples from the remaining data.

To maintain the few-shot setting, every dataset in our benchmark has well-defined training and test sets. To tune hyperparameters, one needs to extract validation data from each training set. For simplicity, we do not perform hyperparameter tuning or model selection (see Section 5), and thus use all of the available few-shot data for training.

We describe our experimental setup in detail, including all models and baselines.

Our experiments show that Splinter dramatically improves performance in the challenging few-shot setting, unlocking the ability to train question answering models with only hundreds of examples. When trained on large datasets with an order of 100,000 examples, Splinter is competitive with (and often better than) the baselines. Ablation studies demonstrate the contributions of both recurring span selection pretraining and the QASS layer.

The remarkable results of GPT-3 Brown et al. (2020) have inspired a renewed interest in few-shot learning. While some work focuses on classification tasks Schick and Schütze (2020); Gao et al. (2021), our work investigates few-shot learning in the context of extractive question answering.

One approach to this problem is to create synthetic text-question-answer examples. Both Lewis et al. (2019) and Glass et al. (2020) use the traditional NLP pipeline to select noun phrases and named entities in Wikipedia paragraphs as potential answers, which are then masked from the context to create pseudo-questions. Lewis et al. (2019) use methods from unsupervised machine translation to translate the pseudo-questions into real ones, while Glass et al. (2020) keep the pseudo-questions but use information retrieval to find new text passages that can answer them. Both works assume access to language- and domain-specific NLP tools such as part-of-speech taggers, syntactic parsers, and named-entity recognizers, which might not always be available. Our work deviates from this approach by exploiting the natural phenomenon of recurring spans in order to generate multiple question-answer pairs per text passage, without assuming any language- or domain-specific models or resources are available beyond plain text.

Similar ideas to recurring span selection were used for creating synthetic coreference resolution examples Kocijan et al. (2019); Varkel and Globerson (2020), which mask single words that occur multiple times in the same context. CorefBERT Ye et al. (2020) combines this approach with a copy mechanism for predicting the masked word during pretraining, alongside the masked language modeling objective. Unlike our approach, which was designed to align well with span selection, CorefBERT masks only single-word nouns (rather than arbitrary spans) and replaces each token in the word with a separate mask token (rather than a single mask for the entire multi-token word). Therefore, it does not emulate extractive question answering. We did not add CorefBERT as a baseline since the performance of both CorefBERT-base and CorefBERT-large was lower than SpanBERT-base’s performance on the full-data MRQA benchmark, and pretraining CorefBERT from scratch was beyond our available computational resources.

We explore the few-shot setting of extractive question answering, and demonstrate that existing methods, based on fine-tuning large pretrained language models, fail in this setup. We propose a new pretraining scheme and architecture for span selection that lead to dramatic improvements, reaching surprisingly good results even when only an order of a hundred examples are available. Our work shows that choices that are often deemed unimportant when enough data is available, again become crucial in the few-shot setting, opening the door to new methods that take advantage of prior knowledge on the downstream task during model development.

This project was funded by the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC HOLI 819080), the Blavatnik Fund, the Alon Scholarship, the Yandex Initiative for Machine Learning and Intel Corporation. We thank Google’s TPU Research Cloud (TRC) for their support in providing TPUs for this research.