Synthetic Target Domain Supervision for Open Retrieval QA

Let $(p,q,a)$ be an MRC example comprising a passage $p$, a question $q$, and its short answer $a$ in $p$. Let $s$ be the sentence in $p$ that contains the answer $a$. In what follows, we train an example generator to produce the triple $(s,a,q)$ given $p$. The answer sentence $s$ is subsequently used to locate $a$ in $p$, as a short answer text (e.g., a named entity) can generally occur more than once in a passage.

To train the generator, we fine-tune BART (Lewis et al., 2020a)—a pre-trained denoising sequence-to-sequence generation model—with MRC examples from open domain datasets like SQuAD (Rajpurkar et al., 2016). The generator $g$ with parameters $\theta_{g}$ learns to maximize the conditional joint probability $P(s,a,q|p;\theta_{g})$. In practice, we (i) only output the first ($s_{f}$) and the last ($s_{l}$) word of $s$ instead of the entire sentence for efficiency, and (ii) use special separator tokens to mark the three items in the generated triple.

Given a target domain passage $p$ at inference time, an ordered sequence $(s_{f},s_{l},[SEP],a,[SEP],q)$ is sampled from $g$ using top-$k$ top-$p$ sampling (Holtzman et al., 2020), which has been shown to yield better training examples than greedy or beam search decoding due to greater sample diversity (Sultan et al., 2020). From this generated sequence, we create positive synthetic training examples for both passage retrieval: $(q,p)$ and MRC: $(p,q,a)$, where $s_{f}$ and $s_{l}$ are used to locate $a$ in $p$. Table 1 shows two examples generated by our generator from a passage in the CORD-19 collection (Wang et al., 2020).

As stated before, we use DPR (Karpukhin et al., 2020) as our base retrieval model. While other competitive methods such as ColBERT (Khattab and Zaharia, 2020) exist, DPR offers a number of advantages in real-time settings as well as in our target scenario where retrieval is only a component in a larger ORQA pipeline. For example, by compressing each passage down to a single vector representation, DPR can operate with significantly less memory. It is also a faster model for several reasons, including not having a separate re-ranking module.

For target domain supervision of DPR, we fine-tune its off-the-shelf open domain instance with synthetic examples. At each iteration, a set of questions is randomly sampled from the generated dataset. Following Karpukhin et al. (2020), we also use in-batch negatives for training. We refer the reader to their article for details on DPR supervision. We call this final model the Adapted DPR model.

For MRC, we adopt the now standard approach of  Devlin et al. (2019) that (i) starts from a pre-trained transformer language model (LM), (ii) adds two pointer networks atop the final transformer layer to predict the start and end positions of the answer phrase, and (iii) fine-tunes the entire network with annotated MRC examples. We choose RoBERTa (Liu et al., 2019a) as our base LM. Given our out-of-domain target setting, we fine-tune it in two stages as follows.

First, the RoBERTa LM is fine-tuned on unlabeled target domain documents, which is known to be a useful intermediate fine-tuning step (Gururangan et al., 2020). This target domain model is then further fine-tuned for MRC, where we use both human annotated open domain MRC examples and target domain synthetic examples, as detailed in Section 3. Additionally, we denoise the synthetic training examples using a roundtrip consistency (Alberti et al., 2019a) filter: an example is filtered out if its candidate answer score, obtained using an MRC model trained on SQuAD 2.0 and NQ, is lower than a threshold $t$ ($t$ tuned on a validation set).

Using the described retrieval and MRC components, we construct our final ORQA system that executes a four-step process at inference time. First, only the $K$ highest scoring passages returned by IR for the input question are retained ($K$ tuned on a validation set). Each passage is then passed along with the question to the MRC component, which returns the respective top answer and its MRC score. At this point, each answer has two scores associated with it: its MRC score and the IR score of its passage. In the third step, these two scores get normalized using the Frobenius norm and combined using a convex combination. The weight in the combination operation is tuned on a validation set. Finally, the answer with the highest combined score is returned.

We select COVID-19 as our primary target domain, an area of critical interest at the point of the writing. We use 74,059 full text PDFs from the June 22, 2020 version of CORD-19 (Wang et al., 2020) document collection on SARS-CoV-2–and related coronaviruses as our retrieval corpus. Each document is split into passages that (a) contain no more than 120 words, and (b) align with sentence boundaries, yielding around 3.5 million passages.

We utilize three existing datasets for COVID-19 target domain evaluation. The first one, used to evaluate retrieval and MRC results separately as well as end-to-end ORQA performance, is COVID-QA-2019(Möller et al., 2020)—a dataset of question-passage-answer triples created from COVID-19 scientific articles by volunteer biomedical experts. We split the examples into Dev and Test subsets of 203 and 1,816, respectively. Since end-to-end ORQA examples consist of only question-answer pairs with no passage alignments, we also create a version of this dataset for ORQA evaluation (Open-COVID-QA-2019 henceforth) wherein duplicate questions are de-duplicated and different answers to the same question are all included in the set of correct answers, leaving 201 Dev and 1,775 Test examples.

Our second dataset—COVID-QA-147 (Tang et al., 2020)—is a QA dataset obtained from Kaggle’s CORD-19 challenge, containing 147 question-article-answer triples with 27 unique questions and 104 unique articles. Due to the small number of unique questions in this dataset, we only use it for out-of-domain MRC evaluation.

Finally, COVID-QA-111 (Lee et al., 2020a) contains queries gathered from different sources, e.g., Kaggle and the FAQ sections of the CDC and the WHO. It has 111 question-answer pairs with 53 interrogative and 58 keyword-style queries. Since questions are not aligned to passages in this dataset, we use it only to evaluate IR and ORQA.

We fine-tune BART for three epochs on the open domain MRC training examples of SQuAD1.1 (Rajpurkar et al., 2016) (lr=3e-5). Synthetic training examples are then generated for COVID-19 from the CORD-19 collection. We split the articles into chunks of at most 288 wordpieces and generate five MRC examples from each of the resulting 1.8 million passages. For top-$k$ top-$p$ sampling, we use $k$=$10$ and $p$=$0.95$. Overall, the model generates about 7.9 million examples.

We use the DPR-Multi systemfrom (Karpukhin et al., 2020) as our primary neural IR baseline. DPR-Multi comes pre-trained on open-retrieval versions of several MRC datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), WebQuestions (Berant et al., 2013), CuratedTrec (Baudiš and Šedivỳ, 2015) and TriviaQA (Joshi et al., 2017). We fine-tune it for six epochs with COVID-19 synthetic examples to train our Adapted DPR model (lr=1e-5, batch size=128). We also evaluate the Inverse Cloze Task (ICT) method (Lee et al., 2019) as a second neural baseline, which masks out a sentence at random from a passage and uses it as a query to create a query-passage training pair. We use ICT to fine-tune DPR-Multi on the CORD-19 passages of Section 3.2, which makes it also a synthetic domain adaptation baseline. Finally, for each neural IR model, we also evaluate its ensemble with BM25 that computes a convex combination of normalized neural and BM25 scores. The weight for BM25 in this combination is 0.3 (tuned on Open-COVID-QA-2019 Dev).

Our baseline MRC model is based on a pre-trained RoBERTa-Large LM, and is fine-tuned for three epochs on SQuAD2.0 and then for one epoch on NQ. It achieves a short answer EM of 59.4 on the NQ dev set, which is competitive with numbers reported in (Liu et al., 2020). For target domain training, we first fine-tune a RoBERTa-Large LM on approximately 1.5GB of CORD-19 text containing 225 million tokens (for 8 epochs, lr=1.5e-4). The resulting model is then fine-tuned for MRC for three epochs on SQuAD2.0 examples and one epoch each on roundtrip-consistent synthetic MRC examples and NQ. For roundtrip consistency check, we use a threshold of $t$=$7.0$, which leaves around 380k synthetic examples after filtering.

We evaluate IR using Match@$k$, for $k\in\{20,40,100\}$ (Karpukhin et al., 2020). For MRC, we use standard Exact Match (EM) and F1 score. Finally, end-to-end ORQA accuracy is measured using Top-1 and Top-5 F1.

Table 2 shows performances of different IR systems on Open-COVID-QA-2019 and COVID-QA-111. BM25²²2Lucene Implementation. BM25 parameters $b=0.75$ (document length normalization) and $k_{1}=1.2$ (term frequency scaling) worked best. demonstrates robust results relative to the neural baselines. While DPR-multi is competitive with BM25 on COVID-QA-111, it is considerably behind on the larger Open-COVID-QA-2019. ICT improves over DPR-multi, indicating that even weak target domain supervision is useful. The proposed Adapted DPR system achieves the best single system results on both datasets, with more than 100% improvement over DPR-Multi on the Open-COVID-QA-2019 Test set. Finally, ensembling over BM25 and neural approaches yields the best results. The BM25+Adapted DPR ensemble is the top system across the board, with a difference of at least 14 points with the best baseline on the Open-COVID-QA-2019 Test set (all metrics), and 8 points on COVID-QA-111.

Upon closer examination, we find that BM25 and Adapted DPR retrieve passages that are very different. For Open-COVID-QA-2019, for example, only 5 passages are in common on average between the top 100 retrieved by the two systems. This diversity in retrieval results explains why they complement each other well in an ensemble system, leading to improved IR performance.

Table 4 shows results on the two COVID-19 MRC datasets. Input to each model is a question and an annotated document that contains an answer. Our proposed model achieves 2.0–3.7 F1 improvements on the Test sets over a SOTA open domain MRC baseline. On the COVID-QA-2019 Dev set, we see incremental gains from applying the two domain adaptation strategies.

Using different pairings of the above IR and MRC systems, we build several ORQA pipelines. Each computes a convex combination of its component IR and MRC scores after normalization, with the IR weight being 0.7 (tuned on Open-COVID-QA-2019 Dev). We observe that retrieving $K$=100 passages is optimal when IR is BM25 only, while $K$=40 works best for BM25+Neural IR.

Table 3 shows end-to-end F1 scores of the different ORQA pipelines. Adapted MRC refers to the best system of Section 4.2 (Table 4 Row 3). Crucially, the best system in Table 3 (last row) uses synthetic target domain supervision for both IR and MRC. In a paired $t$-test (Hsu and Lachenbruch, 2005) of the Top-5 F1 scores, we find the differences with the baseline (Row 1) to be statistically significant at $p$$<$$0.01$.

To investigate if our synthetically fine-tuned COVID-19 models can also help improve performance in a related target domain, we evaluate them zero shot on the BioASQ (Balikas et al., 2015) task. BioASQ contains biomedical questions with answers in the PubMed abstracts. For evaluation, we use the factoid questions from the Task 8b training and test sets, totaling 1,092 test questions. As our retrieval corpus, we use around 15M abstracts from Task 8a. We pre-process them as described in Section 3.1 to end up with around 37.4M passages.

Table 5 shows the BioASQ retrieval results, where the proposed Adapted DPR model outperforms both baselines. Table 6 summarizes the evaluation on the end-to-end ORQA task, where we see similar gains from synthetic training. These results show that synthetic training on the CORD-19 articles transfers well to the broader related domain of biomedical QA.