Embedding-based Zero-shot Retrieval through Query Generation

(Wikipedia Generated Queries) is the dataset synthesized by our QG model. It consists of 110M synthetically generated queries on 22M passages from our English Wikipedia dataset¹¹1We take the official Wikipedia 2016 database dump and split individual Wikipedia articles into passages with maximum 100 words each, respecting sentence boundaries.. See Table 1 for some examples. This dataset will be open-sourced for research purposes.

Kwiatkowski et al. (2019) is an end-to-end question answering dataset constructed from English Wikipedia. We convert Natural Questions into a passage retrieval dataset, keeping queries that have long answers. We discard answers that are not regular paragraphs (e.g., tables, lists). Ultimately, we retain 110,865 of 307,372 queries from the official training set and 4,340 of 7,842 queries from the official development set. Collecting long answers from these queries nets us 84,783 passages. We hold out the 4,340 development queries as the test set.

Dietz et al. (2017) is a dataset created from English Wikipedia for complex answer retrieval, where the queries are generated by concatenating articles and section titles and the ground truth passages consist of the paragraphs within that section. We use the last segment from the predefined 5 segments of the dataset as our test set.

Nguyen et al. (2016) comprises over 1M queries sampled from Bing search query logs. Specifically, we use the MSMARCO Passage Ranking (PR) dataset, which contains 8.8M passages from over 3.5M web documents retrieved by Bing. We use the official development set of 55,578 queries as our test set. We also use the official training set of 532,761 queries and their positive passages to train our query generation model.

Feng et al. (2015) consists of 2,000 evaluation queries on passages sourced from the Insurance Library database.

Hashemi et al. (2020) is a collection of 2,626 open-domain, non-factoid questions sourced from Yahoo! Answers. We use the provided test set of 200 queries with the corpus of over 400K passages for evaluation.

Tsatsaronis et al. (2012) Task B consists of 500 English evaluation queries and reference answers constructed by a team of biomedical experts (BioASQ 7b). We collect approximately 20M non-empty article abstracts sourced from the PubMed database as our passage corpus.

Ahmad et al. (2019) is a benchmark for evaluating sentence-level answer retrieval models. This benchmark consists of two datasets, ReQA SQuAD and ReQA NQ, created from the official training sets of SQuAD and Natural Questions, respectively. Each candidate passage is a sentence concatenated with its context passage and a relevant candidate is the answer sentence concatenated with its context passage.

We begin by finetuning the pretrained BART-large (374M parameters) on MSMARCO, generating queries given a passage. We train BART for 5 epochs with a batch size of 96 and a learning rate of 3e-5 (warmup ratio $0.1$). Once the BART model is trained, we perform generation over English Wikipedia passages to generate 10 synthetic queries for each passage, using nucleus sampling Holtzman et al. (2020) with $p=0.95$, and retaining the top-5 queries based on likelihood scores. The resulting dataset consists of 110M synthetic query-passage pairs, which we call WikiGQ.

We initialize our Siamese/two-tower retrieval models with BERT-base (110M parameters). We then pretrain our retrieval models on WikiGQ. The pretraining is done on 32 Nvidia V100 GPUs with a batch size of 100 per GPU for 10 epochs. We set a max learning rate of 5e-5 with warmup ratio of $0.1$. The pretraining takes about 4 days for the Siamese model and 6 days for the Two-Tower model.

For our baseline models trained on the MSMARCO official training set, we train for 10 epochs on 8 GPUs with a batch size of 50 per GPU and max learning rate of 3e-5 with warmup ratio of $0.1$. Half-precision training²²2https://github.com/nvidia/apex is enabled in all of our experiments.

For our Inverse Cloze Task (ICT) baselines, we follow the process defined in Lee et al. (2019). For each passage, we randomly sample one sentence as a query and remove it from the passage $90\%$ of the time. To ensure fair comparison, we train on the ICT data, where data is generated on-the-fly, for 5 times as many epochs as on the QG synthetic data in order to match the number of training iterations. Specifically, we train our models on QG synthetic data for 10 epochs (50 epochs for ICT) on 8 GPUs with a batch size of 50 per GPU and learning rate 3e-5.

We use the same BART QG model (trained on MSMARCO) to generate domain-specific synthetic data (we do this for several datasets that are not derived from English Wikipedia: InsuranceQA, ANTIQUE, BioASQ). When finetuning our retrieval models on domain-specific synthetic data, we set max learning rate to 3e-5 with warmup ratio of $0.1$ and finetune on 8 GPUs with a batch size of 50 per GPU for 10 epochs (5 epochs for BioASQ).

We consider two variations of retrieval model: the Two-Tower model (without shared weights) and the Siamese model (with shared weights). For each variation we test various embedding sizes for the BERT-base encoders. For simplicity, we perform our ablations on the (non-synthetic) Natural Questions dataset. Fig. 2 shows a monotonic increase in performance as the embedding size grows larger, slowly plateauing around an embedding size of 512.

Recently, several pretraining tasks have been proposed to improve the performance of embedding-based neural retrieval. ICT was first introduced by Lee et al. (2019) with strong performance on open-domain QA tasks. Subsequently, Chang et al. (2020) introduced two new pretraining tasks, Body First Selection (BFS) and Wiki Link Prediction (WLP), in addition to ICT. In this ablation study, we directly compare these pretraining methods against our own.

Beginning with the Natural Questions dataset, we compare models trained with ICT against our own model, trained with synthetic queries. When synthesizing queries, we consider two decoding techniques — beam search and nucleus sampling Holtzman et al. (2020). With beam search decoding, using a beam width of 5, we generate 5 queries for each passage. Using nucleus sampling, with $p=0.95$, we generate 10 independent samples, selecting the top 5 queries based on the likelihood scores. After removing duplicate queries, the total number of generated query-passage pairs is shown in Table 5.

We find that both Siamese and Two-Tower models, pretrained with synthetic queries, significantly outperform their ICT counterparts. We hypothesize that models trained on ICT (where the model attempts to retrieve a passage, given a sentence from that passage) may suffer from this inductive bias when used to perform ranking (where the model must retrieve a passage, given a user query). On the other hand, synthetic queries from our generator are very similar to real user queries. To illustrate this point further, we provide examples in Table 6.

Additionally, we find that training with synthetic queries generated via nucleus sampling outperforms training with queries generated with beam search. While beam search decoding generates strictly more training pairs than nucleus sampling (nucleus sampling can generate duplicates across $k$ independent runs), nucleus sampling tends to result in more diverse queries Holtzman et al. (2020). We show examples in Table 6.

For completeness, we additionally compare our model against the ICT, BFS and WLP pretraining tasks in conjunction. Following Chang et al. (2020), we use all of English Wikipedia during pretraining for both synthetic query generation (i.e., WikiGQ as described in Section 4) and evaluate the resulting models on the ReQA benchmark.

We report our zero-shot retrieval results on the BM25 baseline and models trained on WikiGQ. A summary of the results are shown in Table 7⁵⁵5Our BM25 baseline significantly outperforms those reported in Ahmad et al. (2019); Chang et al. (2020). As described earlier, our BM25 results are obtained from Elasticsearch using default settings. We have communicated with the authors from Ahmad et al. (2019) and they were able to reproduce our results.. As we can see, on both datasets, models pretrained on our synthetic WikiGQ corpus significantly outperform their counterparts pretrained with ICT+BFS+WLP. Notably, the Two-Tower model trained on ICT+BFS+WLP is actually finetuned with $1\%$ of the original training data after ICT+BFS+WLP pretraining. However, our models are trained only on the synthetic WikiGQ data (no real data was used). These results indicate that pretraining with WikiGQ is more favorable than ICT, BFS and WLP for retrieval. Finally, our models pretrained on synthetic data outperform BM25 on ReQA NQ by a large margin. However, our model is unable to consistently outperform BM25 on ReQA SQuAD. We believe that this is due to certain characteristics of the SQuAD dataset. Similar observations have been made in Lee et al. (2019); Karpukhin et al. (2020)⁶⁶6They present two possible reasons. First, the high lexical overlap between SQuAD questions and passages due to the fact that the questions were created by the annotators after seeing the passages. Second, data was collected from only 500+ Wikipedia articles which makes the data highly correlated and suboptimal for models trained on i.i.d samples..

We study the impact of WikiGQ pretraining when we can finetune models on hand-labeled training data. Specifically, we finetune the Siamese models with and without pretrained on WikiGQ on the official training data of two datasets, Natural Questions and InsuranceQA. Our results show that pretraining with WikiGQ improves performance for both Wikipedia-based dataset (Natural Questions) and out-of-domain dataset (InsuranceQA). We detail our results in Table 8.

Finally, we study the sample efficiency of synthetic pretraining with WikiGQ. We evaluate on two datasets, Natural Questions (Wikipedia-based) and InsuranceQA (non-Wikipedia-based). Specifically, we train the Siamese model on synthetic data generated with various fractions of English Wikipedia. Our results show that performance improves monotonically with synthetic dataset size on both Natural Questions and InsuranceQA. However, we witness diminishing returns, and training on 22M passages only confers a slight improvement over 4M passages. The results are shown in Fig. 3.