A Survey for Efficient Open Domain Question Answering

Traditional ODQA models consist of many components such as question processing, document/passage retrieval and answer processing, etc Ferrucci et al. (2010); Baudiš (2015). DrQA Chen et al. (2017) first simplified the traditional multi-component ODQA models into a two-stage retriever-reader framework. Most recent works follow this framework and further supersede the TF-IDF-based retriever, or CNN-based reader, or both modules in DrQA with stronger transformer-based models, such as BERT, T5, BART, etc Yang et al. (2019); Wu et al. (2020); Karpukhin et al. (2020); Guu et al. (2020); Mao et al. (2021); Izacard and Grave (2020); Singh et al. (2021). Further, the reader in retriever-reader ODQA models can be classified into extractive reader and generative reader for their different behaviour in obtaining answers. We summarize those models specifically in the following paragraphs.

Retriever&Extractive-Reader models find the answer by predicting the start and end position of the answer span from the retrieved evidence. The extractive reader is usually completed by BERT-based model and some works integrate this extractive reader into ODQA models to locate exact answers of open domain questions. For example, BERT_serini Yang et al. (2019) and Skylinebuilder Wu et al. (2020) substitute the CNN reader in DrQA with BERT-based reader. Then, ORQA (Open Retrieval Question Answering system) Lee et al. (2019) supersedes both modules in DrQA with BERT-based models. It first pre-trains the retriever with an unsupervised Inverse Cloze Task (ICT) Lee et al. (2019), achieving an end-to-end joint learning between the retriever and the reader. Further, DPR (Dense passage retriever) Karpukhin et al. (2020) directly leverages pre-trained BERT models to build a dual-encoder retriever without additional pre-training. It trains the retriever through contrastive learning as well as well-conceived negative sampling strategies. DPR becomes a stronger baseline in both ODQA (Open Domain Question Answering) and IR (Information Retrieval) domains. Based on DPR, RocketQA Qu et al. (2021) trains the dual-encoder retriever using novel negative sampling methods: cross-batch negatives for mutil-GPUs training, or utilizing well-trained cross-encoder to select high-confidence negatives from the top-k retrievals of a dual-encoder as denoised hard negatives. Meanwhile, RocketQA constructs new training examples from an collection of unlabeled questions, using a cross-encoder to synthesize passage labels. RocketQA-v2 Ren et al. (2021) extends RocketQA by incorporating a joint training approach for both the retriever and the re-ranker via dynamic listwise distillation.

Retriever&Generative-Reader models directly generate free-format textual answers taking the question and retrieved evidence as input. Some ODQA models adopt this generative reader and explore diversified fusion methods between the retriever and the generative reader. For instance, FiD (Fusion-in-Decoder) Izacard and Grave (2021) is one of the typical methods under this framework. It takes the retrieved evidence by BM25 or DPR Karpukhin et al. (2020) as input of the generative reader T5 Roberts et al. (2020). And it aggregates the multiple evidence by concatenating their representations, and decodes the merged representation to generate the answer. FiD-KD Izacard and Grave (2020) integrates knowledge distillation (KD) into FiD to perform iterative training between the retriever and the generator. RAG (Retrieval-Augmented Generation) Lewis et al. (2020b) trains the retriever and the generator in an end-to-end form by viewing the retrieved evidence as a latent variable. Further, EMDR² (End-to-end training of Multi-Document Reader and Retriever) Singh et al. (2021) adopts mutual supervision and performs end-to-end training between a dual-encoder retriever and a T5-based generator.

In summary, for transformer-based ODQA models like DPR, RocketQA and FiD, they may suffer a challenge: training the retriever relies on strong supervision in pairs of questions and positive documents Izacard and Grave (2020). Unfortunately, many ODQA datasets and applications lack adequate such labeled pairs. To this end, many researchers turn to explore end-to-end training between the retriever and the reader Lee et al. (2019); Guu et al. (2020); Singh et al. (2021); Izacard and Grave (2020, 2021). Meanwhile, works Karpukhin et al. (2020); Qu et al. (2021); Ren et al. (2021) focus on constructing more available training example for the retriever by various effective data augmentation or sampling techniques and achieve stronger empirical performance.

The dual-encoder retrievers like DPR, encode for questions and documents independently, ignoring interaction between questions and documents and limiting their retrieval performance Khattab and Zaharia (2020); Humeau et al. (2020); Khattab et al. (2021); Lu et al. (2022). To remedy this issue, Colbert Khattab and Zaharia (2020) add interaction between different embeddings on the top of a dual-encoder, and Colbert-QA Khattab et al. (2021) applies it into ODQA domain to gain better performance.

Retriever-reader ODQA methods generally obtain good performance. However, due to dense representations for corpus passages and longer evidence for answer reasoning, retriever-reader ODQA models normally suffer from a larger index size and a slower processing speed. We will introduce some techniques to reduce the index size and speedup inference in Section 3.

Retriever-only systems tackle ODQA tasks with a single retriever, eliminating reading or generating step.One typical category of Retriever-only ODQA models is phrase-based systems Seo et al. (2019); Lee et al. (2020, 2021b, 2021c), They first split corpus documents into phrases and then directly retrieve related phrases. The phrase with highest relevance to the question is outputted as the predicted answer. The key to phrase-based ODQA methods is how to represent the phrases. DenSPI (Dense-Sparse Phrase Index) Seo et al. (2019) is one of the representative methods for the Retriever-only framework. It represents phrases with both dense and sparse vectors. The dense vectors are obtained by a BERT encoder and the sparse ones are by TF-IDF method. Further, DenSPI+Sparc Lee et al. (2020) achieves dynamic learning of sparse representations through a rectified self-attention mechanism, and uses them to replace the original static ones used in DenSPI. Conversely, DensePhrases Lee et al. (2021b) omits the sparse representations of phrases, and only leverages the dense ones. TQR Sung et al. (2022) further propels the performance of DensePhrases system to a higher level through refining questions at test time.

Different from phrase-based ODQA models, Lewis et al. (2021); Seonwoo et al. (2022) first process a Wikipedia corpus into a knowledge base (KB) of question-answer (QA) pairs, by using a generator model to create QA pairs. Then, based on the QA-pair KB, they handle ODQA tasks by directly retrieving the most similar question and return the answer of the retrieved question as the final answer to the input question. A large KB with 65 million QA pairs, i.e., Probably Asked Question (PAQ), has been created recently by Lewis et al. (2021). Based on PAQ, RePAQ Lewis et al. (2021) is designed to solve ODQA task by retrieving the most similar QA pairs. SQuID Seonwoo et al. (2022) further improves the performance of RePAQ by adopting two dual-encoder retrievers.

To conclude, in retriever-only ODQA models, the omission of the reading/generating step greatly improves the processing speed of answering questions. But there are also a few limitations for Retriever-Only ODQA models. For example, (1) lower performance on average compared to Retriever-Reader ODQA models since less information is considered during answer inference; (2) high storage requirement in terms of indexes for fine-grained retrieval units such as phrases or QA pairs. For instance, the index size of 65M QA-pairs (220GB) are much larger than that of 21M passages (65GB).

Generator-only ODQA models are normally based on single generators, mainly seq2seq generative language models, like T5 Roberts et al. (2020), GPT Brown et al. (2020) and BART Lewis et al. (2020a). They are pre-trained on large Wikipedia corpora and have stored the main knowledge of the corpus in the model parameters. Thus, they can directly generate the answers based on the internal knowledge and skip the evidence retrieval process.

In this way, they obtains low memory cost and short processing time than two-stage systems such as Retriever-Reader and retriever-generator framework. However, the performances of generator-only ODQA methods are temporarily dissatisfied. For example, on Natural Question dataset, the best generator-only method, GAR_generative Mao et al. (2021), achieves only 38.10% on exact match (EM) score, while retriever-reader method R2-D2_reranker Fajcik et al. (2021) can be up to 55.9% (see in Table 2). Another limitation of generator-only ODQA models is the uncontrollability of the results they generated, since generative models distribute corpus knowledge to the parameters in an inexplicable way and hallucinate realistic-looking answers when it is unsure Roberts et al. (2020). Additionally, real-world knowledge is updated routinely, the huge training cost of the generative language models makes it laborious and impractical to keep them always up-to-date or retrain them frequently. Meanwhile, billions of parameters make them storage-unfriendly and hard to apply on resource-constrained devices Roberts et al. (2020).

In this subsection, we delve into the details of efficiency ODQA models. We category them into three classes regarding to the different means of implementing efficiency, i.e., reducing processing time, reducing memory cost and blazing new trails.

This section concludes the core techniques commonly used in existing ODQA models with respect to improve the efficiency. It can be briefly divided into two categories: data-based and model-based techniques. Data-based techniques mainly focus on the reduction of index, which can be downsized from different hierarchies such as number of corpus passages, feature dimension and storage unit per dimension. Model-based techniques try to reduce the model size while avoiding a significant drop of performance. Model pruning and knowledge distillation are commonly used techniques.

Except the different choice of the original corpus, there are also different partition and segmentation strategies. For example, ORQA Lee et al. (2019) and REALM Guu et al. (2020) segment the corpus documents into 13 million blocks, each of which has 288 tokens. DenseSPI Seo et al. (2019), Dens+Sparc Lee et al. (2020) and DensePhrases Lee et al. (2021b) divide corpus documents into 60 billion phrases, each phrase including 20 tokens. The rest ODQA models segment corpus documents into 21 million passages with the length of 100 tokens, leading to a 65GB index Karpukhin et al. (2020); Lewis et al. (2021); Izacard and Grave (2021); Qu et al. (2021).

A comprehensive introduction is illustrated in Table 1. In general, the index size of the corpus is quite large, and the storage of the index is one of the main challenges for ODQA efficiency.

In terms of latency, training time Mao et al. (2021), indexing time Mao et al. (2021), query time Yamada et al. (2021) and reasoning time are normally considered. The metrics Q/s (questions per second) Seo et al. (2019) and FLOPs (floating point operations) Guan et al. (2022) are popular in measuring the total processing latency, where Q/s is the number of questions one ODQA system can answer per second and FLOPs is the number of floating point operations of the model.

In terms of memory, model parameters size, passage corpus size, index size, training data size are important influence factors of memory cost Yamada et al. (2021). We measure the memory consumption for ODQA models using memory unit (bytes) of corresponding data (corpus, index and model) after loading into memory.

In terms of answering quality, EM (Exact Match accuracy) Chen et al. (2017), F1-score, MRR@k (Mean Reciprocal Rank ) Qu et al. (2021), precision@k, Recall@k and retrieval accuarcy@k Karpukhin et al. (2020) are normally used to measure the quality of the ODQA models. Specifically, EM is the percentage of questions for which the predicted answers can match any one of the reference answers exactly, after string normalization Qu et al. (2021). MRR@k is the mean reciprocal of the rank at which the first relevant passage was retrieved Qu et al. (2021).

In this paper, we adopt metrics on latency, memory cost and answering quality to evaluate ODQA models comprehensively. Specifically, we use Q/s to measure the processing speed, use total memory overhead to evaluate the memory cost and use EM score to estimate the end-to-end answer prediction quality.

Table 2 demonstrates a comprehensive comparison⁴⁴4All discussions in this section are developed on the results that are available. of efficiency-related ODQA models from three aspects: memory cost, proceeding speed and answering quality. Specifically, total memory storage, and detailed model size and index size are listed to show details of memory cost. The number of questions can be answered per second (Q/s) demonstrates the processing speed. EM scores on NQ and TriviaQA datasets indicate the answering quality.

With respect to comparison between different frameworks, we can see two-stage methods (retriever-reader)generally obtain better ODQA performances than one-stage methods (i.e., retriever-only and generator-only). The best end-to-end exact match performance on NQ (55.9%) and TriviaQA (74.8%) datasets are obtained by R2-D2+reranker and GAR_extractive respectively. They are both under retriever-reader framework. The second best ODQA performances on NQ (54.7%) and TriviaQA (72.1%) are obtained by UnitedQA and Fid-large+KD_DPR methods, which are also under the two-stage frameworks.

In terms of total memory cost, i.e., the sum of model size and the index size, generator-only systems keep generally low memory overhead. Except GPT-3, the rest of generator-only systems take less then 50GB memory, and five methods out of the eight are less than 5GB. On the contrary, most retriever-only ODQA models require huge memory, normally greater then 200GB. The method DenSPI needs a 2002.69GB memory cost, which is enormous. Retriever-reader ODQA models have a wide range in terms of the memory cost, from 0.31GB to 363.26GB. Overall speaking, Minimal R&R achieves the smallest memory overhead (0.31GB) while DenSPI keeps the largest one (2002.69GB).

In terms of processing speed, which determines how fast one ODQA system can answer a given question, one-stage methods generally achieve higher processing speed than two-stage methods, especially retriever-only systems. Among the eight retriever-only methods, five of them can process more than 20 questions per second (Q/s) and RePAQ_XL and RePQA_base can answer 800 and 1400 questions per second respectively, which is impressive. For the methods with slow processing speed, Fig-large and RAG-seq from retriever-reader framework are the two slowest systems, which process less than 1 question per second.

To conclude, Fig. 4 gives a visual presentation for comprehensive comparison of efficiency-related ODQA models⁵⁵5Here, we only visualize the ODQA models that have results on all the three aspects: memory, processing speed and EM accuracy.. By using NQ evaluation dataset as an example, it illustrates the detailed model size, index size, EM accuracy and processing speed respectively. From Fig. 4, we can see each framework has its own strengths and weaknesses. Retriever-only systems achieve significantly high processing speed, but cost enormous memory storage. Generator-only systems require the least memory storage. However the main concern of them is the answering quality while majority of these systems’ EM scores are less than 30% on NQ datasets. Two-stage retriever-reader systems relatively behave balanced. They achieve high EM accuracy, and obtain moderate memory cost and processing speed.

The total memory cost depends on the model size and the index size.

Specifically, the 65GB index set of dense passage embedding, developed by DPR Karpukhin et al. (2020), is the most commonly adopted index set. It is adopted by 17 methods as we listed in Table 2. Based on this index set, DrQA and GAR_extractive represent passages into sparse vectors, obtained a much smaller index size (26GB) Chen et al. (2017); Mao et al. (2021). DPR+PQ and RePAQ further reduce the index size utilizing product quantization (PQ) technique and compress the size of index Lewis et al. (2021); Karpukhin et al. (2020). DPR+PQ compress the size of index from 65GB to 2GB. RePAQ compresses the size of index from 220GB to 48GB.

On the other side, BPR Yamada et al. (2021) creates an small index less than 2.1GB by integrating the hash-to-learning technique into DPR other than PQ. It also improves the answering performance from 41.6% to 49% on NQ dataset through replacing the BERT-based reader with the ELECTRA-large reader. Meanwhile, Minimal R&R Yang and Seo (2021) establishes the smallest index less than 0.15 GB through multiple techniques including passage filtering, dimension reducing and PQ, with a price of a significant drop of ODQA performance has been paid.

DenSPI+Sparc Lee et al. (2020) and DensePhrase Lee et al. (2021b) smallen the phrase index by pointer sharing, phrase filtering and PQ. However the phrase index is still larger than 1000GB. DensePhrases further cuts down the index size to 320GB by omitting sparse representations and using encoder SpanBERT-base while a relatively high performance remained. SpanBERT-base represents phrases into 768-dim vectors Joshi et al. (2020), compared with 1024-dim ones used in DenSPI+Sparc. DrBoost Lewis et al. (2022) builds an index under 1GB where a passage is represented with a 190-dim vector through the boosting technique and PQ technique.

Most ODQA models are implemented with PLMs that are less than 2GB. A few of ODQA models keep the total model size more than 3GB to achieve higher performance,like FiD-large+KD_DPR Izacard and Grave (2020), RePAQ+FiD_large Lewis et al. (2021), UnitedQA Cheng et al. (2021) and R2-D2_reranker Fajcik et al. (2021). As they employ either larger or more PLMs to focus on improving the performance.

In terms of latency, i.e., processing speed, most ODQA models answer less than 10 questions per second. Retriever-Only ODQA models brings faster processing speed than other three frameworks. Eliminate the step of long evidence reading makes them time efficient. Compared to phrase-base systems, QA-pair based system RePAQ Lewis et al. (2021) and its variants win the fastest inference speed among the listed ODQA models, up to 1400 Q/s. Generator-Only ODQA models also achieve higher Q/s scores than Retriever-Reader ODQA models, as they no need retrieving evidence from a larger corpus which is time-consuming.

One of the goals for efficient ODQA models is to run on low-power machines. Most open domain question answering approaches prove particularly effective when big amounts of data and ample computing resources are available. However, they are computation-heavy and energy-expensive. How can the ODQA system be deployed in low-power devices with limited compute resources and mobile devices is still very challenging. Such machines are often constrained by battery power. They do not usually come with GPUs. Deep learning in low-power machines is worthwhile and is a growing area of research Goel et al. (2020); Cai et al. (2022). So far there is little work about such ODQA models. To deploy DNNs on small embedded computers, it may need to consider multiple factors together to design an efficient ODQA system, such as energy, computation, memory and so on.

To evaluate efficiency of the ODQA models, it seems to be difficult and there are often multiple factors that need to be traded-off against each other. In existing researches, we use accuracy, EM score, F1-score, precision, recall etc. to evaluate effectiveness, however it is not enough. It is also important to establish what resource, e.g., money, data, memory, time, power consumption, carbon emissions, etc., one attempts to constrain Treviso et al. (2022). For example, using specific hardware such as an electricity meter to measure power consumption, which can provide figures with a high temporal accuracy. External energy costs such as cooling or networking should be covered as well. But they are difficult to measure precisely. Besides the power and energy consumption, we should also pay attention to carbon emission. They are normally computed using the power consumption and the carbon intensity of the marginal energy generation that is used to run the program. Low-energy does not mean low-carbon. Last but not the least, financial impact is another key metric in evaluation. Monetary cost is a resource that one typically prefers to be efficient with. Both fixed and running costs affect ODQA application, depending on how one chooses to execute a model. As hardware configurations and their prices form discrete points on a typically non-linear scale, it is worth paying attention to efficient cost points and fitting to these. Implementing pre-emptible processes that can recover from interruptions also often allows access to much cheaper resources. When calculating or amortizing hardware costs, one should also factor in downtime, maintenance, and configuration. Measuring the total cost of ownership (TCO) provides a more useful metric.

Then we also need to consider the trade-off between efficiency and performance. For some extreme cases, for example, low-power machines, there may be conflict between efficiency and performance. How to develop fair metrics is challenging. Furthermore, what if we have some unified metrics for different systems? We compare different efficient ODQA models from different perspectives. It would be convenient to have some standard unified metrics.

There is little work about model bias in efficient ODQA models. Studies of bias in machine learning have become increasingly important as awareness of how deployed models contribute to inequity grows Blodgett et al. (2020). Previous work in bias shows gender discrimination in word embedding Bolukbasi et al. (2016), coreference resolution Rudinger et al. (2018), and machine translation Stanovsky et al. (2019). Within question answering, prior work has studied differences in accuracy based on gender Gor et al. (2021) and differences in answers based on race and gender Li et al. (2020). However, efficient ODQA models often involve redesigning models, or knowledge distillation or small models. It is unclear about the model bias propagation in the efficient ODQA models. How to reduce bias in those systems, including the bias in open domain passage retrieval and closed domain reading comprehension, is a challenge in the advancement of efficient ODQA models.