More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering

Similar works have been done over textual QA benchmarks previously [31][32][9][33][34]. But each of them only covers a few subtasks of textual QA. For example, [31] carefully reviewed the construction methods and the leaderboard information of 22 public MRC datasets; [32] provides detailed statistics and analysis over 54 MRC datasets; [33] mainly focused on ODQA tasks; [9] is a comprehensive survey on English MRC datasets and metrics; [34] is a detailed survey over the Conversational QA tasks. The major difference between our work and previous works is its breadth and granularity. This survey covers three major types of textual QA tasks: MRC, ODQA, and commonsense QA. And we further differentiate the MRC datasets as classical MRC tasks and four novel MRC tasks: conversational QA, multi-hop QA, long-form QA, and cross-language QA.

The rest of the paper is structured as follows:

In section 2, we reviewed 31 MRC benchmark datasets in total. We first introduced the traditional categorization of MRC datasets. Then following this traditional categorization, we reviewed 18 classical MRC datasets. Next, we reviewed 13 novel MRC datasets of 4 different types: conversational QA, multi-hop QA, long-form QA, and cross-language QA. The review of each dataset covers its construction method and statistical information. The information and statistics are summarized in tables.

In section 3, we summarized and reviewed 11 popular textual QA datasets that were used in an Open-Domain Question Answering(ODQA) setting in recent papers. We first summarized the datasets and metrics that were selected to train or test the ODQA systems proposed in recent papers. Then we reviewed datasets, such as Facebook AI’s KILT, that were integrated into a single ODQA benchmark. Multiple feature facets are summarized and compared.

In section 4, we present a spatial type of textual QA task, Commonsense Question Answering. We discussed the uniqueness of this task compared to MRC and ODQA. In total, we reviewed 5 benchmark datasets of this kind. Straightforward examples of each dataset are given along with figures borrowed from the original papers. Information on each dataset is summarized in a table.

In section 5, we summarized the metrics that were used in each textual QA task covered in this paper. A description of and the equations for each metric are provided. The distribution of metrics by answer type is shown in a chart.

In section 6, we summarized the trends in constructing textural QA tasks and propose future directions of textual QA benchmark research.

Finally, in section 7, we conclude our survey.

Next, we will introduce classical MRC benchmark datasets by annotation style.

Next, we will introduce MRC datasets in four novel application scenarios: conversation question answering, multi-hop question answering, long-form question answering, and cross-language question answering. We refer to these tasks as novel MRC.

CuratedTREC[54]) is extended from the TREC QA corpus[2] which was created for ODQA. It contains 2180 question-answer pairs from TREC 1999-2002. In DrQA[22], using English Wikipedia as the external knowledge source, CuratedTREC is used as an ODQA benchmark. In this format, CuratedTREC is used as a benchmark in a wide range of ODQA systems, e,g., R3[24], ORQA[23], REALM[56], DPR[57], RAG[58], T5[59], etc.

WebQuestions[60] was originally designed for Knowledge base QA. It has 4K questions from the Google Suggest API and crowdsourced answers restricted to Freebase entities. In DrQA[22], it was curated and used as an ODQA benchmark by converting the answers from Freebase IDs to entity names and paring them with English Wikipedia as an external knowledge source. Using this format, WebQuestions is used as a benchmark by ODQA systems, e,g., R3[24], ORQA[23], REALM[56], DPR[57], RAG[58], GPT-3[6], T5[59], etc.

WikiMovies[61] contains 96K movie-related question-answer pairs curated from OMDb and the MovieLens database. All the questions in this dataset can be answered using a subset of Wikipedia pages. WikiMovies was used as an ODQA benchmark by [22] pairing it with English Wikipedia. This format is also used in R3[24].

OpenSQuAD is curated from the SQuAD 1.1[43], a popular Stanford MRC dataset annotated by crowd workers using Wikipedia passages. It contains 87K examples for training and 10K for development. A large test set is sequestered by the authors for the leaderboard evaluation only. Discarding context passages and using English Wikipedia, the question-answer pairs in the development set are used to evaluate ODQA systems. This format is used in the following works: R3[24], DenSPI[62], Multi-passage BERT[63], ORQA[23], DPR[57], Fusion-in-Decoder[64], etc. However, [57] argues that SQuAD is not ideal for ODQA because “many questions lack context in the absence of the provided paragraphs”. Following the publication of [57] fewer works make use of OpenSQuAD.

Qusar-T[55] is a large-scale dataset designed for ODQA. It contains 43K trivia question-answer pairs acquired from various internet sources. Paired with English Wikipedia as the external source of knowledge, it is used as one of the benchmark datasets in recent ODQA works like R3[24] and Multi-passage BERT[63].

TriviaQA was proposed by [47] as an MRC dataset containing over 650K question-answer-evidence triples whith 96K unique question-answer pairs collected from trivia websites and on average six evidence documents per question. In ORQA[23], paired with the 2018-12-20 dump of English Wikipedia as an external knowledge source, the unfiltered version of TriviaQA is used as an ODQA benchmark with the evidence documents removed. This setting(denoted as OpenTriviaQA) is also used as a ODQA benchmark in the works of Multi-passage BERT[63], DPR[57] , RAG[58], GPT-3[6], T5[59], Fusion-in-Decoder[64], etc.

Natural Questions is Google’s MRC dataset. Its questions are gathered from user query logs of the Google search engine[48]. Long-form and short-form answers are annotated by crowd workers as a span of words based on the top-5 Wikipedia documents returned by the search engine. It contains 307K (question, Wikipedia page, long answer, short answer) quadruples in the training set and 8K in the development set. ORQA[23] make Natural Questions usable as an ODQA benchmark by only keeping the questions with short answers (with 5 tokens at most), discarding the evidence Wikipedia page, and instead using English Wikipedia as the only external knowledge source. This format (denoted as OpenNaturalQuestions) is also followed by Multi-passage BERT[63], REALM[56], DPR[57], RAG[58], GPT-3[6], T5[59], Fusion-in-Decoder[64], etc.

KILT is a benchmark for Knowledge Intensive Language Tasks proposed by Facebook AI Research[25]. It aims to break the boundaries between datasets for knowledge-intensive language tasks, including Slot Filling, ODQA, Dialogue, Fact Checking, and Entity Linking by mapping them to a unified snapshot of an external knowledge source (2019-08-01 dump of English Wikipedia). Another important feature of KILT is that the format of the KILT benchmark is model-agnostic, which means it supports any system that takes textual input and produces textual output. This feature dramatically reduces the data preprocessing workload when testing across multiple datasets. KILT also tries to provide as much provenance information as possible for all its instances through crowdsourcing. This helps people evaluate whether their system can provide correct evidence for their prediction. Next, we will describe the four ODQA datasets included in KILT, which are KILT-TriviaQA, KILT-NaturalQuestions, KILT-HotpotQA, and KILT-ELI5.

SWAG[26] is a large-scale cloze-style dataset designed for commonsense QA and proposed by researchers from the University of Washington. The task is formulated as, given the first sentence and the first noun of the second sentence, a system needs to select the correct ending of the second sentence from a collection of artificially generated endings. The artificial endings are generated by an RNN-based language model (LM) and then selected by an adversarial model to make sure those artificial endings in the pool cannot be easily identified. Finally, human annotators are hired to check those ending to ensure agreement. The corpus contains 113K multiple choice questions derived from consecutive video captions from two sources, the ActivityNet captions[65][66] and the Large Scale Movie Description Challenge[67]. Figure 7, borrowed from [26], gives examples of SWAG. As it shows, no context information is provided for each cloze-style question forcing systems to learn the underlying commonsense logic of humans solely through QA pairs. System performance is evaluated by accuracy.

CODAH[27] is a more challenging evaluation dataset extended from SWAG, proposed by researchers from Northwestern University. It contains 2.8K multi-choice sentence completion questions. Annotators are educated and rewarded to submit questions that adversarially target the weaknesses of a pre-trained model.

CommonsenseQA[28] is a commonsense QA dataset proposed by researchers from the AI2 for question answering with prior knowledge. Crowd workers created 12K multi-choice questions from ConceptNet[68] to capture common sense beyond associations. The questions in this dataset are supposed to be answerable using only commonsense knowledge. Figure 8, borrowed from [28], gives examples from the CommonsenseQA dataset and shows how they are created based on a ConceptNet subgraph.

ReCoRD[30] is a large-scale Commonsense QA dataset proposed by researchers from Johns Hopkins University and Microsoft Research. It is presented in the form of a cloze-style MRC dataset that contains over 120K instances. Its creators claim that most of the questions require deep commonsense reasoning across the context passages to answer. Figure 9, borrowed from [30], illustrates an example of ReCoRD, where the ’X’ in the query indicates a missing named entity that a system needs to infer from the context passage. The context passages are collected from CNN and Daily mail news articles. Specifically, the first few paragraphs of a news article together with the editor-provided highlights are sampled as the context passages because they usually summarize the article. The questions are constructed from sentences in the rest of the article by replacing a named entity with a placeholder, and usng the named entity as the correct answer. Five carefully designed criteria are applied when selecting the sentences to contract questions. See[30] for more details. EM and F1 measures are used for evaluation.

CosmosQA[29] is a large-scale Commonsense QA dataset proposed by researchers from the UIUC and AI2. It can be also interpreted as a multi-choice style MRC dataset that includes 33K questions that require commonsense reasoning over the associated context passages. Figure 10, borrowed from [29], gives two examples from the CosmosQA dataset. In CosmosQA, most of the correct answers are not explicitly presented in the context passages but can be inferred based on the content of the context. Such a configuration challenges textual QA systems MRC ability and their commonsense reasoning ability. The context passages are collected from a corpus of personal narratives from the Spinn3r Blog Dataset[69]. Based on each context passage, up to two multi-choice questions are created by crowd workers asked to craft questions from four categories: causes of events, effects of events, facts about entities, and counterfactuals. The task is evaluated by accuracy.

Accuracy represents the percentage of questions that a system answered correctly. Let M denote the number of questions a system answered correctly and N denote the total number of questions in the evaluation dataset. Accuracy is defined as:

Exact Match (EM) measures the percentage of predictions that match any reference answers exactly at the token level (in some datasets, each question may have multiple reference answers). In other words, a prediction is counted as correct only when it exactly matches any one of the reference answers to the given question. Let M denote the number of correct predictions, N denote the total number of questions in the evaluation dataset. EM is defined in Eq. 2:

It is worth noting that for span extraction tasks, EM is the same as Accuracy.

The F1 score is the harmonic mean of the precision and the recall. In textual QA, the precision measures the ratio of the number of tokens in a prediction that overlap with the correct answer to the total number of tokens in the prediction. The recall measures the ratio of the number of tokens in a correct answer that have been covered by a prediction to the total number of tokens in the correct answer. Equations of token-level precision and recall for a single prediction are shown in Eq. 3 and Eq. 4 respectively.

where TP, FP, FN denote the token-level true positive, false positive, and false negative respectively. The equation of the F1 score for a single prediction is given in Eq. 5:

When a question has multiple reference answers, the highest F1 score is taken. The average F1 score of all predictions is the final F1 score of the system.

ROUGE[70] is the abbreviation for Recall-Oriented Understudy for Gisting Evaluation. It consists of a set of metrics that were originally proposed to evaluate automatic text summarization. Because it works by comparing the automatically generated text with the human-generated reference text, it is now widely used by textual QA benchmarks especially by those with free-form reference answers.

The most popular ROUGE metrics in textual QA are ROUGE-N and ROUGE-L, which represent a comparison of texts at different granularities. Specifically, ROUGE-N measures the ratio of the number of overlapped n-grams between the generated text and the reference text to the total n-grams in the reference text and is defined in Eq. 6. Similarly, ROUGE-L measures the longest matching sequence of words using the longest common subsequence (LCS) which can automatically include the longest in-sequence n-grams.

BLEU[71] stands for Bilingual Evaluation Understudy, which was originally proposed to evaluate machine translations. Similar to ROUGE, BLEU compares the lexical features between two sequences to words, and it is also widely used for generative textual QA evaluations. Specifically, BLEU is defined as follows:

where $p_{n}$ is called n-gram precision in which $m_{cand}^{gram_{n}}$ is the number of the n-gram in the prediction(candidate) matching the reference answer; $C$ is the candidate answer set; $m_{ref}^{gram_{n}}$ is the number of the n-gram in the reference answer; $w_{t}^{gram_{n}}$ is the total number of the n-grams in the prediction.

Meteor[72] is another automatic metric that was originally proposed for machine translation. It claims to have a better correlation with human judgment. Similar to BLEU, Meteor also consists of two parts: a weighted F score and a penalty factor, which is given by the following equations:

with

where $ch$ is the total number of matching chunks and m is the total number of matched uniforms between the prediction and the reference. The number of chunks decreases when the matches are contiguous which will reduce the penalty. The precision and recall are given as $m/c$ and $m/r$, in which $c$ and $r$ are the lengths of the candidate and reference respectively. Finally, the parameters $\alpha$, $\beta$, and $\gamma$ are tuned to maximize the correlation with human judgments.

HEQ stands for Human Equivalence Score, which was proposed along with the conversational QA benchmark dataset, QuAC[11]. HEQ creators claim that the F1 score can be misleading for questions with multiple answers. HEQ measures the percentage of examples for which the F1 score of predictions exceeds or matches the human F1 score. HEQ has two variants: HEQ-Q (the percentage of questions for which this is true) and HEQ-D (the percentage of dialogues for which this is true for every question in the dialog).

IoU stands for Intersection over Union. It was used with the long-form QA benchmark dataset, NLQuAD[18]. NLQuAD creators claim that F1 score and ROUGE-N have limited use for evaluation of long answers. IoU measures position-sensitive overlap between two spans which is defined as follows:

where $p$ and $t$ are the predicted and target contiguous intervals over the context document, containing the positions of the tokens.

For the question collection methods, we see a trend of collecting real-life user questions from either user logs of search engines or QA websites like Reddit, Stack overflow, etc., instead of constructing cloze-style queries automatically or crowdsourcing questions based on a context passage/document.

For answer collection methods, the free-form answers and span-style answers are the two most popular answer type choices. The free-form answers are usually directly derived from QA websites, while span-style answers are mostly collected through crowd workers, which is more expensive. In addition, we see a tendency towards annotating long answers, thus providing more supportive information to questions. Moreover, in some of the latest datasets, supportive sentences in the context documents are annotated for each answer in order to help increase the interpretability of the system by requiring it not only to return the answer but also these evidence sentences.

For the selection of context of documents or external knowledge sources, we find that Wikipedia is the most popular choice, especially for ODQA tasks. A few MRC datasets choose other domains like news, movies, HealthCare, Twitter, etc.

We believe providing evidence sentences for each answer is a good direction for future textual QA datasets. Besides evidence sentences, we hope to see the use of other forms of information that can support answer selection, e.g., pictures and hyperlinks in the document. This can be of benefit to future research beyond single-model textual QA.

We recommend that instead of continuing to propose new datasets based on Wikipedia articles, researchers should explore other information domains, such as healthcare or social media, each of which has different data quality requirements. For example, in the healthcare domain, people expect high precision, while in the social media domain, people may be more tolerant of misinformation. Different challenges may be constructed based on these features.

For metrics, as we discussed in section 5, different textual QA tasks have various flavors of evaluation measures. But do we really need all of them? We think evaluating and unifying these metrics for textual QA would be a meaningful future work. This not only can reduce the burden when people need to evaluate their models over multiple benchmark datasets but also help people have a better understanding of key features when comparing two answers in natural language. This in turn can help coach the development of textual QA systems. We also believe that more metrics should be evaluated for textual QA tasks. We make this suggestion because we noticed that recently there are many new metrics proposed for text generation tasks, such as the BERT-score[73] and the BLEURT[74]. Both claim to have better alignment with human evaluation than lexical feature-based metrics like ROUGE, BLEU, and Meteor that are widely used in textual QA tasks currently.

Finally, we find that most of the benchmark datasets only have one reference answer for each question, and all of them only require a system to return one answer to each question. In fact, in realistic scenarios, many questions may have multiple answers based on different supportive knowledge. For example, in social media, people may be interested in an event that may not have a certain answer but only a few popular rumors available. Accordingly, new metrics for multiple-answer QA need to be developed.