Fantastic Questions and Where to Find Them:
FairytaleQA– An Authentic Dataset for Narrative Comprehension

Despite a large number of datasets on reading comprehension, few focus on comprehension of narrative text. Table 2 reviews different narrative-related properties of existing popular QA datasets comparing with our proposed FairytaleQA dataset. NarrativeQA  Kočiskỳ et al. (2018) is one of the representative datasets. It was generated by crowd workers who wrote QA pairs according to summaries of books or movie scripts, while the task takers are supposed to answer these questions based on their reading of original books or movie scripts. As such, this dataset is posited to evaluate a person’s understanding of the underlying narrative, with a significant amount of event-related questions Mou et al. (2021). However, NarrativeQA simply instructed crowd-sourced workers to generate questions as if they were to “test students” without using a detailed annotation protocol. It is questionable whether these workers actually had experiences in testing students, and the lack of protocol may have imposed too little control over the coverage of reading sub-skills.

BookTest Bajgar et al. (2016) is an automatically constructed cloze-style QA dataset based on a collection of narrative texts retrieved from Project Gutenberg. The questions were generated by automatically removing a noun or entity in a sentence that has appeared in the preceding context. While cloze-style tests can be a valid instrument for assessing reading comprehension, their validity depends on the careful selection of words to be removed so that filling them in requires proper comprehension  Gellert and Elbro (2013). It is unlikely that automatically constructed cloze tests would meet such standards.

Another dataset, TellMeWhy Lal et al. (2021), aims to facilitate and assess understanding of causal relationships. This dataset contains “why” questions that are relatively challenging, given that they require additional information not directly provided in the text. However, TellMeWhy only addresses one narrative component type (i.e., causal relationship), whereas FairytaleQA provides seven evaluation components. Moreover, TellMeWhy was built upon ROCStories Mostafazadeh et al. (2016) and thus only examine comprehension on incomplete story exerpts, which may have limited the dataset’s ability to assess macro-level summarization and inference making.

There are several benchmarks derived from sources for education purposes (e.g., exams or curricula). RACE Lai et al. (2017) is a large-scale dataset consisting of comprehension questions from English exams for Chinese middle and high school students. RACE uses a mixture of narrative and informational paragraphs. These two genres require slightly different comprehension skills  Liebfreund (2021), and students perform differently based on what genres of text they read  Denton et al. (2015). Mixing these two together in one dataset without annotating the specific genre of each story/question obscures the ability to offer a precise assessment. Moreover, RACE is in multiple-choice format, and paragraphs are usually shorter. These two characteristics may make the RACE dataset less challenging, and recent models have demonstrated close-to-human performance²²2http://www.qizhexie.com/data/RACE_leaderboard.html.

CLOTH Xie et al. (2017) is a cloze-style dataset also collected from English exams with multiple choice fill-in-the-blank questions. CLOTH can be advantageous for educational QG as each question is labeled with the level of reasoning it involves. However, this dataset shares certain limitations inherent to multiple-choice formats  Klufa (2015).

There are some datasets that are designed for assessing narrative comprehension skills but do not use QA as a form of evaluation. Several datasets, such as NovelChapters Ladhak et al. (2020) and BookSum  Kryściński et al. (2021), evaluate models’ comprehension through summarization tasks. However, there have been debates of whether comprehension can be assessed solely through summarization  Head et al. (1989), as summarization poses a high demand on writing that confounds the reading skills intended to be assessed. Two other recent datasets focus on singular specific elements in narratives. The LiSCU dataset Brahman et al. (2021) targets readers’ understanding of characters, and  Sims et al. (2019) propose a dataset for detecting events in narratives. Given their focus on single narrative elements, these two datasets may not provide a comprehensive evaluation of narrative comprehension.

The narrative texts utilized in the dataset are classic fairytales with clear narrative structures. We gathered the text from the Project Gutenberg website³³3https://www.gutenberg.org/, using “fairytale” as the search term. Due to a large number of fairytales found, we used the most popular stories based on the number of downloads since these stories are presumably of higher quality.

To ensure the readability of the text, we made a small number of minor revisions to some obviously outdated vocabulary (e.g., changing “ere” to “before”) and the unconventional use of punctuation (e.g., changing consecutive semi-colons to periods). For each story, we evaluated the reading difficulty level using the textstat⁴⁴4https://pypi.org/project/textstat/ Python package, primarily based on sentence length, word length, and commonness of words. We excluded stories that are at 10th grade level or above.

These texts were broken down into small sections based on their semantic content by our annotators. The annotators were instructed to split the story into sections of 100-300 words that also contain meaningful content and are separated at natural story breaks. An initial annotator would split the story, and this would be reviewed by a cross-checking annotator. Most of the resulting sections were one natural paragraph of the original text. However, sometimes several paragraphs were combined (usually dialogue); and some exceptionally long paragraphs that contained more than one focal event were divided into multiple sections. On average, there are 15 sections per story, and each section has an average of 150 words (Table 4).

Five annotators were involved in the annotation of QA pairs. All of these annotators have a B.A. degree in education, psychology, or cognitive science and have substantial experience in teaching and reading assessment. These annotators were supervised by three experts in literacy education.

We random split the FairytaleQA dataset into train/val/test splits with a QA ratio of roughly 8:1:1. Table 3 shows the detailed statistics of the FairytaleQA Dataset in train/val/test splits.

Overall, the resulting FairytaleQA dataset contained 10,580 questions from 278 fairytale stories. The description of story and question characteristics is presented in Table 4. In FairytaleQA, action and causal relationship questions are the two most common types, constituting 31.6% and 27.8%, respectively, of all questions. Outcome resolution, character, and feeling types each constitute about 10% of all questions. Setting and prediction questions are about 5% each. Our dataset contains about 75% explicit questions and 25% implicit questions (Table 5 for details).

Question Answering (QA) is a straightforward task that our FairytaleQA dataset can contribute to. We leveraged the commonly-used Rouge-L F1 score for the evaluation of QA performances. For each QA instance, we compared the generated answer with each of the two ground-truth answers and took the higher Rouge-L F1 score.

Here in Table 6, we show the QA performance of a few pretrained SOTA neural-model architectures: BERT Devlin et al. (2018), BART Lewis et al. (2019), and DistilBERTSanh et al. (2019). The quality of answers generated by these pre-trained models is on par with each other. Since BART outperformed other model architectures in the QA task of NarrativeQA Mou et al. (2021), we decided to use BART as the backbone for our fine-tuned models.

We report the performance of fine-tuned BART models with the following settings: BART fine-tuned on NarrativeQA, which is the SOTA model reported in Mou et al. (2021), and another BART model fine-tuned on FairytaleQA. We note that for the QA task, the model that was fine-tuned on FairytaleQA dataset performs much better than the model fine-tuned on NarrativeQA by at least 5%. Even the human performance is underestimated here because it is obtained via cross-estimation between two annotated answers, this result still leaves around 12% on both splits between human performance and the model fine-tuned with FairytaleQA, which demonstrates that the QA task is still a challenging problem for existing works on our FairytaleQA dataset. We leave a full large-scale human study for evaluating the accurate human performance to future work.

In terms of the QG performance on FairytaleQA, the task was to generate questions that correspond to the given answers and the context. This task has important empirical applications that in the future, models may help teachers to create questions in the educational settings.

Similar to the QA task, we fine-tuned a BART model to generate a question conditioned on each human-labeled answer and corresponding story section. The generated question is then evaluated with the corresponding ground-truth question. We used ROUGE-L F1 score as the evaluation metric. For this QG task, we compare the models fine-tuned on NarrativeQA, on FairytaleQA, and on both datasets.

Table 7 displays the QG results. The model fine-tuned on FairytaleQA demonstrated a clear advantage on Rouge-L over the model fine-tuned on NarrativeQA. It is worth noting that the model fine-tuned on both NarrativeQA and FairytaleQA performs worse than the model fine-tuned on FairytaleQA only; we would assume that NarrativeQA dataset introduces noises in question type distribution or semantics during the training process.

Further analysis (Table 8) examined the distribution of generated question types according to the beginning word of a question (wh- words). The questions generated by FairytaleQA more closely resembled the pattern of the ground-truth questions, suggesting that our dataset was able to improve the model’s ability to mimic the education experts’ strategy of asking questions that assess the seven elements of reading comprehension.

This result is further supported by qualitative analysis (as seen in examples in Table 9). Compared to the QG model trained with FairytaleQA, the baseline model trained with NarrativeQA dataset tended to generate vague questions that did not build upon specific contextual evidence within the narratives. These kinds of vague questions may not be suitable in educational settings, as improving students’ skills to find text evidence to support their comprehension is a crucial aspect of reading education. The disparity between the two models might be attributed to how the QA-pairs were constructed in these two datasets: while NarrativeQA was constructed by crowd workers who only read the abstract of the stories, FairytaleQA required annotators to read the complete story before developing QA-pairs. As such, it is not surprising that models trained on FairytaleQA dataset could generate questions that are more closely related to the contextual evidence within the original text. In addition, we also observed that the model trained on NarrativeQA tended to generate questions with seemingly more correct grammar but were factually inaccurate (Table 12 Appendix C).