Evaluation of Question Generation Needs More References

Previous works for QG evaluation measure the quality of a generated question $q^{g}$ in regards to a gold reference question $q^{r}$ as $M(q^{g},q^{r})$, where $M$ denotes a similarity metric that is widely used in QG evaluation such as BLEU and ROUGE-L. However, since these metrics suppose only one gold reference, even an appropriate question can be assigned a low score, namely false positive problem.

To deal with this problem, we propose the multi-reference evaluation, where the candidate question $q^{g}$ is compared with multiple references $Q=\{q^{r}_{0},q^{r}_{1},\dots,q^{r}_{N}\}$:

By comparing more diverse gold questions with existing metrics, we can measure the more realistic ability of QG frameworks. Note that, as our method could adopt any similarity-based metrics, we can better gain useful insights from various metrics showing different characteristics of generated questions.

However, as it is impractical to collect such multiple references with human annotators, we leverage the recent large language models, specifically GPT-3 and ChatGPT, such that replace $Q$ with $\hat{Q}$. Given a reference question $q^{r}_{0}$, we augment it with $N$ questions:

Note that we give a gold question $q^{r}_{0}$ only, rather than the pair of context and question as in Liu et al. (2021). It is because the zero-shot QG ability of LLMs is reportedly risky for educational purposes Wang et al. (2022b). We thus use LLMs as a paraphrase generator, which reportedly works well since there is a high correlation between paraphrasing and training paradigms about LLM Chen et al. (2022).

As GPT-3 is inferior to ChatGPT in the zero-shot settings, here we employ the in-context learning ability of GPT-3, where we give three ChatGPT-paraphrased questions questions as a demonstration for GPT-3 like Appendix A. We will further investigate the correctness of the paraphrased questions in experiments (Section 3.6).

To verify the effectiveness of MRE, we use quiz design dataset Laban et al. (2022) for measuring the correlation between automatic question evaluation and human annotation. The quiz design dataset includes 3,164 human-annotated samples, which consist of context, answer, and automatically generated questions. For each sample, the human annotates whether the question is fluent, able to derive the given answer, and fits the context (1) or not (0).

We define the gold human score of a question as the average of the discrete human annotations in $[0,1]$. Then, we select questions with a human score of 1 as the reference question for the given passage. Finally, for the remaining questions, we measure the Pearson correlation coefficient Freedman et al. (2007) and Spearman’s rank correlation coefficient Zar (2005) between the human score and automatic evaluation scores.

Here, as we aim to enhance the existing QG evaluation metrics with multi-reference evaluation, we choose widely used metrics to apply multi-reference evaluation. We apply multi-reference evaluation to BLEU-4 Papineni et al. (2002), ROUGE-L Lin (2004), METEOR Banerjee and Lavie (2005), BERTScore Zhang et al. (2020), BLEURT Sellam et al. (2020). Also, we add RQUGE Mohammadshahi et al. (2022), which is a reference-free QG evaluation metric, as our baseline. We briefly summarize the metrics used in our experiments as follows:

• •

  BLEU-4 Papineni et al. (2002) is a metric that utilizes n-gram precision to evaluate the similarity between a generated text and a reference text. The metric counts the number of occurrences of unigrams, bigrams, trigrams, and four-grams that match their corresponding counterparts in the reference text.

• •

  ROUGE-L Lin (2004) is a metric that utilizes unigram recall to evaluate the similarity between a generated text and a reference text. The metric counts the length of the longest common subsequence as the numerator rather than the exact number of matches.

• •

  RQUGE Mohammadshahi et al. (2022) first predicts answer span with question answering model then computes score with scorer module from given generated question, gold answer, and context. Since RQUGE does not depend on a reference question for evaluation, we only report the correlation of the original RQUGE.

• •

  METEOR Banerjee and Lavie (2005) measures a score by using a combination of unigram-precision, unigram-recall, and fragmentation measures.

• •

  BERTScore Zhang et al. (2020) utilize contextual embeddings for compute token similarity. We report BERTScore based on roberta-large.

• •

  BLEURT Sellam et al. (2020) is a trained metric using a regression model trained on rating data. It combine expressivity and robustness by pre-training a fully learned metric on large amounts of synthetic data, before fine-tuning it on human ratings.

We implemented the paraphrasing frameworks by using two LLMs: OpenAI GPT-3 API Brown et al. (2020) and ChatGPT Webservice OpenAI (2022). For GPT-3, we set the model as "text-davinci-003" and the temperature as 0.5. For ChatGPT, we utilized the default setting since we cannot control it. Our prompts are described in Appendix A. We made 20 examples by using LLMs. For additional comparisons with the fine-tuned paraphrasing model, we also implemented HRQ-VAE Hosking et al. (2022).

As shown in Table 1, we empirically validate the following observations of the advantages of diversified multi-reference evaluation: 1) Our multi-reference evaluation tends to improve the correlation between human score and evaluation metrics. 2) On LLMs, correlation with the human score is high in the order of ChatGPT (0-shot), GPT-3 (3-shot), and GPT-3 (0-shot) paraphrasing framework. Specifically, GPT-3 (3-shot) and ChatGPT paraphrasing framework considerably improve both Pearson correlation and Spearman correlation for all metrics, while paraphrasing with GPT-3 (0-shot) and HRQ-VAE failed at increasing correlations of some metrics.

Also, the increase in correlation through MRE is related to the performance of the paraphrasing framework. As shown in Table 2, the paraphrase of the reference question is better in the order of ChatGPT, GPT-3 (3-shot), and GPT-3 (0-shot). Considering the effect of MRE is also in the same order, we conjecture that the performance of the paraphrasing framework is also important for the effect of MRE. More details in Table 2 are described in Section 3.6.

The assumption of multi-reference evaluation is that most paraphrased questions with LLMs can serve the meaning like the gold questions. We conduct a human study to validate this assumption. For each of GPT-3 (0-shot), GPT-3 (3-shot), and ChatGPT, we sample 50 pairs of reference questions and paraphrased questions and annotate each pair whether the paraphrased questions have the same meaning and have the same answer compared to reference questions. Specifically, we ask two annotators to evaluate with a binary rating (1 for "same" and 0 for "not same"). As shown in Table 2, 92% and 93% of the questions paraphrased by ChatGPT are evaluated as having the same answer and meaning, respectively. In addition, even when paraphrasing with GPT3 3-shot, it has the same meaning and the same answer at a high rate. We refer to Appendix B for more details about human annotation.

For example in E1 in Table 4, one of the texts in paraphrased references matches the generated question. MRE achieves gains over SRE by 1.00 ($0.00\rightarrow 1.00$) on BLEU-4, and we found a positive effect on all other metrics. In E2, the text that received the highest score among paraphrased references differs from each metric. We can observe that MRE works well by showing that you can choose one of the paraphrased references that are measured to be similar for each metric. Moreover, score increases suggest that MRE leads to positive shifts in the metric scores when the human score is 1 (E1, E2). However, the score to utilize MRE cannot be lower than SRE in any example because MRE takes the maximum score for the true reference and paraphrased references. Thus, if the human score is low, it is important to have a small negative effect. One may ask about the risk of MRE giving a higher score than SRE for wrong questions as in E3. However, we argue that it doesn’t weaken the strength of MRE as the gaps between SRE and MRE for wrong questions are relatively smaller than that for correct questions, which we compared in Table 3.