DHP Benchmark: Are LLMs Good NLG Evaluators?

Large Language Models (LLMs) play a crucial role in the field of Natural Language Generation (NLG), advanced wide real-world applications including education Latif et al. (2023), healthcare Yuan et al. (2023), business Teubner et al. (2023), etc. The strong capabilities of LLMs allow them not only to serve as text generators but also increasingly as powerful evaluators of text quality Chiang and Lee (2023); Liu et al. (2023a); Li et al. (2024). Their role as evaluators is crucial for advancements in various applications, such as summarization, story completion, question answering, and translation Li et al. (2024); Wang et al. (2023a). LLMs are expected to serve as NLG evaluators, providing reasonable quality scores based on different quality metrics with specially designed evaluation prompts.

Despite the growing performance of LLMs in evaluation tasks, a significant gap remains in fully comprehending their capabilities in evaluating NLG quality. The question, Are LLMs good NLG evaluators? remains challenging for two main reasons illustrated in Figure 1:

To address these challenges, we introduce a novel DHP benchmarking framework – Discernment of Hierarchical Perturbation – for quantitatively measuring the evaluation capabilities of LLMs without the need for human annotations. We propose the concept of discernment scores, systematically derived from hierarchically perturbed text data and statistical tests. We perturb the reference data using various hierarchical methods, then compare the differences in LLM evaluation scores using the Wilcoxon Signed-Rank Test Wilcoxon (1945). In order to get overall capability results, we introduce the approach of harmonic mean $p$-values and expert weights to combine the results of multiple metrics. The final $p$-value derived can be transformed to Discernment Scores to measure the NLG evaluation capabilities of LLMs. This method allows for a more rigorous and comprehensive evaluation of LLM performance, independent of the response styles of the tested models.

Our study re-establishes six evaluation datasets that encompass four key NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Each dataset is perturbed and leveraged to challenge the LLMs’ evaluative capabilities in distinct ways, providing a robust basis for benchmarking. The datasets include a range of text perturbations, from minor character problems to significant sentence alterations, allowing us to test the potential discernment limits of the LLMs.

Our comprehensive benchmarking, using newly defined quantitative discernment scores, analyzes five major LLM series. This approach uncovers critical insights into their capabilities as NLG evaluators and provides a detailed understanding of their performance. This benchmark reveals important trends and patterns in the capabilities of LLMs, highlighting areas where they excel and where they may fall short.

The DHP benchmark aims to fill the existing gaps by providing a quantitative framework for assessing LLMs’ evaluation capabilities and emphasizing the necessity of considering multiple metrics for accurate and reliable evaluations. We summarize our contributions as follows.

1. (1)

   Design the DHP benchmarking framework: quantitative discernment scores for LLMs as NLG evaluators based on hierarchical perturbation, eliminating the need for human annotations.

2. (2)

   Re-establish six evaluation datasets for four NLG tasks to evaluate the discernment of LLMs.

3. (3)

   Benchmark five series of LLMs to analyze their capabilities of NLG evaluation.

Recent advancements highlight the significant potential of utilizing LLMs as evaluators for a variety of natural language processing (NLP) tasks. Extensive empirical evidence supports this viewpoint, as demonstrated by studies Liu et al. (2023a); Chiang and Lee (2023); Hu et al. (2024); Desmond et al. (2024); Wang et al. (2023a), which assert that the evaluation behaviors of pretrained LLM-based evaluators are well-aligned with those of human preference Liu et al. (2023b) and eliminate the prompt bias Liusie et al. (2024). Another line of work Huang et al. (2024); Zhu et al. (2023); Wang et al. (2023b) finetunes LLMs with specific downstream task datasets to increase the capability as the judges. Despite the great assessment performance of a single LLM, advanced studies involve multi LLM agents Chan et al. (2023); Zhang et al. (2023); Li et al. (2023) or human experts Gao et al. (2024); Li et al. (2024) to further increase the judging capability. However, humans often prefer to remain neutral Leroi et al. (2020), which means a perfect alignment between LLM-based evaluators and human evaluators may still lead to bias and inaccurate judgments.

While using LLM-based evaluators offers a scalable and effective method for approximating human preferences, the significant influence of individual human biases raises concerns about their discernment in assessing LLM-based evaluations. In our work, we focus on developing comprehensive approaches to observe and address this phenomenon.

Previous studies focus on the alignment between human and LLM evaluators, using correlation metrics to gauge the LLMs’ performance in NLG evaluation tasks Liu et al. (2023a); Chiang and Lee (2023). However, these studies often overlook an important variable of evaluators: Response Styles which refer to a respondent’s consistent manner of answering survey questions, regardless of the content Van Vaerenbergh and Thomas (2013). Despite similar levels of professionalism, annotators may assign different scores to the same questionnaire due to differences in age, gender, personality, cultural background, and ethnic group Van Vaerenbergh and Thomas (2013); Hui and Triandis (1989); Kieruj and Moors (2013). Similarly, LLMs, trained on diverse datasets, may also exhibit biases in their responses Salecha et al. (2024). This discrepancy casts doubt on the previous methods used to compare human and LLM scores. Since quality-based scoring often relies heavily on a few experts’ annotations, the final alignment scores tend to favor models that share similar response styles with these specific experts.

We illustrate this with an example of the response styles of five LLMs tasked with annotating quality scores for human reference data from the SummEval dataset Fabbri et al. (2021). We averaged the scores across four metrics for each data point and plotted both the Pearson correlation coefficient ($\rho$) and the average score distributions of the five models. After perturbing the original data by replacing some named entities with fictional ones in the summaries (Fictional Named Entities in Table 1), we repeated the quality evaluation. As shown in Figure 2, all models detected the changes and adjusted their scores accordingly, though their scoring distributions varied significantly. For instance, Llama3 Meta , Mistral Jiang et al. (2023), and Qwen Bai et al. (2023) models assign higher scores to the original data and moderate scores to the perturbed data. In contrast, GPT4-Turbo OpenAI (2023) and Vicuna Chiang et al. (2023) models tend to give moderate scores to the original data and much lower scores to the perturbed data. The variance in the response distributions indicates the presence of bias that can significantly affect alignment ($\rho$), illustrating that alignment is not a direct or credible metric for assessing the ability of LLMs as NLG evaluators. It is crucial to develop a new metric and measurement for evaluation that is not influenced by the evaluators’ biased response styles, ensuring a more accurate and fair assessment of LLM capabilities.

We propose our DHP framework: Discernment of Hierarchical Perturbation. Previous studies overlook the essence of NLG evaluation, i.e., the content-oriented scoring Novikova et al. (2018). In other words, content that is accurate, fluent, and consistent should receive higher scores than content that is inaccurate, disfluent, and inconsistent. Qualified annotators should be able to recognize inappropriate content without additional references and then assign scores, even though the absolute scores may still reflect their biased response styles. The fundamental principle of our assessment is that a qualified LLM evaluator should be able to independently identify issues in perturbed data (which contains some quality issues) and assign relatively lower scores compared to the original reference data during two separate evaluations. This approach does not rely on human scores, thus eliminating the influence of human response styles.

The overall framework is shown in Figure 3. First, for a specific NLG task, we employ a hierarchical perturbation pipeline to transform high-quality reference data into various forms of lower-quality data. Subsequently, an LLM evaluates both the original and perturbed texts respectively using predefined metrics, generating several sets of rating scores. We then conduct a statistical analysis of these scores. For each pair of scores, original and perturbed, we apply the Wilcoxon Signed-Rank Test to determine the differences in their distributions, achieving this with a confidence level expressed as a $p$-value. This test specifically assesses differences in pairwise scores without focusing on absolute values, thereby minimizing the impact of models’ response styles. Following this, we combine the $p$-values from different metrics, incorporating Expert Weights ($EW$) to tailor the aggregated $p$-values to the specific metrics of the corresponding perturbation methods. These combined $p$-values are then transformed into discernment scores, which serve as a direct measure for assessing and comparing the NLG evaluation capabilities of LLMs for this particular task.

We evaluate five series of LLMs with varying parameter sizes: the GPT-series Wang et al. (2023a), which includes GPT3.5-Turbo and GPT4-Turbo; the Llama3-series Meta ; the Vicuna1.5 series Chiang et al. (2023); Mistral-7B Jiang et al. (2023); and the Qwen-series Bai et al. (2023).

The LLMs are evaluated across four NLG tasks using six re-established public datasets: for Summarization, we use SummEval Fabbri et al. (2021) (news articles) and SumPubMed Gupta et al. (2020) (scientific articles); for Story Completion, we select data from Story Cloze Test dataset Mostafazadeh et al. (2017); for Question Answering, we utilize the data and modify the quality metric based on the Answer Equivalence dataset Bulian et al. (2022); and for Translation, we leverage WMT-22 German-to-English and Chinese-to-English general (news) translation subsets Kocmi et al. (2022). To ensure comparability, we select $N=100$ datapoints from each dataset. The quality metrics and perturbation methods are detailed in Table 1.

We present our DHP benchmarking results in Figure 4. By examining the discernment scores achieved by these models, we can gain insights into their competence as NLG evaluators.

We introduce the DHP benchmark to assess the discernment capabilities of LLMs as evaluators across various NLG tasks. Our approach not only provides benchmarking results for LLMs but also establishes a robust framework to evaluate how effectively LLMs can identify quality issues, thus serving as competent NLG evaluators. While most models generally perform well, their performance is significantly influenced by factors such as model size, task type, and dataset complexity. By pinpointing particular weaknesses of LLMs in evaluating NLG tasks, this benchmark aids researchers in improving LLM performance going forward.

The limitation of our DHP benchmark is that it generates discernment scores specific to each NLG dataset. A comprehensive assessment of the general evaluation capabilities of LLMs across all NLG tasks remains an open challenge. Additionally, the benchmark’s current focus on English-related text limits its generalizability across different languages and cultural contexts, potentially affecting its reliability in the general multilingual NLG tasks.