Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model

The intuition is that the generation probability of a piece of text will increase when providing more related and consistent information. On the contrary, the generation probability will drop when conditioned on inconsistent information. Accordingly, we considered three different probability changes in two categories as follows.

Changes with Prior Probability: The prior probability of $\boldsymbol{Y}$ can be estimated by the foundation model $\rm LM(\cdot)$:

$$\boldsymbol{p}_{\boldsymbol{Y}}^{{\rm lm}}={\rm LM}(\boldsymbol{Y})=\{p_{y_{i}}^{{\rm lm}}\}|_{i=1}^{m}$$

(1)

and the sequence-to-sequence probability of $\boldsymbol{Y}$ given $\boldsymbol{X}$ is:

$$\boldsymbol{p}_{\boldsymbol{Y}}^{{\rm s2s}}={\rm LM}(\boldsymbol{Y}|\boldsymbol{X})=\{p_{y_{i}}^{{\rm s2s}}\}|_{i=1}^{m}$$

(2)

If $\boldsymbol{Y}$ is a faithful summary, the sequence-to-sequence probability $\boldsymbol{p}_{\boldsymbol{Y}}^{{\rm s2s}}$ should be larger than the prior probability $\boldsymbol{p}_{\boldsymbol{Y}}^{{\rm lm}}$ as more information consistent to $\boldsymbol{Y}$ is given by conditioning on $\boldsymbol{X}$. Therefore, a faithfulness measurement can be defined as:

$$\mit\Delta_{\boldsymbol{p}_{\boldsymbol{Y}}}^{{\rm prior}}=\frac{1}{m}\sum\nolimits_{i=1}^{m}p_{y_{i}}^{{\rm s2s}}-p_{y_{i}}^{{\rm lm}}$$

(3)

From another point of view, we expect that the generation of $\boldsymbol{Y}$ highly relies on $\boldsymbol{X}$, instead of parametric knowledge stored in $\rm LM(\cdot)$ which is a main resource of hallucinations Ji et al. (2023).

Similarly, a faithful $\boldsymbol{Y}$ can support the contents in $\boldsymbol{X}$. Thus, the differences between the sequence-to-sequence probability of $\boldsymbol{X}$ given $\boldsymbol{Y}$ and the prior probability of $\boldsymbol{X}$ is another reasonable measurement:

		$\displaystyle\boldsymbol{p}_{\boldsymbol{X}}^{{\rm lm}}={\rm LM}({\boldsymbol{X}})=\{p_{x_{i}}^{\rm lm}\}|_{i=1}^{n}$		(4)
		$\displaystyle\boldsymbol{p}_{\boldsymbol{X}}^{\rm s2s}={\rm LM}(\boldsymbol{X}|\boldsymbol{Y})=\{p_{x_{i}}^{\rm s2s}\}|_{i=1}^{n}$
		$\displaystyle{\mit\Delta}_{\boldsymbol{p}_{\boldsymbol{X}}}^{\rm prior}=\frac{1}{n}\sum\nolimits_{i=1}^{n}p_{x_{i}}^{\rm s2s}-p_{x_{i}}^{\rm lm}$

Changes with Conditional Probability: Instead of comparing sequence-to-sequence generation probabilities with prior probabilities, another way is to add more information $\boldsymbol{P}$ besides the input document $\boldsymbol{X}$, leading to an influence on the generation of $\boldsymbol{Y}$. Following She et al. (2023), we simply set $\boldsymbol{P}=\boldsymbol{Y}$. In this way, if $\boldsymbol{Y}$ is inconsistent with $\boldsymbol{X}$, prefixing $\boldsymbol{P}$ will provide additional evidences for generating $\boldsymbol{Y}$, leading to a larger $\boldsymbol{p}_{\boldsymbol{Y}}^{{\rm pref}}$ compared with $\boldsymbol{p}_{\boldsymbol{Y}}^{\rm s2s}$, while a consistent one won’t. Mathematically, the third measurement is:

		$\displaystyle{\boldsymbol{p}}_{\boldsymbol{Y}}^{{\rm pref}}={\rm LM}({\boldsymbol{Y}}|{\boldsymbol{P}},{\boldsymbol{X}})=\{p_{y_{i}}^{{\rm pref}}\}|_{i=1}^{m}$		(5)
		$\displaystyle{\mit\Delta}_{{\boldsymbol{p}}_{\boldsymbol{Y}}}^{{\rm cond}}=\frac{1}{m}\sum\nolimits_{i=1}^{m}p_{y_{i}}^{{\rm s2s}}-p_{y_{i}}^{{\rm pref}}$

We didn’t consider $\boldsymbol{X}$ and $\boldsymbol{Y}$ reversely here. The main reason is that inputting the sequence $[\boldsymbol{P}=\boldsymbol{X},\boldsymbol{Y},\boldsymbol{X}]$ to ${\rm LM}(\cdot)$ is much more costly and may exceed the max sequence length of most models since $\boldsymbol{X}$ is much longer than $\boldsymbol{Y}$, i.e., $n\gg m$.

Goyal et al. (2022) found that high-loss tokens generally correspond to unfaithful contents during training a summarization model. Inspired by this finding and the success of the loss truncation training algorithms Kang and Hashimoto (2020), we think that more attention should be paid to such high-loss (or low-probability) tokens when calculating the faithfulness scores. So, instead of simply averaging the probability changes to get the final score for an $(\boldsymbol{X},\boldsymbol{Y})$ pair, we adopt two operations. First, we take the logarithm of the probabilities before subtraction, which will magnify changes on the low-probability tokens. Second, we re-weight each token based on $\boldsymbol{p}_{\boldsymbol{Y}}^{{\rm s2s}}$ and $\boldsymbol{p}_{\boldsymbol{X}}^{{\rm s2s}}$ correspondingly. We get:

		$\displaystyle{\mit\Delta}_{\boldsymbol{Y}}^{{\rm prior}}=\frac{1}{m}\sum\nolimits_{i=1}^{m}{\rm e}^{p_{y_{i}}^{s2s}}(\log{p_{y_{i}}^{s2s}}-\log{p_{y_{i}}^{lm}})$		(6)
		$\displaystyle{\mit\Delta}_{{\boldsymbol{X}}}^{{\rm prior}}=\frac{1}{n}\sum\nolimits_{i=1}^{n}{\rm e}^{p_{x_{i}}^{s2s}}(\log{p_{x_{i}}^{s2s}}-\log{p_{x_{i}}^{lm}})$
		$\displaystyle{\mit\Delta}_{{\boldsymbol{Y}}}^{{\rm cond}}=\frac{1}{m}\sum\nolimits_{i=1}^{m}{\rm e}^{p_{y_{i}}^{s2s}}(\log{p_{y_{i}}^{s2s}}-\log{p_{y_{i}}^{pref}})$

Finally, FFLM is a combination of these metrics:

$${\rm FFLM}=\alpha{\mit\Delta}_{\boldsymbol{Y}}^{{\rm prior}}+\beta{\mit\Delta}_{\boldsymbol{X}}^{{\rm prior}}+\delta{\mit\Delta}_{\boldsymbol{Y}}^{{\rm cond}}$$

(7)

where $\alpha$, $\beta$, and $\delta$ are weighting parameters in the range of 0 to 1 and $\alpha+\beta+\delta=1$. These three weights can be tuned on a validation set, or set manually as hyper-parameters.

Inconsistency detection regards the faithfulness evaluation as a binary classification problem. In other words, human annotators or automatic metrics only need to recognize whether the summary is faithful to the document or not.

Datasets: The SUMMAC Benchmark Laban et al. (2022) is a benchmark consisting of six summarization evaluation datasets, including CoGenSumm Falke et al. (2019), SummEval Fabbri et al. (2021), FRANK Pagnoni et al. (2021), Polytope Huang et al. (2020), FactCC Kryściński et al. (2020) and XSumfaith Maynez et al. (2020). It standardized these datasets by changing their original labels into a binary label and split each dataset into a validation set and a test set. Most of the original datasets are labeled by three or more annotators, except Polytope and FactCC.

Evaluation Metric: Balanced accuracy Brodersen et al. (2010) is adopted as the primary evaluation metric, which requires binary labels for computation. For approaches with continuous scores, a threshold can be selected via the validation set.

Baselines: We borrowed the baselines from Laban et al. (2022)’s work, including linguistic feature-based metrics NER-Overlap Laban et al. (2021) and DAE Goyal and Durrett (2020), NLI-based metric MNLI-doc Kryściński et al. (2020) and SUMMAC_ZS Laban et al. (2022), QA-based metrics FEQA Durmus et al. (2020) and QuestEval Scialom et al. (2021), prompting with ChatGPT OpenAI (2022) ³³3As Chen et al. (2023) shows that faithfulness evaluation is less reasoning-intensive and chain-of-though Wei et al. (2023) prompting even hurts performances, we only compared with ChatGPT using a vanilla prompt., and two weakly-supervised baselines FactCC-CLS Kryściński et al. (2020) and SUMMAC_CONV Laban et al. (2022). Besides, we implemented the language modeling-based metric BARTScore Yuan et al. (2021) and metrics based on probability changes include CoP She et al. (2023) and HaRiM Son et al. (2022). These three metrics were suggested to use the CNN/DM Nallapati et al. (2016) fine-tuned BART model ⁴⁴4https://huggingface.co/facebook/bart-large-cnn for calculation. We also improved the latter two metrics with our proposal by calculating with a foundation language model, LLaMa, for comparisons.

Faithfulness rating defines the evaluation as a Likert scale coring problem. Annotators or metrics score each summary according to its faithfulness. Generally, the higher, the more faithful.

Datasets: Following Son et al. (2022), we experimented on five different datasets: FRANKCNN and FRANKXSUM from Pagnoni et al. (2021), QAGSCNN and QAGSXSum from Wang et al. (2020), and SummEval Fabbri et al. (2021). For the first four datasets, human judgments were originally done on the sentence level. The faithfulness rating of the whole summary is collected by doing majority voting on each summary sentence among annotators and averaging among sentences. SummEval contains human scores in the range of 1 to 5 in the aspect of consistency. More details are in Table 1.

Evaluation Metrics: Pearson($\gamma$), Spearman($\rho$), and Kendall($\tau$) correlation coefficients are used to measure the alignments between faithfulness ratings annotated by annotators and automatic metrics. The correlations are the higher the better. We consider the summary-level correlations for all datasets. Besides, system-level correlations are calculated on SummEval which contains annotations for 16 extractive or abstractive summarization models.

Baselines: Rouge-2 F1 Lin (2004), Meteor Banerjee and Lavie (2005), BLEU Papineni et al. (2002) and BERTScore F1 Zhang et al. (2019a) are widely-accepted summarization evaluation metrics. We report their best results in Son et al. (2022) by calculating between the summary and the source document. QAGS Wang et al. (2020) is another QA-based metric. Others are the same as the ones for inconsistency detection.

We implemented FFLM with the foundation language model LLaMa Touvron et al. (2023). It contains models with different sizes, where LLaMa7b is selected for our main experiments. We add "TL;DR" between the conditional sequence and the target sequence. The weights in Eq. 7 are determined in $\{0.0,0.1,...,1.0\}$ according to the performance on the corresponding validation set for inconsistency detection. For faithfulness rating, we set $\alpha$, $\beta$, $\delta$ as $0.25$, $0.25$, $0.5$ respectively, with the intuition that the former two are from the same category as introduced in Sec. 2.1. Our experiments are done on a single RTX 3090.

The results on inconsistency detection are in Table 2. Our proposed metric FFLM achieves state-of-the-art performance on 3 datasets including CoGenSum, SummEval, and FRANK, and outperforms ChatGPT on 5 out of 6 datasets from the SUMMAC benchmark except XSumFaith.

Both Polytope and FactCC are only labeled by a single annotator. As a result, their labels may not be as convincing as the other datasets. Although QuestEval, the best QA-based metric, achieves the top-1 accuracy on Polytope, it performs mediocrely on the rest. The weakly-supervised baselines FactCC_CLS and SummaC_Conv are trained with synthetic data constructed with human expertise that may have certain similarities with the FactCC dataset. Therefore, FactCC_CLS shows strong performance on the FactCC dataset while relatively weak on the others including datasets in Table 3, the same as the findings in Laban et al. (2022). Also, that’s why SummaC_Conv shows around 12% significant improvements over our FFLM.

Concentrating on the metrics based on probability changes, zero-shot metrics CoP_BART and HaRiM_BART perform not badly compared with previous SOTA SummaC_ZS, showing the potential of using probability changes for faithfulness evaluation. After introducing the foundation language model, their performances don’t drop in most cases, indicating that fine-tuning with in-domain data is not necessary. However, the leading performance between these two metrics is unstable among datasets. HaRiM_LLaMa outperforms CoP_LLaMa on FRANK and Polytope, while on the rest datasets, the opposite is true. FFLM, as a comprehensive metric, successfully achieves improvements over both of them on 5 out of 6 datasets.

Summary-level results are in Table 3. The results of ChatGPT borrowed from Luo et al. (2023) show its inconsistency improvements among datasets: It doesn’t exceed previous baselines on FRANKCNN, performs similarly on SummEval, and achieves conspicuous gains on FRANKXSUM. Besides, similar to the above analysis for comparisons among probability change-based metrics, our FFLM induces performance gains on 4 out of 5 datasets over CoP_LLaMa and HaRiM_LLaMa, especially on datasets sourced from XSum. Unfortunately, FFLM still lags behind ChatGPT with 175 billion parameters on FRANKXSUM, showing ChatGPT’s strong ability on dealing with highly abstractive summaries. This is also in line with ChatGPT’s favorable performance on XSumFaith in Table 2. After all, FFLM achieves the best scores on FRANKCNN, SummEval, and QAGSXSUM, and performs competitively on the other datasets.

We also report the system-level results on SummEval in Table 5. FFLM performs similarly to ChatGPT according to the Spearman correlation. HaRiM_LLaMa achieves a bit higher Spearman and Kendall correlation than FFLM, while FFLM performs better on Pearson correlation showing better linear correlation with human scores. Moreover, FFLM is more robust than HaRiM_LLaMa on different evaluation settings considering the poor summary-level performance of HaRiM_LLaMa especially for FRANKXSUM and QAGSXSUM in Table 3. Another observation is that although CoP and HaRiM backed on BART perform closely with them backed on LLaMa on the summary-level evaluation, they perform poorly on the system-level evaluation. This can be attributed to the fact that metrics based on CNN/DM fine-tuned BART have inductive bias Son et al. (2022). They tend to prefer summaries generated by abstractive models, while extractive models are generally more faithful. Meanwhile, metrics based on the foundation language model don’t show this bias, leading to the best results for ranking summarization systems.

To recap, our FFLM generalizes well among different task settings and different datasets, showing favorable performance over the baselines. It is backed on LLaMa with only 7 billion parameters and performs competitively with or even outperforms ChatGPT with 175 billion parameters, which is much more efficient for faithfulness evaluation.

We carried out ablation studies of FFLM on faithfulness rating in Table 4. The ablation results on inconsistency detection are in Appendix B.

Ablations on the metric components: We test different combinations of the three probability changes. The results show that ${\mit\Delta}_{\boldsymbol{Y}}^{{\rm cond}}$ is the most powerful component of FFLM. Its combination with ${\mit\Delta}_{\boldsymbol{Y}}^{{\rm prior}}$ ranks first among ablations on both FRANKXSUM and QAGSXSUM. Together with ${\mit\Delta}_{\boldsymbol{X}}^{{\rm prior}}$, our metric FFLM shows over 5% increases in Spearman correlation on QAGSCNN, 1.8% on FRANKCNN and SummEval, without much loss on the other two datasets, records more robust results. Moreover, combining different probability changes induces performance gains in most cases, reflecting the necessity of designing a comprehensive metric(More in Sec 4.4).

Ablations on the metric designs: We use $w$ and $\log$ to annotate the token-level weights and the logarithm operation introduced in Sec 2.2. Both operations contribute to the final FFLM, where $\log$ is more effective for datasets sourced from XSum and $w$ for the others.

Ablations on the combination weights: For the faithfulness rating task where we empirically set the weights $\alpha$, $\beta$ and $\delta$ as 0.25, 0.25 and 0.5, we compared it with the equaling weights, i.e., $\alpha=\beta=\delta=\frac{1}{3}$. FFLM performs relatively better.

By taking a look at the correlations between pairs of the metric components in Figure 6, we can see that the correlations vary among different datasets. None of the pairs show a high degree of correlation, indicating that these components may capture unfaithfulness from different aspects.

To figure out if different probability changes correlate well with different error types in the generated summaries, we take advantage of labels in the FRANKCNN and FRANKXSUM datasets. Pagnoni et al. (2021) divides the factual errors in the generated summaries into three groups. Semantic frame errors(Sem) include errors on the predicate, entities, and additional information about the circumstance. Discourse errors(Disc) consist of coreference errors and discourse link errors. Content verifiability errors(CVer) are closely related to extrinsic hallucinations Ji et al. (2023), containing the out-of-article error and grammatical error. We randomly picked 50 error cases and 10 error cases for each error type from FRANKCNN and FRANKXSUM respectively, and mixed them with the rest faithful summaries. Spearman correlations averaged over 10 times are in Fig. 1.

We observed that ${\mit\Delta}_{\boldsymbol{Y}}^{{\rm cond}}$ captures different errors best, which is accord with the ablation results in Table 4. Comparing among the scores for each $\Delta$ horizontally, we can see that the probability changes with prior probability is good at CVer errors on both datasets, and ${\mit\Delta}_{\boldsymbol{Y}}^{{\rm cond}}$ at Sem errors or Disc errors. The differences among datasets reflect their different characteristics Pagnoni et al. (2021). Summaries in FRANKCNN are made up of multiple sentences, resulting in more diverse and challenging situations for Disc errors than FRANKXSUM with single-sentence summaries. Thus, ${\mit\Delta}_{\boldsymbol{Y}}^{{\rm cond}}$ increases dramatically from 14.2% on FRANKCNN to 41.7% on FRANKXSUM for Disc.

FFLM made further improvements over ${\mit\Delta}_{\boldsymbol{Y}}^{{\rm cond}}$ on both Sem and CVer, showing that combining different probability changes is reasonable and effective in most cases except Discs.

To test FFLM’s performance on different models sizes, we select LLaMa with 3 billion(3B), 7 billion(7B) and 13 billion(13B) parameters ⁵⁵5The corresponding checkpoints from hugging face are openlm-research/open_llama_3b, decapoda-research/llama-7b-hf, and decapoda-research/llama-13b-hf. that are trained on the same data volume with 1 trillion tokens and draw the diagram in Fig. 2 for faithfulness rating datasets. The scores consistently increase from LLaMa-3B to LLaMa-7B across the five datasets, while the improvements are not consistent for LLaMa-13B. Given a certain amount of data, increasing the number of parameters can enhance the model’s language modeling ability and be helpful to faithfulness evaluation. On the other hand, when the model size keeps scaling up, more unexpected biases in the pre-training corpus may be memorized and will hurt the performance. This has also been pointed out by Ranaldi et al. (2023) and Nadeem et al. (2021).

In this way, we think that using larger foundation models may not be the best choice for faithfulness evaluation on summarization, which is also closely related to the research on investigating the optimal model size and dataset size for training foundation language models Hoffmann et al. (2022).

We compare our metric with prompting and instruction-tuning techniques under the same model size in Table 7 for faithfulness rating. Here, LLaMa-7B is the vanilla foundation language model. Vicuna-7B Chiang et al. (2023) and Alpaca-7B Taori et al. (2023) are initialized from LLaMa-7B and instruction-tuned with data collected in different ways. We present the maximum scores for each dataset among different prompts designed by previous works Chen et al. (2023); Gao et al. (2023); Luo et al. (2023). The detailed prompts for each evaluation task are listed in Appendix C.

First, we observe that using models containing 7 billion parameters, FFLM outperforms the prompting approach across different models and datasets. The prompting results here lag behind the performance of ChatGPT dramatically. This leads to the conclusion that the effectiveness of prompting approaches relies highly on much larger models, while our metric FFLM can be a cheaper alternative with smaller models. Second, instruction tuning is important for improving the prompting approach, while is not necessary for our FFLM. It enhances the models’ understanding ability on instruction templates in the prompts by further tuning with relatively small datasets. However, such manually collected datasets may contain unconscious bias and hurt FFLM’s performance.

Faithfulness evaluation metrics can be classified into zero-shot ones and weakly-supervised ones.

Zero-shot evaluation metrics mainly take advantage of the models trained with related natural language tasks. Goodrich et al. (2019) adopted information extraction tools to extract the fact tuples from both the source document and the summary. Tuple mismatches reflect the hallucinations. The intuition behind question-answering-based metrics Wang et al. (2020); Durmus et al. (2020); Scialom et al. (2021) is that identical answers should be generated when asking the same question to a summary and the corresponding document respectively. Natural language inference also shares commonalities with faithfulness evaluation in the way that information in a consistent summary should be entirely entailed by the source document Falke et al. (2019); Mishra et al. (2021); Laban et al. (2022). However, all of these metrics highly rely on the domain-transfer ability of out-of-box models and suffer from error propagation.

Instead, weakly-supervised approaches choose to train classifiers by constructing synthetic in-domain data with heuristics by experts. Different kinds of inconsistency errors are simulated by perturbing the reference document-summary pairs Kryściński et al. (2020); Utama et al. (2022); Yin et al. (2021). The limited heuristic makes it hard to cover all kinds of errors and shows poor generalization ability among datasets Laban et al. (2022).

As language modeling-based metrics Egan et al. (2022); Liu et al. (2022b) receive more attention, another small group of work for faithfulness evaluation computes probability changes with models fine-tuned on summarization datasets She et al. (2023); Son et al. (2022); Xie et al. (2021), showing a biased preference for abstractive summaries. Based on this line of work, we propose FFLM based on the foundation language model. Our zero-shot metric doesn’t require further training with in-domain or synthetic data and shows a strong generalization ability.

With orders of magnitude more parameters and extensive training on large-scale data, large language models (LLMs) Brown et al. (2020); Touvron et al. (2023) have exhibited surprising abilities that may not be observed in previous small language models. The strong capability in language comprehension naturally spurs research in exploring LLMs as better automatic evaluators for various text generation systems Wang et al. (2023).

There are also some attempts of faithfulness evaluation by prompting large models Luo et al. (2023) with different templates and strategies, such as adding detailed definitions Gao et al. (2023) and chain-of-thought Chen et al. (2023). None of these strategies achieve consistent improvements over the original prompt. Besides, neural models are sensitive to the choices of words Chen et al. (2023), resulting in unstable performances(See Appendix D).

Our FFLM takes advantage of the strong capability of LLMs for faithfulness evaluation in a different way and shows competitive performance requiring a much smaller number of parameters than the well-known ChatGPT OpenAI (2022).