Don’t Believe Everything You Read: Enhancing Summarization Interpretability through Automatic Identification of Hallucinations in Large Language Models

In the annotated version of SAMSum used in CONFIT, each generated summary is labeled with the specific type of error it contains. The objective is to leverage these identified error types and apply the token-level annotation to the data following the provided guidelines. This annotation process allows for errors to be marked at a more granular level, providing deeper insights into the distribution and characteristics of errors within the dataset.

The faithfulness tags are defined according to the hallucination categories below –

1. Wrong Reference Error- A pronoun in the generated summary either refers to the wrong or non-existent noun it should be replaced, or when a personal named entity in the summary is used incorrectly instead of a different personal entity mentioned in the reference.

Example: [Reference Summary] Mohit asked Darlene about the test. [Model-Generated Summary] Darlene asked Mohit about the test.

2. Object Error- Factual errors that arise from inaccuracies in either the direct or indirect objects.

Example: [Reference Summary] Tara raised her glass. [Model-Generated Summary] Tara raised her spoon.

3. Circumstantial Error: Circumstantial information (e.g., date, time, location) about the predicate doesn’t match the reference.

Example: [Reference Summary] The USA was founded in 1776. [Model-Generated Summary] The USA was founded in 1767.

4. Other Uncommon Errors - Errors that encompass factual errors resulting from discrepancies in grammatical tense between the generated summary and the reference.

Example: [Reference Summary] The children will go to the library. [Model-Generated Summary] The children went to the library.

5. Not Hallucinated- Tokens in the summary that are not hallucinated.

6. Missing Information- A special tag to be given at the end of sentence token to indicate if a summary suffers from missing information hallucination.

Table 1 shows the faithfulness tags we assign to each category listed above. Every token is labeled with the faithfulness tag corresponding to the category they fall in as defined above.

Six human annotators meticulously annotated 600 conversations with token-level granularity. Upon completion, the resulting analysis revealed that 92.6% of the 10930 tokens remained unaltered, as depicted in Figure 4. Figure 4 visually represents the distribution of errors at the token level. Predominantly, missing information emerged as the most frequent error, occurring 386 times, constituting 3.3% of the tokens. Following closely behind, the wrong reference error occurred 212 times. Additionally, circumstantial errors surfaced in 131 instances, while other uncommon errors occurred 124 times. Notably, object errors proved to be the rarest, presenting a total of 25 occurrences.

Since Zhou et al. (2020) has many hallucinated summaries for each dialog, the test and validation set are prepared by identifying unique dialogues and placing all summaries associated with a dialogue in a single split. Accordingly, for working with this dataset, the distribution for train, validation, and test is (76, 12, 12). Zhou et al. (2020) employed five distinct models for summarization: BART, BART-CONFIT, Pegasus, Pegasus-CONFIT, T5, and T5-CONFIT. Figure 5 shows the distribution of all the labels for each model. During an analysis of the summary tokens produced by these models, a pattern emerged regarding the sources of various errors. Specifically, Pegasus exhibited a higher occurrence of circumstantial errors. T5, on the other hand, demonstrated a tendency to omit information more frequently. Notably, the occurrence of uncommon errors was most prevalent within the outputs of BART. Interestingly, Pegasus presented the least number of object errors, while BART stood out as primarily responsible for wrong reference errors as represented in Figure 5.

For this approach, Zhou et al. (2020) for token-level hallucination detection will serve as a baseline model. This baseline uses binary hallucination detection, as opposed to the proposed approach, which aims for multiclass token-level hallucination detection. In order to employ this baseline in a multiclass context, the gold faithfulness tags are converted to a binary set, mapping all non-O tags to 1s and O tags to 0s. This baseline can be evaluated both with and without gold summaries provided as context for tagging reference summaries.

Previous work focuses on performing a binary classification on tokens in the summary and predicting if a token is hallucinated or not. This is difficult to interpret since it provides no reasoning for why the model deemed a token hallucinated. The annotated tags in this enhanced dataset are generally correlated with certain functions in a sentence and it is more sound to infer what type of hallucination the model believes the token to be.

The proposed approach has an added interpretability factor by assigning each token with a certain hallucination category or as defined above a "faithfulness" tag. This helps in a better interpretation of the reasoning of the model when it classifies a certain token as hallucinated or not.

Given the gold set of "faithfulness" tags for each summary, and the predicted tags, the accuracy, precision, recall, and F1 score for the set of tags are evaluated in the style of seqeval.

Table 3 denotes the precision, recall, f1, and accuracy scores for all the experimentation. The first prompt, placed in the system prompt, simply contained: the names of the available tags, one example of a summary, dialogue, and lastly a string of tags. It was observed that the predicted tag length seldom matched the gold tags, as the model struggled to keep count of the word number even when properly formatted output was produced. Only 10% of the data produced valid output for which F1 could be calculated.

In Prompt 2, when describing tagging, the tags were placed next to each associated word and M was placed next to [EOS]. This fixed the length discrepancies between gold and predicted tags but the model still made mistakes when placing the M tag. Two examples were given. This represented the first real F1 result of 0.36.

Prompt 3 gave 3 more informative examples covering more tags, made tagging a 2 step process of first tagging existing tokens and then checking for missing information, with human-provided explanations for both steps. Additionally, each complete example was moved into the chat history. This combination improved performance by 6 points.

In Prompt 6, the M tag was removed from the system prompt description. The explanations were also altered to be more conducive to token-level tagging and descriptions of each tag were added. These descriptions led to a further 12 point F1 increase.

After attempting to refine the tagging explanations in prompts 3-7, the Prompt 6 explanations were removed in Prompt 8, improving performance by 2 points. Providing explanations with descriptions was found to hurt model performance.

In prompt 9, the tag descriptions were edited to make them more precise. In summary, prompt 9 only contained task and tag descriptions in the system prompt and output examples with tags printed next to tokens in the chat history. The final F1 was 0.71.

All prompts can be seen in the appendix with the best prompt, prompt 9, appended to the top of the tagging section within the appendix.

The baseline performs poorly but the model after fine-tuning performs quite well. However, adding the gold summaries during training does not provide any statistically significant improvement to training without the gold summary. With training, it is seen that the model does not capture trends for any tag except O and M as shown in Figures LABEL:WGCM and LABEL:GCM in the Appendix. GPT tagging was better than training but much slower at scale and more time-intensive for prompt tuning. In the GPT experiments, it was found that asking it to generate tags alongside given summary tokens and then separately consider the Missing Information tag was the best approach. Additionally, explanations often misled the model and more detailed descriptions were more helpful than explanations. In the best prompt, the most frequent error was confusing the W and C tags, according to Figure 8.

The objective of this approach is to use different ways of prompting in state-of-the-art LLMs (GPT-4, LLaMa 2). This is to understand whether a model performs better if it is prompted to take hallucinations into account in the newly generated summary.

To improve the performance of LLMs on the summarization task, LLaMa 2 was fine-tuned on the token-level annotated version of the SAMSum dataset described in the Data section. The fine-tuning was done using Quantized Low-Rank Adaptation of Large Language Models (QLoRA), a lightweight training technique that significantly reduces the number of trainable parameters by introducing several smaller weights in a transformer model, and those are the only weights that are trained.

The exploration encompassed LLaMa2 versions 7B, 13B, and 70B and all versions were sourced from Huggingface. Notably, due to constraints related to GPU capabilities, this model underwent quantization to 4 bits, ensuring feasibility within the computational framework.

The model was fine-tuned using a single Nvidia A100 GPU with 80GB VRAM for 5000 steps on a set of 400 samples from the annotated SAMSum dataset described in the data section. The recommended hyperparameters for LoRA fine-tuning were utilized for this. The loss curve for the fine-tuning is presented in Figure 9. The loss curves for 7B and 13B version can be found in Appendix LABEL:llm_train_loss.

The performance of GPT-4 and LLaMa 2 (both the base model and the fine-tuned variant) in terms of ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-Lsum scores is presented in Table 4. The ROUGE scores are calculated using the gold summaries from the SAMSum dataset. Based on these results, it was found that models generated better summaries when they were asked to generate faithfulness tags for the summaries that they generated as per the defined rules. That said, adding a chain-of-thought protocol by asking it to generate an explanation for the tagging process didn’t cause a significant improvement in performance. However, as chain-of-thought prompting usually goes, explanations tend to make answers more interpretable and, hence more reliable.

While there was no significant change to the model performance after fine-tuning, a lot of that can be attributed to 2 factors:

1. Lack of a big dataset for fine-tuning.
2. Possible loss of precision due to 4-bit quantization.

Alongside that, the smaller variants of the LLaMa 2 model (7b and 13b) would not generate summaries at all as shown in Appendix LABEL:LLaMaoutput; only the 70b model seemed to generate summaries properly.

This study shows that the newly devised method of incorporating hallucination detection into the thought process of the LLM improves its performance in a given task.

Previous works use the proxy model to try and imitate/explain the reasoning followed by the original model. However, Lyu et al. (2023) casts doubt on the ability of the proxy model to accurately reflect the reasoning. Hence, this paper proposes merging the two tasks of summarizing and hallucination tagging into one joint task using only one model.

Figure 10 illustrates the novel approach presented, where the tasks of summarizing and hallucination detection are integrated into a single model. This integration is pivotal for achieving more consistent and reliable outputs. The rationale behind this design is grounded in two main hypotheses –

1. 1.

   Consistency in Reasoning: By positioning the classification layer adjacent to the decoding head, the model can simultaneously generate tags and tokens. This concurrent processing ensures that the reasoning behind both hallucination detection and summary generation is aligned, leading to outputs that are both coherent and faithful to the original content.

2. 2.

   Enhanced Hallucination Detection: The unified architecture allows for the direct application of hallucination detection during the summary generation process. This integration is expected to significantly improve the model’s ability to identify and avoid hallucinations, thereby producing summaries that are more accurate and reflective of the original content.

To empirically validate the proposed approach, the following hypotheses can be tested –

1. 1.

   Integrating summary generation with hallucination detection in a single model will lead to superior hallucination detection. This integration should result in summaries that are not only more accurate but also more faithful to the original content.

2. 2.

   The adoption of a joint loss function, tailored to address both summarization and hallucination detection, will align the model’s training objectives. This alignment is hypothesized to minimize the occurrence of hallucinations, thereby yielding summaries that are more true to the source material.

The BART model, as outlined in Lewis et al. (2019), is utilized as the foundational model for the approach presented. This choice is due to BART’s proven efficacy in producing coherent and contextually relevant summaries, especially when fine-tuned on specific datasets such as SAMSum.

BART’s architecture combines bidirectional and auto-regressive transformers, providing a robust framework for understanding and generating natural language. By fine-tuning BART on the SAMSum dataset, the advanced capabilities of BART in capturing dialog nuances are leveraged, making it an ideal starting point for the proposed enhancements.

Previous works that involve predicting and emitting a human readable explanation perform these tasks sequentially. Camburu et al. (2018) uses a predict-then-explain technique, where the model first predicts then a proxy model generates an explanation for the prediction. On the other hand, Zaidan et al. (2007) uses an explain-predict method, where the explainer model first generates a human readable explanation. This explanation is then used as input to the predictor model on the actual task.

In detecting hallucinations, previous works Zhou et al. (2021) and section 3 utilize a proxy model to help explain/detect hallucinations. Moreover, section 3 allows for a more fine-grained interpretable detection. The novelty in the approach presented is that the extent to which the true reasoning of the model for faithfulness tag generation is reflected is increased.

The use of a single model is proposed that generates a summary token at timestep t and predicts a faithfulness tag at that timestep t. Since the generation of the token and prediction output tag are both simultaneously conditioned on the same computation graph, the model becomes more faithful in the task of detecting hallucinations.

The design presented builds upon the seq2seq transformer architecture detailed in Vaswani et al. (2017), maintaining its core structure of a bidirectional encoder and a left-to-right auto-regressive decoder. The proposed modifications aim to extend the architecture of Section 5.2 to better handle the dual tasks of summary generation and hallucination detection.

The structure of the model, abstracting the common parts in the baselines, is shown in Figure 10. The input to the network is a dialog D, which is a sequence of words $w_{1}$, $w_{2}$, …, $w_{T}$, where T is the length of the dialog. The network consists of two outputs at every time-step t: the summary token $o^{s}$ and the "faithfulness tag" $o^{f}$. The modifications to the baseline model are described below.

The baseline model has a Language Model head, $L^{s}$ that projects the decoder hidden states, $\overrightarrow{\emph{$h_{t}$}}$ into a space, $y^{s}_{t}$ which generates the summary tokens after a softmax operation. Taking inspiration from Zhang and Wang (2016), a linear classification layer, $L_{f}$ is proposed that projects the same decoder hidden states, $\overrightarrow{\emph{$h_{t}$}}$, into a "faithfulness" tag prediction space, $y^{f}_{t}$. Since there are six tag categories, at every time-step t, the $L^{f}$ will project $h_{t}$ into a dimension of six. The "faithfulness" tags $o^{f}$ are finally generated after a softmax operation. Formally,

where $W^{s}$ and $W^{f}$ are transformation matrices for token generation and tag prediction respectively, $b^{s}$ and $b^{f}$ are bias vectors.

In this section, the ideal model output is defined as the primary metric of evaluation, and how the success of the hypothesis would be evaluated is described.

Figure 11 shows the ideal model, which consists of a summary and accurately identified hallucinated tokens with correct "faithfulness" tags assigned to them.

This approach consists of two parts: summaries and faithfulness tags. The evaluation consists of evaluating each part independently.

One of the main objectives is to maintain the summarization quality of the model. Hence, to capture this, existing metrics used to evaluate summaries - ROUGE metrics Lin (2004b) - are utilized.

For evaluating the model’s performance in detecting and categorizing hallucinations by the use of the faithfulness tags, simple precision-based metrics used in token classification tasks would be used. However, since the joint model is fine-tuned, the summaries generated on the test set may not match the gold summaries and hence a gold set of "faithfulness" tags may not exist. For this reason, 10% of the test set will be manually evaluated for the generated faithfulness tags.

Upon evaluating the chart described in Figure 13, the results of the proposed model compared to previous works and the baseline can be seen. Despite using only a small amount of data for fine-tuning the proposed model, a +0.4 improvement in the ROUGE score is observed compared to the other models.

Additionally, it was found that in some cases the proposed model generated better summaries than the gold summaries themselves. This is an interesting result considering the model was trained on limited examples.

Upon human evaluation of the summaries generated by the joint model, an interesting and not uncommon pattern was noticed. In many cases, the summaries produced by the joint model contained more factual information than the human-written gold summaries and the baseline. An example is shown in Figure 12. As seen, the summary generated by the joint model includes additional factual details that enhance the quality of the summary. However, since ROUGE is a reference-based metric, the baseline-generated summary will incorrectly receive a higher score than the joint model’s summary. This raises the question of whether a more robust metric is needed to measure summary quality.