Minimal Evidence Group Identification for Claim Verification

The task of claim verification predicts whether a claim is supported by the presented evidence Thorne et al. (2018); Chen et al. (2023a). A claim verification model is expected to identify the correct evidence pieces (EPs; e.g. evidence sentences or snippets) among tens of retrieved candidate evidence, but a practical challenge lies in that there might exist multiple sets of evidence that verify the claim from different perspectives. Figure 1 shows an example where, given a claim and some retrieved evidence, there exist two different — but both valid — ways of supporting the claim.

While humans can quickly identify mutually redundant EPs, e.g. $e_{1}$ and $e_{3}$ in Figure 1, and propose plausible combinations of EPs as evidence groups (EGs, formally defined in Section 3.1), existing claim verification systems Dagan et al. (2005); Thorne et al. (2018); Wadden et al. (2020); Schuster et al. (2021); Chen et al. (2023a, b) focus only on the relationship between the claim and individual EPs, without considering the co-supporting relationships among EPs. This becomes problematic because the retrieved EPs might be redundant, or an individual EP may only partially support the claim. An EG with redundant EPs makes it more difficult to explain the reasoning for supporting the claim, while an EG composed of partially supporting EPs may still not fully support the claim, resulting in logical flaws. These problematic outputs not only confuse human verifiers, but also hurt the performance of downstream tasks.

In this paper, we introduce the problem of identifying minimal evidence groups (MEGs) from retrieved evidence candidates. Conceptually, an MEG is composed of EPs with the following properties: (1) Sufficiency: each MEG fully supports the veracity of the claim; (2) Non-redundancy: the EPs in an MEG are not redundant with each other; and (3) Minimality: the number of EPs in each MEG is minimal. We formally define the task of MEG identification and show that classic claim verification approaches cannot effectively solve this problem. We propose a simple yet practical approach to decompose it to support prediction and evidence group merging. Our proposed approach significantly outperforms the baseline of directly prompting a large-language model (LLM) by 18.4% and 34.8% absolute precision scores on the WiCE Kamoi et al. (2023) and SciFact Wadden et al. (2020) benchmarks. Finally, we demonstrate the benefit of MEGs for saving computation budget in the downstream task of claim generation.

Classic claim verification Thorne et al. (2018); Chen et al. (2023a) consists of three steps: evidence retrieval, evidence selection, and stance prediction. Evidence retrieval perform coarse-grained filtering of EPs from thousands of candidates. Evidence selection and stance prediction perform fine-grained selection of EPs and predict whether the claim is supported by the selected EPs. MEG identification builds on classic claim verification by restricting evidence selection and stance prediction to predict a minimal group of EPs that fully supports the claim.

To address the problem that claim verification systems Dagan et al. (2005); Wadden et al. (2020); Schuster et al. (2021); Chen et al. (2023b) may predict EPs that only partially support the claim, Laban et al. (2022); Schuster et al. (2022); Kamoi et al. (2023) aggregated individual EPs’ entailment scores into EG scores. However, they did not address the problem of redundancy within an EG; we propose MEG identification to fill this gap.

The closest work to ours is SciFact Wadden et al. (2020), which annotates “minimal evidence sets” for each claim. However, the SciFact shared task does not penalize non-minimal EGs, and consequently models that evaluate on SciFact Pradeep et al. (2021); Li et al. (2021); Zhang et al. (2021); Wadden et al. (2022) are trained on the union of EGs from different human annotators, which is no longer minimal. Similarly, Thorne et al. (2018); Chen et al. (2023b); Kamoi et al. (2023) collect (possibly redundant) EGs from multiple annotators and use their union as training labels. As a result, existing models prioritize EP recall and are not directly comparable to MEG identification models.

The challenge of MEG identification is to find the smallest set of EPs that fully supports the claim. As discussed in Section 2, classic claim verification models treat the EP as the basic unit; they are neither designed nor trained for groups of evidence. Our experiments of prompting directly with LLMs also show poor performance (Table 1, “Direct”)²²2The explicit verification of combinations of EPs reduces from Set Cover and is NP-hard (see proof in Appendix A.).

As Algorithms 1 shows, we decompose MEG identification into two steps: (1) predicting whether a candidate set of EPs fully supports, partially supports, or does not support the claim and (2) bottom-up merging partially supporting groups in search of a fully supporting group. The support prediction Model can be implemented by any reasonable approach, such as prompting LLMs or fine-tuning models like T5 Raffel et al. (2020). When merging groups, we increment the overall group size by 1 at each step. Note that if we only evaluate the base case with $size$=1, this is equivalent to classic claim verification Thorne et al. (2018); Wadden et al. (2020); Schuster et al. (2021); Kamoi et al. (2023).

Based on the definition of MEG (Section 3.1), we derive three principles to prune the problem space for a tractable solution: (1) any superset of an MEG fully supports the claim $c$; (2) any non-empty subset of an MEG partially supports $c$; and (3) if a set of EPs $S$ fully supports or does not support $c$, then $S$ is not a strict subset of any MEG. Therefore, we stop merging sets that are predicted as fully supporting or not supporting to maintain the non-redundancy and minimality of the candidate EP sets. In addition, when choosing a pair of sets to merge, we prune the candidate merge partners for each set using a redundancy checker notRedundant (implemented as a zero-shot LLM prompt; see Appendix C.2). Finally, upon finding a fully supporting set, we stop merging and return all fully supporting sets of the current size.

The non-redundancy of MEGs not only makes the evidence more human-readable, it also improves the performance of downstream applications. Inspired by Chen et al. (2023c), we use WiCE-MEG to highlight the MEG’s minimality and sufficiency properties using claim generation as an example downstream task, with a computation budget measured in the number of words or sentences.

We have addressed the challenging scenario in claim verification where a model is expected to identify a minimal group of evidence pieces (EPs) among a relatively large amount of candidate evidence, and there might exist different sets of evidence that verify the claim from different perspectives. We formally define and study the problem of such minimal evidence group (MEG) identification and show that it can be reduced from a Set Cover-like problem. Our proposed approach achieves significant improvements over direct LLM prompting. Finally, we demonstrate the benefit of MEGs over classic claim verification approaches in downstream applications such as claim generation.

Similar to all prior claim verification works Dagan et al. (2005); Thorne et al. (2018); Wadden et al. (2020); Schuster et al. (2021); Chen et al. (2023a, b), we stress that the MEG identification problem and the MEGs predicted by our proposed approach only consider the relative entailment relationship between the evidence and the claim. In other words, the MEG identification problem and our proposed approach do not guarantee the absolute correctness of the claim or the EPs or EGs themselves. Any future application must be cautious in distinguishing between retrieving evidence that supports the claim, correct or not, and verifying the absolute factual correctness of the claim.