FactKG: Fact Verification via Reasoning on Knowledge Graphs

The most basic type of claim is one-hop, which covers only one knowledge triple. One-hop claims can be verified by checking the existence of a single corresponding triple. In the second row of Table 1, the claim is Supported when the triple (AIDAstella, ShipBuilder, Meyer Werft) exists.

We take the sentences that consist of a single triple in $S_{w}$ as Supported claims. Refuted claims are created by substituting Supported claims in two ways: Entity substitution and Relation substitution. In Entity substitution, we replace an entity $e$ in Supported claim $C$ with another entity $\tilde{e}$ of the same entity type. In order to ensure that the label of the substituted sentence $\tilde{C}$ is Refuted, the entity $\tilde{e}$ should satisfy the following two conditions. i) To select $\tilde{e}$ that is irrelevant to $C$, $\tilde{e}$ is outside 4-hops from all entities in $C$ on DBpedia, ii) the results of NLI ($C$, $\tilde{C}$) and NLI ($\tilde{C}$, $C$) are both contradiction.⁴⁴4We use a natural language inference (NLI) model, RoBERTa-base (Liu et al., 2019) finetuned on the MNLI dataset (Williams et al., 2018). The notation NLI (p, h) represents the result of NLI when p is assigned as the premise and h as the hypothesis. In Relation substitution, we replace a relation in the Supported claim with another relation. We replace the relation of a triple in the claim with another relation that takes the same entity types for the head and tail as the original relation (e.g. currentTeam $\leftrightarrow$ formerTeam). The four groups of compatible relations are listed in Table 6. The overall process of the substitution methods is illustrated in Figure 2.

A claim in the real world can include a mixture of different facts. To incorporate this, we construct a conjunction claim composed of multiple triples. Conjunction claims are verified by the existence of all corresponding triples. In the third row of Table 1, the claim can be divided into two parts: “AIDA Cruise line operated the AIDAstella.” and “AIDAstella was built by Meyer Werft.”. The claim is Supported when all the triples (AIDAstella, ShipOperator, AIDA Cruises), (AIDAstella, ShipBuilder, Meyer Werft) exist. To implement this idea, we extracted sentences consisting of more than one triple from $S_{w}$ and used them as the Supported claims. To create Refuted claims, we use Entity substitution method on these Supported claims.

People may make claims that assert the existence of something (e.g. “She has two kids.”). From the view of a triple, this corresponds to the head or tail missing. To reflect this scenario, we formulate a claim by extracting only {head, relation} or {tail, relation} from a triple. Existence claims are generated using templates and they are divided into two categories: head-relation (e.g. template: {head} had a(an) {relation}.) and tail-relation (e.g. template: {tail} was a {relation}.). Supported claims are constructed by randomly extracting {head, relation} or {tail, relation} in triples from $S_{w}$. The Refuted claims are constructed using the same type of entities as represented in the claim, but with different relations. However, it is possible that unrealistic claims may be generated in this manner. For example, “Meyer Werft had a location.” or “Papenburg was a location.” can be created from the triple (Meyer Werft, location, Papenbug). Hence, we selected 22 relations out of all relations that lead to realistic claims. Templates used for both categories and examples of generated claims are in Table 7.

We also consider multi-hop claims that require the validation of multiple facts where some entities are underspecified. Entities in this claim can be connected by a sequence of relations. For example, the multi-hop claim in Table 1 is Supported if the triple (AIDAstella, ShipBuilder, $x$) and the triple ($x$, location, Papenburg) are present in the graph. The goal is to verify the existence of a path on the graph that starts from AIDAstella and reaches Papenburg through the relations ShipBuilder and location.

Figure 3 shows how a Supported multi-hop claim $C_{M}$ can be generated by replacing an entity $e$ of the conjunction claim $C$ with its type name. First, an entity $e$ is selected from the green nodes. Then, the type name $t$ of the entity $e$ is extracted from DBpedia. However, each entity $e$ in DBpedia has several types $T=\{t_{1},t_{2},...,t_{N}\}$, and it is not annotated which type is relevant when $e$ is used in a claim. So it is necessary to select one of them. For each $t_{n}\in T$, we insert it next to the entity $e$ in the claim $C$ and measure the perplexity score of the modified claim using GPT2-large (Radford et al., 2019). Then we replace $e$ in the claim with the type name that had the lowest score. The Refuted claim is generated by applying Entity substitution to the Supported claim.

For each of the four methods for generating claims, we develop claims that incorporate negations.

One-hop      We use the Negative Claim Generation Model (Lee et al., 2021) which was fine-tuned on the opposite claim set in the WikiFactCheck-English dataset (Sathe et al., 2020).⁵⁵5WikiFactCheck-English consists of pairs of claims, with a positive claim and its corresponding negative claim. To ensure the quality of the generated sentences, we generate 100 opposing claims for each original claim, then only use those that preserve all entities, and contain negations (e.g. ‘not’ or ‘never’). Also, similar to Entity substitution method, we only use sentences whose NLI relation with the original sentences are contradiction bidirectionally. When a negation is added, the label of the generated claim is reversed from the original claim.

Conjunction      The use of negations (i.e., ‘not’) in various positions within conjunction claims allows the generation of a wide range of negative claim structures. We employ the pretrained language model GPT-J 6B (Wang and Komatsuzaki, 2021) to attach negations to the claim. We construct 16 in-context examples, each with negations attached to the texts corresponding to the first or/and second relation. When a negation is added to the Supported claims, all the claims become Refuted. However, when it is added to Refuted claims, the label depends on the position of the negation. When negations are added to all parts with substituted entities, it becomes a Supported claim. Conversely, other cases preserve the label Refuted since the negation is added to a place that is not related to entity substitution. A detailed labeling strategy is described in Appendix D.1.

Existence      The claim is formulated by adding a negation within the templates presented in Section 3.1.3 (e.g. {tail} was not a {relation}.).

Multi-hop      A claim is formulated using the GPT-J with in-context examples, similar to conjunction. The truth of this claim is dependent on the presence of a distinctive path that matches the claim’s intent. For example, the negative claim “AIDAstella was built by a company, not in Papenburg.” is Supported if $x$ exists where the triples (AIDAstella, ShipBuilder, $x$) and ($x$, location, $y$) are in DBpedia and $y$ is not Papenburg. A more detailed labeling strategy is in Appendix D.2.

Using a similar method proposed by Kim et al. (2021), we transform the claim obtained from 3.1 into a colloquial style. For example, the claim “Obama was president.” is converted to “Have you heard about Obama? He was president!”.

We train FLAN T5-large Chung et al. (2022) to generate a colloquial style sentence given a corresponding written style sentence from Wizard of Wikipedia Dinan et al. (2019). However, using sentences generated by the model could have several potential issues: i) the original and generated sentences are lexically the same, ii) some entities are missing in the generated sentences, iii) the generated sentences deviate semantically from the original, iv) the generated sentences lack a colloquialism, as mentioned in Kim et al. (2021). To overcome this, we oversample candidate sentences and utilize an additional filtering process.

First, to make more diverse samples using the model, we set the temperature to 20.0 and generate 500 samples with beam search. i) To avoid generated sentences that are too similar to the original sentences, only sentences with an edit distance of 6 or more from the original sentence are selected among 500 samples. ii) Then, only those that have verbs and the named entities all preserved are selected.⁶⁶6NLTK Bird et al. (2009) POS tagger and Stanza Qi et al. (2020) NER module are used. DBPedia entities are already tagged in each claim, but not all entities exist in the sentence in their raw form, so the NER module is used. iii) Finally, we use bidirectional NLI to preserve the original semantics. Candidate sentences survive when NLI ($O$, $G$) is entailment and NLI ($G$, $O$) is not contradiction where $O$ refers to the original sentence and $G$ the generated sentence. On average, only 41.2 generated sentences survived out of 500 samples. Additionally, in cases where none of the 500 generated sentences pass the filtering process, we include the original claim in the final dataset as a written style claim. Following the filtering process, the AFLITE method Sakaguchi et al. (2019), which utilizes adversarial filtering, is applied to select the most colloquial style sentence among the surviving sentences. We include the selected claim in the final dataset as a colloquial style claim.

A presupposition is something the speaker assumes to be the case prior to making an utterance Yule and Widdowson (1996). People often communicate under the presupposition that their beliefs are universally accepted. We construct claims using this form of utterance. The claims in FactKG are focused on three types of presupposition: factive, non-factive, and structural presuppositions.

Factive Presupposition      People frequently use verbs like “realize” or “remember” to express the truth of their assumptions. The utterance “I remembered that {Statement}.” assumes that {Statement} is true. Reflecting these features, a new claim is created by appending expressions that contain presupposition (e.g. “I realized that” or “I wasn’t aware that”) to the existing claim. We used eight templates to make factive presupposition claims: the details are appended in Table 8.

Non Factive Presupposition      The verbs such as “wish” are commonly used in utterances that describe events that have not occurred. For example, people say “I wish that {Statement}.” when {Statement} did not happen. Claims that are created by the non-factive presupposition method are labeled as the opposite of the original one. We used three templates to make these claims: the templates are appended in Table 8.

Structural Presupposition      This type is in the form of a question that presumes certain facts. We treat the question itself as a claim. For example, “When was Messi in Barcelona?” assumes that Messi was in Barcelona. To create a natural sentence form, only claims corresponding to one-hop and existence are constructed. For the one-hop claim, a different template was created corresponding to each relation reflecting its meaning (e.g. “When did {head} die from {tail}?” for the relation deathCause and “When was {head} directed by {tail}?” for relation director). Existence claims are also generated based on templates (e.g. “When was {tail} {relation}?”) using pairs of head-relation or tail-relation, similar to Section 3.1.3. The templates used are described in Table 9.

In the Claim Only setting, the baseline models receive only the claim as input and predict the label. We used three transformer-based text classifiers, BERT, BlueBERT, and Flan-T5. BERT (Devlin et al., 2018) is trained on Wikipedia from which DBpedia is extracted. So we expect that the model will use evidence memorized in its pre-trained weights (Petroni et al., 2019) or exploit structural patterns in the generated claims (Schuster et al., 2019; Thorne and Vlachos, 2021). BlueBERT (Peng et al., 2019) is trained on biomedical corpus, such as Pubmed abstracts. We use BlueBERT as a comparator for BERT since it has never seen Wikipedia during its pre-training. Flan-T5 (Chung et al., 2022) is an enhanced version of T5 (Raffel et al., 2022) encoder-decoder that has been fine-tuned in a mixture of 1.8K tasks. In all experiments, we fine-tune BERT and BlueBERT on our training set. Different from BERT and BlueBERT, we use Flan-T5 in the zero-shot setting. For this setting, we use “Is this claim True or False? Claim: ” as the prefix. Then, we measure the probability that tokens True and False will appear in the output. Among the two tokens, we choose the one with the higher probability.

In the With Graphical Evidence setting, the model receives the claim and graph evidence as input and predicts the label. The baseline we used is a framework proposed by GEAR (Zhou et al., 2019) that enables reasoning on multiple evidence texts. Since GEAR was originally designed to reason over text passages, we change components to suit KG. The modified GEAR consists of the subgraph retrieval module and the claim verification module. The pipeline of the modified GEAR is illustrated in Figure 4.