PHALM: Building a Knowledge Graph from Scratch
by Prompting Humans and a Language Model

There are several knowledge bases about commonsense, from what appears in the text to what is tacit but not written in the text. ConceptNet (Speer et al., 2017), for example, is a knowledge graph that connects words and phrases by relations. GenericsKB (Bhakthavatsalam et al., 2020) is a corpus describing knowledge of entities in natural language rather than in graph.

In some datasets, commonsense knowledge is collected in the form of question answering. Roemmele et al. (2011) acquire plausible causes and effects for premises as two-choice questions. Zellers et al. (2018) provide SWAG, acquiring inferences about a situation from video captions as four-choice questions. KUCI (Omura et al., 2020) is a dataset for commonsense inference in Japanese, which is obtained by combining automatic extraction and crowdsourcing. Talmor et al. (2019) build CommonsenseQA, which treats commonsense on ConceptNet’s entities as question answering.

In contrast, we propose a method to construct a structural (not QA form) graph based on events.

Regarding commonsense knowledge bases, there are several graphs that focus on events. ATOMIC (Sap et al., 2019) describes the relationship between events, mental states (Rashkin et al., 2018), and personas. Hwang et al. (2021) merge ATOMIC and ConceptNet, proposing ATOMIC-2020.

There are also studies for leveraging context. GLUCOSE (Mostafazadeh et al., 2020) is a commonsense inference knowledge graph for short stories, built by annotating ROCStories (Mostafazadeh et al., 2016). CIDER (Ghosal et al., 2021) and CICERO (Ghosal et al., 2022) are the graphs for dialogues where DailyDialog (Li et al., 2017) and other dialogue corpora are annotated with inferences.

ASER (Zhang et al., 2019) is an event knowledge graph, automatically extracted from text corpora by focusing on discourse. With ASER, TransOMCS (Zhang et al., 2020) aims at bootstrapped knowledge graph acquisition by pattern matching and ranking.

While ConceptNet and ATOMIC are acquired by crowdsourcing, ASER and TransOMCS are automatically built. On one hand, a large-scale graph can be built easily in an automatic way, but it is difficult to obtain knowledge not appearing in the text. On the other hand, crowdsourcing can gather high-quality data, but it is expensive in terms of both money and time.

There is a method that uses crowdsourcing and LLMs together to build an event knowledge graph (West et al., 2022). Although it is possible to acquire a large-scale and high-quality graph, they assume that an initial graph, ATOMIC in this case, has already been available. As stated in Section 1, our proposal is a method to build a commonsense knowledge graph from scratch without such existing graphs.

There have been studies on storing knowledge in a neural form rather than a symbolic form. In particular, methods of considering neural language models as knowledge bases (Petroni et al., 2019; AlKhamissi et al., 2022) have been developed. Bosselut et al. (2019) train COMET by finetuning pretrained Transformers on ATOMIC and ConceptNet, aiming at inference on unseen events and concepts. Gabriel et al. (2021) point out that COMET ignores discourse, introducing recurrent memory for paragraph-level information.

West et al. (2022) propose symbolic knowledge distillation where specific knowledge in a general language model is distilled into a specific language model via a symbolic one. They expand ATOMIC using GPT-3 (Brown et al., 2020), filter the outputs using RoBERTa (Liu et al., 2019), and finetune GPT-2 (Radford et al., 2019) on the filtered ones. Training data for this filtering was created through manual annotation, which is expensive.

We first acquire a small-scale high-quality knowledge graph by crowdsourcing. With Yahoo! Crowdsourcing, specifically, we ask crowdworkers to write events and inferences. In a task, we provide them with 10 shots as a prompt for each event and inference. Note that for inferences, the prompts differ for each relation, as mentioned later. We obtain a graph by filtering the collected inferences syntactically and semantically.

To examine the qualities of the inferences acquired by crowdsourcing, we crowdsource their evaluation. We ask three crowdworkers whether the inferences are acceptable or not and judge their acceptability by majority voting. The evaluation is crowdsourced independently for each relation. The inferences judged to be unacceptable by majority voting are filtered out.

The inferences collected in Section 3.1 are evaluated and filtered as above. The statistics are listed in the middle two columns of Table 1. We also calculated Fleiss’s $\kappa$ ⁶⁶6As a reference, $\kappa\in[0,0.2]$ is considered to be “slight” agreement, $(0.2,0.4]$ is “fair”, and $(0.4,0.6]$ is “moderate” Landis and Koch (1977). as an inner-annotator agreement (IAA) in the evaluation, which is shown in the rightmost column of Table 1. The IAAs were “fair” to weak “moderate”. This is because commonsense is subjective and depends on the individual.

There are several tendencies in the inferences filtered out, i.e., judged to be unacceptable. In some inferences, the order is reversed, as in the triplet $\langle$PersonX sleeps twice, xEffect, PersonX thinks that they are off work today$\rangle$. Others are not plausible, as in $\langle$PersonX surfs the Internet, xNeed, PersonX gets to the ocean$\rangle$.

From a small-scale high-quality knowledge graph acquired in Sections 3.1 and 3.2, we generate a large-scale knowledge graph with an LLM. We use the Koya 39B model of HyperCLOVA JP as an LLM. Both events and inferences are generated by providing 10 shots. The shots are randomly chosen from the small-scale graph for each generation.

Because we use an LLM, some of the generated inferences are invalid. In a previous study, invalid inferences were removed by training and applying a filtering model with RoBERTa. The training data for the filtering model was made by manually annotating a portion of the large graph, which is expensive. In this section, we propose a low-cost filtering method to train filters without human annotation. This is consistent with our claim that a commonsense knowledge graph should be built for each language.

The filtering model is a binary classification model that is given a triplet and outputs whether it is valid or not. Both positive and negative examples are required for training. Since the small-scale graph used as the LLM shots was manually collected, we assume that all triplets in it are valid, and we take them as positive examples. Negative examples are not included, but we create pseudo-negative examples from the triplets in the small graph and adopt them. Details of how to create negative examples are given in Appendix D. The filtering model is constructed by finetuning Japanese RoBERTa⁷⁷7https://huggingface.co/nlp-waseda/roberta-large-japanese on the training data created without human annotation. We use a RoBERTa model finetuned on an NLI task Kurihara et al. (2022) in advance, following the previous study West et al. (2021). See Appendix D for further details on the filtering model.

We obtain a large, high-quality commonsense knowledge graph from scratch by the four procedures described in the previous sections: crowdsourcing acquisition, crowdsourcing filtering, LLM scaling, and filtering with RoBERTa. The statistics of events and inferences generated by HyperCLOVA JP are shown in Table 4. The results of the evaluation and the inter-annotator agreement are also shown in Table 4. Since the filtering threshold is a hyperparameter, we present unfiltered statistics here. A comparison with Table 1 indicates that the quality is as good as those written by crowdworkers. Examples of generated inferences are shown in Table 5.

The generated knowledge graph in Japanese reflects the culture of Japan, such as $\langle$PersonX goes to the office, xNeed, PersonX takes a train$\rangle$ and $\langle$PersonX eats a meal, xNeed, PersonX presses the switch on the rice cooker$\rangle$. This fact indicates the importance of building from scratch for a specific language, rather than translating a similar dataset in a different language, which emphasizes the value of our method proposed in this paper.

In this paper, a small-scale knowledge graph is collected as in Sections 3.1 and 3.2, and a large-scale knowledge graph is generated as in Section 3.3. Here, we examine how effective the filtering for the small graph in Section 3.2 is. As an experiment, we use filtered and unfiltered small-scale graphs as prompts to generate a large-scale graph. Then, we randomly select 500 generated triplets for each relation and evaluate them by crowdsourcing as in Section 3.2. Note that the results for the filtered triplets are the same as Section 3.3.

The ratios of appropriate inferences generated by the LLM with and without filtering for the small graph are shown in Table 6. We experiment with a subset of the graph. For all relations except xNeed, filtering improves the quality of triplets.

To evaluate the filtering model in Section 3.4, 500 triplets in the large graph are manually annotated. The results of applying the filter to the test data are shown in the Figure 3. Since setting a high threshold increases the ratio of valid triplets, it can be said that we have constructed an effective filter at a low cost.

In Section 3.1, on one hand, we asked crowdworkers to describe events and inferences. In Section 3.3, on the other hand, we had an LLM generate them. Here, we compare a small-scale knowledge graph by crowdsourcing and a large-scale one from an LLM, i.e., inferences generated by humans and a computer. Because the relationships between events can be largely divided into contingent and temporal relationships (Bethard et al., 2008), we adopt contingency and time interval as metrics for comparison.

Of the four relations, we focus on xEffect as a representative, which is a typical causal relation. For each head of the triplets acquired by crowdsourcing in Sections 3.1 and 3.2, we generate three tails using the LLM in Section 3.3 and compare them with the original tails. From the 554 heads for xEffect in the small-scale graph, we obtained 586 unique inferences.

Using the constructed knowledge graph, we finetune pretrained models to construct Japanese commonsense generation models. To evaluate inferences on unseen events, 150 triplets are randomly selected as the test set. For pretrained models, we adopt Japanese GPT-2⁸⁸8https://huggingface.co/nlp-waseda/gpt2-small-japanese of the Hugging Face implementation (Wolf et al., 2020).

As GPT-2 predicts the next word, the head and the relation are given as an input, and the model is trained to output the tail. Since the relations are not included in the vocabulary of the pretrained models, they are added as special tokens.

We generate inferences for the head events in the test set using the trained Japanese commonsense generation models and evaluate the inferences manually. Examples of the inference results are shown in Appendix E.

Using crowdsourcing, we evaluate how likely the generated inferences are. We ask three crowdworkers how likely each triplet is and decide by majority vote whether each triplet is valid or not. The result is shown in Table 7. Notably, the performance of GPT-2 exceeds that of HyperCLOVA. Comparing Table 7 with Table 4, which can be regarded as the performance of HyperCLOVA’s commonsense generation, GPT-2 performs better for all relations. This represents the quality of the built graphs and the effectiveness of storing commonsense in the neural model.

Furthermore, as shown in Table 7, the ratio of valid triplets of xNeed and xIntent are lower than xEffect and xReact, respectively. Given that xNeed is a relation that infers past events, xEffect is a future event, xIntent is a past mental state, and xReact is a future mental state, this can be attributed to the fact that we used an autoregressive model, which makes it difficult to infer in reverse order of time.