Syntax Controlled Knowledge Graph-to-Text Generation with Order and Semantic Consistency

KG-to-text generation task has been a hot research topic since the first dataset WebNLG was proposed Gardent et al. (2017). Recent works for solving this task have two main categories. One category is using graph neural networks Marcheggiani and Perez-Beltrachini (2018); Ribeiro et al. (2020b); Li et al. (2021); Cheng et al. (2020) or graph transformers Koncel-Kedziorski et al. (2019) to directly capture the graph structure information and decode into sentences. E.g., the recent work Ribeiro et al. (2020b) forms four different encoder architectures for combining local and global node contexts. ENT-DESC Cheng et al. (2020) introduces multi-graph structure to better aggregate knowledge information. The other category is first linearizing the KG Ribeiro et al. (2020a); Yang et al. (2020); Gardent et al. (2017); Hoyle et al. (2020) and then formulating a sequence-to-sequence generation task with lineared KG nodes as input to generate sentences. E.g., Distiawan et al. Distiawan et al. (2018) utilize a fixed tree traversal order to directly flatten the KG into a linearized representation. The structural information of the KG is not preserved when generating the KG order. In this paper, we follow the second general pipeline with local structure encoded in the order generation process.

A line of KG-to-text models Ribeiro et al. (2020a); Braude et al. (2021) tries to generate sentences conditioned on the KG order, where the order generation is especially important as different KG description orders may result in various generated texts. Most previous works focus on graph traversal-based approaches Flanigan et al. (2016); Gardent et al. (2017); Ke et al. (2021); Li et al. (2021) for KG order generation. Li et al. (2021) proposes to use a relation-biased breadth first search (RBFS) strategy to linearize the KG. These previous graph traversal-based approaches are heuristic without considering the word sequence information in the ground-truth sentences. Inspired by previous work Cornia et al. (2019) which generates image caption with optimal object description order, we extract the sequence information from the ground-truth sentences as supervision and train one order prediction module to generate optimal order. Moreover, our order prediction considers the local graph structure in triplet.

POS tags have been used in various text generation (e.g. image captioning and video captioning) to impose the syntactic constraint. In the neural text generation work Yang and Wan (2021), the authors propose to use POS guided softmax function as the linguistic prior information for modeling the posterior probabilities of next-POS and next-token, in order to increase text generation diversity. In the image caption, Bugliarello et al. Bugliarello and Elliott (2021) claim that incorporating POS tag information in the sentence generation process consistently improves the quality of the generated text. In video captioning, Hou et al. Hou et al. (2019) propose to define the templates of POS tag sequences to represent the syntactic structure of the generated text. In KG-to-text generation task, we not only use the POS tags to ensure the syntactic correctness of the generated text, but also use the POS tags to constrain the positions to copy words from KGs.

Pre-trained language models (PLMs) on massive corpora, such as BERT Devlin et al. (2018), BART Lewis et al. (2019) and T5 Raffel et al. (2019), have achieved superior performance in various natural language generation tasks, including KG-to-text generation task Ribeiro et al. (2020a); Peters et al. (2019); Ke et al. (2021). Ribeiro et al. Ribeiro et al. (2020a) leverage the generation ability of PLMs and use the linearized KG as input to generate texts. We also use the PLM in our method to guarantee the model generalization ability, and at the same time, design extra order prediction and context scoring components to maintain the semantic consistency.

Given the input KG $\mathcal{G}$, which is composed of $\{(h_{1},r_{1},t_{1}),\cdots,(h_{n},r_{n},t_{n})|h_{*},t_{*}\in\mathcal{E},r_{*}\in\mathcal{R}$}, where $\mathcal{E}$ denotes the entity set and $\mathcal{R}$ represents the relation set, the KG-to-text generation task aims to generate a fluent and reasonable text sequence $\mathcal{T}=<t_{1},t_{2},\cdots,t_{k}>(t_{k}\in\mathcal{V})$, where $\mathcal{V}$ denotes the vocabulary. In this paper, we follow the general pipeline Ribeiro et al. (2020a); Ke et al. (2021); Ribeiro et al. (2021) of linearizing the input KG into sequence $\mathcal{G}_{linear}=<g_{1},g_{2},\cdots,g_{m}>$ consisting of $m$ tokens, and then decoding the KG token sequence into sentences.

We propose S-OSC, illustrated in Figure 2, which consists of two main components: one learning-based sorting network for KG, and one copy or prediction selection module for decoding each word in the sentences. The sorting network generates the optimal description sequence for the input KG. Based on the KG order sequence, the sentence decoder generates each word with a certain probability predicted by the copy or prediction selection module to replace the decoded word with the word in the KG. Thus, the model can maintain the word authenticity in the generated sentence compared to the KG. A key innovation in our learning-based sorting network is to utilize the sequence information extracted from the ground-truth sentence to directly supervise the optimal sequence prediction instead of heuristic search in the KG without considering the description sequence prior. Our copy or prediction selection module for decoding each word in the sentences incorporates additional POS syntactic constraint and semantic context consistency scoring function evaluating the semantic fitness of each word in its sliding windows with various sizes. The details of each module in our S-OSC model are illustrated in the following sections.

In this paper, two benchmarks: WebNLG Li et al. (2021) and DART Nan et al. (2020) are utilized to evaluate the performance of our S-OSC model.

Following the previous works Ribeiro et al. (2020a, b); Li et al. (2021) on WebNLG dataset, we adopt four automatic language evaluation metrics for WebNLG dataset, i.e., BLEU-4 Papineni et al. (2002), CIDEr Vedantam et al. (2015), Chrf++ Popović (2015) and ROUGE-L Lin (2004). Following previous works Nan et al. (2020) on DART dataset, in addition to BLEU-4 Papineni et al. (2002), we use four additional automatic evaluation metrics, i.e., METEOR Banerjee and Lavie (2005), MoverScore Zhao et al. (2019), BERTScore Zhang et al. (2019) and BLEURT Sellam et al. (2020).

In the sorting network, the fixed KG order length $N$ is set to 8 and 10 for WebNLG and DART, respectively. The POS generator operates on the POS sequences parsed from ground-truth sentences via NLTK, and trains from BART-Base pretrained model Lewis et al. (2019). For Word-Encoder-Decoder, we follow the code in JointGT Ke et al. (2021) and utilize Bart-Base with self-attention Shaw et al. (2018). The beam search size for generating sentences in inference time is set to 5. We optimize all the parameters under the supervision of the total loss in Eq. 9 using the OpenAI AdamW optimizer. The loss weights $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ in total training loss (Eq.9) are set to 0.7, 0.4, and 0.3, respectively.

We compare our S-OSC model with other state-of-the-art methods on WebNLG and DART. Results on WebNLG are shown in Table 1. It can be observed from Table 1 that our S-OSC model outperforms all the previous methods in three evaluation metrics, i.e., B-4, R-L and Chrf++, except for the CIDEr value being in the second place compared to the best performing CIDEr result reported by the model in Li et al. Li et al. (2021). Note that our S-OSC model contains one learning-based sorting network supervised by the ground-truth order extracted from KG and the ground-truth sentence. We also report our model’s results under the ground-truth order during inference time (denoted as "S-OSC-GT") which serves as the upper bound for our sorting performance. From the comparison of our model with predicted order "S-OSC" and our model with ground-truth order "S-OSC-GT", we can see that our model results with predicted order are close to the results with ground-truth order with the result gaps less than 1.2 points in most metrics, which shows the advantage of our learning-based sorting network for generating KG description order.

Results on DART are shown in Table 2. From previous models’ results, we can see that pre-training on the large corpus (e.g., "T5-Large" and "Bart-Large") brings significant result improvement compared to "Seq2Seq-Att" and "End-to-End Transformer". Our S-OSC model further achieves new state-of-the-art results in all five metrics compared to these pre-training based models (e.g., "T5-Large") with B-4 score improved by 11.35 points, METEOR score improved by 0.03 points, Moverscore improved by 0.1 points, BERTScore improved by 0.01 points, and BLEURT improved by 0.05 points. JointGT outperforms all the previous baseline models. Compared with JointGT Ke et al. (2021), our model can also obtain an improvement of 7.77 points in B-4 score. This result also validates the effectiveness of our model in improving the fluentness and authenticity of the generated sentences with the proposed learning-based sorting network and consistency enhancement under the POS syntactic and semantic context constraints. Similar to the results on WebNLG, our model S-OSC with predicted order can achieve the performance close to that with ground-truth order.

In this section, we conduct extensive experiments on WebNLG to evaluate various factors in the word copy or prediction during sentence generation, e.g., the POS generator and semantic context (SC) scoring. We compare our full S-OSC model with the following variations: without word copy component and just use the predicted word from Word Decoder at each time step (w/o CP), without POS information in copy probability prediction (w/o POS), without semantic context scoring in copy probability prediction (w/o SC), without POS tag information and semantic context scoring in copy probability prediction (w/o POS and SC) and relying on the last hidden states to predict the copy probability as in previous methods Koncel-Kedziorski et al. (2019); See et al. (2017).

From the results in Table 3, we can observe that: (1) By removing the word copy component and directly taking the predicted word from Word Decoder at each time step (w/o CP), the results drop significantly by 3.95 points in B-4, 2.2 points in R-L, 0.41 points in CIDEr and 3 points in Chrf++. (2) We then consider incorporating the copy prediction component, but just rely on the last hidden states to predict the copy probability as in previous methods Koncel-Kedziorski et al. (2019); See et al. (2017) without POS tag information and semantic context scoring in copy probability prediction (w/o POS and SC). The results improve slightly in all the metrics compared with that of "w/o CP". (3) We then evaluate the effectiveness of POS syntactic information and semantic context scores in improving the quality of the generated sentences by removing each of them at a time.

Compared to full S-OSC model, without POS information in copy probability prediction (w/o POS), the results drop by 2.1 points in B-4, 1.5 points in R-L, 0.3 points in CIDEr and 2.4 points in Chrf++. Compared to full S-OSC model, without semantic context scoring in copy probability prediction (w/o SC), the results drop by 2 points in B-4, 1.1 points in R-L, 0.2 points in CIDEr and 1.9 points in Chrf++. Both "w/o POS" and "w/o SC" improve consistently compared to previous copy policy in "w/o POS and SC".

To further reveal more details about our model, we conduct extra ablation studies regarding the following questions. (1) How does triplet structure encoding in the sorting network help the model compared with direct node encoding without triplet structure context? (2) How does our model perform for different triplet numbers in KG? (3) How does the sliding window size in semantic context scoring function affect the model performance?

We conduct the human evaluation on WebNLG to further evaluate the generated text. In this paper, we adopt the same human evaluation criteria as Chen et al. Chen et al. (2019), i.e., Factual correctness including Supp (counting the number of facts that co-exist in the KG and generated text) and Cont (counting the facts in the generated texts missing from or contradicting with KG), Language naturalness including NF (evaluating the accuracy and fluentness of generated sentences). In addition to using absolute score at 5-point metric in NF, we also use relative ranking scores (termed NA). We randomly select 100 knowledge graphs for human evaluation. Five native English speakers volunteer to score all the 100 knowledge graphs. Table 5 reports the results²²2 Cohen’s kappa coefficients for the first two factors are 0.79, 0.83,0.75,0.76.. We can observe that: the generated texts of our S-OSC model are more authentic and consistent with KG than the method in Li et al. Li et al. (2021).

We show two qualitative examples from WebNLG and DART in Figure 7. From the example of WebNLG, we can see that the previous method in Li et al. Li et al. (2021) generates the wrong description order with node 3 and node 2 exchanged positions. This causes the generated sentence from Li et al. Li et al. (2021) containing the wrong text "jusuf kalla" noted in red color and missing the text "Malaysian Indian", while our S-OSC method generates the right order and consistent sentences. In the example from DART, though our S-OSC model generates relative inferior order with node 2 and node 5 exchanged positions, relying on our strong sentence generator with syntactic constraint and semantic consistency constraint, our model is able to generate semantic consistent sentence to the KG with the right words copied from KG. However, in this example, Bart-Large Ribeiro et al. (2020a) still misses some key word description from KG, e.g., "the final fight on 10/28/2014" and "Launch Pad 0", though conditioned on the right predicted order, showing the inferior performance of their copy or prediction module in sentence generation.