Automatic Knowledge Augmentation for Generative Commonsense Reasoning

CommonGen [9] is a challenge in which the task is to generate a sentence that makes sense with an input concept-set by leveraging generative commonsense reasoning. However, it is difficult to derive much commonsense knowledge for the given concept-sets, such as compositional generalization and relational reasoning, from the 67,389 sentences in the CommonGen training set [6, 5]. Additionally, most reference sentences consist of superficial scene descriptions and do not represent any specific relational knowledge between the concepts [9].

To address these issues, we propose a data-centric approach that augments prior knowledge with an additional pre-training session. Our method uses a trained generator that produces semi-golden sentences as patterns to guide generative commonsense reasoning [8, 16]. As the generator, we utilize BART [7], which has the highest average percentage of input concepts that exist in the output sentences in the CommonGen task ($97.35\%$). The generator learns to reconstruct sentences using concepts (e.g., verbs and nouns) extracted from sentences in Wikipedia articles, which have high coverage. Semi-golden sentences with concrete descriptions about the input concept-set or concept-pair set are used in an additional pre-training session to enhance the prior knowledge of the generative language model.

Automatic knowledge generation substantially reduces human effort and ensures the quality of machine-generated data using the results of performance improvements. It can also extend the relational information of given concept-sets, including commonsense knowledge for the intended input query. Moreover, the improvements in performance in auto-regressive models (i.e., a decoder only) [12] and sequence-to-sequence language model [13] (i.e., an encoder–decoder), which have different architectures, demonstrate the strong generalization and model-agnostic nature of this approach.

In this section, we present the process of data construction and experiments using a generator.

First, we extracted 154,892 sentences from a Wikipedia data dump (2021-03-01) that match more than two input concepts via ElasticSearch [4] and BM25 [14, 3]. From the extracted sentences $y$, we defined concept-set $C$, which includes a verb or noun through part-of-speech (POS) tagging. Second, the generator was trained with the reconstructed data using $C=y$ and produced a semi-golden sentence $y^{\prime}$ for an input concept-set $C^{\prime}$. Third, 59,125 concept-pairs and 32,471 unique concept-sets (of size 3 to 5) were extracted from the CommonGen training set and the generator used them as an input sequence to create a semi-golden sentence. Fourth, extracted concept-sets and generated semi-golden sentences were used as pre-training data in the form of $C^{\prime}=y^{\prime}$ for two generative language models, as presented in Table 1. We conducted a case study to demonstrate the impact of our method with GPT2 [12] and T5 [13]. To evaluate the performance, we used the n-gram overlapping metrics BLEU [11], ROUGE [10], and METEOR [2] as well as the concept matching metrics CIDEr [15] and SPICE [1].

As shown in Table 1, all data augmentation cases lead to performance improvements with respect to all evaluation metrics. The results also indicate that there is a difference in performance depending on the combination of input concepts and target models. Augmented concept-pairs include concrete relational knowledge between two concepts and concept-sets have commonsense knowledge for generating sentences by compositionality. GPT2 obtains the best performance improvement when learning both types of machine-generated knowledge through the augmented dataset. In contrast, T5 yields no significant performance difference depending on the form of the augmented dataset, although learning commonsense knowledge with compositionality is slightly more effective. This result shows that there is a variance in the degree of improvement in prior knowledge in a specific area when the model architecture and dataset used for pre-training each language model differ. This experimental result demonstrates that our data-centric approach generates better semi-golden sentences for use as an augmented dataset. It is also possible to produce a machine-generated dataset in the intended format to compensate for the insufficient prior knowledge of each model. Furthermore, this approach is a model-agnostic method and has a positive impact on both auto-regressive and sequence-to-sequence architectures.

We demonstrated how to use natural language generation for data-centric research to reduce the cost of leveraging information retrieval and building a human-annotated dataset. Automatic knowledge augmentation for generative commonsense reasoning is also not limited to a particular model or dataset, but can be applied in different forms as needed. In future work, this method will be highly valuable because it can be extended to other downstream tasks that require commonsense knowledge as supporting evidence.

This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2018-0-01405) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation) and IITP grant funded by the Korea government(MSIT) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques) and Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(NRF-2021R1A6A1A03045425). Heuiseok Lim^† is a corresponding author.