Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models

To conduct the analysis experiments, we select QA-GNN (Yasunaga et al., 2021), a representative approach that integrates PTM with GNN for the commonsense QA task, as the studied model. We adopt the CommonsenseQA (Talmor et al., 2019) and OpenBookQA Mihaylov et al. (2018), two of the most widely used commonsense reasoning benchmarks, for evaluation, with the same data split setting in Lin et al. (2019).

We perform two analysis experiments: one examines the effect of the commonsense KG encoder, and the other one examines the effect of different features in the commonsense KG encoder. To be specific, the two experiments focus on two key questions about commonsense reasoning: (1) what is the effect of the commonsense KG encoder on PTMs? (2) what is the key information within the commonsense KG encoder?

Next, we conduct the experiments and present our findings of commonsense reasoning.

The comparison results are shown in Figure 1. As we can see, using the predictions solely based on the GNN module (i.e., GNN-Pred) can only answer a relatively minor proportion of the questions (no more than 60% in CommonsenseQA). As a comparison, when trained independently (i.e., PTM-Only) or jointly with the GNN module (i.e., PTM-Pred), the PTM module can answer a large proportion of the questions (at least 70% in CommonsenseQA). Furthermore, the incorporation of the GNN encoder is useful to improve the performance of PTMs (PTM-Only v.s. QAGNN). These results show that:

$\bullet$ In the joint PTM-GNN approach, PTM contributes the most to the commonsense reasoning task, which is the key to the reasoning performance.

$\bullet$ Commonsense KG encoder is incapable of performing effective reasoning independently, but can enhance PTM as the auxiliary role.

As shown in Figure 2, we surprisingly find that even after reducing the dimension of node embeddings to 1, the performance of the GNN encoder can be still improved. These results show that node features are not the key information utilized by the GNN encoder. In contrast, removing a considerable proportion of links significantly reduces the performance. From these observations, we can conclude that: The relation features from the CSKG are indeed the key knowledge information that is actually needed by the KG encoder.

Since relation features are shown useful to improve the performance of commonsense reasoning, we consider extracting relation features for better capturing the knowledge semantics from the CSKG. Inspired by KG reasoning studies Lin et al. (2018); Feng et al. (2020b), we construct multi-hop relation paths that connect question nodes with answer candidate nodes on the CSKG to capture the higher-order semantic relatedness among them.

Formally, given the commonsense subgraph $\mathcal{G}^{q,c_{i}}$ for the question $q$ and the answer candidate $c_{i}$, we first extract a set of relation paths within $k$ hops that connect a question concept node $v_{q}\in\mathcal{V}_{q}$ and an answer concept node $v_{c_{i}}\in\mathcal{V}_{c_{i}}$, denoted as $\mathcal{P}^{q,c_{i}}$. Specifically, a path $p\in\mathcal{P}^{q,c_{i}}$ can be represented as a sequence of nodes and relations as $p=\{v_{1},r_{1},\cdots,r_{k-1},v_{k}\}$. Based on the empirical findings in Section 3, we consider a simplified representation for relation paths that removes node IDs but only keeps the relations on a path. To keep the role of each node, we replace a node ID by a three-valued type, indicating this node belongs to a question node (0), answer node (1) or others (2). In this way, a path $p$ can be represented by $p=\{t_{v_{1}},r_{1},t_{v_{2}},r_{2},\cdots,r_{k-1},t_{v_{k}}\}$, where $t_{v}$ is the role type of node $v$. Since we remove explicit node IDs, our model can concentrate on more essential relation features.

Based on the above method, for a question $q$ and an answer candidate $c_{i}$, we extract all the simplified relation paths and count their frequencies among all the paths. We use $\mathcal{F}^{q,c_{i}}=\{\langle p_{j},f_{j}\rangle\}$ to denote all the paths for the question $q$ and the answer candidate $c_{i}$, where each entry consists of the $j$-th path $p_{j}$ and its frequency $f_{j}$. Unlike prior approaches (e.g., QA-GNN), we use such very simple features of relation paths from CSKGs to improve the reasoning capacity of PTMs.

Our KG encoder is built on a full MLP architecture based on simplified relation path features, consisting of a path encoder and a feature aggregator.

In this part, we integrate the above KG encoder with the PTM for commonsense reasoning.

During the training stage, we optimize the parameters of the whole model (including both the PTM and KG encoder) with the cross entropy loss between the predictions and the ground-truth answer (based on the probability distribution $\{\text{Pr}(c_{i}|q)\}_{i=1}^{n}$ ). During inference, we first compute the probability score $\text{Pr}(c_{i}|q)$ for each answer candidate, and then select the highest one as the predicted answer.

For the task of commonsense reasoning, it has become a common approach by integrating PTM with an external KG encoder based on CSKGs. The major difference among these methods (including our approach) lies in the design of the KG encoder. Next, we compare these variants for the KG encoder.

We summarize the comparison between our KG encoder and representative KG encoders in Table 1. We can see that, our approach no longer lies in the node embeddings and the structure of GNNs. Instead, we mainly utilize relation paths as the features of the KG encoder, which is built on a simple MLP-based architecture. Therefore, the number of the model parameters involved in our KG encoder is much smaller than those of existing KG encoders. As will be shown in Section 5, our KG encoder yields better or at least comparable performance compared with existing GNN-based encoders, based on the same configuration for PTMs.

Specifically, our approach can largely reduce the computational costs for encoding the CSKG. For our approach, we need to extract the relation paths from question nodes to all the answer candidate nodes on the CSKG, and it can be efficiently fulfilled via a $k$-hop constrained Depth-First Search Tarjan (1972), which can be pre-computed in offline processing. When the relation paths have been extracted, it is efficient to encode these paths with our MLP architecture. Such a process can be easily paralleled or accelerated by optimized matrix multiplication. In contrast, existing GNN-based encoders rely on iterative propagation and aggregation on the entire subgraph, which takes a much larger computational time cost.

In this part, we introduce the experimental setup.

$\bullet$ CommonsenseQA (Talmor et al., 2019) is a 5-way multiple-choice QA dataset. It is created based on ConceptNet (Speer et al., 2017).

$\bullet$ OpenBookQA (Mihaylov et al., 2018) is a 4-way multiple-choice QA dataset about elementary science questions to evaluate the science commonsense knowledge.

$\bullet$ SocialIQA (Sap et al., 2019b) is a 3-way multiple-choice QA dataset to evaluate the understanding of social commonsense knowledge.

$\bullet$ PIQA (Bisk et al., 2020) is a binary-choice QA dataset about physical commonsense.

$\bullet$ CoPA (Roemmele et al., 2011) is a commonsense inference dataset, to select the most plausible alternative with the causal relation to the premise.

$\bullet$ Fine-tuned PTM directly fine-tunes a PTM without using any CSKG. We use RoBERTa-large Liu et al. (2019) for all tasks. Additionally, we also use BERT-large (Devlin et al., 2019) and AristoRoBERTa (Clark et al., 2020a) for OpenBookQA to evaluate the generality of our KG-encoder.

$\bullet$ PTM+GNN models integrate PTM with additional GNN-based KG encoders. Based on the same PTM (the above baseline), we consider five variants with different KG encoders: (1) Relation Network (RN) (Santoro et al., 2017) using a relational reasoning structure over the CSKG; (2) GcoAttn (Lin et al., 2019) using a graph concept attention model to aggregate entity information from the CSKG; (3) RGCN (Schlichtkrull et al., 2018) extending the GCN with relation-specific weights; (4) MHGRN (Feng et al., 2020a) using a GNN architecture reasoning over the CSKG that unifies both GNNs and path-based models; (5) QA-GNN (Yasunaga et al., 2021) using a GAT to perform jointly reasoning over the CSKG.

For all these methods, we adopt the same architecture and configuration for the PTM, so that we can examine the effect of different KG encoders.

We implement all PTMs based on HuggingFace Transformers Wolf et al. (2020). For all the baselines, we keep the common hyper-parameters as identical as possible and set their special hyper-parameters following the suggestions from the original papers. In our approach, we extract the relation paths with no more than 2 hops between the concept nodes from the question and the answer candidate. We tune the hidden dimension of MLPs from the path encoder in {32, 64, 100}, and the batch size in {32, 48, 60, 120}. The parameters of the model are optimized by RAdam (Liu et al., 2020), and the learning rate of the PTM and the KG encoder is also tuned in {1$e$-4, 1$e$-5, 2$e$-5} and {1$e$-3, 1$e$-2}, respectively. To accelerate the training process, we don’t incorporate Dropout regularization in our model. All the above hyper-parameters are tuned on the development set.

Following previous works Yasunaga et al. (2021); Wang et al. (2021a), we take the results on CommonsenseQA and OpenBookQA as the main experiments to compare different methods. In order to test their robustness to data sparsity, we examine the performance under five different proportions of training data, i.e., $\{5\%,10\%,20\%,50\%,80\%,100\%\}$.

Comparing the results under the full-data setting (i.e., 100% training data), we can see that all the PTM+GNN methods perform better than vanilla PTM (i.e., RoBERTa-large). It indicates that the KG encoder on the CSKG is able to incorporate useful knowledge information to improve PTMs on commonsense reasoning tasks. Additionally, among all the PTM+GNN baselines, QA-GNN performs the best. The major reason is that QA-GNN uses the PTM to estimate the importance of KG nodes and connects the QA context and the CSKG to form a joint graph, which is helpful to improve the reasoning ability on the CSKG. Finally, our method consistently outperforms all the baselines. Our approach incorporates a lightweight MLP architecture as the KG encoder with relation paths as features. It reduces the parameter redundancy of the KG encoder and focuses on the most essential features for reasoning, i.e., semantic relation paths. Such an approach is effective to enhance the commonsense reasoning capacity of PTMs.

Comparing the results under different sparsity ratios of training data, we can see that the performance substantially drops when the size of training data is reduced. While, our method performs consistently better than all baselines. It is because that our KG encoder consists of significantly fewer parameters than those of the baselines, which reduces the risk of overfitting and endows our approach with better robustness in data scarcity scenarios.

The major contribution of our approach lies in the lightweight KG encoder, which can be also used to enhance the commonsense reasoning capacity of various PTMs. To validate it, we examine the performance of our KG encoder when integrated with two other PTMs, i.e., BERT-large and AristoRoBERTa, on OpenBookQA dataset.

As shown in Table 5, the BERT-large and AristoRoBERTa enhanced by our KG encoder perform better than original PTMs. Especially, our KG encoder can improve the performance of AristoRoBERTa by a large margin (with 8.73% improvement). These results show that our KG encoder is a general method to improve PTMs for commonsense reasoning. In contrast, when adapting other KG encoders to these two PTMs, the performance decreases in most cases. It is mainly because these KG encoders have complicated architectures, which may not be easily adapted to other PTMs.

For hyper-parameter analysis, we study the hidden dimension size of the MLP in the path encoder. Concretely, we evaluate our model with varying values of the hidden dimension size on CommonsenseQA and OpenBookQA datasets using RoBERTa-large model. The results are shown in Figure 4. We can see that with the increase of the hidden dimension size, the performance improves at first and then drops to some extent. The possible reason lies in two aspects. On the one hand, a too small hidden dimension size makes the path encoder hard to represent sufficient information from relation paths for commonsense reasoning. On the other hand, a larger hidden dimension size enlarges the parameter number of our KG encoder, which increases the risk of overfitting that may cause performance degradation.

We propose a rather simple KG encoder to effectively utilize the relation features from the CSKG, which first computes the feature values of the relation paths and then aggregates these values as the confidence score of the question and choice from the perspective of KG. In this way, we can generate a table in advance that maps each type of relation path into its feature value that reflects its contribution to the confidence score. Based on this table, it is convenient to directly judge the importance of the relation path and quickly assess the confidence about if the choice is the answer to the question from the perspective of KG. Figure 5 shows some path-value examples on CommonsenseQA dataset. As we can see, the path with a higher value indeed provide more persuasive evidence (e.g., causes and capableof) that indicates the choice is more likely to be the answer to the question. In contrast, the path with a lower value usually represents an ambiguous relationship (e.g., relatedto), which contributes less to the judge of whether the choice is the answer.