COARSE-TO-CAREFUL: SEEKING SEMANTIC-RELATED KNOWLEDGE FOR OPEN-DOMAIN COMMONSENSE QUESTION ANSWERING

Open-domain CommonSense Question Answering (OpenCSQA) is a challenging research topic in AI, which aims at evaluating that if a machine can give the correct answer with the ability to manipulate knowledge like human beings. Questions in OpenCSQA require to be answered with the support of rich background commonsense knowledge. Though it is trivial for human to solve such questions merely based on the question, it is awkward for machines to figure out.

It is notable that emerging Pre-trained Language Models (PLMs), such as BERT [1], can achieve remarkable success on a variety of QA tasks. The outstanding performance of PLMs benefits from the large scale textual corpus [1, 2]. However, commonsense knowledge is often not accessible in the textual form [3]. PLMs also struggle to answer questions that require commonsense knowledge beyond the given textual information from the question. Specifically, they make the judgment with their encapsulated knowledge in unconsciousness manner and emphasize the textual matching between words in the question and answer [4].

There have been several attempts on OpenCSQA regarding leveraging external structured knowledge. Some only incorporate the entity [5] or the relation between words [6, 7] from the knowledge base while ignoring the structural information. Due to the absence of the background document in the OpenCSQA task, it is hard to restrain the boundary of the knowledge to be grounded. Most works search and rank the selected knowledge triples based on the heuristic occurrence score [8, 9] or the score function of the knowledge representation learning [10]. Also, previous works pay no attention to the contextual semantic relevance of the extracted knowledge with the question. Massive external knowledge will bring in noisy and misleading information, and consequently, lead to deviating out of the content of the current question. Therefore, how to introduce knowledge semantically related to questions remains substantially challenging.

To address this issue, we propose a Semantic-driven Knowledge-aware Question Answering (SEEK-QA) framework to manipulate the injection of the relevant knowledge triples in a coarse-to-careful fashion. The advancement of the proposed SEEK-QA framework lies in two folds. First, we design a Semantic Monitoring Knowledge Tailoring (SONAR) strategy which can constrain the selected knowledge triples with the global coarse semantic of the question. It not only denoises irrelevant knowledge input but also benefits upgrading the computing efficiency of the follow-up model. Second, we develop Semantic-aware Knowledge Fetching (SKETCH) module, which capitalizes the careful semantic of question to measure the semantic relevance between the question and knowledge triplets in a hierarchical way.

In this paper, we focus on the typical multiple-choice question answering benchmark CommonsenseQA [11] and consider the structured knowledge base ConceptNet [12] as the external knowledge source. Experiment results demonstrate that SONAR and SKETCH can boost the performance compared with strong baselines. And we exhibit that how injected knowledge influences the judgment of model.

In the CommonsenseQA task, given a question $\mathbf{Q}$ and a set of candidate answers $\mathcal{A}$, the model $\mathcal{M}$ is asked to select the only one correct answer $\mathbf{A}^{\ast}$ from the candidate set. When involving the external knowledge $\mathcal{K}$, the goal of the model can be formalize as: $\mathbf{A}^{\ast}=\mathop{\arg\max}_{A_{i}\in\mathcal{A}}\mathcal{M}(A_{i}|\mathbf{Q},\mathcal{K}_{i})$, where $\mathcal{K}_{i}$ stands for requisite knowledge extracted from ConceptNet of the $i$-th candidate answer. Each knowledge in ConceptNet is indicated in a triple $k=({cn}^{head},r,{cn}^{tail})$, where $cn^{head}$, $cn^{tail}$ are concepts and $r$ is relation between them. We denote extracted knowledge graph of $\mathcal{K}_{i}$ as $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$, where $\mathcal{V}$ represents concepts, and $\mathcal{E}$ stands for relations.

Our SEEK-QA framework, as shown in Fig.1, follows the retrieve-then-answering paradigm that contains two phases: a) retrieve: using the SONAR strategy (Section 3) to filter irrelevant knowledge triples. b) answering: utilizing a QA model equipped with SKETCH module to select the correct answer for a given question.

The main QA model consists of three stages:

Contextual Encoding: We utilize a PLM as the contextual encoder. It takes question $\mathbf{Q}$ and each candidate answer $A_{i}$ ($[\texttt{CLS}]\mathbf{Q}[\texttt{SEP}]A_{i}[\texttt{SEP}]$) as input. The contextual representation of the $t$-th token is denoted as $h_{t}^{c}\in\mathbb{R}^{d_{h}}$, and $d_{h}$ is the hidden dimension of the PLMs encoder. We treat the output of [CLS] ($h_{0}^{c}\in\mathbb{R}^{d_{h}}$) as the global semantic representation of the question-answer pair.

Knowledge Module: The knowledge module, SKETCH, is the core of QA model and will be elaborated in Section 4. It is responsible for encoding and fusing knowledge and generating integrated information $I$ for making final prediction.

Answer Scoring: The last step is to calculate the final score for each candidate answer as: $\mbox{score}(A_{i}|\mathbf{Q},\mathcal{K}_{i})=\mbox{MLP}(I)$, where $I$ will be presented in Section 4.2. The final probability for the candidate answer $A_{i}$ to be selected can be formulated as: $\mathbf{P}(A_{i}|\mathbf{Q},\mathcal{K}_{i})=\frac{\exp\left(\mbox{score}(A_{i})\right)}{\sum_{i^{\prime}=1}^{\left|\mathcal{A}\right|}\exp\left(\mbox{score}(A_{i^{\prime}})\right)}$.

Here, we use cross entropy to calculate the losses between predictions of the model and the ground-truth answer labels.

In this section, we present the SONAR strategy for knowledge selecting and filtering.

During knowledge selection, we first convert question and answer into a list of concepts separately, indicated as $\mathcal{C}n^{q}$ and $\mathcal{C}n^{a}$. We denote the $m$-th concept in question as $cn^{q}_{m}\in\mathcal{C}n^{q}$ and $n$-th concept in answer as $cn^{a}_{n}\in\mathcal{C}n^{a}$. Next, we extract the link paths between question concept and answer concept with the maximal hop to $3$. In the meanwhile, we extract link paths between two arbitrary question concepts. After the knowledge selection, we obtain multiple groups of link paths between each pair of concepts $\langle cn^{q}_{m},cn^{a}_{n}\rangle$.

During the knowledge filtration, we first adopt the knowledge embedding algorithm, i.e., TransE [13], on ConceptNet to obtain the knowledge embedding of concepts and relations. Next, the SONAR rates link paths from three aspects: (1) On the link aspect, we follow operation in [10] that defines the score of one link path as the product of validity score of each triple. (2) On the concept aspect, SONAR represents question with GloVe embedding and operates mean pooling over sentence length to get the question representation. Meanwhile, the concept is represented with its GloVe embedding. SONAR strategy utilizes cosine similarity to measure the semantic relevance score between the question representation and each concept representation, and the final link path score is defined as the average of all concepts’ scores. (3) On the relation aspect, SONAR performs the same operation as the concepts, while the relation is represented with its embedding acquired from the knowledge embedding algorithm. Finally, SONAR removes the link path that does not meet up the bond of thresholds on all three scores and preserves the link path satisfying at least two thresholds. After filtering, the reserved link paths are gathered into a knowledge sub-graph $\mathcal{G}$ that will be taken as the input for the follow-up QA model.

In this section, we elaborate on the workflow of the knowledge module, SKETCH, over the acquired knowledge.

(1) Ablation on the Range of External Knoweledge: We compare the impact of ranges of introduced knowledge extracting with SONAR strategy or filtering knowledge without contextual semantic constraints. As shown in Table 3, filtering knowledge triplets with SONAR results in a better performance comparing with removing semantic constraint. We argue that decreasing the input of irrelevant noisy knowledge can make QA model more focused. Comparing with removing filtering¹¹1i.e. All extracted knowledge is taken as the input to QA model., SONAR strategy also shows an advantage on a wider range of knowledge (Hop=$2$,$1$). That reveals SONAR strategy can help to attract more suitable knowledge for the task and be of benefit to the task performance. (2) Ablation on SKETCH Component: As shown in Table 4, the performance first increases then decays along with the increasing of GATL, and drops when we remove the GAT knowledge encoding. We can assume that GAT can facilitate model to gather information from structured knowledge. When SLS and SUS operations are removed respectively, the performance both gets a decline. It indicates semantic relevance strength helps to distinguish the worth of knowledge triplets. (3) Case Study: As shown in Fig.3, RoBERTa fails in these cases, but our approach makes a correct prediction with the support of the closely related knowledge links. It is notable that the amount of links is greatly reduced with the SONAR strategy while core knowledge links are held. With such requisite knowledge taken as input, the SKETCH model figure out correct answers with great confidence.

In this work, we propose the Semantic-driven Knowledge-aware Question Answering (SEEK-QA) framework, which can manipulate the injection of external structured knowledge in the light of a coarse-to-careful fashion. Experiment results demonstrate the effectiveness of the proposed approach.

We thank the reviewers for their advice. This work was supported by in part by the National Key Research and Development Program of China under Grant No.2016YFB0801003.