CLICKER: Attention-Based Cross-Lingual Commonsense Knowledge Transfer

The CSR task aims to select one from multiple choices that are most reasonable in commonsense given the previous statement or question. For example, the plausible choice of answer to “What is a great place to lay in the sun?” is “beach” rather than “in the basement” or “solar system”. Denoting a set of choices for CSR as $\mathbf{S}^{(j)}$’s, $j\in$ [1,$\dots$, $|C|$], where the number of choices for each input as $|C|$, the goal is to predict the common sense choice:

$\displaystyle\tilde{y}=\text{argmax}_{j}P(y=j|\{\mathbf{S}^{(1)},\dots,\mathbf{S}^{(|C|)}\})$

(1)

Task-adaptive pre-training uses self-attention to enable the XLM-R to learn representations for the CSR task. Specifically, the input is a tokenized utterance, i.e. $\mathbf{S}_{i}=\{\text{[CLS]},q_{i}^{(1)},$ $\dots,q_{i}^{(k)},\text{[SEP]}\}$, where $i\in[1,\dots,N]$, $N$ is the size of the dataset, and $k$ is the length of sequence. A self-attention layer is built on top of the Transformer to obtain attentions toward commonsense knowledge from the Transformer’s pooled output states. The self-attended outputs are optimized by the Cross-Entropy loss through a multiple-choice classifier to select commonsense-reasonable choices. Our model is trained on multilingual CSR datasets containing examples of both English (EN) and German (DE).

In this step, the representation of commonsense knowledge shared across languages is differentiated from the non-commonsense representation using EN and DE datasets in parallel. The inputs are similar to those in Sec 2.2, while inputs with the same semantics in different languages are mapped together. We note here that the parallel datasets are not necessarily restricted to CSR datasets, but can be generalized to any bilingual datasets for mapping semantics of English and the non-English language, e.g. bilingual dictionaries or textbooks.

The output states of the Transformer are pooled and weighted by the self-attention layer followed by a linear projection, being extracted as commonsense embeddings $X_{i}$ and non-commonsense embeddings $\tilde{X_{i}}$, respectively.

$\displaystyle X_{i}=$

$\displaystyle FFN(\text{Attention}(\mathbf{O}_{i}))$

(2)

$\displaystyle\tilde{X_{i}}=$

$\displaystyle FFN(1-\text{Attention}(\mathbf{O}_{i}))$

(3)

where $\mathbf{O}_{i}$ are output hidden states from the last layer of the Transformer for the $i$-th input, and $FFN$ represents a Feed-Forward layer. For brevity, we omit index $i$ in the following equations.

We use $X^{EN}$ and $X^{DE}$ to denote commonsense embeddings of English and German inputs, respectively. Similarly, $\tilde{X}^{EN}$ and $\tilde{X}^{DE}$ represent non-commonsense embeddings. Knowledge mapping is made by measuring the similarities between commonsense embeddings and non-commonsense embeddings. Specifically, we maximize the cosine similarity between English and German embeddings that share the same and valid commonsense knowledge, i.e. $X^{EN_{j^{*}}}$ and $X^{DE_{j^{*}}}$, as in Eq. (4). And we minimize the cosine similarity between $X^{EN_{j^{*}}}$ and $X^{EN_{j}}$, as in Eq. (5). $j^{*}$ is the index of the choice that is reasonable in commonsense, $j\in$ [1,$\dots$, $|C|$] and $j\neq{j^{*}}$. Such that similar commonsense knowledge in both languages is projected into the same position in the semantic representation space.

$$\mathcal{L}_{align}=1-\cos(X^{EN_{j^{*}}},X^{DE_{j^{*}}})$$

(4)

	$\displaystyle\mathcal{L}_{diff}$	$\displaystyle=\sum_{j=1,j\neq{j^{*}}}^{|C|}(\text{max}(0,\cos(X^{EN_{j^{*}}},X^{EN_{j}}))$		(5)
		$\displaystyle+~{}\text{max}(0,\cos(X^{DE_{j^{*}}},X^{DE_{j}})))$

On the other hand, the non-commonsense embeddings represent knowledge unrelated to cross-lingual commonsense. Assuming the correct choice and other incorrect choices associated with the same question share similar non-commonsense knowledge, we maximize the intra-language cosine similarity of non-commonsense embeddings. Moreover, the correct choice of different languages should share the same non-commonsense knowledge so that we maximize inter-language cosine similarity jointly, as defined in Eq. (6).

	$\displaystyle\mathcal{L}_{nc}$	$\displaystyle=\sum_{j=1,j\neq{j^{*}}}^{|C|}(1-\cos(\tilde{X_{i}}^{EN_{j^{*}}},\tilde{X_{i}}^{EN_{j}}))$		(6)
		$\displaystyle+\sum_{j=1,j\neq{j^{*}}}^{|C|}(1-\cos(\tilde{X_{i}}^{DE_{j^{*}}},\tilde{X_{i}}^{DE_{j}}))$
		$\displaystyle+1-\cos(\tilde{X_{i}}^{EN_{j^{*}}},\tilde{X_{i}}^{DE_{j^{*}}})$

All the losses above and the Cross-Entropy loss are optimized as the joint training objective of the cross-lingual CSR. We use output commonsense embeddings $X^{DE_{j^{*}}}$ and $X^{DE_{j}}$ to calculate the Cross-Entropy loss.

Finally, our model is fine-tuned by the training objectives similar to Sec 2.3 for evaluating CSR on the multiple-choice question-answering (QA) and the clause-selection tasks, leveraging parallel CSR datasets of English (EN) and German translated from English (EN_DE) as inputs. Different from previous steps, each input of XLM-R is the concatenation of a question and a choice of answer which are then split into tokens with additional special ones, i.e. $\mathbf{S}_{i}=\{\text{[CLS]},q_{i}^{(1)},\dots,q_{i}^{(m)},\text{[SEP]},\text{[CLS\_Q]},a_{i}^{(1)},\dots,$ $a_{i}^{(n)},\text{[SEP]}\}$, where [CLS_Q] is the beginning special token of the answer spans, $q_{i}$ and $a_{i}$ are tokens of the question and answer, and $m$, $n$ are numbers of question and answer tokens, respectively.

Table 1 shows the test accuracy of baselines and CLICKER models for CSR in German. Different combinations of losses are applied in experiments for optimizing commonsense differentiation. We observe consistent improvements with our three-step framework by extracting commonsense knowledge with self-attentions (i.e. CLICKER - base) on both datasets compared to baselines.

Results show that the align loss further improves the base CLICK model on X-CSQA. And the non-commonsense (nc) loss is proved effective on both datasets. The best performance on X-CSQA is achieved when using the align loss with or without the diff loss, which shows that lining up embeddings in English and German with the same commonsense knowledge dominates the performance of CSR. Besides, the model with align and nc loss is slightly inferior to the model with nc loss only on X-CSQA. On X-CODAH, our CLICK models perform the best with the nc loss which maximizes the cosine similarity of non-commonsense embeddings, improving 1.6% on accuracy.

Our models address the alignment of extracted embeddings with various combinations of objectives. The fact that align+nc loss is not as good as nc loss alone suggests a conflict between aligning the commonsense embeddings and aligning the non-commonsense embeddings. This can be explained as both objectives aiming to maximize the cosine similarity of embeddings, making it harder for the model to discern different commonsense knowledge in them. From the best accuracy achieved on two datasets, we conjecture the quality of commonsense embeddings (optimized by align and diff losses) dominates CSR on X-CSQA, while non-commonsense embeddings (optimized by nc loss) dominates that on X-CODAH. The reason for this may be extracting commonsense knowledge for clause selection in X-CODAH is more challenging than multiple-choice QA in X-CSQA, whereas separating the non-commonsense embeddings help the multiple-choice classifier understand the commonsense portion with less noise. We also observe that using align and nc losses together is not the best practice according to our experiments. It suggests that jointly optimizing both objectives makes it more difficult for the multiple-choice classifier to predict correctly, as correct choices are pushed closer to incorrect ones.

Commonsense v.s. Non-commonsense. To investigate the effectiveness of our learned commonsense embeddings, we evaluate the accuracy of our CLICKER models on the X-CSQA dev set predicted by commonsense embeddings or non-commonsense embeddings. As seen in Table 2, the performance of commonsense embeddings is significantly better than that of non-commonsense embeddings. It is as expected, as our models are trained with cross-lingual CSR objectives to discern commonsense embeddings, while maximizing the similarity of non-commonsense embeddings. Non-commonsense embeddings can induce confusion for CSR, such that combining both embeddings performs worse than using commonsense embeddings only.

Does self-attention imply commonsense knowledge? We assume that self-attentions in our models can appropriately attend to tokens that affect the plausibility of commonsense. Figure 2 is the heatmap of the attention head in the self-attention layer evaluated on an example “What do you need to be before you can dream?” from X-CSQA. It’s noteworthy to see that attention weights are given more to commonsense-related tokens, such as “before”, “dream” and “sleep” tokens. A similar phenomenon is observed on X-CODAH as well. These tokens are weighted to generate commonsense embeddings and help our model improve accuracy and interpretability of reasoning commonsense knowledge.