Attentive Tree-structured Network for Monotonicity Reasoning

After both the premise and the hypothesis are encoded through the tree-LSTM, each tree’s hidden states from the nodes are concatenated into a pair of matrices $H_{p}$ and $H_{h}$ and passed to a self-attentive aggregator. The aggregator contains a multi-hop self-attention mechanism (Lin et al., 2017). A sentence has multiple components such as groups of related words and phrases to form an overall context, especially for long sentences. By performing multiple hops of attention, the model can get multiple attentions that each focus on different parts of the sentence. Given a matrix $H$, the self-attention mechanism performs multiple hops of attention and outputs an annotation matrix $A$ which consists of the weight vector from each hop. $A$ is calculated from a 2-layer multi-layer perceptron (MLP) and a softmax function. Below is the equation to calculate $A$:

The annotation matrix $A$ is then multiplied by the hidden state matrix $H$ to obtain a context matrix: $M=AH$. In the model, there will be a pair of context matrices $M_{p}$ and $M_{h}$. A batch dot product and a tanh function is then applied to the context matrices with a trainable weight to obtain a pair of output $F_{p}$ and $F_{h}$ matrices:

To aggregate $F_{p}$ and $F_{h}$, we follow Conneau et al. (2017)’s generic NLI training scheme, which includes three matching methods: (i) a concatenation of $F_{p}$ and $F_{h}$, (ii) an absolute distance between $F_{p}$ and $F_{h}$, and (iii) an element wise product of $F_{p}$ and $F_{h}$. Results from the three methods are then concatenated to $F_{r}$ as the factor of semantic relation between the two sentences which can measure how close the two vector representations of the sentence pair are in the target space. This relatedness information will help the classifier to determine whether the hypothesis is entailed by the premise.

Six different types of training data are used to train our model. Initially, we used the HELP dataset (Yanaka et al., 2019b) to train our model. HELP is a dataset for learning entailment with lexical and logical phenomena. It embodies a combination of lexical and logical inferences focusing on monotonicity. HELP consists of 36K sentence pairs including those for upward monotone, downward monotone, non-monotone, conjunction, and disjunction. Next we trained our model with the Multi-Genre NLI Corpus (MNLI) dataset (Williams et al., 2018). MNLI contains 433k pairs of sentences annotated with textual entailment information. That dataset covers a wide range of genres of spoken and written language. The majority of the training examples in that dataset is upward monotone. In order to provide more balanced training data, we combined a subset of the MNLI dataset with the HELP dataset to reduce the effect of the large number of downward monotone examples in the HELP dataset, we call this combined training data HELP+SubMNLI. The fourth training data contains both the HELP+SubMNLI training data and the training set for simple monotonicity from Richardson et al. (2019)’s Semantic Fragments. The fifth training data contains both the HELP+SubMNLI training data and the training set for hard monotonicity from Semantic Fragments. Finally, the last training data contains the HELP+SubMNLI training data and the training set for simple and hard monotonicity from Semantic Fragments.

To validate our model’s ability for monotonicity reasoning and to evaluate its performance on upward and downward inference, the Monotonicity Entailment Dataset (MED) was used (Yanaka et al., 2019a), which is designed to examine a model’s ability of performing monotonicity reasoning. MED contains 5382 premise-hypothesis pairs including 1820 upward inference examples, 3270 downward inference examples, and 292 non-monotone examples. The sentences in MED cover a variety of linguistic phenomena, including lexical knowledge, reverse, conjunction, disjunction, conditional and negative polarity items. We removed sentence pair with the label ”contradict” from MNLI dataset since the test dataset MED and the training dataset HELP do not contain the label ”contradict”. We furthermore tested our model on the simple and hard monotonicity fragments test sets from Semantic Fragments.

Word embeddings are a common way to represent words when training neural networks (Mikolov et al., 2013). To train our model we used Stanford’s pre-trained 300-D Glove 840B vectors (Pennington et al., 2014) to initialize the word embeddings. The Stanford Dependency Parser (Chen and Manning, 2014) was used to parse each sentence in the dataset. The model is trained with the Adam optimizer (Kingma and Ba, 2014) which is computationally efficient and helps a model to quickly converge to an optimal result. A standard learning rate for Adam, 0.001, is also used. Dropout with a standard rate of 0.5 is applied to the feed-forward layer in the self-attention aggregator and the classifier to reduce the over-fitting of the model. For the number of hops of the self-attention, we used the default 15 hops. The metric for evaluation is accuracy based. The system is implemented using a common deep learning framework, PyTorch and is trained on a GPU for 20 epochs.

In this section, we evaluated our model’s ability of performing monotonicity reasoning. Table 1 shows a comparison of the performance of different models on the Monotonicity Entailment Dataset (MED), including our model. The data for all models except for ours was developed byYanaka et al. (2019a) who developed the MED dataset. Our model achieves an overall accuracy of 75.7% which outperforms all other models, even a state-of-art language model like BERT. Table 1 shows the ability of different models on performing upward and downward inference. Our attentive tree model performed better on downward inference than other models with an accuracy of 74.5% . Our model’s performance on upward inference outperforms other models except BERT. However, the upward inference accuracy of our model (81.4) is very close to the accuracy of BERT (82.7). We believe the good performance on upward and downward inference is due to considering parse tree information. Furthermore, the accuracy on upward inference increased significantly when trained with a combination of HELP and MNLI (HELP+SubMNLI) then trained only with HELP; the accuracy increased from 55.7 to 81.4 while the downward accuracy did not change much. Such phenomena suggests that adding MNLI to HELP does reduce the effect of the large number of downward monotone examples in the HELP dataset and thus improve the model’s ability on upward inference.

To demonstrate the robustness of our model, we experimented with training the model on various datasets. First, the model was trained on the HELP dataset alone. The overall accuracy was 66.0%, which outperformed other models from Table 1 except BERT trained with HELP+SubMNLI and our model trained with HELP+SubMNLI. Even on downward inference alone our model outperforms all other models with an accuracy of 72.6% except our model trained with HELP+SubMNLI. This result indicates that with a rich set of downward monotone examples, the model can learn to better predict a downward inference problem.

We then trained a model with the MNLI dataset alone. It contains a large amount of upward inference examples and only a rare number of downward inference examples. The result shows that the model generalized to the training data, and had an accuracy of 58.6% which is still higher than most models from Table 1. Interestingly, the model’s performance on downward inference is still better than its performance on upward inference, even though the training dataset contains a large number of upward monotone examples. This suggests that the model is immune to significant change of training data possibly due to the multiple dropout layer added to the aggregator and the classifier which forces a the model to learn more robust features. As Table 1 show, comparing to BERT trained with MNLI along, our model trained with MNLI along has better performance on downward inference than BERT’s performance from Yanaka et al. (2019a).

Finally, we trained our model on a combination of the MNLI dataset and the HELP dataset (HELP+SubMNLI). Because of the large number of upward training examples in MNLI, we suspected that the combination would alleviate the effects of this distortion and as such increase the accuracy for upward inference. We selected 20% of the complete MNLI dataset due to the long training period. As the results in Table 1 show, our model still performs well on downward inference with 74.5% accuracy, it also showed significant improvements on upward inference with an accuracy of 81.4% . The overall performance also increased substantially to 75.7% . Compared to the results of BERT trained with HELP+MNLI from Yanaka et al. (2019a), our model performs better on both upward inference and downward inference, and achieves a higher overall accuracy. The result validates our hypothesis that training on a combination of upward and downward monotone sentences can help the model achieve good performance on both upward and downward monotone, and that the use of AttentiveTreeNet is a good choice.

To further evaluate which part of the model contributed the most for monotonicity reasoning, we performed several ablation tests on the model. The ablation tests were trained with HELP and HELP+SubMNLI separately and the models were evaluated on the MED dataset. The results are shown in Table 2. We will focus our evaluation on the HELP+SubMNLI data.

For ablation test 1, we removed the self-attentive aggregator and built the feature vector for classification right after the tree-LSTM encoder. As Table 2 (–Self-Attentive Aggregator) shows, performance of the model trained on HELP+SubMNLI shows a significant, 6.6 percentage point drop in overall accuracy, a 10.9 percentage point drop in upward inference accuracy and a 7.6 percentage point drop in downward inference accuracy. The results of this test suggest that the self-attentive aggregator is an important component of the model that cannot be removed.

For ablation test 2, we replaced the tree-LSTM encoder with a standard LSTM encoder. Here, we see an even larger drop in performance. As Table 2 (–Tree-LSTM) shows, performance of the model trained on HELP+SubMNLI shows a large, 17.1 percentage point drop in overall accuracy, a 26.7 percentage point drop in upward inference accuracy and a 14.1 percentage point drop in downward inference accuracy. Based on the results, replacing tree-LSTM with standard LSTM has significant negative impact on the model’s monotonicity reasoning performance. Thus, tree-LSTM is a major component of the model that cannot be replaced.

For ablation test 3, we compared two matching methods for aggregating the two sentence vectors. In our final model (Full Model w/ vector-concat), we updated the matching method by following the generic NLI training scheme (Conneau et al., 2017). In it, we concatenate the two sentence vectors with an absolute distance and an element-wise product as the input vector for the classifier. We compared the performance to our original model (Full Model w/ mean-dist) which contains the tree-LSTM encoder, the self-attentive aggregator, and the concatenation of an absolute distance, an element-wise product, and a mean distance as the input vector for the classifier. For this ablation test, the results from Table 2 (Full Model w/ mean-dist) are mixed, yet important. While the overall accuracy decreases just slightly, by 2.7 percentage points and the downward inference accuracy only decreases by 0.8 percentage points, the accuracy for upward inference decreases by a significant 12.5 percentage points. We believe that these results justify the use of concatenation of the sentence vector pair.

Overall, the removal of the Tree-LSTM encoder affected the model’s performance most. Thus, we conclude that the Tree-LSTM encoder contributes the most to the model’s performance on monotonicity reasoning.

To check if our pre-trained model can be generalized to other monotonicity dataset, and to see if the model can be easily trained to master the new dataset while retaining its performance on the original benchmark, we conducted some additional testings on the model. We tested our pre-trained models on the Semantic Fragments test dataset which provides a more in-depth test for an NLI model’s performance with semantic phenomena, see (Richardson et al., 2019). Since our model focuses on monotonicity reasoning, we only selected the simple and hard monotonicity fragments for testing. Additionally, since our models are pre-trained on datasets that only contain two labels: ”Entailment” and ”Neutral”, we removed sentence pairs with the third label ”contradict” from the test dataset.

Table 3 shows the results of our testing. While we show the results for both, the HELP and HELP+SubMNLI data sets, we will focus our discussion again on the data obtained with the HELP+SubMNLI data set.

The top portion of Table 3 shows that the model trained on just HELP+SubMNLI performs poorly on the simple and hard monotonicity fragments. This performance is on par with other state-of-art model’s, see (Richardson et al., 2019).

The first middle portion on Table 3 shows the results of our model’s performance when only the simple training fragments were added to the HELP+SubMNLI training set. As the data shows, the model masters the simple monotonicity reasoning tests, does well on the hard monotonicity reasoning tests and retains its accuracy on the original benchmark MED.

The second middle portion of Table 3 shows the results of our model’s performance when only the hard training fragments were added to the HELP+SubMNLI training set. In this case, the model masters the hard monotonicity reasoning tests, does well on the simple monotonicity reasoning tests and again retains its accuracy on the original benchmark MED.

The bottom portion of Table 3 shows the results of our model’s performance when both the simple and hard training fragments were added to the HELP+SubMNLI training set. As the results show, the model masters both the simple and hard monotonicity reasoning tests while retaining its accuracy on the original benchmark MED.

Overall, the results show that the model trained on the fragments can be generalized to both simple and hard monotonicity reasoning.