Mind Reasoning Manners: Enhancing Type Perception for Generalized Zero-shot Logical Reasoning over Text

The logical reasoning task, which aims at testing the reasoning capability of models, has aroused wide interest. Several representative datasets have been proposed, such as ReClor [13] and LogiQA [16]. Current methods on the logical reasoning task can be divided into graph-based methods and data-based methods. The former focuses on the construction of the text graphs and leverage the node connection to model the logical relations. Among this category, DAGN [7] is the first work to split the text into EDUs and perform the reasoning with graph neural networks [17][18], but it only builds a chain-type graph and ignores the long distance interaction between nodes. FocalReasoner [8] proposes to attend to the fact triplets [19] within the text and constructs a supergraph based on the extracted triplets. However, it lacks the modeling of the logical information in the text. To fully explore the logic, AdaLoGN [10] is proposed to construct the adaptive neural-symbolic system to improve the performances. However, its reasoning process is complex and costly. Considering the above drawbacks, Logiformer [9] stresses much importance on both the causal relations and the co-occurrence relations in a two-branch graph transformer network. Yet it still lacks the perception of different question types, which limits the zero-shot reasoning capability of the model. The latter one is the data-based methods. These works aim to improve the performance through the data augmentation strategy. One of the representatives is LReasoner [11], which extends the symbolic expressions by logical rules and templates. In addition, MERIt [12] designs a meta-path-guided contrastive learning method to facilitate the training process, by utilizing the extra data. However, both of them fail to model the reasoning type of each question and lack the application value in zero-shot setting.

Current machine reading comprehension (MRC) tasks [20] can be categorized into two settings: full-data setting and low-resource (i.e., few-shot, zero-shot) setting.

Full-data setting has aroused wide concerns in the area of MRC in recent years. Popular benchmark SQuAD [21] [22], which is sourced from from Wikipedia articles, contains over 100,000 questions. It forms an abundant database for the model training. Similarly, HotpotQA [23] includes 113K questions with a variety of reasoning strategies. It stresses more on the multi-hop reasoning abilities of the model. RACE dataset [24] is specially designed to improve the reading skills for both middle school and high school students, which contains about 9.7K examples. Also, some domain-specific datasets like NewsQA [25], TextbookQA [26], PubMedQA [27], aim at the area of journalism, education and biology respectively. All of them have rich training examples to improve the model capability. However, such full-data setting encourages the models to depend on the ideal scenes with abundant training data, which may not fit in some real cases.

Based on such concern, some previous works focus on the low-resource setting MRC, including few-shot or zero-shot setting. For example, to evaluate the robustness of the model, [28] reconstructs the data from knowledge sources for zero-shot commonsense QA. Also, [29] proposes a neuro-symbolic approach to boost the performance of zero-shot commonsense QA. Both of the methods are not exposed to the training data, forming a zero-shot setting. However, they rely on the extra knowledge bases, which limits their applications. There are also some works related to the few-shot QA. In [30], a new pretraining strategy is proposed to explore the realistic few-shot setting. Also, [31] tackles the few-shot challenge on the task of Visual Question Answering. However, all of the above methods simply obtain the few-shot splits according to the amount, which treat all the samples equally.

To sum up, current MRC benchmarks mainly focus on the common types of questions. However, in reality, it is more challenging when faced with some uncommon types of questions. To make up for these drawbacks, we attend to the reasoning types in the logical reasoning tasks and propose the first benchmark for zero-shot logical reasoning based on type attributes.

The zero-shot logical reasoning datasets are split based on ReClor [13] without shuffling. Considering the situation in reality, it is easier to learn to reason on common types of samples while struggle with the rare ones. To this end, we design three sampling strategies, namely amount, randomness and difficulty.

For amount, we select top-k reasoning types as the seen types, merely by the amount. It can be seen as a simple implementation to filter the uncommon types of samples.

For randomness, we arrange the reasoning types in descending order of amount and select the seen ones based on the geometric distribution. The discrete form of the geometric distribution is,

where $k$ is the sorted index of the reasoning type, $p$ is the hyper-parameter set to 0.1 in our implementation. In this random setting, the type with more training samples has a higher probability to be selected, which is in parallel with the real situation.

For difficulty, we first rank the difficulty of the reasoning types, based on their performance with RoBERTa-Large single model. On one split, we select some of the most difficult reasoning types as the seen ones. On the other split, we select a part of the easiest types as the seen ones.

In total, 6 zero-shot splits are obtained (2 for each strategy) for ReClor. The details of the splits are presented in Table I.

We propose the zero-shot setting for the logical reasoning task. Given the context $C_{i}$, question $Q_{i}$ ($\emph{type}(Q_{i})\in\mathcal{T}$) and option set $A_{i}$ of the $i^{th}$ question, the model is required to predict the correct answer $a\in A_{i}$. In the definition, $\emph{type}(Q_{i})$ denotes the reasoning type of the question $Q_{i}$ and $\mathcal{T}$ is the type set of the zero-shot splits. For each zero-shot split, a part of the types are sampled as the seen types, while others are viewed as the unseen ones. Only the questions of seen types exist during training and they consist of the training examples. The type set seen during the training stage and the set for the test stage are $\mathcal{T}^{train}$ and $\mathcal{T}^{test}$ respectively, thus they satisfy $\mathcal{T}^{train}\cup\mathcal{T}^{test}=\mathcal{T}$.

To be closer to the real scenes, we consider the generalized zero-shot setting. That is to say, the test scope is not limited to the unseen types. To this end, we employ two metrics for the generalized zero-shot setting, Test-All and Test-Unseen. The former one denotes the exact match results on the full-data test split, which contains both seen and unseen types, i.e., $\mathcal{T}^{train}\subseteq\mathcal{T}^{test}$. The latter one is the exact match results only on the unseen types of the test split, which ignores the performance on the seen types, i.e., $\mathcal{T}^{train}\cap\mathcal{T}^{test}=\emptyset$. In this way, we expect to achieve comprehensive assessments of the model capabilities.

Also, we are going to verify the necessity of the zero-shot setting from the pilot experiments. For comparison with each split, we randomly sample the same amount of training examples on both the seen and unseen types, forming the comparison group. Take the zero-shot split v1 as an example, it has $2,190$ seen samples for training. Thus, its comparison training group also includes $2,190$ examples, but are distributed over all types. The purpose of the pilot experiments is to observe the performance changes of the two metrics on the test split.

In the implementation, we utilize the RoBERTa single model [32] to conduct the reasoning and maintain the same hyper-parameters of the zero-shot splits with comparison groups. Especially, to avoid the noise brought by the random sampling, we conduct the comparison experiments five times with different random seeds. Final results are the average values of the five experiments.

We select the results on four of the split versions for illustration (i.e., v1-v4), shown in Fig. 2. The first and the third column in blue are the performance of the comparison group, on the full-data test and the unseen types of test respectively. The second and the fourth column represent the performance on the zero-shot setting. With the same number of training examples, training only on seen types (zero-shot setting) witnesses obvious drops on the performance, especially on the unseen types of the test split.

Such observations are consistent among all the zero-shot splits. Zero-shot pilot experiments based on reasoning types uncover the obvious drawback of the current full-data setting. In another word, it verifies the necessity to propose a new benchmark for the generalized ZsLR.

One of the common practices for MCQA problems is to concatenate three sequences as inputs: context, question and option. But it is insufficient for the zero-shot setting. There exist two main drawbacks: 1) the modeling of the reasoning type is implicit in the sequence; 2) it is difficult to bridge the interaction between context and options via the question sequence in the middle since it is not natural for the language model (LM). To this end, this paper introduces the heuristic reconstruction to the inputs based on the type extraction.

To address the first drawback, we are required to label the reasoning type of each question based on the limited inputs. Inspired by LReasoner [11], we propose a simple but effective type extractor through keywords. The procedure of the heuristic type extractor is presented in the pseudo-code of Algorithm 1.

Since the ReClor dataset only makes the reasoning types on the test split public, we are required to collect the classification method from it. Before the extraction, we conclude the keywords and phrases for each reasoning type, forming the keywords base $\mathcal{B}$. Also, we include the maximum window size W and the question sentence as the inputs. In Line 1, we first split the question into words and form the word sequence $S$. Then we start the iteration in the descending order of the window size. We slide the window over the question sequence to obtain the sub-sequence (Line 5). Then we match the sub-sequence $C$ with keywords base $\mathcal{B}_{k}$ for each reasoning type $k$ to derive the number of exact matches (Line 7). After each window size iteration, we judge the exit condition (Line 11). If there exists a unique type with the maximum number of matches, we label the type as the ground truth. Meanwhile, if we can not extract the type at the last round of iteration, we label the instance as Others type (Line 16, 17).

Then, we convert the type index of each question into the natural language (e.g., Implication, Conclusion). Meanwhile, to equip the LM with the type information, we add a type-related prefix at the beginning of the sequence:

For the second drawback, one intuitive idea is to reconstruct the inputs of question and option sequences, transforming them into a single sequence in a declarative form, defined as Q-A pair. In another word, we are required to fill the option into the proper position in the question. According to our observation, the question sentence is led by a trigger span. For example, in the question Which one of the following most weakens the arguments above ?, the span Which one of the following serves as the trigger. Then, we can replace the trigger span with the option sequence to obtain the Q-A pair, formulated as [Option] most weakens the arguments above.

Therefore, the core of the heuristic strategy is to pinpoint the precise position of the triggers. Although the trigger span is similar to the basic form of Which of the following, it is not always the same. To further improve the diversity of the trigger base, we propose two heuristic strategies:

• •

  Combination: We predefine a set of basic words (i.e., {which, one, of, the, following}), and then randomly combine all or parts of them to form the new triggers. For instances, of which the following and which of following are two newly constructed triggers.

• •

  Tolerance: More trigger spans are not limited to the combination of the predefined words. To this end, we propose a tolerance strategy to contain the extra words within the trigger span. For example, in the trigger of the following claims, which, we include the extra word claims to the span.

In this way, the trigger base can be expanded to adapt to the complex situations in reality. So far, the declarative Q-A pair is obtained.

Concatenating the context sequence $C_{i}$ and Q-A pair sequence $P_{i}$ of the $i^{th}$ question, we send it into the pre-trained LM. In this paper, we employ the RoBERTa-Large model [32] to obtain the token-level representations as follows,

where the tokens $\{c_{0},c_{1},\ldots,c_{|C_{i}|}\}$ make up $C_{i}$, and $\{p_{0},p_{1},\ldots,p_{|P_{i}|}\}$ make up $P_{i}$. The representation of $C_{i}$, $P_{i}$ and the whole sequence $S_{i}$ (concatenation of prefix, $C_{i}$ and $P_{i}$) can be obtained through the mean pooling strategy,

According to the previous analysis, the context and Q-A pair may interact differently under different reasoning types. Therefore, we consider to build the topology structure of the two parts respectively and finally learn their interactive semantic. For clearer illustration, we take the context part as an example and present the graph construction process in Fig.4.

Firstly, we split the text sequence into units (e.g., U1-U6 in Fig. 4) based on the predefined explicit connectives or punctuation (marked in blue in Fig. 4). These text units function as the nodes during the graph reasoning process. Therefore, two node sets for the context and Q-A pair parts are obtained respectively.

To form the dual subgraphs of the input sequence, we are also required to define the edge relations, including a) order relation and b) overlap relation. The former one models the original text order, where nodes are connected in sequence. It forms a chain-type structure which maintains the position perception. The latter one models the similarity between nodes, where nodes with more vocabulary overlaps or similar semantics will be connected. In the implementation, we will transform the text units into word sets, where the stopwords are excluded. Through the calculation of the overlap ratio between sets, we connect the node pairs that exceed the threshold.

In this way, the independent topology of the context part and the Q-A pair part are formed. To make the interaction of the two parts learnable, we add a global node to link all the extracted nodes. The global node is initialized with $\mathbf{h}_{S_{i}}$ and aggregates the two-way information flow. Therefore, this node representation is expected to contain the type semantic to improve the zero-shot performance.

Further, to overcome over-smoothing in graph neural networks [33] and fully conduct node interactions, we employ the Graph Transformer Network [34, 9]. Let $\mathcal{N}$ be the node set in one graph, with the number of $|\mathcal{N}|$. We feed the node sequence into the Transformer architecture [35]. For each layer $l$, the updated information $\mathbf{e}_{j}$ of the node $\mathcal{N}_{j}$ is formulated as follows:

where $\mathcal{N}\backslash\{\mathcal{N}_{j}\}$ denotes the set without $j^{th}$ node.

The core of the equation is the computation for the weighted attention $\alpha_{j,k}$. It is obtained by the two matrices query $\mathbf{Q}$ and key $\mathbf{K}$:

where $\alpha_{j,k}$ is one of the elements in the matrix $\mathbf{A}$. $d$ is the dimension of the hidden states. $\mathbf{M}\in\mathbb{R}^{\mathcal{|N|}\times\mathcal{|N|}}$ is the attention bias matrix to encode the structural semantic of the whole graph, which is the adjacent matrix of the graph.

Thus, the final feature of the global node $\mathbf{h}_{g}$ is updated through the graph reasoning network, containing the global semantic of the text sequence. Before predicting the correct answer, we concatenate $\mathbf{h}_{g}^{(j)}$, $\mathbf{h}_{C_{i}}^{(j)}$, $\mathbf{h}_{P_{i}}^{(j)}$ and $\mathbf{h}_{S_{i}}^{(j)}$ to obtain the score of the $j^{th}$ option followed by softmax:

where $\mathbf{W}\in\mathbb{R}^{4d\times 1}$ denotes the linear projection. Same with most of the MCQA methods, we adopt the cross-entropy loss function for the optimization:

In the upper part of the architecture, the interactive semantic of the context and Q-A pair is learned through one global node. Besides the implicit awareness of the type, we also expect the model to distinguish the ground-truth type from the negative ones for zero-shot logical reasoning.

Based on the heuristic extractor mentioned above, the reasoning type of each example can be derived. For each reasoning type, the detailed description in the form of natural language is utilized (listed in Table II).

Feeding all 17 reasoning type descriptions into the LM (i.e., Sentence-BERT [15]), we obtain the sentence-level embeddings, which are fixed during training.

For each example, we propose to attend to the type information through contrastive learning. The final representation of the global node $\mathbf{h}_{g}$ serves as the anchor. The ground-truth type functions as the positive sample, while others are negative samples, with the embedding of $\mathbf{h}_{gt}$ and $\mathbf{h}_{n}$ ($n$ is the index of the negative samples) respectively. Our purpose is to close the distance between the global node $\mathbf{h}_{g}$ and the positive sample $\mathbf{h}_{gt}$, while distancing the negative samples.

For simplicity, we model the score of two vectors using Hadamard product. That is:

where $\mathrm{score}^{+}$ and $\mathrm{score}^{-}_{n}$ represent the score of the positive sample and the scores of the negative ones respectively. We employ the margin loss as the auxiliary optimization function for each example:

where $\gamma>0$ controls the difference between positive and negative scores. We select the maximum score of the negative one for the loss computation.

To maximize the joint optimization performance of the two loss functions, we set a trade-off coefficient $\alpha$ and transform the final loss function into:

In this way, the global semantic for reasoning reinforces mutually with the type-aware semantic, which benefits the zero-shot performance. In addition, it provides the interpretability for the test process, which will distinguish the correct reasoning type from others.

Currently, there exist two datasets in the logical reasoning task, ReClor [13] and LogiQA [16]. Since the LogiQA datasets contain only 5 reasoning types, we consider the ReClor dataset in the zero-shot experiments. And we take both of them into account for the full-data setting to prove the generalization capability of TaCo. The detailed information of the two datasets is presented below.

We include all the previous SOTA baselines in the main paper for comparison, as well as the results from some classical language models.

In this paper, all experiments run on a single GPU of Tesla A100. We employ RoBERTa-large model [32] as the text encoder for all previous methods for fair comparison. For each split of the zero-shot setting, we train on the seen types and select the best epoch on the seen types of the development set for the test. We rerun other baselines on the zero-shot setting with the original configuration and maintain their reported results on the full-data setting. As for our proposed model, the hyper-parameters keep the same for both settings. The training epoch is set to 30 and the batchsize is fixed to 1. For optimization, we take Adam [38], with the peak learning rate as 5e-6. The number of heads and layers in the graph transformer is set to 6 and 4 respectively. $\gamma$ in the margin loss is tuned as 12. The loss trade-off $\alpha$ is 0.2. The selected five random seeds for the comparison groups are {42, 12, 23, 234, 1234}. Additionally, some important hyper-parameters are tuned for the best within a search scope. The details are included in the Table III.

The model TaCo is designed to bridge the gaps of the zero-shot logical reasoning setting, thus we are going to evaluate its performance. We present the results of 6 zero-shot splits in Table IV. We rerun five strong baselines on the zero-shot setting, including (1) BERT-Large, (2) RoBERTa-Large, (3) graph-based representative method DAGN [7], (4) data-based representative model LReasoner [11], (5) SOTA model Logiformer [9].

It shows that TaCo outperforms these strong baselines with large margins on all the zero-shot splits. Compared with the previous SOTA model Logiformer, the averaged improvements among all the splits are 3.80% and 4.54% respectively for the two metrics Test-All and Test-Unseen. And compared with all suboptimal results on the two metrics, TaCo still has great superiority of 2.50% and 3.65% respectively. To explore the respective roles of the two metrics, Test-Unseen witnesses the greater improvements among all the splits, especially 6.14% and 5.04% over the sub-optimal methods in split v1 and v4 respectively. It illustrates that TaCo performs better on tackling the unseen types of samples than the current methods, while maintaining the superior performances on the seen ones. In addition, among all the zero-shot splits, the performances within each method vary greatly. For example, the previous SOTA Logiformer shows good performances in v2 but struggles a lot in v1. It proves that the benchmark provides diverse distributions of the data splits and encourages the consistent performances of the model. And considering the experiment results, TaCo can function as a strong baseline for the future work on the ZsLR benchmark.

To illustrate the effectiveness of each module of TaCo in the zero-shot setting, we conduct the following ablation studies. The results are listed in Fig. V

Detailedly, in part a), w/o type prefix and w/o reconstruction are related to the heuristic input reconstruction, where the former one ablates the type-oriented prefix, and the latter one maintains the simple concatenation of the question and option. Generally, the heuristic type prefix and input reconstruction bring more improvements on the metric of Test-Unseen with an average of 3.95% and 3.26% respectively respectively on all six splits. It shows that type-oriented prefix is more effective than input reconstruction in most cases (i.e., v2-v5).

In part b), w/o graph reason ablates the whole module of graph construction and reasoning, and replaces the global node feature with the pooled output of the whole sequence. Since the reasoning process is the core of this task, it contributes most to the performance on both the metrics, with the average gains of 2.58% and 4.23%. While w/o global node simply replaces the global node feature with the pooled representation of all the other nodes. It can be regarded as a small part of graph construction and reasoning. From the results, it works on the splits v2, v4 and v6. As the training set of these three splits are obviously larger, it can be concluded that global node is especially effective with more training samples.

In part c), we ablate the type-aware contrastive learning module. From the experiments, the modeling of reasoning types does great help to the unseen types of samples. In the unseen types of v1, v2 and v4, the margin loss brings significant improvements, with gains of 4.81%, 4.17% and 3.70% respectively.

Overall, the three key modules (i.e., type prefix, input reconstruction, type contrast) proposed to enhance the type perception all have positive effects on the model performances, especially on the unseen parts of the test split.

Further, we present the visualization of different selections for hyper-parameters. All the hyper-parameters are selected in the full-data setting, since the zero-shot splits are diverse and hard to unify. The first one is the number of layers and the number of heads in the graph transformer network, shown in Fig. 5. To make a comprehensive comparison, we search both the head number and the layer number within {3,4,5,6}. As can be seen from the heatmap, the darker color represents the higher performance. TaCo reaches the optimal performance when the head number is 6 and the layer number is 4.

In addition, we present the model performances under different margins $\gamma$. The search scope of $\gamma$ is within {8,10,12,14,16}. And we select the optimal performance on the test split. The optimal value of $\gamma$ is 12.

Compared with all the current methods, TaCo achieves competitive results. In comparison with the SOTA model Logiformer, TaCo is better on the metric of Dev and Test-E with 0.60% and 2.27% gains. And it is only slightly lower than SOTA with 0.20% in the test split. We argue that in the full-data setting, the ideal distribution of types leads to the weakening of the type modeling. Combined with previous experiments on the zero-shot setting, though Logiformer does good at over-fitting the full-data setting, it loses some generalization capability. From this perspective, TaCo also shows great superiority on the model generalization, performing consistently for both settings.

Further we conduct the ablation studies to analyze the effectiveness of the proposed modules in the full-data setting. From the results, the graph reasoning contributes most to the performance on the test split. Also, we only witness 0.40% improvements for the consideration of type-aware margin loss. Since in the full-data setting, the train and test splits share the same distributions, the effects of the reasoning types are reduced. Therefore, the slight gain of the contrastive learning is reasonable. In all, the generalization capability of TaCo is well verified.

To discuss the generalization capability of TaCo, we further conduct some experiments on another logical reasoning dataset LogiQA[16]. Following the method utilized for ReClor, we also label each example of LogiQA with one of the 17 reasoning types, and consider the zero-shot setting. We apply the above mentioned strategies to obtain three zero-shot splits (i.e., z1, z2 and z3) for LogiQA.

In comparison, we take some previous logical reasoning models into account, including two typical language model BERT-Large and RoBERTa-Large, the graph-based method DAGN and previous SOTA model Logiformer. We rerun the zero-shot setting for each baseline.

From the results shown in TABLE VIII, TaCo shows its superiority in most cases, only except the Test-Unseen metric of z1 split. Specifically, compared with previous SOTA Logiformer, TaCo achieves an average gain of 1.89% and 0.89% on the Test-All and Test-Unseen respectively. In general, experiments on the LogiQA dataset prove the good generalization capability of TaCo, which satisfies our expectations.