TC-GAT: Graph Attention Network for Temporal Causality Discovery

The task of causality extraction, which endeavors to discern the causal relationship between entities or events within texts, has elicited significant attention among scholars. Initial explorations of causality extraction focused on lexical and syntactic features, causal cues such as “because” and “result in”, statistical features of event occurrence, and the construction of a term causality network from web-based text collections [5, 6, 7, 9, 18, 19]. These methods primarily emphasized the computation of similarities in sentence syntax-dependent structure and the introduction of novel causal feature words. However, more recent research [10] has leveraged the contextual relevance-enhancing capabilities of the BERT [11] to achieve remarkable results, despite neglecting the influence of temporal factors on entities or events.

Exploration for causality extraction through the lens of time relations garners considerable attention. Pioneering works such as Bethard [12] and Rink [13] establish a close correlation between time relations and causality, with the subsequent proposal of the utilization of TimeML labeling [14] to annotate causal events. Unfortunately, they are limited by the progress of machine learning then. CATENA [15] adopts filtering rules grounded in time relations, while CAEVO [29] grows a temporal graph in a multi-sieve manner, where predictions are incrementally added through successive sieves, either needs to maintain a rules bank or updates a temporal graph constantly. CaTeRS [16] leverages a large corpus of story-type events to determine the time-related causality judgments, and TCR [17] infers the causality of events based on specified rules.

While these methods represent a range of approaches to causality extraction, many fall short in terms of their ability to harness the power of temporal relations for prediction or the capabilities of deep learning. This paper seeks to address this gap by presenting a novel approach based on joint temporal relationship reasoning, with a focus on utilizing existing classifications of time.

The pre-trained language model BERT [11] has gained significant prominence in recent times for its demonstration of efficacy in comparison to traditional one-way language models and shallow splicing methods. The masked language model (MLM) approach utilized by BERT has been shown to provide more semantic information within generated language representations.

Our work harnesses the capabilities of BERT by incorporating sentence vectors as inputs to the model, yielding contextual features that serve as word embeddings. The representation of each token in sentence S is obtained through the amalgamation of its respective token embedding, segment embedding, and position embedding. This is achieved by utilizing the sentence encoding of the form $[CLS,E_{S},SEP]$, where CLS signifies the initiation of the sentence and SEP serves as an indicator of the sentence’s conclusion.

The BiLSTM model serves as the fundamental structure in our approach to extracting contextual information. By combining the outputs of this model with the Graph Attention Network (GAT) and a temporal relationship matrix, we are able to obtain feature vectors that encapsulate both contextual and temporal relationships. This synergistic utilization of the BiLSTM model and additional components enhances our ability to capture the intricacies of the data at hand, enabling the derivation of more nuanced insights.

We present a novel approach that leverages the power of Graph Attention Networks (GAT) and the causal relationship adjacency matrix to construct a causal relationship-based graph attention network. This network, named C-GAT (Causal Graph Attention Networks), is based on the causal knowledge graph and can effectively capture the causal relationships in a sentence. We first input the word embedding vectors into a Bi-LSTM network to obtain the feature vectors, which encapsulate the contextual information of the words. Then, we integrate the causal relationship graph into the network by obtaining the adjacency matrix that reflects the causal relationships between the words. Finally, the feature vectors and the adjacency matrix are input into the GAT to obtain the output, which accurately captures the causal relationships.

This innovative approach to modeling causal relationships provides significant advantages over conventional methods and contributes to a deeper understanding of the causal relationships in natural language processing tasks.

In this paper, we delve into the interplay between time and causality, specifically examining the implications of temporal relationships on causality extraction. Despite the axiomatic truth that the occurrence of entities must be grounded in the temporality of events, the presence of such relationships does not necessarily imply a causal relationship between them. As such, the indiscriminate consideration of temporal information can pose a significant hindrance to accurate causality assessment. To address this challenge, we introduce a novel equilibrium mechanism that endeavors to mitigate the pernicious influence of time relationships by learning context-sensitive attention weights. These weights are based on a number of trade-off factors that we consider, which are carefully designed to strike a balance between the utilization of temporal information and the preservation of causality judgment accuracy. The trade-off factors are as follows:

The trade-off coefficient g is used to obtain the result vector after the trade-off:

Where $h_{I}^{{}^{\prime}}$ and $H_{B}ERT$ are the eigenvalues obtained by time reasoning and BERT, W, and b are the model parameters.

During training, we take the cross-entropy loss function as our loss function,

Where $n=2$, $x_{i}$ is the value of the i-th dimension of the real label one hot form, $P(x_{i})$ is the value of the i-th dimension of the output prediction vector.

The coarse-grained experimental results of the TC-SemEval2010-Task8 dataset are shown in Table III, and Table IV shows the results of the TC-Altlex dataset, the Macro-F1 values of the TC-GAT in the two datasets are 89.97% and 90.97%, respectively, and the accuracy is also the highest, which is better than others.

The empirical results of this study demonstrate that the inclusion of temporal relationships within both the C-GCN model and the TC-GAT model proposed herein has led to a significant improvement in the extraction of causal relationships. This attests to the viability of this line of inquiry. On the one hand, the TC-GAT model leverages graph attention for temporal relationships to concentrate on words imbued with temporal significance, and on the other hand, it employs a causal relationship knowledge graph to focus on words with a potential causal connotation, further augmented by the integration of both BERT and BiLSTM to capture the contextual information of words. Such synergistic combination engenders a more robust extraction of causal relationships. Furthermore, the comparison between the C-GCN model and the TC-GAT model reveals that the implementation of an equalization mechanism effectively counteracts the adverse effects associated with the temporal relationships.

To maintain a level of impartiality, all hyper-parameters are held constant while the innovative additions are integrated and retrained. The proposed model employs BERT as an embedding encoder of contextual features, with the incorporation of T-GAT to calculate the temporal attention weight, and C-GAT to compute the causal attention weight. The final addition of an equalization mechanism, referred to as Equilibrium, serves to balance the impact of the temporal relationship, culminating in the complete expression of the methodology.

In the ablation experiment, Table  V demonstrates the results of evaluating each constituent module’s efficacy, yielding positive outcomes on both datasets. This confirms the validity of the proposed method across both datasets. In the first experiment, the removal of BERT as the embedding encoding context feature was tested, resulting in a decrease of 2.69% and 4.38% in Macro-F1 value in the two datasets, respectively, thereby attesting to BERT’s efficacy in capturing contextual information. Subsequently, the removal of the equalization mechanism based on the original model resulted in a relatively modest drop in performance by 0.02% and 1.89%, however, instability in prediction was observed when comparing the results of labels C and E, thus highlighting the importance of the balance mechanism in equilibrating the weight outcomes of T-GAT and C-GAT. The performance of T-GAT and C-GAT models was then evaluated in the two datasets, yielding improved performance, thereby affirming the time relationship’s ability to enhance causal relationship extraction accuracy, as well as the impact of the causal knowledge graph. However, it is important to note that the performance of these modules is inferior to the overall model, and the disparity between the P-value and R-value in the T-GAT model highlights the deleterious effect of the temporal relationship.

The generalization capabilities of the C-GAT module of the proposed model were evaluated through an exploratory investigation. To this end, the entire dataset of SemEval2010-Task8 was selected for comparison, taking into consideration the constraint imposed on the longest text of 50 words and the consequent removal of sentences exceeding this limit. The statistics of the retained sentences are presented in Table 6.

The comparative evaluation of the C-GAT model proposed in this study against Flair, BiLSTM-LAN, BERT-CRF, and (Char)BiLSTM-CRF, reveals the superiority of the former in terms of coarse granularity, as exhibited in Table 7. The Macro-F1 value of the C-GAT model stands at 75.37%, surpassing that of the BERT-CRF model, which indicates the efficacy of BERT in representing contextual and semantic features of text and the capacity of CRF in incorporating inter-label relationships into the model. The utilization of multiple embeddings in Flair contributes to the refinement of semantic features, while the hierarchical attention per word in the LAN module enables the capture of long-term label dependencies.