TacoERE: Cluster-aware Compression for Event Relation Extraction

The document clustering aims to split document into intra- and inter-clusters, where intra-clusters aim to enhance the relations within the same cluster, while inter-clusters attempt to model the related events at arbitrary distances. Our observation is that predicting the relations for event pairs relies on a limited content rather than the entire document. For example in Figure 1, just based on sentence S2 and S3, the relation between event originated (e2) and reaching (e3) can be deduced. Moreover, sentences in document with the same cluster are more probably to involve event relations, such as the cluster of development process with blue color (sentences S2-S4) in Figure 1 has three relations. As a result, we split the $D$ into $K$ mutually independent intra-clusters at sentence level. To obtain the intra-clusters, arbitrary clustering methods, such as traditional machine learning or deep neural networks, can be applied. In our experiments, we directly utilize the effective and widely used K-means algorithm Guan et al. (2022); Rakib et al. (2020). Specifically, we extract multiple features to enhance the clustering, including cluster words extracted by LDA model, trigger words, tf-idf, and sentence representation encoded by RoBERTa.

$$J=\sum^{k}_{i=1}\sum_{j=1}p_{ij}||v_{j}-u_{i}||^{2}$$

(1)

where $J$ is objective, $v_{j}$ is the feature of sentence $s_{j}$, $u_{i}$ is center of i-th cluster, and $p_{ij}$ is indicator.

Beyond individual intra-cluster, among different intra-clusters also occur with event relations which can be seen in Figure 1, relation between event cyclone (e1) in background and destroyed (e7) in impact. Thus, we fuse any two intra-clusters as inter-clusters. The order of sentences in intra- and inter-clusters will follow their original order in $D$. And the intra- and inter-clusters can involve all possible relations in $D$.

The cluster summarization aims to simplify and highlight important text content of clusters for mitigating information redundancy and event distance. Intra- and inter-clusters directly group the sentences with related sub-topics. It will inevitably contain redundant information and ignore coherence, which may hinder the performance Gao et al. (2021). Text summarization as a technique can effectively simplify and spotlight important text content, which would solve this problem to some extent. To this end, we further utilize a summarization model to generate summaries $C^{a}$ and $C^{r}$ for intra- and inter-clusters respectively, where $\{C^{a},C^{r}\}\in C$. In this paper, we utilize a transformer-based Vaswani et al. (2017) encoder-decoder framework for summarization. In particular, a pretrained language model as the encoder to learn the contextual representation of input, and a transformer-based decoder is utilized to generate its summary word-by-word.

The relation prediction aims to construct the event relation prediction process by giving the text content from cluster summarization. We first need to obtain the contextual representation of each token in the document $D$, $C^{a}$, and $C^{r}$ respectively. Take $D$ as an example, we leverage pre-trained language model RoBERTa Liu et al. (2019) as the encoder. Since event mention/trigger words perhaps contain multiple words, i.e., take place, and individual word in document may be split into sub-words by wordpiece, i.e., word “summarization” will be split into three sub-words “sum”, “mar”, and “ization”, we adopt LogSumExp pooling method Zhou et al. (2021) over all its mentions (sub-words) embeddings in the last encoding layer as the event representation. Then, we can obtain the overall representation of event pairs in $D$. For cluster summaries $C^{a}$ and $C^{r}$, we extract the events which appear in original document $D$.

Event pairs of $C^{a}$ and $C^{r}$ all appear in $D$, which means one event pair may occur multiple times. Different from existing work Adhikari et al. (2019a); Beltagy et al. (2020) which aggregate all identical event representations to get the final event pair representations, for the same event pair, we select the most relevant one, and the detailed selection process is as follows: (1) for the same event pair, the priority of event pair in $C^{r}$ is higher than $C^{a}$; (2) for the event pair which not in $C^{a}$ and $C^{r}$, we directly use the representation in $D$. Finally, a two layers feed-forward network with softmax is adopted to learn the classes probability based on the event pair representations. For training objective of relation prediction $\mathcal{L}_{rp}$, we use cross-entropy function as follows:

$$\mathcal{L}_{rp}=-\sum_{i\neq j}\sum_{\mathcal{R}}\{r_{ij}\mathit{log}P_{ij}+(1-r_{ij})\mathit{log}(1-P_{ij})\}$$

(2)

where $P_{ij}$ is the probabilities between events $e_{i}$ and $e_{j}$, and $r_{ij}\in\mathcal{R}$.

The Training Phase describes both Joint Training and Pretraining for Cluster Summarization, where Joint Training aims to jointly optimize the cluster summarization and relation prediction, while Pretraining for Cluster Summarization aims to teach model to do better content representation.

Performance-based Reward $\mathcal{R}^{per}(C)$ aims to use the performance as the direct training signal. We calculate the reward $\mathcal{R}^{per}(C)$ based on the performance for all document event pairs. In particular, for each event pair $(e_{i},e_{j})$, $\mathcal{R}^{per}(C)$ = 1 if relation prediction model calculates the true relations between $e_{i}$ and $e_{j}$, and 0 otherwise.

Summary-based Reward $\mathcal{R}^{ec}(C)$ aims to encourage the correlation between the clusters and its summaries to train cluster summarization. The motivation of $\mathcal{R}^{ec}(C)$ is that summary expresses important information about the text and they are consistent at the semantic level Guan et al. (2021b). In Section 2.1, we split document into intra- and inter-clusters. Intra-clusters are independent of each other and cover the content of the entire document. Thus, we only adopt the intra-clusters and its summaries to calculate the reward $\mathcal{R}^{ec}(C)$. Similar to existing work Pasunuru and Bansal (2018); Paulus et al. (2018), we utilize the popularly known automatic evaluation metric for summarization, ROUGE Lin (2004), as reward function. However, this metric mainly focuses on phrase matching/n-gram overlap while assuming equal contributions from each word. Addressing these issues, we introduce salient function to give higher weight to the trigger words.

$$P=\frac{\sum_{k}(\mathit{LCS_{\cup}}(C^{a}_{k},T^{a}_{k})+\sum_{l}\eta(w_{kl}))}{|D|}$$

(3)

$$R=\frac{\sum_{k}(\mathit{LCS_{\cup}}(C^{a}_{k},T^{a}_{k})+\sum_{l}\eta(w_{kl}))}{\sum_{k}|C^{a}_{k}|}$$

(4)

$$\mathcal{R}^{ec}(C)=\frac{(1+\sigma^{2})RP}{R+\sigma^{2}P}$$

(5)

where $C^{a}_{k}$ is the summary of $k$-th intra-cluster $T^{a}_{k}$, $\mathit{LCS_{\cup}(\cdot)}$ is the union longest common subsequence, $\sigma$ is defined in Lin (2004), and $\eta(\cdot)$ is function to measure whether $w_{kl}$ is a trigger word.

After obtaining reward $\mathcal{R}^{per}(C)$ and $\mathcal{R}^{ec}(C)$, the overall reward can be calculated as $\mathcal{R}(C)=\alpha\mathcal{R}^{per}(C)+\beta\mathcal{R}^{ec}(C)$, where $\alpha$ and $\beta$ are trade-off parameters. Following existing work Man et al. (2022), we minimize the negative expected reward $\mathcal{R}(C)$ over the possible choices of summaries:

$$\mathcal{L}_{sum}=-\mathbb{E}_{C^{\prime}\sim P(C^{\prime}|e_{i},e_{j},D)}[\mathcal{R}(C^{\prime})]$$

(6)

Then, the gradient can be further formalised by utilizing the one roll-out sample method.

$$\nabla\mathcal{L}_{sum}=-(\mathcal{R}(C)-\theta)\nabla\mathit{log}P(C|e_{i},e_{j},D)$$

(7)

where $\theta$ is used to reduce variance.

Datasets  We conduct experiments on three datasets: MAVEN-ERE (Wang et al., 2022), EventStoryLine (Caselli and Vossen, 2016) and HiEve (Glavas et al., 2014). MAVEN-ERE is a unified large-scale human-annotated ERE dataset, which contains 4,480 documents, 112,276 events, 57,992 causal relations, and 15,841 subevent relations in total. We split the data into train, dev, and test sets in 2,913, 710, and 857 documents as in Wang et al. (2022). EventStoryLine (i.e., version 0.9) contains 258 documents, 22 topics, 5,334 events, and 5,655 causal relations. Following existing works Gao et al. (2019); Tran Phu and Nguyen (2021), we use the last two topics as development set, and the other 20 topics are verified with a 5-fold cross-validation. For HiEve, the dataset contains 100 documents, 3,185 events, and 3,648 subevent relations. Similar to Zhou et al. (2020); Wang et al. (2020), we split the 100 documents into 60 training, 20 validation, and 20 testing.

Following the existing works Gao et al. (2019); Zhou et al. (2020); Tran Phu and Nguyen (2021), we use standard Precision (P), Recall (R), and F1-score (F1) as the metrics.

For LLMs implementation, we use the model API provided by OpenAI, where “gpt-4”, “gpt-3.5-turbo”, and “text-davinci-003” refer to models GPT-4, ChatGPT, and Text-Davinci-003 respectively.

For small-scale PLMs, we first select BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019). In addition, we also experiment with three strong and relevant models: (1) Hierarchical Adhikari et al. (2019b) which uses a pretrained model to encode different chunks of the document, and sets an additional BILSTM model to aggregate representations; (2) SIEF Xu et al. (2022) which proposes to randomly remove the useless sentences for prediction; (3) SCS-EERE Man et al. (2022) which selects a sentence set for each event pair for prediction.

The evaluated LLMs include: (1) GPT-4 which is an advanced and improved iteration of the GPT series, demonstrating human-level performance and significant enhancements in various aspects; (2) ChatGPT which is an advanced conversational AI model, is able to provide contextually relevant and coherent responses aligned with human expectation; (3) Text-Davinci-003 which is a variant of GPT-3.5 series, offering improved performance over GPT-3 through further instruction tuning.

We report the results of causal relation on MAVEN-ERE and EventStoryLine, respectively. Subevent relation performance is evaluated on MAVEN-ERE and HiEve, respectively.

Performance of causal relation  The results are shown in Table 1, TacoERE (PLMs) means the implementation of our compression-then-extraction method with PLMs. Experimental results demonstrate that TacoERE (PLMs) outperforms all the baselines on both MAVEN-ERE and EventStoryLine datasets. We also have the following four observations: (1) compared with pretrained model BERT, TacoERE (PLMs) achieves 4.2% improvements of F1-score on MAVEN-ERE, and 3.6% on EventStoryLine data; (2) compared with Hierarchical, which models different chunks of document, TacoERE (PLMs) achieves improved performance. This validates the effectiveness of cluster summarization, which can further mitigate information redundancy and event distance; (3) compared with SIEF, we note that TacoERE (PLMs) achieves better performance, even though SIEF is designed for modeling the important content by removing sentences from the original document. This indicates that our method equipped with compression-then-extraction can effectively alleviate the problem of long-range dependencies; (4) SCS-EERE performs better than Hierarchical and SIEF. It adopts a straightforward idea that selects a set of sentences for each event pair. However, this might cost too much computation time when applied to large-scale datasets, and we directly predict all event relations in a document at one time.

Performance of subevent relation  Table 2 presents the detailed results on MAVEN-ERE and HiEve, and our method again achieves better performance than all the baselines. Our method improves upon the pretrained model BERT by 3.8% and 4.5% in terms of F1-score on both datasets, respectively. In all, such performance on Table 1 and 2 clearly demonstrates the benefits of compression-then-extraction on event relation extraction.

To better understand the contributions of our method on long-range dependencies, we show the performance under different word distances between related events in Figure 3.

Compared with existing methods, our method achieves a certain improvement in dealing with long-range dependencies. We can also find: (1) as the word distance continuously increases, the improvement of our method over the baselines shows an overall monotonic upward trend; (2) the overall F1-score on event pairs with long distance is much lower than that of short ones, indicating the challenge of long-range dependency; (3) particularly, compared with the two baselines which directly models the entire document (RoBERTa), or compresses the document by removing irrelevant sentences (SIEF), our method obtain a larger improvement, especially when the word distance of related events is greater than or equal to 4, which can further indicate that our TacoERE (PLMs) can better handle the long-range dependencies.

In this section, we try different strategies to obtain the cluster and evaluate the performance to verify the effectiveness of our document compression. We consider the following two strategies: (1) “OneSum” means only generating one summary for each document; (2) “AvgSum” refers to splitting document evenly into chunks based on sentence, and then generating summary for each chunk.

The results are shown in Table 3, and we have the following three observations: (1) compared with the other two methods, i.e., OneSum and AvgSum, our proposed TacoERE (PLMs) achieves the best performance; (2) AvgSum obtains second-best results. However, the chunks are independent to each other in their setting, which means they ignore the relation information among chunks. This may prevent it from achieving better performance; (3) for the metric EP, OneSum only preserves 27.3% events of the document, which is due to the fact that summary mainly focuses on the important content of the document and the length is relatively short compared to the input document. As a whole, model performance gradually gets better as more events are preserved.

In addition to the document compression strategy, we also conduct experiments to ablate intra-clusters (w/o intra-clusters), inter-clusters (w/o inter-clusters) and cluster summarization (w/o summarization) to understand their contributions. The results are shown in Table 4. We can observe that: (1) compared the results of w/o (without) intra-clusters with w/o inter-clusters, we can find that the overall model works better. The use of intra- and inter-clusters helps the model to better understand the event relations both within and among the sentences; (2) the performance of TacoERE drops on the three variations, which proves that both variations have contributed to the overall performance; (3) removing w/o intra-clusters causes a sharp performance drop compared to w/o inter-clusters.

The aforementioned series of experiments has highlighted the efficacy of our TacoERE, based on PLMs, in enhancing ERE performance. To further validate its effectiveness and robustness, we conduct extensive experiments on LLMs. Illustrated in Figure 2, our framework primarily comprises three key components: Document Clustering, Cluster Summarization, and Relation Prediction. We introduce TacoERE (LLMs), which directly leverages LLMs to implement these three components. To thoroughly validate our approach, we configure the task to enable relation prediction between event pairs individually. For testing, we randomly sample 50 documents from MAVEN-ERE, resulting in 646 causal relations. We compare TacoERE (LLMs) with three variants: (1) Document, which involves utilizing the entire document to predict relations; (2) Sentence Pair, which entails using sentences containing the event pairs to predict relations; (3) Document Clustering, which involves using sentences within a cluster to predict relations.

The results are shown in Table 5, we notice that our TacoERE (LLMs) achieves the best performance across all three models, with improvements of 11.2% and 9.1% on ChatGPT and GPT-4, respectively. We also have the following four observations: (1) from model perspective, GTP-4 achieves the highest F1 score of 41.9%, followed by ChatGPT; (2) compared with Document, Sentence Pair obtains better results, indicating that predicting relation between events does not depend on the whole document; (3) compared with Document and Sentence Pair, Document Clustering achieves improved results, indicating that our method can reduce redundant information while retaining useful information for relation prediction; (4) compared with Document Clustering, our TacoERE (LLMs) achieves the best performance, suggesting that reducing redundant information and shortening event distance can further facilitate the improvement of performance.

We display a case in Figure 4 to quantitatively analyze the model prediction by our TacoERE (LLMs) and different comparison modules, such as Document, Sentence Pair, and Document Clustering. TacoERE (LLMs) means we use the content from Cluster Summarization to predict relations. We can see, for each event pair, the prediction does not rely on the whole document. Some dependency information may be ignored by using Document Clustering for prediction. Our proposed TacoERE (LLMs) with compression-then-extraction is an effective means to enhance event relation extraction.

Event Relation Extraction is a challenging task in natural language processing, especially for events that are scattered in different sentences Gao et al. (2019); Chen et al. (2022). Recently, deep learning based methods are becoming the mainstream Cao et al. (2021); Xu et al. (2021), and extensive explorations have been made, such as joint reasoning methods which extract multiple relations simultaneously Ning et al. (2018); Han et al. (2019), and graph-based methods which use event mentions as nodes and model document as graph Tran Phu and Nguyen (2021); Fan et al. (2022); Guo et al. (2023). However, these works use document as input, which cannot well handle the long-range dependency problem. In contrast, we improve event relation extraction by processing document in advance with cluster-aware compression.

Currently, a series of LLMs have been developed, such as GPT series, LaMDA Thoppilan et al. (2022), and PaLM Chowdhery et al. (2022), and have achieved remarkable performance in various fields. Among them, GPT series, i.e., GPT-4 and ChatGPT, is undoubtedly the most popular work. Thus, to verify the effectiveness of our method, we conduct extensive experiments on them.

With the development of deep learning and the increasing demand for generation quality, increasing studies are focusing on controlled text summarization Dou et al. (2021); Guan et al. (2021a), such as designing copy mechanism to directly copy important word from input See et al. (2017), extracting actual fact triples for modeling Cao et al. (2017), extracting templates from the training data to guide the summarization generation Wang et al. (2019). However, these methods typically design an additional module or using existing third-party content selectors. On the contrary, our method adopts the events in document as intermediate representation to better guide the summary generation.