Only One Relation Possible?
Modeling the Ambiguity in Event Temporal Relation Extraction

In our work, we adopt pre-trained language models BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019) as the encoder module. Taking the text sequence $\mathbf{S}$ and an event pair $(e_{1},e_{2})$ as input, it computes the contextualized representation for the event pair. Following Zhong and Chen (2021), we insert typed markers to highlight two event mentions in the given text at the input layer, i.e., <E1:L> $e_{1}$ <E1:R>, and <E2:L> $e_{2}$ <E2:R>. Additionally, inspired by Wadden et al. (2019), we add cross-sentence context, i.e., one sentence before and after the given text, expecting pre-trained language models to capture more comprehensive information. We feed the new sequence $\tilde{\mathbf{S}}$ into the encoder and obtain the event pair representation:

$$\mathbf{h}_{e_{1},e_{2}}=MLP_{1}([\mathbf{x}_{\hat{e_{1}}}||\mathbf{x}_{\hat{e_{2}}}])$$

(1)

where $\mathbf{x}$ is the output representation, $\hat{e_{i}}$ is the indices of $e_{i}$ in $\tilde{\mathbf{S}}$, [·||·] is the concatenation operator, and $MLP_{1}$ is a multilayer perceptron.

Different from single-label classifications, which directly choose the relation of the highest probability from $\tilde{R}$, we adopt a multi-label classifier to better characterize Vague. Taking the event pair representation $\mathbf{h_{e_{1},e_{2}}}$ from encoder as input, our classifier will calculate the probability distribution over well-defined relation set $R$ and a threshold value $T$. Inspired by Zhou et al. (2020), we adopt a learnable, adaptive threshold for our classifier to make decisions. Therefore, we obtain the probability distribution $P(R|e_{1},e_{2})$ and $T_{e_{1},e_{2}}$ with:

$$P(R|e_{1},e_{2})=MLP_{2}(\mathbf{h}_{e_{1},e_{2}})$$

(2)

$$T_{e_{1},e_{2}}=MLP_{3}(\mathbf{h}_{e_{1},e_{2}})$$

(3)

where $P(R|e_{1},e_{2})\in\mathbb{R}^{|R|}$ and $T_{e_{1},e_{2}}\in\mathbb{R}$.

Therefore, the temporal relationship of event pair $(e_{1},e_{2})$ can be deduced from $P(R|e_{1},e_{2})$ and $T_{e_{1},e_{2}}$. Specifically, if only one relation whose probability is higher than the threshold, we consider this well-defined relation as the final prediction. While if more than one relation’s probability is higher than the threshold, we think the temporal relation of the event pair is Vague, and we can provide detailed information on exactly what relations cause the ambiguity. As the example shown in Figure 1, we infer the temporal relation as Vague due to the high possibility of both relation Before and After. There are also some cases that all relations’ probability are lower than the threshold, and we map it to Vague as well, which can be explained as the model does not have enough confidence in any relation for the event pair.

When training the multi-label classifier, we need the gold labels of each event pair to calculate the training loss. However, this is not trivial in our case. For the instances labeled as Vague, we do not exactly know their original possible well-defined relation labels, e.g., Before and Include for the example sentence in Table 1. Therefore, we need a new mechanism to identify those possible well-defined relations behind Vague to make sure proper training for our model, i.e., providing a proper reward to those original well-defined relation labels which result in Vague. In our approach, we first utilize $P(R)$ to speculate a dynamic Confusion Set, which consists of the potential relations underlying Vague. Then relations in Confusion Set are regarded as gold labels of Vague at the current stage, and are used in training loss calculation.

We evaluate our model in three public temporal relation extraction datasets. Detailed data statistics of each dataset are reported in Appendix A.

Following Zhang et al. (2022), we utilize the symmetry property of temporal relations to expand our training set for data enhancement. For example, if the temporal relation of an event pair $(e_{1},e_{2})$ is Before, we add the event pair $(e_{2},e_{1})$ with relation After into the training set. We do not expand the validation set and test set for a fair comparison.

We use micro-F1 score as the evaluation metric following the previous works Zhang et al. (2022); Mathur et al. (2021). Besides, we implement a baseline model, which adopts the same encoder as our model, and use an MLP as a single-label classifier over $\tilde{R}$, for a better comparison of our approach.

Table 3 compares the performance of our approach with previous works and our enhanced baseline. Our model makes a significant improvement based on the baseline model on all three benchmarks, and outperforms current state-of-the-art models by 0.6% on TBDense and delivers a comparable result on MATRES. By modeling the latent composition of Vague, our model can effectively leverage the hidden information about well-defined relations, leading to an enhanced understanding of temporal relations.

As Vague accounts for nearly half of the training set of TB-Dense, which is far more than the proportions in MATRES and UDS-T, our model can explore more information from Vague when trained on TB-Dense. Therefore, compared to baseline model, our model shows the largest improvement by 2.6% F1 score on TB-Dense with BERT-Base as encoder, while 1.5% and 1.0% F1 improvements on MATRES and UDS-T, respectively. Experiments with RoBERTa-Large also show a similar trend, which indicates the efficacy of our model in extracting concealed information from Vague to facilitate temporal relation prediction.

Our approach is still effective when provided with better event pair representations. As the encoder is changed from BERT-Base to RoBERTa-Large, our model shows a stable improvement on all benchmarks compared to the baseline. Due to our approach’s exclusive focus on the classifier module, independent from the majority of previous works that concentrate on enhancing the encoder to acquire superior event pair representations, we believe that our approach can be effortlessly integrated into their models as a more effective classifier, and achieve better performance.

The key of our approach is how we incorporate the mystery of Vague into the design of $CS$. To explore the impact of $CS$, we implement the model variant METRE w/o $CS$, where $CS$ is set to empty. Besides, we add the penalty on the relation $r\notin CS$ in another model variant METRE w. Pnt, to evaluate the negative effect of the penalty mechanism. Experimental results are shown in Table 4.

$CS$ is composed of two parts: Top2 relations and confusion relation of Top1 relation. So when faced with Vague, our model learns sufficient knowledge about at most three well-defined temporal relations. By leveraging such helpful information from Vague, our model can have a better understanding of these relations. Therefore, we compare the performance of our model on every relation to baseline model, and report their F1 scores on TB-Dense in Table 5. On all well-defined relations, our model shows higher F1 scores than baseline. This result proves that, with a stronger comprehension of temporal relations, our model is more confident in making decisions on a certain well-defined relation, and can discriminate them from each other better.

The F1 score of Vague is slightly decreased, and we found that it mainly results from its decrease in recall score, which is also reported in Table 5. We can observe that, except Vague, the recall scores increase greatly in all other relations, which is an encouraging indication that our model tends to recognize well-defined temporal relations between event pairs rather than predicting them as Vague. Such a capability can enhance the utility of our model in downstream tasks, such as timeline construction, by providing more definitive decisions.

When defining $CS$, we assume confusion relation is an important factor in causing the ambiguity of Vague. Thus we carefully check what relations are brought into $CS$ as a confusion relation. We calculate the proportion of each class as a confusion relation in $CS$, and compare it to their proportion in the training set, as shown in Figure 2. The minority class, i.e. Include, Is included and Simultaneous, which only account for 29.0% in the training set, makes up of 81.4% in the confusion relation part of $CS$. While the majority class, i.e. Before and After, only has a proportion of 18.6%. This phenomenon may result from imbalanced label distribution in the dataset. As this exists in most temporal relation benchmarks, the model tends to predict the majority class with higher probability, so more minority class can be brought in by the majority class through confusion relations. Although it provides a valuable opportunity for the information of minority classes to be learned, we still need to figure out whether the confusion relations play an active role in explaining the intrinsic composition of Vague.

Therefore, we compare the performance of both majority class and minority class to our baseline model. Table 2(b) shows that, the minority class achieves a greater improvement of 4.6% F1 score and 20.3% relative improvement compared to baseline, while majority class only achieves 1.0% and 1.7% improvement respectively. The positive effect on the performance of minority class demonstrates the effectiveness of confusion relations, which can reasonably explain where the uncertainty of an event pair’s temporal relationship comes from. Besides, by providing more opportunities for our model to explore and learn the knowledge of minority classes, it enables us to better understand the minority class, and alleviates the imbalanced label distribution issue to some extent.

Event pairs with temporal relation Vague contain sufficient information about well-defined relations. In our approach, our model learns the meaning of a well-defined relation not only from itself, but also from Vague. The importance of supplementary information provided by Vague could be more obvious when the training resources are limited. Therefore, to evaluate our model’s capability in effectively exploring sufficient information from training data, we conduct experiments with variable amounts of training data, ranging from 5% to 50%, while keeping the test set unchanged. In low resources scenarios, our model outperforms baseline model by 7.1% on average F1 score. Figure 3 illustrates the experiment results of the F1 scores of baseline model and our model on TB-Dense.

We can observe that, our model stably outperforms baseline model in every portion of the training set. This result indicates that our model is more capable of capturing the information in Vague efficiently and making full use of it. When the smaller size of the training set is used, such as 5% to 30%, our model shows a larger difference from baseline. This phenomenon mainly results from the different amounts of information models can obtain. Since there are only a few well-defined relations in the small training set, single-label classification-based models are struggling to recognize them correctly. However, our model’s capability in leveraging Vague to learn knowledge of well-defined relations plays an important role, which helps to capture effective information from the limited training set more comprehensively and efficiently.

In addition, due to the imbalanced label distribution, minority classes like Include and Is included are hardly included in the small training set. Therefore, single-label classification-based methods may have difficulty in making predictions on these relations. Our method, on the other hand, can explore their information through a large amount of Vague, and achieves higher average F1 scores of 4.7% and 6.2% on Include and Is included respectively.

Treated as a multi-label classification task, our model is able to predict relations composition of Vague. To better evaluate the prediction accuracy, we check the consistency between our prediction of Vague and the annotation results from humans. As we mentioned in Section 4.1, every event pair’s relation is determined by 3 annotators, so we can calculate the overlap of the prediction of our model with the 3 different annotation results. Specifically, if we make a correct prediction on Vague, we choose the Top$K$ relations and judge whether these $K$ relations all conform to the annotation results. The precision scores are shown in Table 6.

Our model outperforms baseline by 5.2%, 14.4% and 8.3% in Top1/2/3 precision respectively, demonstrating a greater capability of our model in predicting the intrinsic composition of Vague. Given the precision of random ranking, we calculate the relative precision value as $Pr_{ours}/Pr_{random}-1$ and $Pr_{base}/Pr_{random}-1$. As $K$ increases, our model shows a larger improvement in relative precision than baseline, from 1.30 times to 1.84 times, which indicates that our advantage is more obvious when the model is asked to provide full information of Vague.