Cross-document Event Coreference Search: Task, Dataset and Modeling

Recent within-document coreference resolution models Lee et al. (2018); Joshi et al. (2019); Kantor and Globerson (2019); Wu et al. (2020), were inspired by the end-to-end model architecture introduced by Lee et al. (2017). In particular, two distinct components were adopted in those works, which were shown to be effective in detecting mentions and their coreference relations, both in the within-document and cross-document Cattan et al. (2020) settings. In our proposed model, we similarly adopt those two components to better represent coreference relations, in the coreference search settings.

The CoreSearch passage collection which contains manually annotated event mentions was created by importing the validated portion of the WEC-Eng Eirew et al. (2021) dataset (§2.1 and Table 1).

Specifically, we merged the WEC-Eng validated test and development set coreference clusters into a single collection of 522 none-singleton clusters (“Non-Singleton Culsters” in Table 1 and “Clusters” in Table 2). We then split the clusters between CoreSearch train, development and test sets. Each cluster contains passages that form our annotated passage collection.

Those passages will serve the roles of queries and of positive retrieved coreferring passages.

In order to collect a large collection of passages for challenging and realistic retrieval, we generate negative passages (i.e., destructing passages) using two resources: (1) The entire WEC-Eng train set, which is not manually validated, though quite reliable; (2) By extracting the first paragraph of any Wikipedia article not containing a hyperlink to any of the CoreSearch annotated passages, and hence are unlikely to corefer with any of them (Table 2).

We observe that our annotated data is characterized by two prominent types of coreference clusters: Type-1 - clusters containing only passages with event mention spans that include the event time or location (e.g., “2006 Dahab bombings”, “2013’s BET Awards”), and Type-2 - clusters that are comprised partly of passages as in Type-1, as well as passages containing mention spans without any event identifying participants (e.g., “the deadliest earthquake on record”, “BET Awards”, “plane crash”). Naturally, Type-2 clusters will create queries/passage examples with a higher degree of difficulty. Identifying coreference for Type-2 clusters is indeed challenging in our dataset, because WEC-Eng includes a multitude of event mentions which are similar lexically but do not corefer (e.g., different earthquakes) Eirew et al. (2021), requiring a model to identify event coreference using the arguments in the surrounding context.

To measure the distribution of cluster types within CoreSearch, we randomly sampled 20 clusters and found 90% are of type-2, demonstrating the challenging nature of the CoreSearch data. Table 3 illustrates examples of queries extracted randomly from five type-2 clusters.

The CoreSearch dataset is built on top of the English version of WEC (WEC-Eng). Consequently, since WEC is adoptable to other languages with relatively low effort Eirew et al. (2021), and the process for deriving CoreSearch from it is simple and fully automatic, the CoreSearch dataset may be adopted to other languages as well with very similar effort (as for WEC).

In order to accommodate our setup of mention-directed search and to better signal the model to be aware of the query event mention, we edit the query by marking the span of the mention within the query passage by using boundary tokens. Given the query event mention span $m_{i}=[q_{i}^{k},q_{i}^{k+1},\ldots,q_{i}^{k+l-1}]$, we append the boundary tokens to obtain the final edited query ($m_{i}$ denotes the sequence of the mention’s tokens):

For implementing the text encoders $E_{Q}(\cdot)$ and $E_{P}(\cdot)$, we employed the SpanBERT⁵⁵5DPR originally used BERT Devlin et al. (2019) as their query and passage encoders. Joshi et al. (2020) model as our query and passage encoders. SpanBERT is an appealing encoder, as it was pre-trained for better span representations, rather then the individual tokens, and was also shown to be more effective for coreference resolution tasks Joshi et al. (2020); Wu et al. (2020).

During our preliminary experiments, we observed that both the additional event mention marking as well as replacing BERT with SpanBERT contributed significantly to the performance over our dataset.

We construct our positive and negative examples by iterating sequentially through every training set event coreference cluster $C_{j}=[m_{1},m_{2},\ldots,m_{|C_{j}|}]$, where $m_{i}$ denotes an event mention surrounded with its context (the entire passage). Given each event mention $m_{i}$ acting as a query $q_{i}$, we construct one positive coreference example for each of the remaining $|C_{j}|-1$ coreferring event mentions in the cluster. Then, for each such positive example, we first construct one “challenging" negative example by selecting randomly one of the top-20 passages returned by the BM25 retrieval model for the corresponding query. In addition, for each query in a training batch, we create additional (“easier") in-batch negative examples by taking the “challenging" passages of all other queries in the current batch, similarly to Karpukhin et al. (2020).

Let ${\mathcal{D}={\langle q_{i},p^{+}_{i},p^{-}_{i,1},\ldots,p^{-}_{i,n}\rangle}^{m}_{i=1}}$ be the CoreSearch training set. Similarly to Karpukhin et al. (2020), the goal is to optimize the negative log likelihood loss of the positive passage, which is based on the contrastive loss:

We train the two separate encoders using a maximum query size of 64 tokens for the query encoder. In order to cope with memory constrains, we limit the maximum passage size given to the passage encoder to 180 tokens. Batch size is set to 64. We train our model using four 12GB Nvidia Titan-Xp GPUs.⁶⁶6We leveraged and modified the implementations in the Haystack framework of the DPR and BM25 models: https://github.com/deepset-ai/haystack

We train the single cross-encoder using a maximum sequence size of 256 tokens, in order to cope with memory constraints. We use up to 64 tokens from the surrounding query mention context (which in many cases take less then 64 tokens) for query representation, and concatenate the passage context using the remaining available sequence. In case the passage context length exceeds available sequence size for passage representation, we segment the passage using overlapping strides of 128 tokens, creating additional passage instances with the same query. The batch size is set to 24, and both $\texttt{FFNN}_{m}$, $\texttt{FFNN}_{a}$ use a single hidden layer set to 128. We train the models using two 12GB Nvidia Titan-Xp GPUs.

All models parameters are updated by the AdamW Loshchilov and Hutter (2019) optimizer, with a learning rate set to $10^{-5}$ and a wight-decay rate of 0.01. We also apply a linear scheduling with warm-up (for 10% of optimization steps) and dropout rate of 0.1. We train all models for 5 epochs and consider the best performing ones over the development set. At inference, we set the retriever top-$k$ parameter to 500.

Following common evaluation practices, we set $k$ to 10, expecting that the topmost correct result should appear amongst the top 10 results (that is, no credit is given if the topmost correct result is ranked lower than 10).

We report recall at ${k\in\{10,50\}}$ for the end-to-end model evaluation, assessing recall in two prototypical cases where the user might choose to look at rather few or rather many results. For the retriever model we report recall at ${k\in\{10,100,500\}}$, illustrating the motivation for the $k=500$ cutoff point that we chose (beyond which there were no substantial recall gains).

The mAP metric assesses the ranking quality of all correct results within the top-k ones, measured for $k\in\{10,50\}$, as measured for recall.⁷⁷7We use mAP rather than Normalized Discounted Cumulative Gain (NDCG), because the latter requires a scaled gold relevancy score for each query result. mAP applies a similar ranking evaluation criterion, but is suitable for binary relevancy scores, which is the case in our coreference setting.

We use the above metrics with the additional question answering (QA) measurements of Exact Match (EM) and token level F1 score⁸⁸8Using the official SQuAD evaluation script. with the reference answer after minor normalization as in Chen et al. (2017); Lee et al. (2019); Karpukhin et al. (2020).⁹⁹9We note that QA measurements only take into consideration lexical matches. However, equal lexical representation does not necessarily imply a coreference relation (for example, the mention plane crash can appear twice in the same passage, each time referring to a different plane, thus denoting different events). To that end, we add a necessary constraint limiting relevant answers only to those where the answer index within context intersects with the gold mention span.

Table 4 summarizes the retriever performance results over the CoreSearch test set. Our retriever model surpasses the BM25 method (see further details in Appendix A.1) by a large margin on every metric (Table 4, BM25 versus $\mbox{Retriever-S}^{+}$). It should be noted that BM25 is considered a strong information retrieval model Robertson and Zaragoza (2009), also compared to recent neural-based retrievers Khattab and Zaharia (2020); Izacard et al. (2021); Chen et al. (2021); Piktus et al. (2021). We observed this phenomenon during our experiments, as the underlying DPR retriever (i.e., BERT without boundary tokens), yielded poor results on our settings, surpassed by the BM25 model on all measurements by a significant gap (Table 4, BM25 versus Retriever-B).

Table 5 presents our end-to-end system results applied over the CoreSearch test set. We found that both of the reader models (E2E-DPR and E2E-Integrated) present appealing performance given different measurement aspects we now describe.

We conclude from the recall results (R@10 and R@50) that the $\mbox{E2E-DPR}^{-}$ model is an effective re-ranking model, ranking almost all relevant passages extracted by the retriever within the top 50 results (86.62% out of maximum of 87.12% ranked by the $\mbox{Retriever-S}^{+}$ model at top-500). The EM and F1 results indicate that the E2E-Integrated model gains better mention extraction capabilities compared to both E2E-DPR models (by 1.5% EM and 1.2% F1 compared to $\mbox{E2E-DPR}^{+}$).

Finally, the MRR and mAP results indicate that the E2E-Integrated model overall performs better then both E2E-DPR models at ranking relevant passages at higher ranks (indicated by MRR@10, mAP@10 and mAP@50 in Table 5). In particular, we find that the MRR@10 results are especially appealing (90.06%), showing the model predominately ranks a relevant passage at the first or second position.

Finally, Table 6 illustrates a sample of the E2E-Integrated top-2 model results, on a sample of queries containing mention spans not including event arguments, randomly sampled from five Type-2 CoreSearch clusters (§3.1). The table Illustrates the model effectiveness in returning relevant passages and the coreferring mention within them.

We observed that on rare occasions the model returns a relevant passage (and a coreferring mention) marked as negative in the dataset. We sampled 15 queries and manually validated their top-10 answers. We found that from 58 negative results, only 1 was a false negative, indicating that indeed this phenomenon is rather rare and insignificant. Such false negatives can originate either from the WEC-Eng training set (§2.1), or from our destructing passage generation (§3). Notice that, such false negatives can only have a deflating effect on results.

In both our retriever and reader experiments, we found that adding the span boundary tokens around the query mention, provides a strong signal to the model. In our retriever experiments, while most of the gain to performance was originated by replacing the BERT model with SpanBERT (Retriever-B and $\mbox{Retriever-S}^{-}$ in Table 4), applying boundary tokens significantly improved performance further all across the board ($\mbox{Retriever-S}^{+}$ in Table 4).

However, in our end-to-end model experiments, we observed that applying boundary tokens will help the model mostly to improve at span detection, while less so at re-ranking ($\mbox{E2E-DPR}^{-}$ and $\mbox{E2E-DPR}^{+}$ in Table 5).

Our main motivation for replacing the DPR reader passage selection method (Eq. 5), with a coreference scoring one, was to create a better passage selection mechanism for re-ranking. Indeed, this modeling prove efficient both at re-ranking, as well as at mention detection, as indicated by the E2E-Integrated model results in Table 5.