Cross-document Event Identity via Dense Annotation

Widely studied in the literature, with datasets curated for both within and cross-document tasks. ACE 2005 Walker et al. (2006), OntoNotes Weischedel et al. (2013), and TAC-KBP Mitamura et al. (2017) are commonly used benchmarks for within-document coreference. For cross-document coreference, ECB+ Cybulska and Vossen (2014) is a widely popular benchmark and is an extended version of the original ECB dataset Bejan and Harabagiu (2008). ECB+ suffers from a major limitation with coreference annotations restricted to only the first few sentences in the documents. However, CDEC is a long-range phenomenon, and there is a need for more densely annotated datasets.

Many other datasets have since been curated for the task of CDEC. Some related works include, MEANTIME Minard et al. (2016), Event hoppers Song et al. (2018), Gun Violence Corpus (GVC) Vossen et al. (2018), Football Coreference Corpus (FCC) Bugert et al. (2020), and Wikipedia Event Coreference (WEC) Eirew et al. (2021). However, most CDEC systems are still evaluated primarily on ECB+. Additionally, all of these datasets do not account for the quasi-identity nature of events.

Though compiled from Wikinews, CDEC annotations in the MEANTIME corpus were limited to events with participants from a pre-defined list of 44 seed entities. While the FCC corpus was also crowdsourced, the annotation unit was an entire sentence instead of a single event mention. WEC corpus uses hyperlinks from Wikipedia but primarily handles referential events. In this work, we use open-domain events and treat an event mention as our annotation unit. We collect coreference links across all the mention pairs from all the underlying document pairs.

Recasens et al. (2011) postulated entity coreference as a continuum, with identity, non-identity and near-identity relations. In a follow-up work Recasens et al. (2012), they identify near-identity relations using the disagreement between annotators. They say subjects are not fully aware of the near-identity behavior, therefore making direct annotation collection hard. The continuum idea has since extended to events Hovy et al. (2013). Determining if two event mentions are identical is not a trivial decision. It depends on the arguments of the mentions (often underspecified in the local context), the semantics of the mention, and the document context. In this work, we are specifically interested in cross-document coreference. Wright-Bettner et al. (2019) studied the impact of the subevent relationship on quasi-identity, but a more general annotation framework is missing. Accurately capturing event identity is critical to CDEC dataset construction and the subsequent modeling. Therefore, we qualitatively study this phenomenon by collecting supplementary information with each coreference link.

To facilitate the redistribution of the documents under an open license, we prioritized collecting the documents from publicly available news sources. We chose Wikinews for three key reasons. First, the news articles were sourced from trusted news outlets and reported impartially. Second, these articles are available under an open license (CC BY 2.5), allowing easy redistribution. Finally, each article is human-labeled with categories (e.g., Disaster and accidents, Health, Sports, etc.),⁵⁵5https://en.wikinews.org/wiki/Wikinews:Categories_and_topic_pages as we describe later, this meta-information plays a significant role in our dataset collection. We use the July 1st, 2020 dump of English Wikinews, which contains a total of 21k titles (or articles/documents). These news articles are timestamped from November 2004 to July 2020. Annotating coreference between every document pair in Wikinews is infeasible. Therefore, we first identify groups of related news articles. Articles within a given group usually describe a part of a developing news story or storyline.

To identify these latent storylines, we first construct an undirected Wikinews graph ($W$) with articles as nodes and add an edge between two nodes if one is mentioned under the “Related News” section in the other. We then identify cliques ($C_{W}$) (i.e., fully connected sub-graphs) in the Wikinews graph, which constitute our potential set of storylines. While the articles within each clique are related, we also want to minimize the relatedness of articles across cliques. Therefore, we construct a new graph ($M$), where each clique ($\in C_{W}$) is a node, and an edge is added between two nodes if the two cliques are not disjoint or if any two articles in the two cliques share an edge in the Wikinews graph ($W$). Finally, we extract maximal independent sets from $M$ that correspond to separate storylines. Among the multiple feasible maximal independent sets, we optimize for maximum overlap in Wikinews categories of articles within each clique.

This algorithm satisfies two requirements of a CDEC dataset. First, within each storyline, all articles are related to each other. Second, articles from different storylines aren’t adjacent in the Wikinews graph ($W$); thereby, they are very likely unrelated.

For this work, we narrow our focus only to articles in the “Disaster and Accidents” category on Wikinews.⁶⁶6https://en.wikinews.org/wiki/Category:Disasters_and_accidents Following the terminology of prior work, our dataset constitutes of a single topic (Disaster and accidents) and 55 subtopics (individual storylines). We restrict CDEC annotations to subtopics that contain 3 or 4 documents. Our algorithm aims for completeness of the CDEC dataset by maximizing for intra-subtopic and minimizing inter-subtopic coreference.

To annotate the event mentions in the above-collected documents, we first run a combination of mention detection systems. Specifically, we use the OpenIE system Stanovsky et al. (2018) from AllenNLP Gardner et al. (2018) and an open-domain event extraction system Araki and Mitamura (2018). The former is effective at extracting verbal events, whereas the latter is good at nominal events. In contrast to most prior work, we do not restrict the mentions to specific event types or salient events. We believe it is important to study all underlying events to achieve a complete understanding of the corpus. Since the quality of mention identification is critical to our CDEC dataset, we ask an expert to go through the automatically identified mentions and add/edit/delete mentions using the Stave annotation tool Liu et al. (2020).⁷⁷7the expert annotator is an author of this work.

Table 1 presents the overall statistics of our document corpus. Our documents are $\sim$14.6 sentences long, comparable to prior work, ECB+ (16.6), GVC (19.2), and FCC (34.4). However, our documents are significantly more dense in terms of event mentions. Our documents contain $\sim$41 mentions (on avg.), much higher compared to prior work, ECB+ (15.3), GVC (14.3), FCC (5.8). Given the dense nature of our documents, we appropriately design our annotation task and interface.

We use the responses to follow-up questions for qualitatively analyzing cases of partial identity. We consider a link to be a case of partial identity if a strict majority of annotators indicate one of the following. First, there is an inclusion relationship between corefering mentions. Second, the two overlap in place, time, or participants. With this screening methodology, we found $\sim$32% of the total CDEC links to be candidates for partial identity (Table 1). We qualitatively analyze the dataset and identify three types of partial identity, 1. Membership, 2. Subevent, and 3. Spatiotemporal continuity. Table 2 illustrates each type with examples from our compiled dataset.

An event mention $E_{a}$ is a member of event mention $E_{b}$. Consider the two sentences, 1a, and 1b. The mention ‘fire’ (1a) denotes a specific wildfire, whereas ‘wildfires’ (1b) denotes a group of wildfires, including the one in 1a. The concept of partial identity often challenges the transitivity assumption of coreference. For instance, the mentions [smaller] ‘aftershocks’ (2b) and [7.1] ‘aftershock’ (2a) share no identity, thereby, non-coreferential. However, both the mentions partially corefer with [60] ‘aftershocks’ from 2c.

An event mention $E_{a}$ is a subevent of event mention $E_{b}$. This behavior can be seen in the coreference between the ‘crash’ event from 3a, and the ‘accident’ event from 3b. While the ‘accident’ event involves many individual events, derailed, caught fire, spill chemical, and release fumes, it partially corefers with the event ‘crash’ from 3a that likely refers only to the derailment. Similarly, consider the case of the Boston Marathon Bombing in examples 4a and 4b. The ‘bombing’ event from 4a refers to the whole incident, whereas the ‘explosions’ in 4b refers to specific subevents of the ‘bombing’.

The identity of an event can continuously evolve over space and time. Consider the two mentions, ‘storm’ and ‘Hurricane’ from Table 2 (5a, 5b). At a high level, these mentions are corefering because they denote the same event (storm Richard). However, the expressions of this event differ slightly across the two documents. In the former, it’s a storm (with 70mph winds) having an impact in Honduras, whereas, in the latter, it’s a hurricane (with 90mph winds) impacting Belize. Similar behavior is visible with the [Haitian cholera] ‘outbreak’ event from Figure 1. The outbreak gradually evolves, with growing infection (2600 $\rightarrow$ 3000 $\rightarrow$ 4000) and deaths (200 $\rightarrow$ 259 $\rightarrow$ 292). In both of these examples, we observe the event changes gradually and is always continuous in both space and time dimensions.¹⁸¹⁸18We borrow the term spatiotemporal continuity from the Philosophy literature. It describes the properties of well-behaved objects Wiggins et al. (1967). A similar treatment for entities is presented in Recasens et al. (2011).

In line with prior work on entities Recasens et al. (2011), we believe identity and coreference of events to be a continuum. Our dataset already includes many instances of partial identity to support this hypothesis. The above-described cases of partial identity (membership, subevent, and spatiotemporal continuity) will pose new challenges to future dataset collection efforts. We believe our annotation workflow and guidelines will be of use to future work.

In this section, we establish a clear case for tackling partial identity within the coreference resolution task. However, in practical settings, the boundaries between full, partial, and null identities remain fuzzy. As seen in our analysis on the inter-annotator agreement, humans find it hard to identify cases of partial coreference. In the downstream coreference resolution task, users are primarily interested in knowing if two given mentions share an identity or not. Therefore, we propose to view both full and partial identity under a single coreference label (‘coreference’) and contrast them against cases with no shared identity (‘non-coreference’). Compared to prior datasets, this presents new challenges in tackling partial identity within the ‘coreference’ label.

For our first baseline, we implement the traditional lemma-match baseline. We use spacy’s large model²⁰²⁰20en_core_web_lg from https://spacy.io to extract the head lemma of the event mentions, and consider two mentions corefering if the lemma’s match. Following Upadhyay et al. (2016), we also experiment with a Lemma-$\delta$ baseline. In our experiments, we found the best dev performance with $\delta$=0, resolving to a simple lemma baseline. This could be due to our assumption of access to gold subtopic information.

As a second baseline, we implement BERT-based cross-encoder model. The input consists of a pair of sentences with both mentions highlighted using special tokens to indicate the start and end of mention spans (<E>, </E>). We first concatenate the two event-tagged sentences (with [SEP] token) and pass it through a bert-base-uncased encoder. We then perform mean pooling on the event start tags (<E>), and pass the pooled embedding through a linear classification layer to predict coreference vs. non-coreference. For training the cross-encoder, in addition to the positive coreference pairs, we generate two types of negative mention pairs. For the first type, we collect non-coreference mention pairs from sentences that have a coreference link between a different mention pair. For the second type, we extract non-coreference mention pairs from random sentence pairs between the documents. During training, we use a dataset ratio of 1:5:5 (positive:negative-I:negative-II). We use huggingface transformers Wolf et al. (2020), and train the model using AdamW Loshchilov and Hutter (2019) with an initial learning rate of 2e-5. We also use a linear warmup scheduler, with 10% of training steps for warmup. We finetune the # epochs and positive:negative dataset ratio during the development stage (5-fold cross-validation) and use the best configuration when training on the entire train set.

Table 3 presents the results of our baselines. For model development, we perform 5-fold cross-validation on the training set (40 subtopics). To report the results on the held-out test set (15 subtopics), we train the model’s best configuration on the entire training set. We report precision, recall, and F1 scores of the coreference label averaged on five different runs. The lemma baseline only achieves an F1 score of 48.2, indicating that the proposed dataset is lexically diverse. The cross-encoder improves upon the lemma baseline, especially on the recall. Upon inspection of development set predictions, we observe two possible error cases for the cross-encoder model. First, the model struggles at the cases of partial identity (‘explosion’ vs. ‘incident’ and ‘evacuate’ vs. ‘evacuations’). This drawback of cross-encoder indicates that the model requires a deeper understanding of event identity. Second, the cross-encoder model is often limited by the information available in a single sentence. It is known the event arguments are often underspecified in the local context Ebner et al. (2020); therefore, increasing the context to a paragraph or the entire document might help improve the performance.