Event Linking: Grounding Event Mentions to Wikipedia

We first collect all hyperlinks $(hypertext,\ title)$ in Wikipedia text, which links a hypertext to a Wikipedia title. Then, we map the FreeBase type of Wikipedia titles to FIGER types (Ling and Weld, 2012), and all titles with a type “Event” are regarded as event titles. All the hypertexts linked to these event titles are regarded as event mentions.

Because same event mentions in one Wikipedia page are hyperlinked only once, and editors tend to hyperlink more nominal mentions than verb mentions, verb mentions are highly limited in Wikipedia. To balance the size of verbs and nominals, we use SpaCy Part-of-Speech model²²2https://spacy.io/usage/linguistic-features##pos-tagging to keep all verb mentions, and sample the same size of nominals. To prevent models from overfitting, we design hard and easy cases for verbs and nominals:

Verbs: We classify each verb mention mainly by whether the surface form (S) of the verb is seen in training data, and whether the gold event title (T) is seen in training data. If both S and T are seen in training data, we call it Seen Event. If T is seen in training data, but S is new, we call it Unseen Form. If T is never seen in training data, we call it Unseen Event. Under this setting, “Seen Event" is regarded as easy cases, and the other two are hard cases. Because of the limited size of verb mentions, all the event titles with fewer than or equal to 5 verb mentions are used as “Unseen Event".

Nominals: We classify each nominal mention mainly by its surface form similarity to the gold title. We calculate the Jaccard similarity between the nominal mention and the gold title by taking 3 grams of the surface form. If the similarity is lower than 0.1, we think it is a hard nominal; otherwise, it is an easy nominal. Then we sample same numbers of hard and easy cases.

We sample 2,500 lead paragraphs from The New York Times Annotated Corpus (Sandhaus, 2008), which contains New York Times articles from 1987 to 2006. We first use an off-the-shelf verb and nominal SRL model³³3https://cogcomp.seas.upenn.edu/page/demo_view/SRLEnglish to extract event mention candidates, and then we use Amazon Mechanical Turk to annotate the corresponding Wikipedia title of the predicted mention candidates. To ensure the quality of the annotation, we design our annotation process in two rounds:
First round. Annotators need to answer whether they think the predicted mention is an event or not. If they think it is an event, then they need to find the corresponding Wikipedia title, otherwise submit “Nil”. Each mention is annotated by three annotators. If all of them submit “Nil”, we include this event mention as a “Nil” example in the final test data. To prevent annotators from simply submitting “Nil”, 10% of the event mentions are the relatively easy cases from the Wiki data and we know their answers. We randomly insert them into the input data for AMturk (i.e., annotators are unaware of that) to evaluate the accuracy of the annotator. Only the annotation from annotators with an accuracy higher than 90% will be accepted.
Second round. This round verifies the annotated results in the first round. Each mention with the annotated title is verified by another three annotators. They need to read the page, and figure out whether it introduces the mention. If the majority vote for “yes”, we include it in the final test data. Because of majority voting, some annotations that not all the annotators agree would be included. The inter-agreement is 63.74 Fleiss’ kappa.

Event linking in the news domain is more challenging than that in the Wikipedia domain because of the following reasons:

(i) News articles describe an event at a different granularity as how Wikipedia does, usually with more details. For example, here is a piece of news about “Iraq_War”: "A contractor working for the American firm Kellogg Brown & Root was wounded in a mortar attack in Baghdad." The event “wounded” here is a very small event in Iraq War, but it is what daily news would report. On the other hand, the event mention that links to “Iraq_War” in Wikipedia domain is: "When touring in Europe, the US went to war in Iraq." The different granularity in representing events makes the task slightly different in two domains. Event linking in Wikipedia domain is more like event coreference, while event linking in news domain is mixed with more sub-event relation extraction.

(ii) As analyzed in Section 2, event linking is challenging because some event mentions may only exist in the subsection of the correct page, and the correct title is not consistent because of the event hierarchy. However, these problems mainly happen in the news domain. First of all, the Wikipedia hyperlinked mentions usually have their own pages instead of just existing in subsections. In news domain, we annotate events that only exist in subsections of a Wikipedia page. Second, in Wikipedia domain, the gold title of same event mentions is usually consistent. For example, all of the event mentions “drafted” of football players link to “National Football League Draft” instead of the page of the specific player. However, the annotation standard of NYT is not always consistent with Wikipedia hyperlinks. For example, annotators would link event mentions about sports player draft to the page of the specific player instead of the general concept page “National Football League Draft”. These problems make data annotation and model evaluation in news domain very challenging.

Because of the reasons claimed above, we think that, for some cases in news domain, the correct answer is multiple titles instead of just one title. Ranking the annotated title to the second place may be because the top one is also correct. To relax the evaluation metric here, for news domain, we also report the number of Accuracy@5, which means that if the annotated title is ranked in the top 5 candidates, we think it is correct.

A key difference between entity and event is that the context of an entity is more diverse than the context of an event. For example, when the entity “China” is mentioned in a sentence, it is unclear what entities or what events probably would also be mentioned together. However, if a verb like “invade” is used to represent the event “Battle of France” in a sentence, it is very likely that entities like “Germany”, “Italy” and “France” will also be mentioned. This shows that an event is defined by its arguments, and these arguments, with a large chance, will also be mentioned in the local context because the verb itself cannot refer to any event. Given this observation, we think that the entities in the local context of the event mention should overlap with the entities in the correct Wikipedia page, and these entities can be used to help the model better represent events. To embed these entities information explicitly to the event representation, we use similar method as how Vyas and Ballesteros (2021) embed entity attributes information to the entity representation.

Event mentions: To represent event mentions in local context, we first use an off-the-shelf Named Entity Recognition model ⁴⁴4https://cogcomp.seas.upenn.edu/page/demo_view/NEREnglish trained on 18-type OntoNotes dataset (Weischedel et al., 2013) to extract the entities around the event. We simply define the context window by 500 characters around the event mention. After predicting all the entities $e_{i}$ with their type $t_{i}$, we represent the event mentions by:

	$\displaystyle r_{1}$	$\displaystyle=[\mathrm{CLS}]\ \mathrm{ctxt}_{l}\ [\mathrm{M}_{s}]\ \mathrm{m}\ [\mathrm{M}_{e}]\ \mathrm{ctxt}_{r}$		(1)
	$\displaystyle r_{2}$	$\displaystyle=[\mathrm{t}_{1_{s}}]\ \mathrm{e}_{1}\ [\mathrm{t}_{1_{e}}]\ \cdots[\mathrm{t}_{n_{s}}]\ \mathrm{e}_{n}\ [\mathrm{t}_{n_{e}}]$		(2)
	$\displaystyle r_{m}$	$\displaystyle=r_{1}\ [\mathrm{SEP}]\ r_{2}\ [\mathrm{SEP}]$		(3)

where $\mathrm{m}$, $\mathrm{ctxt}_{l}$, $\mathrm{ctxt}_{r}$, $\mathrm{e}_{i}$ are tokens of event mention, the context on the left of the mention, the context on the right of the mention and predicted entities. $[\mathrm{M}_{s}]$ and $[\mathrm{M}_{e}]$ are special tokens to tag the start and end of the event mention. $[\mathrm{t}_{i_{s}}]$ and $[\mathrm{t}_{i_{e}}]$ are special tokens to tag the start and end of the entity whose type is $t_{i}$. $r_{m}$ is the final representation of event mentions.

Title: To represent Wikipedia titles, since important entities are already hyperlinked in the page contents, we take the first ten hyperlinked spans as entities, and represent the title by:

	$\displaystyle r_{3}$	$\displaystyle=[\mathrm{CLS}]\ \mathrm{title}\ [\mathrm{TITLE}]\ \mathrm{description}$		(4)
	$\displaystyle r_{4}$	$\displaystyle=\mathrm{h}_{1}\ [\mathrm{SEP}]\ \mathrm{h}_{2}\ [\mathrm{SEP}]\ \cdots\ [\mathrm{SEP}]\ \mathrm{h}_{n}$		(5)
	$\displaystyle r_{t}$	$\displaystyle=r_{3}\ [\mathrm{SEP}]\ r_{4}\ [\mathrm{SEP}]$		(6)

where $\mathrm{title}$, $\mathrm{h}_{i}$ and $\mathrm{description}$ are tokens of the title, hyperlinked spans, and the content of the Wikipedia page. We simply take the first $2,000$ characters as the description. $[\mathrm{TITLE}]$ is the special token to separate the title and the description. $r_{t}$ is the final representation of Wikipedia titles.

Similar to Wu et al. (2020), we first use a bi-encoder architecture to efficiently generate candidates, and use a cross-encoder architecture, which requires more computations, to rank the candidates.

Candidate Generation. We use a bi-encoder architecture to train the candidate generation model. We use two independent BERT transformers (Devlin et al., 2019) to encode the representation of event mentions $r_{m}$ and Wikipedia titles $r_{t}$, and use the output of the two $\mathrm{[CLS]}$ tokens in $r_{m}$ and $r_{t}$ as the event mention vector $v_{m}$ and the title vector $v_{t}$. Then, we maximize the dot product between the vectors of event mentions $v_{m}$ and the correct title $v_{t}$ in a batch with randomly selected negatives. At inference time, representations of all the titles are cached, and for each event mention, we calculate the dot products between its representation and the representation of all the titles, and titles with higher scores will become candidates.

Candidate Ranking. For each event mention, we take 30 candidates from the candidate generation model as the training data for the ranking model, and use a cross-encoder architecture to train the candidate ranking model. We concatenate the representation of event mentions $r_{m}$ and titles $r_{t}$, use one BERT transformer to encode the concatenated representation, and use the output of the $\mathrm{[CLS]}$ token as the final vector $v$. Then we maximize the dot product between the vector $v$ of the correct title and an additional linear layer $W$.