Anatomy of OntoGUM—Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms

OntoNotes is a human-annotated corpus with documents annotated with multiple layers of linguistic information including syntax, propositions, named entities, word senses, and within-document coreference (Weischedel et al., 2011; Pradhan et al., 2013). It covers three languages---English, Chinese and Arabic. The English subcorpus has 3,493 documents and $\sim$1.6 million words annotated for coreference. WikiCoref, which is annotated for anaphoric relations, has 30 documents from English Wikipedia Ghaddar and Langlais (2016), containing 7,955 mentions in 1,785 chains, following OntoNotes guidelines.

The Georgetown University Multilayer (GUM) corpus (Zeldes, 2017) is an open-source corpus of richly annotated texts from 12 genres, including 168 documents and over 150K tokens. Though it originally contains more coreference phenomena than OntoNotes using more exhaustive guidelines, it also contains rich syntactic, semantic and discourse annotations which allow us to create the OntoGUM dataset described below. The syntactic annotations, which consist of manually annotated Universal Dependencies trees de Marneffe et al. (2021), are particularly useful in analyzing and converting the GUM corpus. We also note that due to its smaller size (currently about 10% the size of the OntoNotes coreference dataset), it is not possible to train SOTA neural approaches directly on this dataset while maintaining strong performance.

As mentioned above, GAP is a gender-balanced labeled corpus of ambiguous pronoun-name pairs, used for out-of-domain evaluation but limited in coreferent types and genre. Several other comprehensive coreference datasets exist as well, such as ARRAU Poesio et al. (2018) and PreCo Chen et al. (2018), but these corpora cannot be used for out-of-domain evaluation because they do not follow the OntoNotes scheme. Their conversion has not been attempted to date.

Prior to the introduction of deep learning systems, the coreference task was approached using deterministic linguistic rules (Lee et al., 2013; Recasens et al., 2013) and statistical approaches (Durrett and Klein, 2013, 2014). More recently, three neural models achieved SOTA performance on this task: 1) ranking the candidate mention pairs (Wiseman et al., 2015; Clark and Manning, 2016a), 2) modeling global features of entity clusters (Clark and Manning, 2015, 2016b; Wiseman et al., 2016), and 3) end-to-end (e2e) approaches with joint loss for mention detection and coreferent pair scoring Lee et al. (2017, 2018); Fei et al. (2019). The e2e method has become the dominant one, gaining the best scores on OntoNotes. To investigate differences between deterministic and deep learning models on unseen data, we evaluate the two approaches on OntoGUM.

We first run the deterministic system (dcoref, part of Stanford CoreNLP, Manning et al. 2014) on the OntoGUM benchmark, as it remains a popular option for off-the-shelf coreference resolution. As a rule-based system, it does not require training data, so we directly test it on OntoGUM’s test set. However, POS tags, lemmas, and named-entity (NER) information are predicted by CoreNLP, which does have a domain bias favoring newswire. The system’s multi-sieve structure and token-level features such as gender and number remain unchanged. We expect that the linguistic rules will function similarly across datasets and genres, notwithstanding biases of the tools providing input features to those rules.

Combining the e2e approach with a contextualized LM and span masking is the current SOTA on OntoNotes. The system utilizes the pretrained SpanBERT-large model, fine-tuned on the OntoNotes training set. Hyperparameters are identical to the evaluation of OntoNotes test to ensure comparable results between the benchmarks. We note that while we choose the SOTA system as a ‘best case scenario’, most off-the-shelf neural NLP toolkits (e.g. spaCy) actually use somewhat simpler e2e models than SpanBERT-large, due to memory/performance constraints.

The last rows in each half of Table 3 give overall results for the systems on each benchmark. e2e+SpanBERT encounters a substantial degradation of 15 points (19%) on OntoGUM, likely due to lower test set lexical and stylistic overlap, including novel mention pairs. We note that its average score of 64.6 is somewhat optimistic, especially given that the system receives access to gold speaker information wherever available (including in fiction, conversation and interview, some of the better scoring genres), which is usually unrealistic. dcoref, assumed to be more stable across genres, also sees losses on OntoGUM of over 18 points (30%). We believe at least part of the degradation may be due to mention detection, which is trained on different domains for both systems (see the last three columns in the table). These results suggest that input data from CoreNLP degrades substantially on OntoGUM, or that some types of coreferent expressions in OntoGUM are linguistically distinct from those in OntoNotes, or both, making OntoGUM a challenging benchmark for systems developed using OntoNotes.

Both systems degrade more on specific genres. For example, while vlog (with gold speaker information) fares well for both systems, neither does well on textbook, and even the SOTA system falls well below (or around) 60s for the news, whow and textbook genres. This might be surprising for vlog, which contains transcripts of spontaneous unedited speech from YouTube Creative Commons vlogs quite unlike OntoNotes data; conversely the result is less expected for carefully edited texts which are somewhat similar to data in OntoNotes: OntoNotes contains roughly 30% newswire text, and it is not immediately clear that GUM’s news section, which comes from recent Wikinews articles, differs much in genre. Examples (5)--(5) illustrate incorrectly predicted coreference chains from both sources and the type of language they contain.

I’ve been here just crushing ultrasounds … I’ve been like crushing these all day today … I got sick when I was on Croatia for vacation. I have no idea what it says, but I think they’re cough drops. (example from a radiologist’s vlog, incorrect: ultrasounds $\neq$ cough drops)

The report has prompted calls for all edible salt to be iodised … Tasmania was excluded from \lfbox[border-style=solid,none,solid,solid,border-color=black]the study - where a voluntary iodine for- \lfbox[border-style=solid,none,solid,none,border-color=black]tification program using iodised salt in \lfbox[border-style=solid,solid,solid,none,border-color=black]bread, is ongoing (newswire example, incorrect span and coref: [the study - where a voluntary...])

These examples show that errors occur readily even in quite characteristic news writing, while genre disparity by itself does not guarantee low performance, as in the case of the vlogs whose lanugage is markedly different. In sum, these observations suggest that accurate coreference for downstream applications cannot be expected in some common well edited genres, despite the prevalence of news data in OntoNotes (albeit specifically from the Wall Street Journal, around 1990). This motivates the use of OntoGUM as a test set for future benchmarking, in order to give the NLP community a realistic idea of the range of performance we may see on contemporary data ‘in the wild’.

We also suspect that prevalence of pronouns and gold speaker information produce better scores in the results. Table 4 ranks genres by their e2e CoNLL score, and gives the proportions of pronouns, as well as score rankings for span detection. Because pronouns are usually easier to detect and pair than nouns (Durrett and Klein, 2013), more pronouns usually means higher scores. On genres with more than 50% pronouns and gold speakers (vlog, interview, conversation, speech, fiction) e2e gets much higher results, while genres with few pronouns ($<$30%) have lower scores (academic, voyage, news). This diversity over 12 genres supports the usefulness of OntoGUM, which can evaluate the genrealizability of coreference systems.