Free the Plural: Unrestricted Split-Antecedent Anaphora Resolution

Single-antecedent coreference resolution has been extensively studied. In the pre-neural period, both rule-based [Lee et al. (2013] and statistical models [Soon et al. (2001, Björkelund and Kuhn (2014, Clark and Manning (2015] were developed to resolve single-antecedent coreference resolution. Recently, ?; ?) first introduced a neural network-based approach to solving coreference in a non-linear way. ?) integrated reinforcement learning to let the model, optimising directly on the B³ scores. ?) proposed a neural joint approach for mention detection and coreference resolution. Their model does not rely on parse trees; instead, the system learns to detect mentions by exploring the outputs of a bilstm. After the introduction of contextual word embeddings such as elmo [Peters et al. (2018] and bert [Devlin et al. (2019], the ?) system has been greatly improved by those embeddings [Lee et al. (2018, Kantor and Globerson (2019, Joshi et al. (2019, Joshi et al. (2020] to achieve SoTA results. But none of those SoTA systems can resolve split-antecedent anaphors.

There is substantial research on resolving split-antecedent anaphors both in linguistics [Kamp and Reyle (1993] and psychology [Murphy (1984, Sanford and Lockhart (1990, Kaup et al. (2002, Patson (2014], but only a few early computational models [Eschenbach et al. (1989]. This work was primarily concerned with explaining preferences and restrictions on split antecedent anaphora: for example, in (2.2a) they can refer to Michael, Peter and Maria, or to Peter and Maria, but not to Michael and Maria; or in (2.2b) there seems to be a preference for they to refer to Peter and Maria [Kaup et al. (2002]. Proposals differed, e.g., on whether complex reference objects are created immediately or after encountering the anaphor.

• (5)
  • .

    Michael met Peter and Maria in the pub. They had a great time.

  • .

    Michael watched Peter and Maria in the pub. They had a great time.

Only two recent computational treatments of this type of anaphor exist [Vala et al. (2016, Zhou and Choi (2018]; they are discussed in Section 6.

Much research has focused on resolving under-resourced tasks via semi-supervised learning [Pekar et al. (2014, Yu and Bohnet (2015, Kocijan et al. (2019, Hou (2020] and shared representation based transfer learning [Yang et al. (2017, Cotterell and Duh (2017, Zhou et al. (2019].

Our baseline is a simplified version of the SoTA coreference architecture by ?), further developed by ?). In this model, mention detection and coreference are carried out jointly, but here we only use the coreference part, since we evaluate our model with gold mentions.

Our baseline system first creates representations for mentions using the output of a bilstm. The bilstm takes as input the concatenated embeddings of both word and character levels. For word embeddings, GloVe [Pennington et al. (2014] and bert [Devlin et al. (2019] embeddings are used. Character embeddings are learned from a convolution neural network (CNN) during training. The tokens are represented by concatenating outputs from the forward and the backward lstms. The token representations $(x_{t})_{t=1}^{T}$ are used together with head representations ($h_{i}$) to represent mentions ($M_{i}$). The $h_{i}$ of a mention is obtained by applying attention over its token representations ($\{x_{b_{i}},...,x_{e_{i}}\}$), where $b_{i}$ and $e_{i}$ are the indices of the start and the end of the mention, respectively. Formally, we compute $h_{i}$, $M_{i}$ as follows:

$$\alpha_{t}=\textsc{ffnn}_{\alpha}(x_{t})$$

$$a_{i,t}=\frac{exp(\alpha_{t})}{\sum^{e_{i}}_{k=b_{i}}exp(\alpha_{k})}$$

$$h_{i}=\sum^{e_{i}}_{t=b_{i}}a_{i,t}\cdot x_{t}$$

$$M_{i}=[x_{b_{i}},x_{e_{i}},h_{i},\phi(i)]$$

where $\phi(i)$ is the mention width feature embeddings. Next, we pair the mentions with candidate antecedents to create a pair representation ($P_{(i,j)}$):

$$P_{(i,j)}=[M_{i},M_{j},M_{i}\circ M_{j},\phi(i,j)]$$

where $M_{i}$, $M_{j}$ is the representation of the antecedent and anaphor, respectively, $\circ$ denotes element-wise product, and $\phi(i,j)$ is the distance feature between a mention pair. To make the model computationally tractable, we consider for each mention a maximum 250 candidate antecedents as we observed from the arrau corpus, most of the antecedents can be retrieved within the 250 candidates window size.

The next step is to compute the pairwise score ($s(i,j)$). Following ?), we add an artificial antecedent $\epsilon$ to deal with cases of non-split-antecedent anaphor mentions or cases when the antecedent does not appear in the candidate list during the training. We do not use $\epsilon$ during test time as we use gold split-antecedent anaphors. We compute $s(i,j)$ as follows:

$$r(i,j)=\textsc{ffnn}(P_{(i,j)})$$

$$s(i,j)=\frac{1}{1+e^{-r(i,j)}}$$

At test time, the system will generate two to five antecedents according to their $s(i,j)$ scores. The upper threshold is based on the observation that the vast majority of split-antecedent anaphors in arrau have no more than 5 antecedents. To generate the antecedents, we first rank the candidates by their $s(i,j)$ scores in descending order. We add up to 5 top candidates that have a $s(i,j)$ score above 0.5.⁴⁴4We use the gold single-antecedent clusters to ensure the selected antecedents belong to distinct gold cluster. If less than two candidates were selected, we add top two candidates to the predictions regardless of their scores.

Since the number of examples of split-antecedent anaphora in arrau is small, we deployed four auxiliary corpora created from either the crowd annotated Phrase Detectives (pd) corpus or the gold annotated arrau corpus to improve the performance of the system.

The pd corpus was created using the Phrase Detectives game, whose players are asked to find the antecedent/split-antecedents closest to the mention in question [Poesio et al. (2019]. The corpus comes with all raw annotations and silver labels aggregated using the Mention-Pair Annotation model [Paun et al. (2018]. We created our first two auxiliary corpora from the latest version of the pd corpus⁵⁵5https://github.com/dali-ambiguity/Phrase-Detectives-Corpus-2.1.4 by using different aggregation methods. The arrau corpus consists of texts from four very distinct domains: news (the rst subcorpus), dialogue (the trains subcorpus), fiction (the pear stories), and medical / art history (the gnome subcorpus). Its annotation scheme covers the annotation of referring (including singletons) and non-referring expressions; coreference relations including split antecedent plurals and generic references; and non-coreferential anaphoric relations including discourse deixis and bridging references. We create the other two auxiliary corpora from the arrau corpus. The rest of the subsection describes our auxiliary corpora in detail.⁶⁶6The ParCorFull corpus [Lapshinova-Koltunski et al. (2018] also includes split-antecedent annotations, but the number of split-antecedent examples are too small to be used in our experiments.

Silver Labels (pd-silver) For our first auxiliary corpus, we simply added to our training data the split-antecedent anaphora examples from the pd corpus. We used the silver labels that come with the corpus and extracted 507 split-antecedent anaphors (see Table 1). This nearly doubled the size of our training data. We assessed the quality of the silver labels by comparing it against the gold annotated subset of the pd corpus;⁷⁷7A subset of the pd corpus comes with additional gold labels annotated by experts. the silver labels have a relatively good quality (62.9% F1), recalling 68.8% of the gold split-antecedent anaphoric links and with a precision of 57.9%.

Raw Crowd Annotations (pd-crowd) The second auxiliary corpus was created by extracting all split-antecedent examples from the raw annotations in pd to maximise recall. After extracting all split-antecedent annotations, we used majority voting as our aggregation method when players did not agree on split-antecedent annotations. In this way, we extracted 47.7k split-antecedent annotations associated with 6.2k mentions (Table 1). The quality of this extraction method was evaluated on the gold portion of the pd corpus as well; the resulting dataset has a recall of 91.7%, which fulfils the goal of this setting. As expected, the corpus is noisy, with a precision of 11.1% and an F1 of 19.7%. We manually checked the false-positive examples, finding they are mainly due to three types of mistakes: single-antecedent coreference (the coreference chains were annotated as the split-antecedent), bridging reference (not required to be annotated), and other annotation mistakes. The first two types of mistakes are not harmful to our task as our third and fourth auxiliary corpora are created using those types of relations.

Element-of Bridging References (element-of) arrau is also annotated with bridging references, and one of the bridging relations covered by the annotation, element-of (and its inverse) are very closely related to the task of resolving split-antecedent plurals. Element-of is the relation between a new singular entity and a plural entity introduced in the discourse, as in (3.2a), or between a previously introduced singular entity and a new plural entity, as in (3.2b).

• (6)
  • .

    There are two supermarkets in our village, but one is very small. (element-of)

  • .

    Yet another small bookshop just opened in our village. Our independent bookshops are our main attraction. (element-of-inverse)

Since the proposed system uses a pairwise approach, the relations between split-antecedent anaphors and their antecedents are established by multi-links between anaphors and individual antecedents. These are element-of relations, but differ from the bridging case in two respects. First, the plural relations are an inverse version of the element-of relations where the antecedent is an element of the anaphor. Second, split-antecedent coreference is coreference: the union of all antecedents has the same denotation as the anaphor, unlike in bridging. Nevertheless, the element-of bridging relation is close enough to be potentially useful for our task. We therefore created a third auxiliary corpus by extracting element-of bridging relations from arrau. In total we extracted 1059 training examples (see Table 1).

Single-antecedent anaphors (single-coref) Our last auxiliary corpus using single-antecedent anaphors. The main reason for using single-antecedent anaphors as supporting dataset is that single-antecedent anaphors are very common: e.g., in arrau train we only have 500 split-antecedent anaphors, but 30k single-antecedent anaphors (see Table 1). This gives us a much larger corpus than all other auxiliary corpora proposed earlier. Using a large auxiliary corpus allows our system to learn a better mention and pairwise representations that might be beneficial for our under-resourced task.

Training with multiple corpora is challenging, especially when the auxiliary corpus is noisy. In this paper, we evaluate our system with three different training strategies to maximise the performance on split-antecedent anaphora resolution.

Concatenation (concat) The first and simplest strategy is to use the auxiliary corpus as additional training data by concatenating it with the main corpus. We configured training to train on documents from the main and auxiliary corpus in turn, with 50% of the time on the main corpus. By doing so, we make sure the system will not overfit the auxiliary corpus.

Pre-training (pre-train) Our second strategy was to first pre-train the system on the auxiliary corpus, and then fine-tuning the model on the main corpus to fit our task. Such a strategy works well when the auxiliary corpus is noisy as the fine-tuning step will only be trained on the gold annotations.

Corpus Annealing (annealing) Our last strategy was inspired by ?)’s teacher annealing proposal. ?) use teacher annealing to enable smoother learning. The multi-task model initially learns from the predictions of the single-task model, but training gradually switches to gold labels by a weighted loss function. In this paper, we configured our system to initially learn from the auxiliary corpus, and using a linearly decreasing ratio of training with the auxiliary corpus. Instead of using a weighted loss as done by ?), we used the ratio to control the source of our training documents (main or auxiliary). By doing so, the learning process smoothly switches from the auxiliary corpus to the main corpus, training 100% on the main corpus when the end of training is reached.

Following ?), we optimise our model on the marginal log-likelihood of all correct antecedents. We consider an antecedent correct if it is from the same gold single-antecedent coreference cluster gold$(i)$ as the gold antecedent lists. We also use the gold single-antecedent clusters to extend the split-antecedent anaphor list during training: i.e., mentions in the same single-antecedent cluster of a split-antecedent anaphor are considered as split-antecedent anaphors. The extension boosts the number of split-antecedent anaphors in the training data by 79% to 908. We compute the losses as follows:

$$log\prod_{j=1}^{N}\sum_{\hat{y}\in Y(j)\cap\textsc{gold}(i)}r(\hat{y},j)$$

in case mention $i$ is not a split-antecedent anaphor or $Y(j)$ (the candidate antecedents) does not contain mentions from $\textsc{gold}(i)$, we set gold$(i)=\{\epsilon\}$.

We first applied all three training strategies to our auxiliary corpora to find the best training strategy for each auxiliary corpus. We used lenient F1 scores on the development set to select the strategy most suitable for each individual corpus. As illustrated in Table 2, our baseline model trained only on arrau train already achieves a reasonably good F1 score for this task (58.2%). Starting with a strong baseline, our system enhanced by the auxiliary corpora achieved substantial improvements of up to 11.3%.

Among the training strategies, pd-silver works best when using the concat method. This makes sense as pd-silver contains split antecedent examples annotated using the same annotation scheme as arrau. The pd-crowd corpus is much noisier, but despite containing a large amount of false positives, it achieves better F1 scores than pd-silver. This confirms our hypothesis that a higher recall of split-antecedent examples is important, and the false positive examples –mainly single-antecedent anaphors and bridging relations–do not harm the results. Both pre-train and annealing are suitable strategies for the pd-crowd corpus, with the former slightly better. The element-of corpus works best in a pre-train setting, with a large improvement of 6.2% when compared to the baseline even though the corpus only contains a small number of examples (1k). The large improvement confirmed our hypothesis that element-of bridging relations are closely related to split-antecedent relations. Finally, the single-coref corpus achieved the best scores with all three training strategy, but the largest improvement of 11.3% is achieved by training with annealing method. As the single-coref corpus has a substantially larger number of examples when compared to all other auxiliary corpus used in this paper, it seems likely that this might be an important reason for its usefulness. Overall, our auxiliary corpora and training strategies showed their merit for enhancing the performance on split-antecedent anaphora resolution; we will further discuss this in later sections.

We then evaluated our models, each trained using the best training strategy for that corpus, on the test set. Since our paper reports the first result on split-antecedent anaphora resolution on arrau, we compare our system with various baselines. Following ?), we created naive baselines: recent-m and random. recent-m assigns to an anaphor the $m$ antecedents (from distinct single-antecedent clusters) that are closest. random assigns all candidate antecedents random probabilities; the antecedents are selected using the same method as the one used in our trained models.

Table 3 shows the results on the test set. The naive baselines achieved a maximum lenient F1 score of 28.9% when using the $4$ most recent antecedents. When using strict evaluation, most naive baselines perform really badly. This poor performance of the naive baselines confirms the difficulty of our task. Our neural baseline trained solely on arrau train achieved a lenient F1 of 58.6%, more than double the best result of the naive baselines. With strict evaluation, the same model achieved 22.7%–6 times the best score of naive baselines, but still low. Using auxiliary corpora improved the performance of the neural model by a minimum of 3.2 and 8.2 p.p. when evaluated in a lenient and strict setting, respectively. But our best model, trained using single-coref auxiliary information, achieved a lenient F1 of 69.6% and a strict accuracy of 42.7%, which is 11% and 20% better than our neural baseline.

We further evaluated using combinations of auxiliary corpora–i.e., using the pd-crowd, element-of, and single-coref corpora, and using either the pre-train or annealing strategies– e.g., using pd-crowd for pre-train, then fine-tune the model with single-coref corpus using annealing strategy. In total, we evaluated three combinations (see Table 3). The best result, achieved by combining all three auxiliary corpora, was 0.4 p.p. and 0.9 p.p. better than the results achieved by using single-coref alone in a lenient and strict evaluation, respectively.

Number of Antecedents We compared our best model with the neural baseline, using both lenient and strict scores and also considering the number of split-antecedents. For this analysis, we evaluated both models on the concatenation of test and development set to collect more examples.

As shown in Table 4(a), 82.6% of the anaphors in the dataset have two antecedents; the rest have three or more antecedents. With anaphors that have two antecedents, our best model achieved improvements of 9.5% (lenient) and 10.9% (strict), respectively. With anaphors that have more antecedents, our best model achieved even larger improvements for both lenient (13.2%) and strict (15.1%) scores. Overall, our best model outperforms the baseline by large margins in all evaluations. System prediction examples for both systems can be find in Table 5.

Size of the Auxiliary Corpus Our single-coref auxiliary corpus achieved much larger improvements than all other corpora evaluated in this paper. A simple explanation would be that this is because the single-coref corpus is substantially larger. So, to understand the impact of the auxiliary corpus size on our task, we further trained our model with auxiliary corpora of different size. The examples are randomly selected from our single-coref corpus. Table 4(b) shows our results on the development set. When using 1k examples from the single-coref corpus, the lenient F1 is 4.2% lower than element-of’s 64.3% which suggest the element-of corpus is more effective when the number of training examples is similar. When compared with pd-crowd, the same amount of the gold annotated single-antecedent coreference examples (6k) achieved broadly the same score. Adding more training examples results in an increase in lenient scores. The strict scores follow a similar trend up until 2/3 of the examples are used (20k). Overall, the auxiliary corpus size is an important factor for the final results.