Does Recommend-Revise Produce Reliable Annotations?
An Analysis on Missing Instances in DocRED

Relation Extraction (RE) is an important task which aims to identify relationships held between entities in a given piece of text. While most previous methods focus on extracting relations from a single sentence Lin et al. (2016); Zhang et al. (2018), recent studies begin to explore RE at document level Peng et al. (2017); Zeng et al. (2020); Nan et al. (2020); Huang et al. (2021); Zhang et al. (2021), which is more challenging as it often requires reasoning across multiple sentences.

The rapid development of document-level RE in the past two years has benefited from the proposal of DocRED Yao et al. (2019), the first large-scale and human-annotated dataset for this task. Noticeably, longer documents introduce an unprecedented difficulty in annotating the relation instances: as the total number of entities dramatically increases in accordance to text length, the expected number of entity pairs surges quadratically, intensively increasing the workload to check relationships between every pair. To address this problem, Yao et al. (2019) applies a recommend-revise process: in the recommendation phase, a small set of candidate relation instances is generated through distant supervision; then, annotators are required to revise the candidate set, removing the incorrect relation instances and supplementing the instances not identified in the recommendation phase.

Shifting the construction process from scratch to an edit-based task, it seems that the recommend-revise scheme cuts down the effort of annotating by a large margin. However, whether the quality of the annotation maintains a reliable standard in practice remains in doubt. To what extent can the accuracy of annotation be sacrificed due to the automated recommendation? And, how does the provided recommendation affect the behaviours of the annotators in the revision phase? Moreover, what are the real effects on the models trained on a dataset annotated with this scheme?

To answer these questions, we aim to provide a thorough comparison between careful annotations from scratch and the annotations under the recommend-revise scheme. We randomly select 96 documents from DocRED and ask two experts to relabel them from scratch independently. After annotating, the two experts come to a consensus of gold labels via discussion. This revised dataset is publicly available at https://github.com/AndrewZhe/Revisit-DocRED, and we hope it can be used to evaluate the model’s performance on real data distribution¹¹1While we cannot guarantee that the relabeled data is totally error-free, we believe the quality is high enough to be approximated as a real distribution because each entity pair is examined by two annotators.. With the help of these annotations, we discovered three sobering issues regarding the effects of the recommend-revise scheme:

(1) A noticeable portion of relation instances is left out, and the distributional bias in the recommendation output is inherited, even after the revision process. It is not surprising that recommendations alone fail to recognize all the relation instances, since RE models are far from perfect. Ideally, these unidentified instances should be added by human annotators during the revision phase. However, it turns out that $95.7\%$ of these missing instances are still left out even after revision. Furthermore, while the recommendations from distant supervision favor instances associated with popular entities and relations in the source Knowledge Base (Wikidata), this bias is still maintained and inherited even after human revision, leaving less popular relations and entities to be neglected.

(2) Worryingly, we find the models trained on DocRED have low recall on our relabeled dataset and they also inherit the same bias towards popular relations and entities. We train recent models on DocRED and test them with the dataset relabeled by us. We notice that all models have much lower recalls on our dataset than previously reported on DocRED due to the numerous false negatives in training data, and those models are also biased to popular entities and relations. Further investigation reveals that the models’ bias comes from the training set by comparing different strategies of negative sampling. Since one straightforward real-world application of relation extraction is to acquire novel knowledge from text, a RE model would be much less useful if it has a low recall, or perform poorly on less popular entities and relations.

(3) The recommendations actually also impacts the behaviors of annotators, making them unlikely to supplement the instances left out. This is the underlying reason for the two concerns above. We argue that the revision process fails to reach its goal, since it puts the annotators in a dilemma: while they are supposed to “add” new instances left out by the recommendations, finding these missing instances may force the annotators to thoroughly check out the entities pair-by-pair, which is time-consuming and against the goal of this scheme. As a result, annotators can hardly make effective supplementation and would tend to perform the easier goal of validating existing relation instances.

The major challenge for annotating document-level RE datasets comes from the quadratic number of potential entity pairs with regard to the total number of entities in a document. As reported by Yao et al. (2019), a document in DocRED contains 19.5 entities on average, thus rendering 360 entity pairs with potential relationships. Therefore, for the 5,053 documents to be annotated, around 1,823,000 entity pairs are to be checked. Such workload will be around 14 times more than TACRED Zhang et al. (2017), the biggest human-labeled sentence-level RE dataset. Therefore, exhaustively labeling relations between each entity pair involves intensive workload and does not seem feasible for document-level RE datasets.

To alleviate the huge burden of manual labeling, Yao et al. (2019) divides the annotation task into two steps: recommendation and revision. First, in the recommendation phase, Yao et al. (2019) takes advantage of Wikidata Vrandecic and Krötzsch (2014) and an off-the-shelf RE model to collect all the possible relations between any two entities in the same document. This process is automated and does not require human involvement. Then, during the revision phase, the relations that exist in Wikidata or are inferred by the RE model for a specific entity pair will be shown to the annotators. Rather than annotating each entity pair from scratch, the annotators are required to review the recommendations, remove the incorrect triples and supplement the missing ones.

Table 1 shows the statistics and comparison of $\mathbf{D_{Scratch}}$, $\mathbf{D_{Recommend}}$ and $\mathbf{D_{Revise}}$ on the 96 randomly-selected documents in DocRED.

To investigate if RE models trained on such data will likewise learn the same bias, we train and select RE models on the recommend-scheme-labeled dataset, $\mathbf{D_{Revise}^{Train}}$ and $\mathbf{D_{Revise}^{Valid}}$ and then assess the models’ performance on the real data distribution, $\mathbf{D_{Scratch}}$. The construction process of $\mathbf{D_{Revise}^{Train}}$ and $\mathbf{D_{Revise}^{Valid}}$ is the same as $\mathbf{D_{Revise}}$, while the former is actually the original train set and the latter is the validation set in DocRED excluding the 96 documents in $\mathbf{D_{Revise}}$. In those settings, we examine the performance of recent models: BiLSTM Yao et al. (2019), GAIN-BERT_base Zeng et al. (2020), SSAN-Roberta_large Xu et al. (2021), ATLOP-Roberta_large Zhou et al. (2021) and DocuNet-Roberta_large Zhang et al. (2021). The last three models are the most competitive ones for DocRED currently, while the others are shown to make sure that our analysis can generalize to models of smaller sizes.

Finally, we move on to discuss another more implicit influence of the recommend-revise scheme on the annotators’ aspect. As discussed in Section 4.1, while we expected the revision process to help supplement the instances left out, it turns out that an incredibly low number is added indeed. Given that the annotators are trained to accomplish the revision task, we wonder why they still fail in such a uniform manner. We would like to argue that it is the nature of the revision process that puts the annotators in a dilemma, where they have to choose between a huge effort and insufficiency of supplementation.

Recall that there is a distinct difference in the settings of examining a labeled relationship and supplementing an unidentified relationship. For the former, annotators are required to find evidence for a recommended relation instance and remove it if there is conflicting or no evidence. This process only requires checking a single entity pair and collecting the information related to the two specific entities. However, this is not the case for supplementing a possible, unidentified relation instance, which can exist between any entity pair. There is no clear range of searching or indicating information; all they can do is to check pair-by-pair, just like what they do from scratch. This puts annotators in an awkward dilemma, especially when they understand the motivation of this scheme: if they are to be fully responsible for the missing instances at large, they will always have to complete the thorough pairwise checking one by one; however, this would make the whole process of the recommend-revise scheme meaningless in return, as it’s just like a practice from scratch.

The harsh requirements of supplementing push annotators to overly rely on the recommendation results and simply examine them. This is especially worth worrying about in real practice, where annotators are recruited to complete a certain number of annotations, and typically paid according to the estimated number of hours or the total number of instances they devote to the annotationDraws et al. (2021). Under this dilemma, it is a natural result that they are especially unmotivated to carry out the exhaustive checking for supplementation in order to get a reasonable pay in the given time.

In fact, we observe an interesting phenomenon that annotators largely tend to just pick some most obvious missing instances, convince themselves that they have accomplished the supplementation, and simply move on to the next document. This can be seen in Figure 5, where we compare the distributional characteristics of the successfully supplemented instances ($\mathbf{D_{Revise}}$ - $\mathbf{D_{Recommend}}$) and all the missing instances in general ($\mathbf{D_{Scratch}}$ - $\mathbf{D_{Revise}}$). Sub-figure (a) shows the accumulative statistics of the position of the head entity’s first appearance in the document. We can see that the instances added by annotators in DocRED exhibit an extremely obvious tendency to occur earlier in the text, where more than 70% added instances are in the first 3 sentences. In contrast, all missing relation instances as a whole are almost distributed in every part of the document uniformly. This reveals the interesting fact that humans typically tend to pick up the relations where the entities in it are mentioned earlier in the document. Sub-figure (b) further compares the minimum distance between the mentions of the head and tail entities of one relation instance. We once again see the interesting fact that annotators have a strong tendency to add the “most easily identifiable” instances where the head and tail entities are quite close. Specifically, the proportion of entity pairs mentioned in just one single sentence (Interval=0) is around 20% for all missing facts, but is as high as 45% for the ones chosen by annotators to be supplemented. This tells us how annotators naturally avoid burdens of reading brought by longer intervals, which possibly indicates more complicated inference with multiple sentences.

From these observations, we see that there exist clear patterns among the very few instances added by human annotators. This reveals a serious fact that annotators are intentional in “pretending” to be supplementing with the least possible effort. Given the consensus behavior of annotators and the very limited number of additional, it is most likely that the nature of the annotation task pushes the annotators to this embarrassing dilemma of adding and abandoning. Thus, we propose a call to the NLP community that researchers should always be aware that annotation schemes, like the recommend-revise scheme, can have a direct impact on the annotation workers, affecting their willingness and behaviors, and thus have a deeper influence on the collected data.

We can summarize all these problems mentioned above in the annotation with a concrete case in DocRED shown in Figure 6. The figure depicts the annotations associated with the entity Michael Imperi, as well as the relation that is added in revision. Let’s first focus on the red edges, which indicate the relation triples that are neither recommended nor supplemented by human. Regrettably, half of the total 18 relation triples remain missing, and just one triple is added during revision (the green edge). Compared with black edges, which indicated correctly annotated instances, the red edges are more likely to be associated with less popular entities. For example, "Sopranos" [4] and "Law & Order" [7], two popular series with at least 100K+ comments on IMDB, and about 200 edges in Wikidata, are connected with "Michael Imperioli" [0] with the relation "cast member" in the annotation, but "Detroit 1-8-7" [17] and "Mad Dogs" [20], supposed to hold the same relation to "Michael Imperioli" [0], are missed. In the text, all these series appear in similar circumstances, and the only difference is the latter ones are not recommended to the annotators, essentially because of their less popularity (less than 10K comments on IMDB, and less than 50 edges in Wikidata). We can also see the effect on the popularity of relations in the connection between [7] and [9]. "present in work" and "characters" should occur symmetrically according to the definitions, but the latter one is missed in the recommendation. Correspondingly, in Wikidata, the latter relation has 19057 links, which is less than the former’s 82250 links. The last point to notice is the only green edge between Louis Fitch [15] and ABC [16], which is not recommended, but supplemented by annotators. Among all the missed instances in the recommendation, the annotators only supplement this one, which is easy to identify in the text due to both the head and tail entities being mentioned in the same sentence. This is consistent with our analysis of annotators’ behavior above.

With the advance of deep learning models, the annotation sometimes becomes the bottleneck for a machine learning system. Recently, analyzing the annotation quality has received increasing attention. Northcutt et al. (2021) collects and analyzes the label errors in the test sets of several popular benchmarks, showing label errors are ubiquitous and destabilize machine learning benchmarks. More specific to RE task, Alt et al. (2020) addresses the annotation problems in TACRED Zhang et al. (2017), a popular sentence-level RE dataset. They find label errors account for 8% absolute F1 test error and more than 50% of the examples need to be relabeled. Stoica et al. (2021) expands this research to the whole dataset, resulting in a complete re-annotated version, Re-TACRED, and conducts thorough analysis on the models’ performance. Our work differs from them in that we delve into the nature of document-level RE task, and especially explore how the error is systematically introduced into the dataset through recommend-revise scheme.

Methodologies to solve incomplete annotations for information extraction tasks have been widely discussed in previous works. Different from classification tasks, information extraction requires annotators to actively retrieve positive samples from texts, instead of just assigning a label for a given text. The problem is also attributed to the use of distant supervision Reiplinger et al. (2014) where the linked KG is not perfect. Some works apply general approaches like positive unlabeled learning Xie et al. (2021); Peng et al. (2019) or inference learning Roller et al. (2015). Task-specific models are also designed, like Partial CRF Tsuboi et al. (2008) for NER Yang et al. (2018), and novel paradigm for joint RE Xie et al. (2021). However, none of them examine the distribution bias in the training data, and those methods are not validated in the context of the document-level RE task.

Prevalent effective methods on document-level RE include graph-based models and transformer-based models. Graph-based models like Zeng et al. (2020) and Zhang et al. (2021) are designed to conduct relational reasoning over the document, and transformer-based models Zhou et al. (2021); Xu et al. (2021) are good at recognizing long-distance dependencies. However, all previous models treat unlabeled samples in the dataset as negative samples, and do not concern the problems in annotations. We believe our analysis and re-annotated dataset will help future work focus more on the discrepancy between the annotation and real-world distribution, instead of just overfitting the dataset.

In this paper, we show how the recommend-revise scheme for DocRED can cause bias and false negative issues in the annotated data. The flaws of dataset affect the model’s recall on real data and also teach the model the same bias in training data. As this scheme cannot reduce the human labor essentially without the loss of annotation quality, more efficient strategies for annotation are to be explored. On the other hand, considering that building a reliable training set for document RE is extremely expensive, it is also a meaningful topic that how to alleviate the dataset shift problem Moreno-Torres et al. (2012) by injecting appropriate inductive bias into the model’s structure, instead of inheriting the bias in the training data. We believe the in-depth analysis provided in this paper can benefit future designs of document-level RE models, and our Scratch dataset can serve as a fairer test set.

This work is supported in part by National Key R&D Program of China (No. 2020AAA0106600) and NSFC (62161160339). We would like to thank the anonymous reviewers and action editors for their helpful comments and suggestions; thank Weiye Chen for providing feedback on an early draft. For any correspondence, please contact Yansong Feng.

This work focuses on quality checking and re-annotations of DocRED, a publicly available dataset constructed from Wikipedia pages. All source documents and types of relationships are provided and utilized in the original DocRED dataset, and no additional annotation rule that may involve unexamined ethical concerns was introduced. Annotators receive a competitive pay of 100 yuan per hour (more than 4 times the local minimum wage) under the approval of the institute, and both the annotation and discussion stage count towards the paid working time. Annotators are required to read the ACM Code of Ethics before annotating and report any document that violates the code. These documents are removed from the sampled documents. However, there may still be sentences or entities that are from Wikipedia pages with potentially improper content. The possible adoption of such content is not a decision of the authors, and all content in the dataset does not reflect the views or stances of the authors. The resulting re-annotations from the agreement of two expert annotators form a decent approximation of the gold labels, but may still not be the ground truth due to natural error rates. Further use of the dataset should be aware of the limitations and other possible issues, and we are not responsible for issues in further model training processes using our data.