On Releasing Annotator-Level Labels and Information in Datasets

Obtaining multiple annotator judgements on the same data instances is a common practice in NLP in order to improve the quality of final labels Snow et al. (2008); Nowak and Rüger (2010). Cases of disagreement between annotations are often resolved through majority voting, averaging, or adjudication in order to derive a single “ground truth”, often with the aim of training supervised machine learning models. However, in relatively subjective tasks such as sentiment analysis or offensiveness detection, there often exists no single “right” answer (Alm, 2011). Enforcing such a single ground truth in such tasks will sacrifice valuable nuances about the task that are embedded in annotators’ assessments of the stimuli, especially their disagreements Aroyo and Welty (2013).

Annotators’ socio-demographic factors, moral values, and lived experiences often influence their interpretations of language, especially in subjective tasks such as identifying political stances (Luo et al., 2020), sentiment Díaz et al. (2018), and online abuse Waseem (2016); Patton et al. (2019). For instance, feminist and anti-racist activists systematically disagree with crowd workers in their hate speech annotations Waseem (2016). Similarly, annotators’ political affiliation is shown to correlate with how they annotate the neutrality of political stances (Luo et al., 2020). A potential adverse effect of majority voting in such cases is that it may sideline minority perspectives in data.

In this paper, we analyze annotated data for eight different tasks across three different datasets to study the impact of majority voting as an aggregation approach. We answer two questions:

• •

  Q1: Does aggregated data uniformly capture all annotators’ perspectives, when available?

• •

  Q2: Does aggregated data reflect certain socio-demographic groups’ perspectives more so than others?

Meaningfully addressing such issues in multiply-annotated datasets requires understanding and accounting for systematic disagreements between annotators. However, most annotated datasets often only release the aggregated labels, without any annotator-level information. We argue that dataset developers should consider including annotator-level labels as well as annotators’ socio-demographic information (when viable to do so responsibly) when releasing datasets, especially those capturing relatively subjective tasks. Inclusion of this information will enable more research on how to account for systematic disagreements between annotators in training tasks.

NLP has a long history of developing techniques to interpret subjective language Wiebe et al. (2004); Alm (2011). While all human judgments embed some degree of subjectivity, some tasks such as sentiment analysis Liu et al. (2010), affect modeling Alm (2008); Liu et al. (2003), emotion detection Hirschberg et al. (2003), and hate speech detection Warner and Hirschberg (2012) are agreed upon as relatively more subjective in nature. As Alm (2011) points out, achieving a single real ‘ground truth’ is not possible, nor essential in case of such subjective tasks. Instead, we should investigate how to model the subjective interpretations of the annotators, and how to account for them in application scenarios.

However, the current practice in the NLP community continues to be applying different aggregation strategies to arrive at a single score or label that makes it amenable to train and evaluate supervised machine learning models. Oftentimes, datasets are released with only the final scores/labels, essentially obfuscating important nuances in the task. The information released about the annotations can be at one of the following four levels of information-richness.

Firstly, the most common approach is one in which multiple annotations obtained for a data instance are aggregated to derive a single “ground truth” label, and these labels are the only annotations included in the released dataset (e.g., Founta et al. (2018)). The aggregation strategy most commonly used, especially in large datasets, is majority voting, although smaller datasets sometimes use adjudication by an ‘expert’ (often one of the study authors themselves) to arrive at a single label (e.g., in Waseem and Hovy (2016)) when there are substantial disagreements between annotators. These aggregation approaches rely on the assumption that there always exist a single correct label, and that either the majority label or the ‘expert’ label is more likely to be that correct label. What it fails to account for is the fact that in many subjective tasks, e.g., detecting hate speech, the perceptions of individual annotators may be as valuable as an ‘expert’ perspective.

Secondly, some datasets (e.g., Jigsaw (2018); Davidson et al. (2017)) release the distribution across labels rather than a single aggregated label. In binary classification tasks, this corresponds to the percentage of annotators who chose one of the labels. In multi-class classification, this may be the distribution across labels obtained for an instance. While this provides more information than a single aggregated label does (e.g., identifies the instances with high disagreement), it fails to capture annotator-level systematic differences.

Thirdly, some datasets release annotations made by each individual annotators in an anonymous fashion (e.g., Kennedy et al. (2020); Jigsaw (2019)). Such annotator-level labels allow downstream dataset users to investigate and account for systematic differences between individual annotators’ perspectives on the tasks, although they do not contain any information about each annotators’ socio-cultural backgrounds. Finally, some recent datasets (e.g., Díaz et al. (2018)) also release such socio-demographic information about the annotators in addition to annotator-level labels. This information may include various identity subgroups the annotators self-identify with (e.g., gender, race, age range, etc.), or survey responses from the annotators that capture their value systems, lived experiences, or expertise, as they relate to the specific task at hand. Such information, while tricky to share responsibly, would help enable analysis around representation of marginalized perspectives in datasets, as we demonstrate in the next section.

In this section, we investigate how the aggregation of multiple annotations to a single label impact representations of individual and group perspectives in the resulting datasets. We analyze annotations for eight binary classification tasks, across three different datasets: hate-speech Kennedy et al. (2020), sentiment Díaz et al. (2018), and emotion Demszky et al. (2020). Table 1 shows the number of instances, annotators and individual annotations present in the datasets. For hate-speech and emotion datasets, we use the binary label in the raw annotations, whereas for the sentiment dataset, we map the 5-point ordinal labels (-2, -1, 0, +1, +2) in the raw data to a binary distinction denoting whether the text was deemed positive or negative.¹¹1We do this mapping for the purposes of this analysis, where we are focusing on binary tasks. Ideally, a more nuanced 5-point labeling schema will be more useful. While the emotion dataset contains annotations for 28 different emotions, in this work, for brevity, we focused on the annotations for only the six standard Ekman emotions Ekman (1992) — anger, disgust, fear, joy, sadness, and surprise. In particular, we use the raw annotations for these six emotions, rather than the mapping of all 28 emotions onto these six emotions that Demszky et al. (2020) use in some of their experiments.

Another argument in favor of retaining annotator-level labels is their utility in modeling disagreement during training and evaluation. Prabhakaran et al. (2012) and Plank et al. (2014) incorporated annotator disagreement in the loss functions used during training to improve predictive performance. Cohn and Specia (2013) and Fornaciari et al. (2021) use a multi-task approach to incorporate annotator disagreements to improve machine translation and part-of-speech tagging performance, respectively. Chou and Lee (2019) and Guan et al. (2018) developed learning architectures that model individual annotators as a way to improved performance. Wich et al. (2020) show the utility of detecting clusters of annotators in hate-speech detection based on how often they agree with each other. Finally, Davani et al. (2021) introduce a multi-annotator architecture that models each annotators’ perspectives separately using a multi-task approach. They demonstrate that this architecture helps to model uncertainty in predictions, without any significant loss of accuracy or efficiency. This array of recent work further demonstrates the utility of retaining annotator-level information in the datasets for downstream modeling steps.

Building models to predict or measure subjective phenomena based on human annotations should involve explicit consideration for the unique perspectives each annotator brings forth in their annotations. Annotators are not interchangeable– that is, they draw from their socially-embedded experiences and knowledge when making annotation judgments. As a result, retaining their perspectives separately in the datasets will enable dataset users to account for these differences according to their needs. We demonstrated that annotation aggregation may unfairly disregard perspectives of certain annotators, and sometimes certain socio-demographic groups. Based on our analysis, we propose three recommendations aimed to avoid these issues:

We thank Ben Hutchinson for valuable feedback on the manuscript. We also thank the anonymous reviewers for their feedback.