Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree

Disagreement among data annotators can reveal nuances in NLP tasks that lack a simple ground truth, such as hate speech detection. For instance, what one group of annotators deems acceptable might be considered offensive by another. The current standard for resolving such disagreement, aggregation via majority voting, casts aside variance in annotator labels as noise, when in subjective tasks this variance is key to understanding the perspectives that arise from the annotators’ individuality and backgrounds.

To address this problem, recent research has explored alternatives to majority voting. Most notably, studies have taken the approach of predicting the ratings of individual annotators Davani et al. (2022); Fleisig et al. (2023); Gordon et al. (2022). We aim to improve the prediction of rating behavior, guided by the following questions:

• •

  Does incorporating annotator information via collaborative filtering, embedding-based architecture, or in-context learning improve downstream rating predictions?

• •

  What annotator information best informs toxicity rating predictions? Do demographics provide useful information beyond what survey information can provide?

We proposed and tested a neural collaborative filtering (NCF) module, an embedding-based architecture, and an in-context learning (ICL) module for individual rating prediction. First, we incorporated NCF to the classification head of a RoBERTa-based model Liu et al. (2019). Embedded annotator information¹¹1The annotator information used is a combination of demographic information, survey information, and annotator rating history. was combined with a separate embedding of annotators’ rating history to predict individual annotator toxicity ratings. Secondly, we used embedding models to encode annotator information, which was then used to predict toxicity ratings. Lastly, we prompted LLMs such as Mistral (Jiang et al., 2023) and GPT-3.5 (Brown et al., 2020) to study different ways of integrating annotator information.

Our findings indicate that while NCF does not outperform baseline models, ICL and our embedding-based architecture  improve performance, with the embedding-based architecture  significantly outperforming all other approaches tested. In addition, our research on the effectiveness of demographic information as a feature indicates that imputing demographics from survey data performs similarly to using direct demographic inputs, suggesting that survey responses already capture the relevant demographic information for rating prediction. This suggests that, on this task, demographics have little predictive power beyond what survey information provides.

Our work is fundamentally motivated by the need for alternatives to majority-vote label aggregation in NLP tasks. Pavlick and Kwiatkowski (2019) find that disagreement among annotators is partially attributed to differences in human judgment. Basile et al. (2021) underscore the importance of the consideration of a system’s output over instances where annotators disagree.

Newer work in this field aims to directly model individual annotator rating behavior. Davani et al. (2022) employ a multi-task based approach, where predicting each annotators’ judgment is a subtask to their larger architecture. Fleisig et al. (2023) use a RoBERTa-based model to predict an individual annotators’ ratings. Gordon et al. (2022) put together a jury of annotators, predicting individual judgments.

For the individual annotator rating prediction task, Deng et al. (2023) create individual annotator embeddings and annotation embeddings. This idea of learning embeddings based on user-specific data has been applied in various domains successfully, e.g., imitation learning (Beliaev et al., 2022) or recommendation systems (Biyik et al., 2023).

Collaborative filtering (CF) learns user embeddings based on their past behaviors (Bokde et al., 2015). He et al. (2017) show that neural collaborative filtering (NCF) offers better performance than more naive CF implementations. This motivates our NCF approach to learning annotator embeddings. Intuitively, this approach would be effective in learning deeply rooted preferences and behaviors of annotators. Thus, we hypothesized that this method would more accurately predict individual annotator ratings.

Several recent approaches use sociodemographic traits of individual annotators to learn for the rating prediction task (Fleisig et al., 2023; Davani et al., 2022), but Andrus et al. (2021) warn that legal and organizational constraints, such as privacy laws and concerns around self-reporting, often make collecting demographic data challenging. Gupta et al. (2018) suggest using semantically related features in the absence of sensitive demographic data. For instance, in the absence of gender information, (Zhao et al., 2019) use other demographic features – age, relation, and marital status – for their prediction task. This work motivates our objective of incorporating auxiliary annotator information (survey information and annotator history) in the prediction task.

Lastly, Orlikowski et al. (2023) challenge the utility of demographic information, since they do not find strong evidence that explicitly modeling demographics helps to predict annotation behavior. In concurrent work, Hu and Collier (2024) argue that there is an inherent limit to how much predictive power can be provided by demographics. Their findings indicate that while incorporating demographic variables can provide modest improvements in prediction accuracy, these gains are often constrained by the relatively low variance explained by these variables. This motivates our final objective, studying the efficacy of demographics as a useful mediating variable for rating prediction.

Our approach includes creating three separate modules based on neural collaborative filtering (NCF), an embedding-based architecture, and in-context learning (ICL). We evaluate each approach’s efficacy in predicting annotator rating behavior. The latter two modules are used to investigate our second research question; we integrate different ablations of annotator information as input to the rating prediction models to study their effect on toxicity rating prediction.

We used Kumar et al. (2021)’s dataset to evaluate the performance of our rating prediction modules. This dataset consists of sentences rated for toxicity (0 = least toxic, 4 = most toxic). Each sentence has been labeled by 5 annotators and each annotator has labeled 20 distinct sentences. For each annotator, the dataset contains their rating behavior; demographic information (race, gender, importance of religion, LGBT status, education, parental status, and political stance); and survey information, e.g., their preferred forums, social media, whether they have seen toxic content, if they think toxic content is a problem, and their opinion on whether technology impacts peoples’ lives.

For ablations, we took distinct combinations of annotator information (rating history, demographics, survey information) along with the text to be rated, assessing the impact of each on the model’s performance. To study whether demographics are a necessary feature for predicting annotator ratings, we also used a separate model to predict annotator demographics using rating history and survey information and applied these predicted demographics as input for our ablations.

For all three methods, we used Mean Absolute Error (MAE) of predicting individual annotators’ ratings as the evaluation metric, allowing us to quantify the performance of different model configurations.

Our three approaches predicted annotators’ toxicity ratings on a scale from 0 to 4, based on both textual data and various combinations of annotator-specific information (demographics, survey responses, rating history). We also examine how well these models handle predicted demographic data rather than using the ground truth demographic values for each annotator. This helps to assess the data efficiency and effect of demographics as an input to the rating prediction task.

For our ablations that studied the improvement on rating predictions, we compared our results to previous baselines that predicted ratings of annotators using the same dataset.

Q1: Does incorporating annotator information via collaborative filtering, the embedding-based architecture, or in-context learning improve downstream rating predictions?

Our embedding-based architecture  outperformed all other experiments with an MAE of 0.61; the best ICL approach (with Mistral) reached an MAE of 0.69. Both the ICL approach and embedding-based architecture  outperform the most recent baseline for the dataset Fleisig et al. (2023) and the embedding-based architecture  matches the best previous MAE on this dataset (Gordon et al., 2022). The best-performing models use all available annotator-specific information as input (annotator demographics, survey information, and historical rating data). At its best, our ICL configuration with Mistral had an MAE of 0.69 (using annotator demographics, survey information, and historical rating data). The NCF approach had consistently poorer results, with a best MAE of 0.79 when including all annotator-specific information.

When creating the NCF architecture, we tested several variations. We first created a baseline from which we compared different outputs of our NCF module. Evaluating the finetuned RoBERTa model with all annotator-specific information as input along with the text to be rated yielded a baseline MAE of 0.81. We experimented with integrating embeddings through dot product vs. concatenation, freezing RoBERTa during the training process, and placing the collaborative filtering task in different parts of the RoBERTa architecture. Our best performing model froze the pretrained RoBERTa model, used concatenation, and placed the collaborative filtering piece in the classification head. However, it was only able to achieve an MAE of 0.80, not significantly improving on our baseline.

Our embedding-based architecture consistently outperformed other approaches on every ablation, suggesting that a hybrid feature-extraction approach most effectively uses annotator-specific information in rating predictions.

Q2: What annotator information best informs toxicity rating predictions? Do demographics provide useful information beyond what survey information can provide?

Incorporating demographic information improves performance over using only survey information, rating history, or both across ablations. However, we find that much of this gap can be compensated for by distilling demographic information out of survey information. Compared to the text-only baseline, incorporating predicted demographics with survey information and annotator history achieved MAE reductions of 10.26% with Mistral, 8.64% with GPT-3.5, 11.84% with text-embedding-3-small, and 12% with text-embedding-3-large. Replacing true demographic information with predicted demographic information results in nearly as strong performance for Mistral, GPT-3.5, and text-embedding-3-small.

Incorporating predicted demographics alongside survey information and annotator history notably improves accuracy. This occurs despite the fact that the accuracy of predicted demographics varies widely (highest for race and gender, but near-random for some demographics; see Table 4). Although the true demographics are somewhat helpful, annotator ratings can be effectively predicted without direct demographic data. This finding suggests that detailed demographic data may not be especially useful as a feature in individual rating prediction, beyond what can be inferred from individual preferences in survey responses.

Predicting Demographics. The performance of predicting demographics was evaluated across various configurations (Table 4). The baseline approach incorporating only survey information achieved the highest accuracies, with 47% for race and 63% for gender. Combining survey information with text slightly reduced the performance, potentially indicating the noise that the text to be rated added. The majority class approach is indicated as a baseline comparison to highlight the performance improvements for the different categories.

Our findings indicate that successively incorporating annotator demographics, rating history, and survey information improves performance for nearly all configurations tested (Table 1). Overall, the comprehensive model incorporating demographics, annotator history, and survey data consistently outperformed other configurations, demonstrating the value of integrating multiple data sources for demographic and rating predictions.

Leveraging the embedding-based architecture and ICL methods substantially improved toxicity rating predictions. NCF, by contrast, was not a competitive method for predicting ratings. Incorporating annotator information significantly enhances model performance. The best-performing embedding-based architecture achieved the lowest MAE of 0.61 by integrating demographics, annotator history, and survey data. This suggests that personalized predictions based on individual annotator preferences can lead to more accurate outcomes. Meanwhile, the ability to predict some demographics from survey information, and the fact that these imputed demographics nearly match performance with the true demographics, suggest that although demographics are helpful, individual annotator ratings can be predicted effectively without demographic data. This finding suggests that some differences in annotator opinions may be best captured by modeling individual preferences rather than demographic trends. In addition, the effectiveness of our embedding-based architecture suggests that it could help to inform future frameworks for annotator rating prediction.

While our study advances the accuracy of annotator rating predictions, several limitations exist. The generalizability of our findings is limited to English text from the U.S. and Canada, which hinders applicability in other linguistic and cultural contexts. The integration of detailed annotator information poses ethical and privacy risks and can amplify existing biases in the data. Additionally, the complexity and computational demands of our models challenge scalability and interpretability. Future research should address these issues to enhance the robustness and fairness of predictive models in subjective NLP tasks. It should also focus on expanding these methods to other domains and exploring the ethical implications of incorporating inferred data for predictions. By continuing to refine these approaches, we can develop more accurate and reliable models that better capture the complexities of human behavior and preferences.

We found that individual ratings can be predicted well without demographic information. This is helpful in that it permits individualized rating prediction without collecting demographic information. Unfortunately, that does not mean the ratings are predicted independent of demographic information: in fact, we also found that survey information is a close enough proxy that demographics can be predicted with substantially better than random accuracy, especially for race and gender, off of survey information responses. Incorporating these predicted demographics further improves accuracy. However, our finding thus uncovered the potential privacy issue that collecting seemingly innocuous survey information data carries the risk of revealing annotator demographics. This suggests that future research in this area must proceed with caution: collecting or inferring demographic information improves prediction accuracy, but risks tokenism (where opinions within a demographic group are assumed to be homogeneous). Instead, future research could identify survey information questions that help to improve rating prediction but do not risk revealing annotator demographics.