Investigating Bias In Automatic Toxic Comment Detection: An Empirical Study

We experiment with two widely popular architectures for text classification: long-short-term-memory networks (LSTMs) and convolutional neural network (CNNs) as described below:

• •

  LSTM: In our work, we use a single layer bidirectional LSTM network that uses a forward LSTM and a backward LSTM to process the input from both directions. The hidden state vector is obtained by concatenating forward ($\vec{h_{f}}$) and backward ($\overleftarrow{h_{b}}$) hidden vectors. To obtain forward and backward state, we experiment with three options:

  • –

    Final hidden state: In this approach, we simply use the hidden state of last token of the utterance.

  • –

    Mean and Max pool: In this approach, we concatenate the average and maximum of all hidden states over all tokens in the utterance. This approach is particularly intended towards long utterances where past context information is lost.

  • –

    Attention Mechanism: Attention is one of the most powerful strategy in deep learning models to identify the most significant features (tokens in an utterance) that is helpful for the downstream task. In this work, we use compute attention score ($a_{i}$) for each token in the utterance [10]. This attention score is used as for weighted combination of hidden states ($h_{u}$):

    $$a_{i}=\frac{exp(tanh(W_{a}h_{i}))}{\sum_{j=1}^{n}exp(tanh(W_{a}h_{j}))}$$

    (1)

    $$h_{u}=\sum_{i}^{n}a_{i}h_{i}$$

    (2)

• •

  CNN: We use three layer 1d-CNN model [4] i.e, the model contains three convolution + pooling layer before the feeding the representations to feed forward layer for final predictions. Max-pool operation is used at pooling layer to aggregate features from different filters.

In addition to training a classifier network for toxcitiy comment classification, we also predict toxicity sub-types present in the dataset: severe toxicity, obscene, threat, identity attack, and insult. To train such model, we share the utterance input and representation layer keeping different classification layer for the two tasks at hand. The rationale behind such setup is that to improve the accuracy of detecting toxic comments along with predicting toxicity sub-types. Joint learning setup can act as inherent regularizer for the other tasks due to share representation layer making a multi-task setup a competitive choice for training the model for toxicity classification.

Each utterance $u$ is represented as a matrix of $t\times d$ vectors, where $t$ is the number of tokens in the utterance and $d$ is the embedding dimension. We use two pre-computed embedding lookups from glove vectors [6] and fasttext embeddings [3] to generate document-embeddings fed to the classification network at the input layer. In the training setup, we evaluate the impact of frozen embeddings against making the embedding layer trainable.

The primary downstream task at hand is toxic comment classification which is a binary classification task. The prediction layer $t$ for binary classification task can be represented as:

Additionally, in multi-task setup, we have toxicity sub-type classification as auxiliary task where model needs to predict the toxicity sub-types across six labels with one or more labels being correct. Thus, toxicity sub-type classification is a multi-label classification task which can be represented as:

Since we have a binary and multi-label classification at hand, we use a cross-entropy (CE) loss:

For primary toxicity comment detection task, we use ROC-AUC as the evaluation metric, referred as Toxicity AUC from hereon. In addition to this, to evaluate the models against the proposed research questions, we use three sub-metrics as per the task evaluation framework ²²2https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/overview/evaluation:

• •

  Subgroup AUC: To calculate this metric, the test set is restricted to only those datapoints that contains a particular identity subgroup. A lower subgroup AUC value would mean that the model does a poor job at distinguishing the toxic vs non-toxic comments wherever specific identity is mentioned.

• •

  BPSN AUC: BPSN or Background Positive, Subgroup Negative represents those datapoints where the comments are toxic while subgroup is absent or comments are non-toxic while identity subgroup is mentioned in the comment. A low BPSN AUC value signifies the case where model confuses the non-toxic comments that mentions identity with toxic comments that don’t. In other words, model predicts higher toxicity score than it should for the comments that mentions an identity subgroup.

• •

  BNSP AUC: For BNSP or Background Negative, Subgroup Positive AUC, the test set is restricted to the datapoints where toxic comments mentions the identity while non-toxic comments do not. A low BNSP AUC would refer to the scenario where model predicts lower scores for the toxic comments containing identity subgroup than it should.

The final evaluation metric, called as generalized AUC is measured as:

where, $M_{p}$ is called as pth power mean function, $m_{s}$ is subgroup bias metric for identity group $s$ and $N$ is the number of identity groups. A value of p=5 is choosen as per the task evaluation methodology which enourages to focus on subgroup bias in addition to toxicity AUC scores.

In this work, we use dataset provided in Jigsaw Unintended Bias in Toxicity Classification Kaggle competition. The challenge is designed to build a model that recognizes toxicity and minimizes unintended bias with respect to mentions of identities. The dataset contains more than 1.8M datapoints each annotated by upto 10 annotators. The toxicity score for any comment is determined by the fraction of annotators who have marked a comment as toxic. Each comments was further marked with toxicity sub-types under the categories: severe toxicity, obscene, threat, identity attack, and insult.

In addition to toxicity labels, annotators are also asked to label identity subgroups mentioned in the dataset. A sample annotation from the corpus looks like this:

Comment: Continue to stand strong LGBT community. Yes, indeed, you’ll overcome and you have.
Toxicity Labels: All 0.0
Identity Mention Labels: homosexual_gay_or_lesbian: 0.8, bisexual: 0.6, transgender: 0.3 (all others 0.0)

As per the competition guidelines, we binarize all labels by assigning a label of 1 wherever $score>=0.5$. The dataset distribution across identities is mentioned in Table 1. In the preprocessing step, we remove all special (non-alphanumeric) characters and emoticons. We also truncate any comment to a maximum of 200 sequence length.

We conduct experiments with varying hyperparameters. For model type we used $CNN$ and $LSTM$ which are state of the art for many text classification tasks (prior to transformers era). To study the impact of model size on the performance, we identify the architecture on 3 sizes $S$,$M$,$L$ depending on the number of hidden layers/filters of the model. For $S$, we use $64$ hidden layers/filters in the LSTM/CNN model. For $M$ and $L$, we use $256$ and $512$ hidden layers/filters in the LSTM/CNN model respectively. We use two embedding models $glove$ and $fasttext$ to understand the dependency of the embedding vectors on the bias. We also experimented with the freezing the embedding layer unlike the most suggested option of fine-tuning the embedding. We train them in half the experiments and we froze them in half the experiments to see the biases in the pretrained embedding model. We perform the experiments for 5 epochs each and use early stopping to choose the model with best validation loss. We stop training the model if validation loss does not improve over most recent three epochs. For optimization, we use cross entropy loss as specified in Eqn. 5 trained with Adam optimizer with a learning rate choosen over 1e-3, 1e-4 and 1e-5 for respective models. We code the model in Pytorch framework and the model is trained on a K-80 GPU.

For model training, we do not use any code repository directly. However, our evaluation scripts are inspired from the available codes on Kaggle platform.

Table 2 present the results pertaining to research questions on how does model size, architecture and frozen embedding layer affects the model performance and the unintended bias towards identity groups. Based on results, here are the key observations:

• •

  Model Size: Model with largest parameters i.e, model size L leads to best performance for both model architectures (LSTM: 0.9634 and CNN: 0.9614). Additionally, model L has best bias metric results too (LSTM: 0.9346 and CNN: 0.9296).

• •

  Model Architecture: LSTM model with attention leads to better results as compared to CNN for both toxic classification and the associated bias. However, the difference is marginal as evident from detailed results in Table 2.

• •

  Frozen Embedding: The embedding layer, if frozen, leads to sub-optimal results for each respective comparisons on word embedding (fasttext, glove) or model sizes. On average statistics, freezing embedding layer downgrades the model performance by 0.36% on AUC metric and 0.48% on bias metric. On a deeper look, the degradation is higher (0.6%) for CNN models as compared to LSTM models.

While it is not strictly consistent, the results also show that an increase in model performance for toxicity comment classification also leads to an improvement in bias metric with a few exceptions. Out of top five models selected purely on the basis of AUC score, except for top two ranks, the models stack in the same order on bias metric as well. One rationale for this correlation could be the fact that generalized AUC computation factors on AUC score and thus an increase in AUC leads to improvement in generalized AUC. Decoupling this metrics may be a future task that could be looked into.

Since dataset comes with annotation for mention of identity subgroups, we measure subgroup specific biases as reported in Table 3 for nine identity groups that contain more than 500 examples in the test set (as specified in the competition rules). The subgroup identities at hand are: black, white, make, female, homosexual, christian, jewish, muslim, psychiatric. When the subgroup AUC are calculated for these identities, following observations could be gathered:

• •

  black and homosexual are the identity groups having the highest bias from the models. Along with that, even white ranks among the identity groups facing the most bias from automatic toxicity detection.

• •

  Toxicity detection models are able to identify toxicity for male, female, christian identities with considerably less bias.

• •

  Embedding source be it glove or fasttext does not lead to distinctive results overall as the scores obtained by the respective models are very similar to each other (Avg. AUC per model in Table 3). However, take a granular look at the results, we observe that fasttext embedding leads to less biased models in 8 out of 9 identity groups.

• •

  Almost all identity groups have consistent BPSN AUC scores but BNSP AUC scores vary significantly for different groups (Figure 2). black, white, homosexual have particular a widened gap between the two scores suggesting that model predicts lower toxicity scores for these groups.

In an effort to mitigate unintended bias in the automatic toxicity detection systems, we attempt to leverage the toxicity sub-types annotations to train a multi-task system that jointly learns toxicity binary as well as sub-type labels with a shared representation layer. The results are reported in Table 4. The observations based on the reported results are:

• •

  Multi-task learning setup distinctively leads to better model performance across all reported metrics inclusive of toxicity AUC, subgroup AUC and generalized AUC.

• •

  The reported gain is 0.26% on toxicity AUC, 0.14% on avg. subgroup AUC and 0.12% on generalized AUC metrics for LSTM model.

• •

  Both LSTM and CNN model benefits from the multi-task setup suggesting that such methods could be a potential improvement over vanilla models that could additionally help mitigate unintended model bias.

• •

  Figure 3 shows the impact of multi-task training on BPSN AUC score which reflects the false positivity ratio. The graph demonstrates that MTL training helps the model reduce false positivity ratio on six out of nine identity groups. For some of the identity groups such as make, christian, psychiatric, this gap is wide enough to consider MTL contributing to significant improvement in reduction of FP rate for these groups.