RECAST: Interactive Auditing of Automatic Toxicity Detection Models

There is a growing desire to moderate and remove toxic language from social media and public forums, as a result of increasing online interactions [5]. For example in a 2015 user survey of the online social network platform reddit, 50% of people who wouldn’t’t recommend reddit cited hateful or offensive content and community as the reason why, however on the other hand 35% of complaints from extremely dissatisfied users were about heavy handed moderation and censorship[3]. This balance of issues is further complicated by the high volume of interaction on these platforms, making manual moderation often not tractable and leading to the development of automatic toxicity detection models such as the Google Perspective API [7].

However, the existing body of research highlights potential flaws in the underlying language models for toxicity detection systems. For example, several word embedding models and their training language datasets exhibit biases towards certain subgroups such as gender and race [2, 9]. Some widely deployed NLP models, specifically BERT, also tend to overlook simple linguistic structures like negation, reducing their effectiveness [4]. A serious problem is that it is difficult to fix the potential erroneous model outputs due to a lack of the interpretability of NLP models.

These flaws in language models persist in toxic statement detection systems, especially for end users. Therefore users of online forums that use toxicity detection systems based on black-box NLP models might question how their language is being examined. Nonnative speakers may inadvertently write language that could confuse the system and be marked as toxic. Without tools designed for actual end users to audit what is being detected and make actionable changes to their language, people are disempowered to participate in discourse online. Furthermore, without an ability to detect when a model is falsely flagging language due to either linguistic limitations or social biases, the work of finding and correcting bias are left entirely to the unrepresentative population of machine learning researchers and software engineers. Therefore the ability to audit these models must be provided to end-users affected by them.

We address these challenges by developing an interactive tool called Recast, which allows for the interrogation of toxicity detection models through counterfactual alternative wording and attention visualization. This design does not require any expertise in machine learning, but enables users to visualize their sentence through the eyes of the algorithm. Recast currently supports analysis of a fine-tuned BERT model on the Jigsaw Toxicity dataset [6]. Our ongoing work presents the following contributions and vision:

Recast is an interactive system (Figure 1) that allows users to input text and view the toxicity of the overall sentence, along with which words most contribute to output score. Recast displays a score between 0 and 100, which represents the probability that the model will classify the input as toxic. Users can view highly attended words, edit the text, select suggested alternate wordings, and watch the toxicity score dynamically update. Its design is purposefully simple, mirroring the text interaction techniques of underlining to note where editing is required and selecting listed alternatives which end users are familiar with from common software like Microsoft Word or Grammarly. This accessibility allows RECAST to effectively communicate the complexities of toxicity detection models using a visual language users are already fluent in.

Since the purpose of Recast is to provide power to end-users, an important feature to include is an ability to flag when the model gets it wrong. These examples can be used to provide researchers with data for retraining their models, and provide an avenue of recourse for people adversely affected by the errors in the model. Therefore the statements about the accessibility and usability of Recast must be validated empirically by a user study. To this end, we plan on evaluating end-users’ capacity to reduce toxicity in a sample text given Recast. In preparation for a full study we have run a small pilot on 18 participants through Amazon Mechanical Turk with approval from the Institutional Review Board, where we found that users given Recast rated the usefulness of the tool in reducing toxicity on a 5 point scale (higher being more useful) an average of 4.4, compared to 3.6 for a control group. While these results are not highly statistically significant, they help justify further exploration into this tool.

Recast takes steps towards increased transparency for black-box NLP models that are responsible for moderating large swaths of the internet. By enabling users to interact with text input, view alternative wordings for toxic sentences, and identify potential biases, Recast provides insights about models to those people actually affected by them, and allows everyone to participate online in both a less toxic, and more fair, environment.

This work was supported in part by NSF grants IIS-1563816, CNS-1704701, NASA NSTRF, gifts from Intel (ISTC-ARSA), NVIDIA, Google, Symantec, Yahoo! Labs, eBay, Amazon.