Facilitating Fine-grained Detection of Chinese Toxic Language:
Hierarchical Taxonomy, Resources, and Benchmarks

More and more people have acquired information from social media platforms where posts containing toxic language are also rampant. Toxic language is viewed as a rude, disrespectful, or unreasonable comment that is likely to make someone leave a discussion Dixon et al. (2018). Due to its negative impact on individuals and society, toxic language has been rapidly recognized as an increasing concern Silva et al. (2016). Recently, researchers have tackled the problem of toxic language detection using the techniques of natural language processing, making great progress in many languages. Mou et al. (2020); Cao and Lee (2020); Tekiroglu et al. (2020); Founta et al. (2018); Zhou et al. (2021a); Mathew et al. (2021); Caselli et al. (2020); Hanu and Unitary team (2020).

In contrast, the relevant research on Chinese toxic language detection has lagged significantly Jahan and Oussalah (2021). There are two key issues that have been overlooked. First, existing studies Deng et al. (2022); Jiang et al. (2022); Zhou et al. (2022) lack a fine-grained annotation of textual toxic types, resulting in hate speech being conflated with general offensive language. Compared to hate speech, general offensive language does not attack groups with special social attributes, and it is just used to emphasize emotions in many cases Wang et al. (2014). Like Exp. 1 in Table 1, the insult "fuck" can be considered as a modal particle to express surprise. Since there is no equivalence between general offensive language and hate speech, it is crucial to determine their boundary conditions Davidson et al. (2017).

In addition, most studies on toxic Chinese language only concentrate on detecting direct and explicit bias and offense. And they lose sight of indirect expressions including implicit hatred (e.g., stereotype and irony) ElSherief et al. (2021) and reporting experiences of discrimination Chiril et al. (2020). Due to the absence of direct swear words, these indirect toxic samples are obviously harder to be filtered ElSherief et al. (2021). To further illustrate the distinction of several expressions, a few examples are listed in Table 1. Meanwhile, compared to English, Chinese has richer variants of profanity with implicit toxic meaning Zhang (2010); Sohn and Lee (2019), which brings challenge to research on toxic language detection. However, the existing insult lexicon fails to cover these terms. An example is "fairy" in Exp. 3, which itself is a positive word and is used here to implicitly attack women. Due to the significance of lexical knowledge to detect toxic language Wiegand et al. (2021); Hartvigsen et al. (2022), it is important to construct an insult lexicon containing implicit toxic terms.

To fill these gaps, we facilitate fine-grained detection of Chinese toxic language. To distinguish hate speech from general offensive language and analyze the expressions of samples, we first introduce Monitor Toxic Frame, a hierarchical taxonomy. Based on the taxonomy, the posts are progressively divided into diverse granularities as follows: (I) Whether Toxic, (II) Toxic Type (general offensive language or hate speech), (III) Targeted Group, (IV) Expression Category (explicitness, implicitness, or reporting). After taking several measures to alleviate the bias of annotators, we then conduct a fine-grained annotation of posts, including both direct and indirect toxic samples. And ToxiCN dataset is presented, which has 12k comments containing sexism, racism, regional bias, and anti-LGBTQ.

For the convenient detection of toxic language, we construct an insult lexicon against different targeted groups. It contains not only explicit profanities but also implicit words with toxic meanings, such as ironic metaphors (e.g., "fairy"). To exploit the lexical feature, we further present a migratable benchmark of Toxic Knowledge Enhancement (TKE), enriching the text representation. In the evaluation phase, several benchmarks with TKE are utilized to detect toxic language, demonstrating its effectiveness. After that, we analyze the experimental results in detail and offer our suggestions for identifying toxic language. The main contributions of this work are summarized as follows:

• •

  We present a hierarchical taxonomy, Monitor Toxic Frame, to progressively explore the toxic types and expressions of samples from diverse granularities.

• •

  Based on the taxonomy, we propose ToxiCN, a fine-grained dataset of Chinese toxic language. It divides hate speech from offensive language, including samples with not only direct offense but also indirect expressions.

• •

  We present an insult lexicon, and a Toxic Knowledge Enhancement benchmark to incorporate the lexical feature. We evaluate its performance at different levels and conduct an exhaustive analysis.

Toxic Language Detection. Toxic language detection is a high-profile task in the field of natural language processing. Recently, most researchers have utilized methods of deep learning based on the pre-trained language model to tackle this problem Mou et al. (2020); Cao and Lee (2020); Tekiroglu et al. (2020); Mathew et al. (2021); Zhou et al. (2021a). Two re-trained BERT Devlin et al. (2019), HateBERT Caselli et al. (2020) and ToxicBERT Hanu and Unitary team (2020), have been specifically proposed to detect toxic language. Davidson et al. (2017); Founta et al. (2018); Mathew et al. (2021) attempted to distinguish hate speech from offensive language, while ElSherief et al. (2021); Hartvigsen et al. (2022) explored the benchmark of implicit and latent hate speech and Chiril et al. (2020); Pérez-Almendros et al. (2020) considered testing for reporting related to hate speech and behavior. Meanwhile, for the construction of the toxic language dataset, there have been some studies focused on how to improve the reliability of the annotation process to alleviate the subjective bias of annotators Waseem and Hovy (2016); Zeinert et al. (2021).

Linguistic Research of Chinese Toxic Language. Chinese toxic language has been researched extensively in language studies and sociolinguistics. Zhang (1994) analyzed Chinese vernacular novels and summarized the common rhetorical methods of offensive language. According to Wang and Liu (2009); Li et al. (2020), insults are more easily expressed through variants. Due to the lack of morphological markers, authors often make sentences based on language sense and consensus Zhang (2004), expressing hatred more concisely compared to Indo-European Zhang (2005). Zhang (2010); Sohn and Lee (2019) compared insults in English and Chinese and discovered that Chinese has a richer variety of profanity due to its unique culture and linguistics. These linguistic features bring challenges to Chinese toxic detection. Recently, some Chinese toxic language datasets have been constructed Deng et al. (2022); Jiang et al. (2022); Zhou et al. (2022). However, they fails to separate hate speech from general offensive language, and overlooks toxic samples containing indirect expressions. Besides that, it lacks the construction of the lexicon containing implicit profanities. In this work, we fill these gaps to facilitate fine-grained detection of Chinese toxic language. Here we list Table 2 to compare these studies with our ToxiCN.

We divide the insult lexicon built in the process of annotation into five categories according to the attacking object. The lexicon includes sexism, racism, regional bias, anti-LBGTQ, and general swearwords, referring to the swear words that can be used to offend any group. The final dictionary contains 1,032 terms, including not only explicit vulgar words but also implicit insults. In addition, as the Internet generates a wealth of new insults every year, it is vital to explore their origins. Therefore, for the convenience of the follow-up research, we further analyze the rules for the derivation of Internet swear words from the proposed insult lexicon. Specifically, we briefly summarize them in terms of both surface features and their actual meaning. More related terminology notes and examples of profanity are illustrated in Appendix C.

Surface Features. To circumvent the filtering mechanism, netizens change the original insults and create new variants with similarities in glyphs and pronunciation Chen (2012); Zhang (2010), which are called deformation and homophonic, respectively. In addition, Chinese characters are at times replaced with other language codes in some profanities Li et al. (2020), creating code-mixing words or abbreviations.

Actual Meaning. Internet users often introduce implicit terms to attack or defame the target groups, including usage of metaphor and irony Chen (2012). Besides that, some borrowed words also contain specific prejudices, which are used in implicit toxic comments Shi (2010). Compared to variants based on surface features, these terms with deep semantics have to be detected with background knowledge.

In view of the significance of lexical knowledge to detect toxic language, we propose a migratable benchmark of Toxic Knowledge Enhancement (TKE), incorporating the lexical feature to enrich the original sentence representation. The illustration of TKE is shown in Figure 2.

For a given sentence $S=\{x_{1},x_{2},...,x_{n}\}$, each token $x_{i}$ is embedded as $w_{i}\in\mathbb{R}^{d}$, which is a vector representation in $d$-dimensional space. Inspired by Zhou et al. (2021a), we design a toxic embedding to introduce lexical knowledge. Specifically, we first employ the n-gram to determine whether $x_{i}$ is a subword of an insult, and if so, its attacked group is further indicated. Then, we randomly initialize the group category representation as $C=(c_{0},c_{1},...,c_{m})$, where $c_{i}\in\mathbb{R}^{d}$, $c_{0}$ refers to non-toxic term and $m$ is the number of categories of the insult lexicon. In this work, $m=5$. Based on the $c_{i}$, we further propose the definition of toxic embedding $t_{i}$ of $x_{i}$:

Since the element-wise addition of multiple linear vector representations is an efficient way to fully incorporate various information Mikolov et al. (2013), we utilize this method to integrate toxic and word embedding. The enhanced representation of $x_{i}$ is $w_{i}^{\prime}=w_{i}+\lambda t_{i}$, where $\lambda\in[0,1]$ is a weighting coefficient to control the ingestion of toxic knowledge. The ultimate sentence embedding of $S$ is $\{w_{1}^{\prime},w_{2}^{\prime},...,w_{n}^{\prime}\}$, which is the input of the connected sequence encoder. Due to its convenience, TKE can be migrated to any pre-trained language model (PLM).

Due to the rampant dissemination of toxic online language, an effective detection mechanism is essential. In this work, we focus on Chinese toxic language detection at various granularities. We first introduce a hierarchical taxonomy Monitor Toxic Frame to analyze the toxic types and expressions. Based on the taxonomy, we then propose a fine-grained annotated dataset ToxiCN, including both direct and indirect toxic samples. Due to the significance of lexical knowledge for the detection of toxic language, we build an insult lexicon and present a benchmark of Toxic Knowledge Enhanced (TKE), enriching the representation of content. The experimental results show the effectiveness of TKE on toxic language detection from different granularities. After an error analysis, we suggest that both knowledge introduction and bias mitigation are essential. We expect our hierarchical taxonomy, resources, benchmarks, and insights to help relevant practitioners in the identification of toxic language.

Despite the fact that some measures have been implemented to minimize bias in labeling, we are still explicitly aware that our dataset may contain mislabeled data due to differences in the subjective understanding of toxic language by the annotators. Meanwhile, due to the limitation of data coverage, our benchmark is not practical for all types of toxic comments.

For reasons of intellectual property, we only capture the comments rather than the full text, which affects the actual semantics of the sentence to some extent. Besides that, non-textual features are not taken into account in this work, such as images and meta information about publishers. In future work, we will further research span-level and multi-modal toxic language detection.

We strictly follow the data use agreements of each public online social platform and double-check to ensure that there is no data relating to user privacy. The opinions and findings contained in the samples of our presented dataset should not be interpreted as representing the views expressed or implied by the authors. We hope that the benefits of our proposed resources outweigh their risks. All resources are for scientific research only.

This research is supported by the Natural Science Foundation of China (No. 62076046, 62006034). We would like to thank all reviewers for their constructive comments.