bitsa_nlp@LT-EDI-ACL2022: Leveraging Pretrained Language Models for Detecting Homophobia and Transphobia in Social Media Comments

Twenty first century social media has become the epicenter of polarized opinions, arguments, and claims. The ease of information access not only benefits fruitful discussions but also facilitates phenomena such as hate speech and cyber bullying.

Recently organized workshops and shared tasks have fostered discussions around detection of hate speech, toxicity, misogyny, sexism, racism and abusive content (Zampieri et al., 2020; Mandl et al., 2020). While research in processing and classifying offensive language in social media is vast (Pamungkas et al., 2021), there is very little work on detecting sexual orientation discrimination in particular. More so, compared to resource-rich languages such as English and Japanese, Indic languages such as Tamil and Malayalam are scarce in well-annotated data. Although advancements in large multilingual models have promoted cross-lingual transfer learning in Indic languages Dowlagar and Mamidi (2021), there have not been any visible attempts to censor homophobia and transphobia. The perception of the subject matter as being taboo prohibits advancements in data collection, annotation and analysis.

Curbing sensitive online content is imperative for preventing harm to mental health of the community as well as avoiding divide between minorities. These reasons have contributed towards the need of moderating social media comments spreading any form of hatred towards the LGBTQIA+ population.

While both - the detection of homophobia/transphobia and the corresponding research in Indic languages - is underserved and low-resource, another factor contributing to the difficulty in processing social media texts is code-mixing - a phenomena in which multilingual speakers switch between two or more languages in a conversation with the aim to be more expressive. Popular language models tend to perform adversely when applied to code-mixed text and hence newer techniques need to be adopted to handle this situation (Doğruöz et al., 2021).

The pretraining and fine-tuning paradigm has taken extensive advantage of transformer based large multilingual models which perform well in cross-lingual scenarios. In this paper we explore the performance of a number of such models when fine-tuned on a dataset for detecting homophobia and transphobia. Surprisingly, our experiments also show that these multilingual models exhibit reasonably accurate performance on code-mixing tasks, even without any previous exposure to code-mixing during pretraining.

The remainder of the paper is organized as follows: Section 2 talks about the previous related work in this domain. Section 3 gives a detailed explanation of the methods used in the system and Section 4 describes the corresponding experimental settings. We mention the detailed results in Section 5, conduct an ablation study in Section 6 and conclude our discussion with Section 7.

To the best of our knowledge no prior work identifying either homophobia or transphobia directly exists in recent literature. However, offensive language detection, in general, in Dravidian languages has been the focus of multiple research works in the past (Chakravarthi et al., 2021a; Mandl et al., 2020).

Baruah et al. (2021) at HASOC-Dravidian-CodeMix-FIRE2020 trained an SVM classifier using TF-IDF features on code-mixed Malayalam text and an XLM-RoBERTa based classifier on code-mixed Tamil text to detect offensive language in Twitter and YouTube comments. Sai and Sharma (2020) fine-tuned multilingual transformer models and used a bagging ensemble strategy to combine predictions on the same task.

Saha et al. (2021) developed fusion models by ensembling CNNs trained on skip-gram word vectors using FastText along with fine-tuned BERT models. A neural classification head was trained on the concatenated output obtained from the ensemble.

A number of approaches have been deployed to tackle code mixing in Indic languages as well, since multilingual transformer models lack the complexity to extract linguistic features directly from code switched text. Vasantharajan and Thayasivam (2021) used a selective translation and transliteration technique to process Tamil code-mixed YouTube comments for offensive language identification. They converted code-mixed text to native Tamil script by translating English words and transliterating romainzed Tamil words. Similar technique was used by Upadhyay et al. (2021) and Srinivasan (2020).

This shared task was formulated as a multiclass classification problem where the model should be able to predict the existence of any form of homophobia or transphobia in a YouTube comment. The entire pipeline consists of two main components - a classification head on top of different popular models based on the transformer architecture, and a data augmentation technique for oversampling the English dataset. These components have been explained in further detail ahead.

In this section we review the setup needed to reproduce the experiments.

The metric used to rank system performances is macro-averaged F1-score. It is calculated as the (unweighted) arithmetic mean of all the per-class F1-scores.

where $i$ is the class index and $N$ is the number of classes

Tables 5,  7 and  9 list the macro-averaged Precision, macro-averaged Recall and macro-averaged F1-score for various PLMs tested on English, Tamil and code-mixed Tamil-English development dataset respectively.

Similarly Tables 6,  8 and  10 list the corresponding metrics achieved by the final submissions on English, Tamil and Tamil-English test dataset as released by the organizers.

The tables also provide baseline metrics for each track based on the method explained in Section 4.4.

In this section we discuss the effect of preprocessing and data augmentation (DA) on the model performance.

The dataset as described in Section 4.4 is highly skewed towards the Non-anti-LGBT+ content class. Hence it makes sense to compare the performance of a majority classifier with that of the models submitted for evaluation.

We train a dummy classifier based on most-frequent strategy and tabulate the results (macro-averaged Precision, Recall and F1-score) in Table 11. We deliberately use the un-augmented version of preprocessed English dataset to show the show the performance of the majority classifier without handling class imbalance.

The poor performance is a consequence of the extreme class imbalance which we aim to solve by data augmentation. However, not all DA techniques prove to be effective for all NLP tasks. Thus we also analyze the effect of preprocessing and DA on the performance of transformer models.

Table 12 analyzes the efficacy of EDA as a DA technique for handling class imbalance in our English dataset. It also divides a line between the performance of the model on the stock dataset v/s one that has been preprocessed.

We observe that preprocessing (de-emojification in all three tracks and de-punctuation in the case of only English) does not increase the macro-averaged F1 score for English and Tamil. Infact it reduces the score by a small margin. However, we notice a significant improvement in the case of code-mixing.

We also observe that EDA is not an efficient DA technique as it fails to handle the class imbalance. Transformer models were able to successfully predict with higher precision and recall in the absence of any augmentation and with limited samples.

Homophobia and transphobia have not been the focus of many umbrella hate speech detection tasks. We examined the ability of pretrained large transformer-based models to detect homophobia and transphobia in a corpus of YouTube comments written in English and Tamil. Experimental results demonstrated that multilingual BERT performed the best on both language tasks, and the code-mixed task as well, without being exposed to any code-mixing beforehand. This can be attributed to its capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks.

From Section 6 we also observed that the effect of preprocessing was largely dependent on the choice of language setting. This makes sense considering the difference in underlying language constructs. Tamil, for instance, does not make use of standard English-based punctuation marks. On the other hand, we conclude that the choice of an effective DA techniqe depends on the underlying task and the data source. Social media data often lacks linguistic purism and hence, token perturbations such as those introduced by EDA did not help.

In the future, we would like to adopt a more aggressive DA technique such as that involving text generation (text In-filling, generating typos) or an auxilliary dataset (kNN, LM decoding). We would also like to evaluate the effect of translation and transliteration on code-mixed text classification.

We would like to acknowledge the efforts of the workshop organizers in effecting positive social change through AI by conducting such shared tasks. We also thank the reviewers for their time and insightful comments.