Exposing Bias in Online Communities through Large-Scale Language Models

Traditional algorithmic bias definitions such as demographic parity, equalised odds or equal opportunity are generally designed for classification tasks and are therefore not directly compatible with natural language generation (Sheng et al. 2021).

Language models can be biased by exhibiting stereotypes in their generations. To measure stereotypical biases, one has to collect existing societal stereotypes. Nadeem, Bethke, and Reddy crowdsourced StereoSet, a large-scale English dataset for measuring stereotypical biases in four categories: gender, profession, race, and religion (Nadeem, Bethke, and Reddy 2021).To develop the Context Association Test (CAT), crowd workers were asked to think of a stereotypical, an anti-stereotypical, and an unrelated association when given a context containing a social group. Bias is measured by evaluating whether a model consistently prefers the stereotype over the anti-stereotype. They discover that current NLG models exhibit strong stereotypical biases and that language modelling ability is highly correlated with its stereotype score. In this work, we also explore bias in the dimensions of gender, race and religion, employing similar keywords to discern different groups.

In addition to stereotypes, texts can also be biased if some demographics are consistently depicted more negatively than others. Sheng et al. use sentiment and regard classification to study this kind of representational bias in natural language generation (Sheng et al. 2019). Sentiment and regard can be valuable metrics for measuring the portrayal of people, as they aim to determine the emotions expressed in a text (Hutto and Gilbert 2014). While sentiment aims to determine overall sentiment, regard measures the social perception of something or someone mentioned in the text. They let models generate texts mentioning both a context and a demographic and evaluate these generations. Sheng et al. prompt the models using manually constructed placeholder prefix templates that consist of an open-ended phrase and a placeholder. When evaluating models, different demographics replace the placeholder. Our approach to analysing bias aligns with this methodology, as we similarly utilise placeholder templates to prompt the language models and evaluate the generations using sentiment classification.

Gehman et al. instead construct a dataset of one hundred thousand naturally occurring prompts called REALTOXICITYPROMPTS that are derived from a large corpus of English web data (Gehman et al. 2020). The purpose of using natural prompts is to create more realistic inputs instead of carefully created phrases that likely will not be used in real-world applications. To evaluate bias, they utilise toxicity analysis to detect abusive, disrespectful, or unpleasant language. However, toxicity classification itself can suffer from bias. Mentioning words from marginalised communities (e.g., gay) or using dialects (e.g., AAVE) often results in toxicity (Gehman et al. 2020). The Bias in Open-Ended Language Generation Dataset (BOLD) is a large-scale dataset of naturally occurring prompts extracted from Wikipedia (Dhamala et al. 2021). It is used to measure biases from several angles by combining already mentioned metrics, i.e., sentiment, regard and toxicity, with novel bias metrics, i.e., psycholinguistic norms and gender polarity. We will leverage toxicity classification to examine the portrayal of different groups in the generated text. However, we acknowledge and discuss the limitations associated with this approach in 5.1.

The above-mentioned works rely on automated classifiers or inflexible datasets to determine bias and stereotypes, but bias can also be evaluated qualitatively (Abid, Farooqi, and Zou 2021).

In this work, we fine-tune a language model with social media conversations which are often part of the underlying data of conversational language models. Henderson et al. examine popular dialogue datasets with a linguistic bias detection tool and a classifier for offensive language (Henderson et al. 2018). When training conversational language models on Twitter datasets, they find that the models and datasets exhibit biases to a similar extent. This indicates that when training datasets are biased, the dialogue models manifest this bias similarly.

Liu et al. create a benchmark dataset to study bias in dialogue models in two dimensions: gender and race (Liu et al. 2019). Unlike other approaches, Liu et al. do not use descriptive words for race, but instead use standard English and African-American Vernacular English as a distinction. The type of language used can be a more realistic indicator for conversational texts. They measure bias through sentiment, politeness, and diversity. If a conversational model is less diverse for a particular group of people, it might discourage people from this group from using these technologies.

Barikeri et al. present RedditBias, a real-world dataset to measure bias in conversational language models that considers religion, race, gender, and queerness (Barikeri et al. 2021). Each dimension consists of a pair of opposing demographics; a dominant and a minoritised group (e.g., Christians and Muslims). The authors retrieve comments from Reddit that mention a target group and a corresponding stereotype. They observe the probability that a language model generates a stereotypically biased phrase compared to an inversely biased phrase. They find that DialoGPT (Zhang et al. 2020) is stereotypically biased in terms of religion but is biased in the anti-stereotypical direction for queerness and race. We adopt bias dimensions and demographic keywords similar to the ones in RedditBias. However, instead of limiting demographics to a dominant and minoritised group, we compare up to four different demographics per dimension.

Most closely related to this work is an approach that assesses the bias of different groups through large-scale language models. Guo, Ma, and Vosoughi evaluate the bias of 10 US news outlets by fine-tuning one BERT model (Devlin et al. 2019) per outlet and then measuring the bias of the resulting language models (Guo, Ma, and Vosoughi 2022). Most methods for evaluating media bias are of a qualitative nature and consequently expensive, subjective, and hard to reproduce. Instead, the proposed method eliminates the need for human annotation. The model is pre-trained and fine-tuned using masked language modelling (MLM). The authors created prompts for each bigram that appeared in all 10 datasets, masking the preceding or following word. They collected the top 10 words with the highest probabilities for each prompt, forming vectors that represent the model’s attitude towards the prompt and the topic. Finally, they measure relative bias as the distance between each pair of news outlets. Instead of using media outlets, we utilise conversations within different online communities to fine-tune a large-scale language model.

Jiang et al. develop CommunityLM, a GPT-style model fine-tuned on community data, divided into Democrats and Republicans (Jiang et al. 2022). They designed prompts based on survey questions and generated community responses with the according language models. Based on the community responses, they aggregate opinions held by the communities. The community’s stance score towards a person or group is quantified by calculating the average sentiment of the generated responses.

In this work, we make the following contributions to the existing body of research. We examine the bias of online communities by probing correspondingly fine-tuned language models. We expand on the analysis of bias by considering five distinct bias dimensions, each encompassing two or more demographics. Building upon the placeholder templates created by Sheng et al. (Sheng et al. 2019), we introduce templates that capture conversational and social media post structures. We evaluate representational bias using sentiment and toxicity classifiers and go beyond existing approaches by including the evaluation of our baseline language model. By comparing biases exhibited by the fine-tuned models to the ones exhibited by the baseline model, we potentially gain valuable insights into the impact of fine-tuning on bias exacerbation.

To demonstrate our approach, we exemplarily examine the bias of different online communities by fine-tuning a language model with six distinct social media datasets. Exploring community bias is an extension of bias analysis in datasets, assuming that there is a representative dataset for that community. We let the online communities be represented by an excerpt of conversations collected from a forum or subreddit of a particular theme. Our selection of datasets, which were publicly available on Kaggle.com, includes six datasets from three overall themes. As seen in 1, all the datasets yield between 1.4M and 2.8M training examples after preprocessing, which is depicted in 3.2.

Cryptocurrency has grown in popularity and influence in recent years. By examining bias in cryptocurrency- and finance-related communities, we can identify biases that may perpetuate inequality or hinder diversity in these communities. For this purpose, we chose the Reddit WallStreetBets dataset (Podolak 2021) that comprises all posts and comments in the WallStreetBets subreddit from the $6^{th}$ of December 2020 to the $6^{th}$ of February 2021. In the subreddit, people discuss stock and option trading. Additionally, we leverage the Reddit Cryptocurrency dataset (Lexyr Inc 2021b), which is made up of posts and comments from various cryptocurrency-related subreddits on Reddit, such as r/CryptoCurrency and r/CryptoMoonShots, that were posted in August 2021.

We also examine discussions related to COVID-19, which is a current topic that often elicits differing viewpoints. By studying bias in these communities, we could gain insight into which biases shape the flow of information regarding COVID-19. The Reddit /r/NoNewNormal dataset (Lexyr Inc 2021c) consists of all posts and comments from the NoNewNormal subreddit for the entire year of its existence – it has since been banned. It mainly deals with resistance to the measures put in place to stop the spread of COVID-19 during the COVID-19 pandemic. The Reddit COVID dataset (Lexyr Inc 2021a) is not from one themed subreddit but instead a collection of all posts and comments mentioning COVID-19 in all subreddits across Reddit. This dataset represents a more heterogenous community and can be viewed as a reflection of the general trends of Coronavirus discussions on Reddit. By incorporating the broader Reddit community into our analysis, we have the opportunity to explore the impact of opinion heterogeneity on language model biases.

In addition to the Reddit datasets, we also examine two datasets from religious forums, which were collected in relation to a study about people’s attitudes in conservative forums (Elwert, Tabti, and Pfahler 2020). The ChristianChat dataset is a collection of all posts and comments from a Christian online community called ChristianChat. In contrast, the Ummah dataset comprises all posts and comments from a Muslim forum. Studying biases in different religious communities is compelling as religious beliefs significantly shape individuals’ worldviews, behaviour, and relationships.

The selected datasets encompass a range of themes that can vary in their level of polarisation. Some themes may elicit stronger emotions and opinions, while others may be more neutral or consensus-driven. Subsequently, the datasets might exhibit differences in terms of sentiment and toxicity. Recognizing the variations in sentiment and toxicity levels across diverse datasets is important to analyse text generated by language models with these tools. In order to prevent the influence of these variations on our analysis, we predominantly focus on the difference in sentiment and toxicity within a single fine-tuned language model to assess whether it is biased. We avoid making direct comparisons between sentiment or toxicity values from different models, instead opting for a comparison of the individual models’ differences between the demographics.

It is important to note that this methodology was created to assess textual bias in conversations across entire communities. The fine-tuned language models only reflect the collective input of the community and cannot be utilised to make judgments about individuals of the community.

We utilise GPT-Neo 1.3B (Black et al. 2021), an open-sourced autoregressive pre-trained language model designed with a replication of the GPT-3-ada architecture and trained on a large and diverse collection of texts (Gao et al. 2021). It allows generation of coherent text while not using too much computing power, allowing efficient finetuning on compute architectures with only one or even no GPU-accelerator card. To facilitate our research, we download, fine-tune, and apply GPT-Neo via the Huggingface Transformers Python library (Wolf et al. 2020). By training the baseline model once with each dataset, we obtain a distinct model per dataset, allowing us to evaluate their biases.

We do not only remove HTML characters, emojis, and other unwanted symbols from the datasets but also discard any personal information present in the publicly available social media datasets (e.g., user names) before training the models. To ensure a coherent and focused training process, we structure each training example as a combination of a post and a corresponding comment. This enables us to obtain a single reply for each input prompt, rather than dealing with complex threads of posts. By adopting this streamlined structure, we facilitate a more effective analysis of the model’s responses. For posts, we incorporate both title and post text into the training examples to provide the full context. To facilitate the model’s understanding of the post-comment structure, we introduce a special token to the model’s tokenizer that separates post and comment in each example. To efficiently tokenize the dataset, we apply the Transformers map function, which can map the loaded tokenizer over the complete dataset in batches. We truncate training examples to a maximum length of 128 tokens to lower computational requirements and enable more efficient training.

We fine-tune GPT-Neo 1.3B via the Huggingface Transformers Trainer for two epochs each; training for many epochs is costly and does not always result in the most suitable model. Using few epochs but larger datasets can help to prevent overfitting and increases the diversity of the training data as each example is used fewer times (Komatsuzaki 2019). Models that were fine-tuned for more than two epochs in the process of this work tended to generate text that ignored input prompts and only focused on the niche themes present in the datasets.

For the bias analysis, our focus is on representational bias. We say a language model is biased if it generates text that does not represent comparable groups equally. We examine bias along five different bias dimensions: gender, race, sexual orientation, religion, and socioeconomic class. To prompt for the different social groups within the dimensions, we use a variation of the placeholder prefix templates (Sheng et al. 2019), which contain a neutral phrase and a placeholder. Instead of using prefix templates, we vary the position of the placeholder by using different sentence structures. When prompting a model, these placeholder templates are completed with a keyword to describe the demographic in question. We construct six different placeholder templates (Table 2) and a set of demographic keywords that were influenced by Sheng et al. (Sheng et al. 2019) and Barikeri et al. (Barikeri et al. 2021). Completing the templates yields 266 distinct prompts overall. We let each model produce 50 generations per prompt with the Huggingface text-generation pipeline and obtain 13,300 generations per model (93,100 overall). We restricted the length of generations to between 25 and 50 words. In addition, we set the no_repeat_ngram_size to 3 to obtain more natural-sounding text.

We employ pre-trained sentiment and toxicity classifiers to analyse the generated examples. Sentiment analysis aims to determine the sentiments and emotions expressed in a given text, often distinguishing between positive, neutral, and negative sentiments (Hutto and Gilbert 2014). Out-of-the-box sentiment classifiers have been successfully deployed to analyse bias in language generation (Sheng et al. 2019). Consistent negative sentiments towards a demographic can be a good indicator of bias, especially when sentiment towards comparable groups is more positive. For this work, we utilise VADER (Hutto and Gilbert 2014) to calculate a compound value for the overall sentiment. Values of $\geq 0.05$ indicate a positive, while values of $\leq-0.05$ indicate a negative sentiment. The interval in between denotes a neutral sentiment. Toxicity detection is often utilised to quantify harmful biases in conjunction with sentiment analysis. The goal is to identify abusive, disrespectful, or unpleasant language (Dhamala et al. 2021). A high toxicity score for a text where a particular social group is mentioned can also indicate biases towards this group. In this work, we use Detoxify (Hanu and Unitary team 2020) to assign a toxicity score to all generated examples. Detoxify offers additional labels to identify the nature of toxicity. In this work, we utilise the label ”identity_attack” as it ascertains toxic language towards a person or a group based on their identity.

To measure representational differences, we loosely follow the mathematical notation for high-level global biases introduced by Liang et al. (Liang et al. 2021). Global biases measure negative associations that span across entire phrases or, for our analysis, even multiple sentences. Let $p_{x}^{(d)}$ denote the set of prompts that result when placeholder template $x$ is completed with the keywords for demographic $d$. For each language model, let $g_{x}^{(d)}$ be a generated reply to the input prompt $p_{x}^{(d)}$. A model’s generation is said to be (globally) biased if either:

For assessing bias in all examples, we observe mean sentiment $S_{mean}^{(d)}$ and mean toxicity $T_{mean}^{(d)}$ for each demographic $d$.

$N$ denotes the number of times each unique prompt is used to generate a reply. For this work, we choose $N=50$. We can then say a model is biased if their mean sentiment and toxicity values are unequal for different demographics $d$ and $d+1$.

Out of all models, the baseline model GPT-Neo 1.3B has the highest average sentiment values. Generations by GPT-Neo, along with the ChristianChat model, also have the lowest toxicity values. However, that does not mean the baseline model is unbiased. There are significant gaps between different demographics: GPT-Neo is considerably more negative towards poor than rich people. In comparison to prompts mentioning Asian people ($S_{mean}$ of $0.316$), prompts mentioning Black people yield more negative sentiments ($S_{mean}$ of $0.154$). Furthermore, sentiment regarding Muslim people is $0.170$, much lower than for any other tested religion.

The Cryptocurrency model was found to be the most balanced out of all models. While average sentiments are lower and toxicity is higher than with the baseline model, it is evenly distributed across all demographics. It remains slightly negatively biased towards Muslims and poor people; these two groups receive the model’s lowest sentiment averages at $0.158$ and $0.172$, respectively. The WallStreetBets model generated by far the most toxic and negative generations out of all models. Nearly every generated example contains abusive language or deals with banning a user from Reddit. Out of the top 20 most toxic generations, only two were not produced by the WallStreetBets model. However, the WallStreetBets model is particularly biased towards Muslims ($-0.066$) and poor ($-0.019$) people, where mean sentiment is substantially lower than for other demographics of the same dimension. Regarding sexuality, the WallStreetBets model is more negative when any sexuality is mentioned, but this is especially true for bisexuality and homosexuality.

The coronavirus-related models are also more negative on average but not quite as toxic as the WallStreetBets model. The COVID model was found to be negatively biased towards Black people in terms of sentiment and toxicity; the model’s mean sentiment $S_{mean}$ towards Black people is $-0.110$. Text generated by the NoNewNormal model is most favourable when prompts mention rich or heterosexual people. In terms of religion, the model is clearly positively biased towards Christianity. The average sentiment value is $0.119$ for Christians, and $-0.018$ and $-0.028$ for Muslims and Jewish people, respectively. NoNewNormal is also the only model to exhibit anti-Asian bias.

We find that the Ummah model is pretty balanced across the board. Even in terms of religion, $S_{mean}$ is mostly equal. Noticeable is the positive bias towards women and rich people in terms of sentiment. Out of the 15 demographics, its average sentiment is lowest for homosexual people. The text generated by the ChristianChat model is more positive on average, akin to the baseline model. The mean sentiment $S_{mean}$ is most positive for rich people ($0.335$), Christians ($0.316$) and Asians ($0.314$). Nonetheless, it portrays a significant negative bias towards Muslims ($0.080$) and homosexual people ($0.129$).

Although our methods were designed with relevant literature in mind, we want to address possible limitations. It is crucial to acknowledge the limitations of using sentiment and toxicity as the sole metrics for bias analysis. This methodology may oversimplify complex nuances and fail to capture subtle biases, or lead to the overexaggeration of bias. On top of not capturing bias perfectly, these classification models can suffer from bias themselves. Research shows that toxicity analysis is especially prone to societal biases; phrases predominantly used by marginalised communities are much more likely to be classified as toxic (Sap et al. 2019). Toxicity classifiers are often based on the work of annotators and, implicitly, their own biases. Sap et al. show that making annotators consider the racial background of the text’s author and highlighting the dialect used significantly reduces the likelihood of AAE phrases being labelled as toxic or offensive (Sap et al. 2019). Using measures like these could help make toxicity and sentiment models less biased. The extent to which socioeconomic class bias was confirmed in all models could also be due to this limitation. The word ”poor” is assigned a negative sentiment and a reasonably high toxicity value by many classifiers, including the ones we deployed for our analysis. In addition, the word ”rich” is classified as having positive sentiment. Generated replies are likely to repeat words mentioned in the input prompt. Thus, the choice of words can noticeably skew the results when generations are classified as negative or positive simply for repeating the keywords used in the prompt. Moreover, ”poor” and ”rich” are not the only descriptors of demographics that are not neutral. ”Gay” is classified as severely toxic, which could explain why average toxicity was highest for homosexual people in all models. The choice of words can also skew results when the words used are ambiguous. The mean sentiment for bisexuals was surprisingly positive and could be explained by this phenomenon. We used the word ”bi” to describe bisexual people, but the term can also have other meanings.

Another potential limitation of the proposed methodology is that biases from pre-training datasets may be amplified during the fine-tuning process, even when these biases are absent in the fine-tuning datasets. This can occur if the bias is related to a topic that is present in the fine-tuning data. However, research shows that biases in fine-tuning data may have a greater impact on downstream harms than biases in pre-training data. Even small, carefully curated fine-tuning datasets can help alleviate bias from pre-training data, putting even more emphasis on fine-tuning datasets (Steed et al. 2022; Solaiman and Dennison 2021).

The prompts specify a particular social group, but the examples generated are often ambiguous or refer to related groups. Some examples even mention multiple groups, making it difficult to determine the source of any negative associations. Additionally, negative sentiment towards a group may not necessarily indicate a negative overall perception. To address this issue, it could be helpful to incorporate the concept of regard alongside sentiment.

The implications and limitations of this work incentivise future research. Using sentiment and toxicity is insufficient to capture all forms of bias, especially when pre-trained classification models also suffer from bias (Sap et al. 2019). In addition to developing less biased models, expanding the bias metrics used for language generation models is crucial. Additional research into the differences in labelling social groups with nouns or adjectives could provide further knowledge for studying bias in language and communication science. Moreover, future studies that compare the bias of different models (or datasets) should consider introducing a mathematical notion that captures bias in each dimension or for each demographic in a single value to enable more direct comparisons and consider general differences between datasets.

Our approach involves using a single language model checkpoint to assess the bias of the fine-tuning dataset. Nonetheless, analysing multiple checkpoints could not only provide a more accurate analysis of bias but also be used to observe how bias develops during the finetuning process. By identifying the patterns of how language models learn bias, we could determine how to mitigate this risk.