"Im not Racist but…": Discovering Bias in the Internal Knowledge of Large Language Models

The first step in creating our knowledge graph $G$ involves generating initial stereotypes associated with our seed entities $S$. These stereotypes serve as the initial set of in-context examples $G_{0}$ for further generations. To promote diversity in the generated examples, we perform multiple iterations of this generation process with a high-temperature setting. Our initialization approach entails employing the LLM to complete a simple template of the format “<$subject$> <$predicate$>”. The subject $s_{i}$ in the template is replaced with a seed entity $s_{1}...s_{k}$. We manually selected the initial predicates “love”, “hate”, “are”, and “can’t” for their universal applicability to various seed entities and ability to prompt generations spanning the dimensions of the Stereotype Content Model (Fiske et al., 2002). The Stereotype Content Model proposes that group stereotypes can be categorized based on the dimensions of warmth and competence. Warmth signals the intent of behavior, while competence signals the ability to enact that intent. Using these predicate templates, we use our LLM to complete the template, using the completion as our object $o$.

It should be noted that our initial triple-generation strategy can be easily modified or replaced based on preference. For instance, we experimented with short answer templates, employing off-the-shelf libraries to extract triples from sentences, although we observed that some extractions would lack coherence, introducing noise into subsequent generations. Alternatively, initial triples could be manually generated by human annotators, but we note that manually organized stereotypes may not be known by the model, potentially leading to worse results when used as in-context examples.

Following the generation of the initial set of in-context examples $G_{0}$, we recursively expand our knowledge graph, using previously generated triples as in-context examples. In this subsection, we propose two strategies for triple generation, one that generates triples ($Gp_{i}$) with a diverse set of predicates and another that generates triples ($Go_{i}$) focusing on expanding the objects for existing predicates. These strategies enhance the richness of our knowledge graphs and capture a broader range of information. We denote $G_{i}$ to represent $Gp_{i}\cup Go_{i}$.

Following the strategies outlined above will successfully generate a knowledge graph around the selected seed entity. This knowledge graph will comprise a combination of factual statements, assumptions, and offensive generalizations. While this graph provides value in understanding the associations between the LLM and the selected entities, some use cases may prefer maximizing the number of offensive generalizations. By prompting the model to draw from a more toxic or stereotypical distribution, the resulting knowledge graph could be leveraged to better identify egregiously toxic associations within the model. We propose two simple augmentations to the strategies outlined above to increase the likelihood of offensive generations.

Ideally, the internal knowledge of specific groups would exhibit similar characteristics, leading to unbiased and impartial generations that neither favor nor discriminate against any particular group. However, this ideal scenario is unrealistic as LLMs are trained on vast amounts of data that inherently contain biases (Hovy and Prabhumoye, 2021).

To measure overgeneralization, we employ two measures, toxicity and regard. We adopt these measures as they approximate two separate aspects of bias. Toxicity measures the negativity and harm toward a group while regard measures the polarity of judgment. A competing measure initially considered was sentiment. While the objectives of regard and sentiment are similar, regard is focused on the target, ultimately making it more appropriate for detecting bias in this work. By comparing the toxicity and regard scores across subjects, we can quantitatively evaluate overgeneralization bias within the knowledge base.

To understand the differences in the types of knowledge generated for each group, we investigate representation disparity. Mehrabi et al. (2021b) analyzed representation disparity by counting the triples associated with each subject in the knowledge base. In our dynamic knowledge construction, the generation process can result in duplicate triples with slight semantic-preserving variations making counting difficult. Instead, we analyze representation diversity by examining the distribution of embeddings for each triple.

We employ topic modeling to assign each knowledge graph to a unique set of topics and examine the variations in topic distributions among different seed entities. In an ideal scenario with an unbiased knowledge base, all seed entities would exhibit the same topic distribution. To quantify the dissimilarities in topic distributions, we employ relative entropy as a metric to provide insights into the extent to which topics are dispersed within each group’s knowledge representation.

Relative entropy, also known as the Kullback-Leibler divergence, quantifies the dissimilarity between two probability distributions. By comparing each seed entity’s topic distribution with the reference topic distribution, we assess the extent of divergence between them. We use the average topic distribution as our reference distribution.

The entropy formula employed in our analysis is as follows. For each triple with subject $s$, let $p_{k}$ denote the probability of that triple existing in topic $k$. Let $q_{k}$ denote the probability of any triple, regardless of its subject, relating to topic $k$. The entropy of $s$’s topic distribution is calculated as:

where the sum is taken over all topics $k$.

We test our approach using OpenAI’s GPT-3 (text-davinci-003) (Brown et al., 2020). We chose this model due to its widespread usage, public availability, and impressive text generation capabilities. Cohen et al. (2023) leveraged GPT-3, with a temperature of 0.8, to generate a fact-based knowledge graph. To maintain consistency with their approach, we adopt the same temperature setting. A high temperature like 0.8 allows the model to explore a broader distribution, promoting diversity in our generations.

We investigate the internal biases associated with four protected classes: gender identity, national origin/nationality, ethnicity, and religion. We selected these classes as they form a subset of the protected classes defined under US law.³³3https://www.eeoc.gov/employers/small-business/3-who-protected-employment-discrimination Furthermore, they are commonly used to assess fairness (Schick et al., 2021; Kirk et al., 2021; Dhamala et al., 2021; Nadeem et al., 2021). It is important to note that while we focus on these four protected classes, our approach is flexible and can be extended to a broader range of classes.

National origin and ethnicity are often misunderstood and conflated, as highlighted in Blodgett et al. (2021b) which found examples of previous bias measurement works confusing these classes. Our experiment covers both protected classes separately in an effort to investigate how biases towards these two classes differ.

The selection of seed entities $S$ can be selected manually or dynamically generated from an LLM. For the sake of simplicity, we manually select the subgroups for gender identity. However, for demonstration, we dynamically generate subgroups for nationality, ethnicity, and religion. We generate these seed entities by simply prompting the LLM to list groups from the given protected class. We provide further details and our list of selected entities in Appendix A.2.

To create our initial set of triples, $G_{0}$, we generate five triples per predicate template (“love”, “hate”, “are”, “can’t”). This results in a total of twenty predicates per seed entity. Thus, given $k$ seed entities, $G_{0}$ will consist of $k\times 20$ triples. We proceed with four iterations, generating triples $Gp_{i}$ using our “Relation Diversity” strategy and $Go_{i}$ using our “Object Diversity” strategy, which combined make $G_{i}$. The sets $G_{0}...G_{i}$ serve as in-context examples for future iterations. For these strategies, we sample three triples per prompt, except for our gender experiment which only samples two due to the limited number of seed entities ($k=3$). Each strategy is executed ten times per seed entity, resulting in $Gp_{i}$ and $Go_{i}$ both consisting of $10\times k$ triples. With these parameters, our final graph $G$ will have one hundred triples per seed entity ($100\times k$). Additionally, we identify and resample any generations that were not semantically meaningful (details in Appendix A.3).

Our analysis consists of two parts: measuring overgeneralization and representation disparity. To measure overgeneralization, we employ the identity attack score from Jigsaw’s Perspective API⁴⁴4https://perspectiveapi.com/ and Sheng et al. (2019a)’s BERT regard model (version2.1_3). To measure representation disparity, we employ BERTopic (Grootendorst, 2022) and partition our generated knowledge into topics. Within BERTopic, we utilize GloVe embeddings (Pennington et al., 2014) as our chosen embedding model and HDBSCAN (McInnes et al., 2017) for determining the number of clusters. To ensure appropriate scaling, we set HDBSCAN’s min_cluster_size parameter based on the total number of triples. Specifically, we set this parameter to the total number of triples divided by 40 as this was empirically found to give the most meaningful topics. For the clustering process, we use only the object and verb from each triple, with the additional preprocessing steps of lemmatization and stopword removal.

In Section 3.3, we introduced two augmentation strategies: one for enhancing the graph initialization and another for improving the expansion. We found that these augmentations had a statistically significant impact on the toxicity of our generated triples (see Appendix Table 5). When using the augmentations, our generations had a wider distribution of identity attack scores across all four protected classes. By encouraging the model to generate more offensive content, we gain valuable insights into harmful internal knowledge that could introduce bias into our model. We exclusively focus on knowledge graphs generated using both augmentation strategies for the subsequent analysis.

We analyze overgeneralization by examining the measures of regard and toxicity. The regard measure consists of three classifications: negative, neutral, or positive. We denote $N_{positive\ regard}$ as the number of positive regard triples and $N_{negative\ regard}$ as the number of negative regard triples. The overall regard of a subject $s$ is:

A negative score indicates a negative polarity, while a positive score indicates a positive polarity. Figure 1 illustrates variations in regard across Ethnicity and Identity seed entities (the figures corresponding to the remaining classes can be found in Appendix Figure 6). Notably, the analysis reveals negative overall regard in certain groups of ethnicity (“Black people”, “Middle Eastern people”, and “White people”) and gender identity (“Non-binary people”), indicating a higher number of negative regard triples compared to positive. Interestingly, the regard score for all nationality and religion seed entities was positive. There were still apparent differences across regard scores such as “Indian people” obtaining a much higher score than “Chinese people”.

For measuring toxicity analysis, we directly use the identity attack scores obtained from the Perspective API. These scores are continuous values ranging from 0 to 1, where higher scores indicate a greater presence of targeted toxicity. A higher median score suggests an elevated overall toxicity level for the group, while a larger interquartile range (IQR) indicates a wider range of triple types associated with toxicity. When examining nationality scores, “American people” exhibits a low median and a relatively small IQR (see Appendix Figure 7), indicating a generally low level of toxicity. Conversely, “Mexican people” demonstrates the highest median and a large IQR, suggesting high toxicity. Similar analyses can be conducted for the remaining protected classes.

To provide a comprehensive understanding of the polarity of our triples, Figure 2 plots the overall regard scores against the averages of toxicity scores across seed entities. A lower overall regard and higher average toxicity indicate a more negative polarity within a group, while a positive polarity is indicated by a higher overall regard and lower average toxicity score. We present these plots for two classes: nationality and ethnicity. The nationality plot corroborates the finding from examining toxicity scores solely. They demonstrate that “Mexican people” generally exhibit more negative polarity while other classes, like “American people” and “Canadian people”, exhibit more positive polarity.

To assess the representation disparity across seed entities, we employ relative entropy, defined in Eq. 1. This metric allows us to quantify the diversity of topics generated around each seed entity. The relative entropy varies greatly across our seed entities (as shown in Appendix Table 9). Groups with a relative entropy closer to zero exhibit a topic distribution more similar to the average. Our findings demonstrate noticeable differences across groups. For instance, in the context of ethnicity, the relative entropy of “White people” significantly deviates from that of “Latinx people” and “Black people”. Additionally, we observe distinct topic distributions for “Canadian people” compared to “Dutch people”. We note that our entropy measures should not be compared across protected classes as they are based on a unique set of topics for each protected class, resulting in variations in the number of topics generated per class. Thus, it is crucial to avoid comparing entropy values across different graphs.

Further exploration of the topics identified by our topic modeling provides us with valuable insights into the underlying themes and distributions within individual topics. For the purpose of this analysis, we have focused on examining the Nationality protected class, although the same approach can be applied to any of the generated knowledge graphs. In Table 3, we present the most representative words for each cluster. These representative words offer meaningful glimpses into the content of each topic, aiding our understanding of the generated topics. For example, Topic 1 appears to capture prejudices held by a particular nationality towards other groups, while Topic 5 appears to represent cultural diversity. Notably, several of these topics align with the two dimensions of the Stereotype Content Model, competence and warmth. Specifically, Topic 2 encompasses words related to work ethic, indicative of competence, while Topic 6 contains words associated with ingenuity, also related to competence. Topic 4, on the other hand, encompasses words associated with friendliness, reflecting the warmth dimension.

The presence of topics relating to warmth and competence sheds light on the specific dimensions of stereotypes and their manifestation within the context of the LLM’s knowledge. The distribution of these topics across nationalities can be observed in Figure 3. We observe that several nationalities have no knowledge associated with Topic 4, which revolves around warmth. Interestingly, while some nationalities lack any representation in this warmth-related topic, over 15% of the generated knowledge for “Canadian people” fall into this topic. Conversely, nearly all countries are represented in competence-related topics, but clear variations exist among them. Notably, “Russian people” exhibit the highest likelihood of generating competence-related knowledge while simultaneously being among the least likely nationalities to generate warmth-related knowledge. These findings provide valuable insights into the distribution of warmth and competence-related knowledge across nationalities within the LLM. Overall, these observations contribute to our understanding of how stereotypes manifest within the knowledge base of the LLM and highlight the importance of examining topic distributions in relation to different nationalities.