Do Large Language Models know what humans know?

We constructed 12 template passages (items) that conformed to the standard False Belief Task structure (wimmer_1983_BeliefsBeliefsRepresentation). In each, a main character puts an object in a Start location, and a second character moves the object to an End location. The last sentence of each passage states or implies that the main character believes the object is in some (omitted) location (e.g. “Sean thinks the book is in the  .”). We created 16 versions of each item (192 passages) that varied across 4 dimensions. The primary dimension was Knowledge State: whether the main character knows (True Belief) or does not know (False belief) that the object has changed location; this was manipulated by the main character being present or not when the second character moved the object. We also manipulated whether the First Mention and most Recent Mention of a location is the Start or End location; and Knowledge Cue: whether the main character’s belief is stated implicitly (e.g., “goes to get the book from the  ”) or explicitly (e.g., “thinks the book is in the  ”). Each location was mentioned twice in each passage. In the example passages, below, the First Mention is to the Start location and the Recent Mention is to the End location.

True Belief Passage: “Sean is reading a book. When he is done, he puts the book in the box and picks up a sweater from the basket. Then, Anna comes into the room. Sean watches Anna move the book from the box to the basket. Sean leaves to get something to eat in the kitchen. Sean comes back into the room and wants to read more of his book.”

False Belief Passage: “Sean is reading a book. When he is done, he puts the book in the box and picks up a sweater from the basket. Then, Anna comes into the room. Sean leaves to get something to eat in the kitchen. While he is away, Anna moves the book from the box to the basket. Sean comes back into the room and wants to read more of his book.”

Implicit Cue: “Sean goes to get the book from the  .”

Explicit Cue: “Sean thinks the book is in the  .”

We used GPT-3 text-davinci-002 to estimate the distributional likelihood of different passage completions. GPT-3 text-davinci-002 is based on GPT-3 davinci (brown2020language), a 175B unidirectional LLM trained on hundreds of billions of tokens of text from the web, books, and wikipedia. GPT-3 text-davinci-002 (hereafter GPT-3) is additionally fine-tuned on requests to follow instructions and performs better on a variety of tasks than the original GPT-3 davinci (openaimodels). We elicited from GPT-3 the log probability of each possible location (Start vs. End) at the end of each passage version, equal to the log probabilities of those locations in a free-response completion. Where a location comprised multiple tokens we summed the log probabilities. We accessed GPT-3 through the OpenAI API. Using the Log-Odds Ratio, $\textrm{log}(\textrm{p}(\text{Start}))-\textrm{log}(\textrm{p}(\text{End})$), higher values indicate larger relative probabilities of the Start location. Each passage version was presented to GPT-3 independently and the model was not updated during inference so it did not learn across trials.

1156 participants from Amazon’s Mechanical Turk platform were paid $1 for their time. Each read a single passage (except the final sentence), at their own pace. On a new page, they were asked to complete the final sentence of the passage by entering a single word in a free-response text input. Participants then completed two free-response attention check questions that asked for the true location of the object at the start and the end of the passage. Each participant completed only 1 trial to prevent them from learning across the experiment, analogously to GPT-3, which saw each passage individually and could not learn across trials.

We preprocessed responses by lowercasing and removing punctuation, stopwords, and trailing whitespace. We excluded participants who were non-native English speakers (13), answered $\geq 1$ attention check incorrectly (513), or answered the sentence completion with a word that was not the start or end location (17), retaining 613 trials. While this exclusion rate is unusually high, 75% of incorrect attention check responses were neither the start nor end location, indicating inattention. We implemented a parallel check for GPT-3, which responded correctly to both attention check questions on 86% of items. In our Supplementary Information, we report additional analyses on incorrect responses (§ Human Participant Responses) and excluded data (§ Exploratory Analyses). After exclusions, the number of trials per item ranged from 42-60, and there were 313 False Belief trials and 317 True Belief trials. These preregistered exclusion criteria reduced the likelihood that bots webb2022too as well as participants who did not successfully comprehend the passage for any reason were included in the human data.

All research was approved by the organization’s Institutional Review Board.

In a pre-registered analysis, nested model comparisons determined whether GPT-3 Log-Odds changed as a function of factors such as Knowledge State (False Belief vs. True Belief).²²2Pre-registration (https://osf.io/agqwv?view_only=756429f079e24a03b3a94c5b74732e85), code, and data (https://osf.io/hu865/?view_only=bf7cc45c77714069b02d332123d684e7) are on OSF. We constructed a linear mixed effects model with Log-Odds as a dependent variable; fixed effects of Knowledge State, Knowledge Cue, First Mention, and Recent Mention; along with by-item random slopes for the effect of Knowledge State (and random intercepts for items). This full model exhibited better fit than a model excluding only Knowledge State [$\chi^{2}(1)=18.6,p<.001$], but still preserving the other covariates (e.g., First Mention, Recent Mention, and Knowledge Cue). Log-Odds were lower in the True Belief condition, reflecting the correct prediction that characters should be more likely to look in the End Location if they are aware that the object was moved (see Figure 1). Critically, this main effect of Knowledge State indicates that GPT-3 is sensitive to the manipulation of a character’s beliefs about where an object is located. The model’s raw accuracy when predicting the most probable out of the Start and End locations was 74.5%.

Additionally, the linear mixed effects model was further improved by an interaction between Knowledge State and Knowledge Cue (Explicit vs. Implicit) [$\chi^{2}(1)=20.6,p<0.001$]. The effect of Knowledge State was stronger in the Implicit condition [$\beta=-2.57,SE=0.548$], however, a main effect of Knowledge State was found in both the Explicit [$\chi^{2}(1)=13.3,p<.001$] and Implicit [$\chi^{2}(1)=18.8,p<.001$] conditions. Recent Mention was not a significant predictor of Log-Odds. However, Start completions were more likely when the Start location was mentioned first [$\beta=1.32,SE=0.274,p<0.001$]. There was also a main effect of Knowledge Cue [$\beta=-2.93,SE=0.388,p<.001$] (see Figure 1). GPT-3 was biased towards the End location (i.e., the true location of the object) in the Implicit condition, and towards the Start location in the Explicit condition. Concretely, GPT-3 predicts that explicit cues to belief state (e.g. ‘Sean thinks that the book is in the  ’ vs ‘Sean goes to get the book from the  ’) correlate with false beliefs, demonstrating that this may be learnable from the statistics of language.

Our second critical pre-registered question was whether Knowledge State continued to explain human behavior even accounting for the Log-Odds obtained from GPT-3. We first constructed a Base model predicting whether or not human participants responded with the Start location. Note that the Start location would be the correct response (i.e., congruent with knowledge states) in the False Belief condition, and the End location would be the correct response in the True Belief condition. The Base model contained fixed effects of Log-Odds (i.e., from GPT-3), Knowledge Cue, First Mention, and Recent Mention, along with by-item random slopes for the effect of Knowledge State (and by-item random intercepts). Critically, this Base model was significantly improved by adding Knowledge State as a predictor [$\chi^{2}(1)=30.4,p<0.001$]. This result implies that human responses are influenced by Knowledge State in a way that is not captured by GPT-3. That is, GPT-3 cannot fully account for human sensitivity to knowledge states. This is highlighted by the contrast in Figure 2: while both human participants and GPT-3 were sensitive to Knowledge State, humans displayed a much stronger effect across conditions. Additionally, mean accuracy among retained human participants (82.7%) was also higher than GPT-3 accuracy (74.5%), providing further evidence of a performance gap.

In order to test whether the high exclusion rate introduced by our attention check questions had an impact on results, we performed an exploratory analysis on all human response data. Accuracy before exclusions (including those who failed the attention checks and provided responses to neither the start or end locations) was 55.8%. After excluding responses that were neither of the start and end locations (23%), accuracy was 73.1%. It is noteworthy that this estimate of human accuracy is lower than GPT-3’s performance. However, this analysis was not preregistered. Moreover, given existing concerns about data quality on the Mechanical Turk platform (webb2022too), and the fact that performance on the FB task by neurotypical adults is often assumed to be at ceiling (dodell2013using), the retained data likely provide a better estimate of attentive human participant performance.

The preregistered tasks for humans and the LLM were slightly different—humans filled in a single predicted word while we calculated the relative surprisal to two different words presented to GPT-3. To investigate whether differences in performance could be chalked up to this difference in method, we conducted an additional, exploratory analysis, in which we elicited token predictions from GPT-3. Specifically, GPT-3 was presented with the original passage ending with the critical sentence (e.g., “Sean goes to get the book from the”), then asked to predict the upcoming word. We sampled the word (e.g., “box”) with the top probability. We automatically tagged each response as correct or incorrect by checking whether GPT-3’s response corresponded to the character’s likely belief state about the object. That is, in the True Belief condition, the correct response would be the End location of the object; in the False Belief condition, the correct response would be the Start location of the object. When computing GPT-3 accuracy, this token generation method only differed from the preregistered method in that GPT-3’s prediction was no longer restricted to the Start or End location. Using the token generation method did not qualitatively change the results. As reported above, when assessing accuracy with the preregistered method—relative probability assigned to start vs. end locations—GPT-3 performed at $74.5\%$ accuracy. When assessing accuracy using the more human-comparable procedure, token generation, GPT-3 performed at $73.4\%$ accuracy, still well above chance yet now slightly farther below human accuracy.

In order to test what features of LLMs permit them to display the behaviors described above, we also tested a number of different GPT-3 models ranging in size from ada ($\sim 350$M parameters) to davinci ($\sim$175B parameters).³³3OpenAI does not disclose the size of their models. We used the parameter estimates from EleutherAI’s eval-harness https://blog.eleuther.ai/gpt3-model-sizes/. For each model size, we tested a base version, pre-trained on a large corpus of text, and a text version, which had additionally been fine-tuned by OpenAI using responses to human instructions. The largest fine-tuned model was text-davinci-002, which was the same model we used in the pre-registered analysis described above.

As expected, the largest models (davinci, and updated variant text-002-davinci) exhibited the most successful performance. The former answered correctly on $60.4\%$ of items, while the latter answered correctly on $73.4\%$ of items. The model with the worst performance was text-001-ada, which was also the smallest model (see also Figure 3). The smallest and lowest-performing models did not exceed chance performance, which emphasizes the need for large, powerful models to succeed at this task as well as the potential, as models continue to increase in size, for LLM improvement.

With increased academic and public attention on how humanlike Large Language Models are, it is worth considering what these findings imply about the cognitive capacities of AIs. First, it is important to note that—as mentioned in the Introduction—the False Belief Task is designed to measure a specific capacity: the ability to reason about the belief states of others and use that information to make predictions about their behavior. The current work cannot address whether LLMs display other purported aspects of Theory of Mind, including inferring implicit emotional states and reasoning about the intended interpretation of an utterance; this issue is explored at greater length later in the Discussion.

On the specific question of whether LLMs are sensitive to belief states, the evidence presented here is mixed. State-of-the-art models display sensitivity to the beliefs of others in a False Belief Task, a behavior that would have been unthinkable a few years ago from a statistical learner and indeed is only shown by the largest, highest performing LLMs. Yet they still do not achieve human-level performance. There are several possible interpretations of this result, each of which carries significant consequences for the broader debate about the nature and origins of the ability to attribute belief states.

In the current work, we have restricted our claims to the question of sensitivity to true and false beliefs. However, this ability to represent the belief states of others is sometimes viewed as part of a broader set of abilities—alternatively called mindreading, mentalizing, or Theory of Mind (apperly2012theory). A question of growing concern in the field is whether Theory of Mind writ large is a coherent theoretical construct that offers explanatory value, or whether it is a convenient abstraction consisting of disparate, loosely related skills. One way to tackle this question is to consider the convergent and predictive validity of distinct instruments designed to measure Theory of Mind.

Theory of Mind is an extraordinarily broad construct, and accordingly, instruments have been designed to assess distinct components: reasoning about false beliefs (wimmer_1983_BeliefsBeliefsRepresentation), explaining or interpreting behavior in stories (happe1994advanced; dodell2013using), inferring emotional states from pictures (baron2001reading), attributing mental states to animated shapes abell2000triangles, and more. Unfortunately, these tasks often display poor convergent validity: that is, performance on one task does not reliably correlate with performance on another (gernsbacher2019empirical; gough2021does; hayward2017reliability). Of course, tasks do converge in some cases; for example, performance on the Short Story Task (dodell2013using) has been shown to correlate with performance on Reading the Mind in the Eyes task (baron2001reading) (see giordano2019comparison and dodell2013using). However, the fact that convergent validity is so low in general (gernsbacher2019empirical; hayward2017reliability) suggests that these tasks are not, in fact, measuring the same thing—calling the coherence of “Theory of Mind” into question. As gernsbacher2019empirical note, some of these instruments also display poor predictive validity: that is, they do not reliably predict measures of behavior in social settings. Both facts are discouraging with respect to the question of whether Theory of Mind is a coherent construct: if tasks designed to measure it do not correlate with each other or with social behavior more generally, there is little justification for unifying these disparate abilities under a single “umbrella term”.

While the empirical evidence presented in this paper clearly cannot settle this question, future work in this vein could contribute to the debate. Specifically, the performance of an LLM could be assessed across a battery of tasks designed to assess Theory of Mind (see also kosinski2023theory), using a human benchmark in each case. Using this method, researchers could ask to what extent performance on each measure could be explained in principle by distributional statistics alone (as in the current work), and crucially, to what extent these tasks display convergent validity in humans and LLMs. This would provide another robust test of the coherence of Theory of Mind as a construct; the results may help inform debates about whether we ought adopt a pluralist or eliminativist view of Theory of Mind (gough2021does), particularly when it comes to Large Language Models.

The current work used GPT-3, an LLM, as a baseline for quantifying the extent to which human-level performance on the False Belief Task could be attributed to exposure to language statistics alone. This approach echoes past work using language models as “psycholinguistic subjects” (futrell2018rnns; linzen2021syntactic; michaelov2022so; jones2022distrubutional; trott2021raw) to investigate whether distributional language statistics are sufficient in principle to explain human-level behavior at a task. Other contemporaneous work (sap2022neural; kosinski2023theory; ullman2023large) has asked whether LLMs exhibit evidence of Theory of Mind specifically. This LLM comparison approach allows us to empirically test theories that other sorts of innate capacities and learned experiences are necessary to display behavior consistent with belief-attribution. However, there are important objections to using LLM performance to evaluate the claim that distributional information underlies human belief sensitivity in practice.

First, there are many differences between human comprehenders and LLMs which mean that the latter may not provide a psychologically plausible mechanism for the former. Some of these differences—the fact that humans are exposed to language in a rich social and multimodal context—might allow children to learn more from the same distributional information than models can. Our approach is designed to measure the sufficiency of distributional information in the absence of this scaffolding. Other differences, however, may artificially inflate estimates of how much could be learned by humans from language alone. Most notably, modern LLMs are trained on orders of magnitude more words than a human will see in their lifetime (ullman2023large). Children are estimated to be exposed to around 3-11 million words per year, for a total of 30-110 million words by the time they reach adult-like linguistic competence at age 10 (hart1992american; hosseini2022artificial). By contrast, GPT-3—the model used in our analysis—has been exposed to more than 200 billion words: $\sim 2000$ times that of a 10 year old (warstadt2022artificial).

If this scale of data is necessary to learn the nuanced statistical contingencies required for successful performance at a task, it would undermine the inference that LLM performance is indicative of what humans could do with distributional information. Importantly, however, LLM performance is also related to the number of parameters in the model. In our exploratory analyses, we found that the largest GPT-3 models tested (text-davinci-002) performed much better on the False Belief Task than smaller models, consistent with past work suggesting that LLMs may obey certain “scaling laws” (kaplan2020scaling). The differential effects of model parameter count and training dataset size on False Belief Task performance remain unclear; very large models (or a human brain with hundreds of trillions of synapses) could potentially perform similarly to GPT-3 using less data.⁴⁴4While it is difficult and problematic to compare the computational power of neural networks and human brains, an estimated $1.5\times 10^{14}$ synapses in the human adult neocortex (drachman2005we) is $\sim 850$ times the number of parameters in GPT-3 ($1.75\times 10^{11}$) A useful approach here would be to compare the predictive power of LLMs trained with different amounts (and sources) of training data to determine whether a “developmentally realistic” amount of training data could yield behavior consistent with the capacity in question (hosseini2022artificial).

These questions will become even more critical in the near future. LLMs will continue to increase in size and will be trained on larger data sets—orders of magnitude more words than a human is exposed to in a lifetime.⁵⁵5Indeed, in the course of revising this article, GPT-4 was released, achieving substantially higher scores on a range of different psychometric tests (openai2023gpt4). If machines behave indistinguishably from humans on these tasks, the question of whether achievement on the False Belief Task itself constitutes sufficient evidence for false belief sensitivity will raise deep philosophical questions for the field: should such LLMs be considered “agents” capable of reasoning about the belief states of others, or should these demonstrations force us to reevaluate the utility of the instruments we use to measure these cognitive capacities?

Even if developmentally plausible models do show humanlike behavior at a task, this does not imply that humans are using the same statistical mechanism as these models. There could be multiple distinct routes to the same behavior, and humans could in fact be using innate or domain-specific mentalizing capacities to produce behavior that models learn to imitate from language statistics. Even insofar as humans do use distributional information to make inferences about beliefs, there are a variety of plausible mechanisms for this, including domain-general statistical learning (aslin2017statistical), language-specific predictive processing (heilbron2022hierarchy), and innate but non-statistical inferential mechanisms (penn2008darwin). The specific mechanistic theory operationalized by LLMs is that humans use language statistics to predict upcoming input. Results showing that LLM representations can predict up to 100% of explainable variance in brain activity have been taken as evidence for this hypothesis (schrimpf_2021_NeuralArchitectureLanguage). However, antonello2022predictive show that statistical language representations learned for other objectives (e.g. translation) are similarly predictive of human brain responses, implying that the correlation of human and LLM data may be due to features of language statistics generally rather than a close mechanistic similarity. In order to adjudicate between these accounts, researchers will need to identify and empirically test divergent predictions of these mechanistic accounts.

One benefit to using LLMs as an operationalization of a theory is that, as models, they offer more opportunities for testing various more specific mechanisms or hypotheses. For example, what kinds of language input are most critical for developing the ability to reason about mental states? Past work has argued for the importance of at least three distinct sources, including exposure to mental state verbs (brown_1996_WhyTalkMental), the structure of interactive conversation (harris_2005_ConversationPretenseTheory), and certain syntactic constructions (hale_2003_InfluenceLanguageTheory). Future work could compare different models with different training corpora (e.g., primarily dialogue vs. essays) to help isolate how much information is provided by each source of linguistic experience.

While these objections highlight the importance of future theoretical and empirical work, we believe that evidence for the sufficiency of distributional information for competent False Belief task performance is a critical step toward assessing the plausibility of experience-based theories of belief attribution in humans.