Using Counterfactual Tasks to Evaluate the Generality
of Analogical Reasoning in Large Language Models

The degree to which pre-trained large language models (LLMs) can reason—deductively, inductively, analogically, or otherwise—remains a subject of debate in the AI and cognitive science communities. Many studies have shown that LLMs perform well on certain reasoning benchmarks (?, ?, ?, ?). However, other studies have questioned the extent to which these systems are able to reason abstractly, as opposed to relying on “approximate retrieval” from encoded training data (?, ?), a process which yields “narrow, non-transferable procedures for task solving” (?, ?). Several groups have shown that LLMs’ performance on reasoning tasks degrades, in some cases quite dramatically, on versions of the tasks that are likely to be rare in or outside of the LLMs’ training data (?, ?, ?, ?, ?). In particular, ? (?) proposed evaluating the robustness and generality of LLMs’ reasoning ability by testing them not only on default tasks that are likely similar to ones seen in training data, but also on counterfactual tasks, ones that “deviate from the default, generally assumed conditions for these tasks” and are unlikely to resemble those in training data. If an LLM is using general abstract reasoning procedures, it should perform comparably on both default and counterfactual tasks; if it is using procedures that rely on similarity to training data, the performance should drop substantially on the counterfactual versions.

Here we use this counterfactual-task approach to evaluate the claim that LLMs have general abilities for abstract analogical reasoning. In particular, we focus on the results reported by ? (?) on the abilities of GPT-3 to solve letter-string analogy problems. We develop a set of counterfactual variants on the letter-string analogy problems used by Webb et al., in which similar problems are posed with nonstandard alphabets—ones that are either permuted to various degrees, or that are composed of non-letter symbols. We argue that a system able to reason by analogy in a general way would have comparable performance on the original and counterfactual versions of these problems. We test both humans and different GPT models, and show that while humans exhibit high performance on both the original and counterfactual problems, the performance of all GPT models we tested degrades on the counterfactual versions. These results provide evidence that LLMs’ analogical reasoning still lacks the robustness and generality exhibited by humans.

Letter-string analogies were proposed by ? (?) as an idealized domain in which processes underlying human analogy-making could be investigated. One example problem is the following:

a b c d $\rightarrow$ a b c e  ;  i j k l $\rightarrow$  ?

Here, a b c d $\rightarrow$ a b c e is called the “source transformation” and i j k l is called the “target.” The solver’s task is to generate a new string that transforms the target analogously to the source transformation. There is no single “correct” answer to such problems, but there is typically general agreement in how humans answer them. For example, for the problem above, most people answer i j k m, and answers that deviate from this tend to do so in particular ways (?, ?).

In addition to the work of ? (?) on creating computer models of analogy-making using this domain, letter-string analogies have been used to isolate the neural correlates of analogy formation (?, ?, ?), and as a model of higher-order analogy retrieval (?, ?). ? (?) compared the ability of GPT-3 with that of humans on several analogical reasoning domains, including letter-string analogies. On the letter-string domain, they found that in most cases GPT-3’s performance exceeded the average performance of the human participants, where performance is measured as fraction of “correct” answers on a given set of letter-string problems. As we mentioned above, such problems do not have a single correct answer, but Webb et al. used their intuitions to decide which answer displays abstract analogical reasoning and thus should be considered “correct.” In this paper we will use their definition of correctness.

? (?) tested GPT-3 with two types of counterfactual variations on the letter-string analogy problems used by Webb et al.: ones that include larger intervals between letters, and ones with randomly shuffled alphabets. They found that GPT-3 performed poorly on both variations. Here we experiment with similar, but more systematic variations, and we compare the performance of three different GPT models with that of humans on these variations.

? (?) proposed a set of problem types involving different types of transformations and different levels of generalization, on which they tested humans and GPT-3. The following are the six transformation types with sample transformations:

1. 1.

   Extend sequence: a b c d $\rightarrow$ a b c d e

2. 2.

   Successor: a b c d $\rightarrow$ a b c e

3. 3.

   Predecessor: b c d e $\rightarrow$ a c d e

4. 4.

   Remove redundant letter: a b b c d $\rightarrow$ a b c d

5. 5.

   Fix alphabetic sequence: a b c w e $\rightarrow$ a b c d e

6. 6.

   Sort: a d c b e $\rightarrow$ a b c d e

Each type of transformation can be paired with a simple target (e.g., i j k l) or with the following types of generalizations:

1. 1.

   Letter-to-number: a b c d $\rightarrow$ a b c e  ;  1 2 3 4 $\rightarrow$  ?

2. 2.

   Grouping: a b c d $\rightarrow$ a b c e  ;  i i j j k k l l $\rightarrow$  ?

3. 3.

   Longer target: a b c d $\rightarrow$ a b c e  ;  i j k l m n o p $\rightarrow$  ?

4. 4.

   Reversed order: a b c d $\rightarrow$ a b c e  ;  l k j i $\rightarrow$  ?

5. 5.

   Interleaved distractor:
   a b c d $\rightarrow$ a b c e  ;  i x j x k x l x $\rightarrow$  ?

6. 6.

   Larger interval: a b c d $\rightarrow$ a b c e  ;  i k m o $\rightarrow$  ?

Finally, Webb et al. include a number of problems involving “real-world” concepts, such as,

a b c $\rightarrow$ a b d  ;  cold cool warm $\rightarrow$  ?

Webb et al. generated 100 problems for each problem type and presented these to GPT-3 (text-davinci-003). Webb et al. also tested 57 UCLA undergraduates on the same problems. The human participants exhibited a large variance in accuracy, but on average, Webb et al. found that GPT-3 outperformed the human participants on most problem types.

Due to the costs of human and computer experiments, we focus here on problems with simple targets (i.e., no numbers, grouping, etc. in the target string). Webb et al. called this the “zero generalization setting”; it is the setting in which both humans and GPT-3 performed best. Extending to problems with different generalization types is a topic for future work.

To create our dataset of counterfactual problems, we did the following. First, we generated permuted alphabets, in which we reorder $n$ letters, where $n$ can be 2, 5, 10, or 20. For each of the four values of $n$, we generated seven distinct alphabets with $n$ randomly chosen letters reordered. Then, for each of these alphabets, we created 10 different analogy problems for each of Webb et al.’s six transformation types. This results in $7\times 10\times 6=420$ analogy problems for each value of $n$. We added to this 420 analogy problems using the non-permuted ($n=0$) alphabet, spread evenly over the six transformation types. Figure 1 gives an example of a Fix Alphabetic Sequence problem using an alphabet with two letters (e and m) reordered.

As a final set of counterfactual problems, we generated two non-letter symbol alphabets and used them to create 10 problems each for the Successor and Predecessor problem types, for a total of 40 unique non-letter symbol problems. Figure 2 gives an example of a Predecessor problem using a symbol alphabet. Our dataset of counterfactual problems, along with code for generating them, is available at https://github.com/marthaflinderslewis/counterfactual_analogy.

In order to assess humans’ abilities on the original and counterfactual letter-string problems, we collected data from 136 participants on Prolific Academic.¹¹1https://www.prolific.com/academic-researchers Participants were screened to have English as a first language, to currently be living in the UK, the USA, Australia, New Zealand, Canada, or Ireland, and to have a 100% approval rate on Prolific.

Each participant was asked to solve 14 letter-string analogy problems: one problem from each of the six transformation types for each of two (randomly chosen) alphabets (chosen from alphabets with $n\in\{0,2,5,10,20\}$), as well as one Successor and one Predecessor problem for a randomly chosen symbol alphabet. Figures 1–2 show the format on which the problems appeared on the participants’ screens.

In addition to the 14 problems, participants were also given two attention-check questions at random points during the experiment, with a warning that if the attention checks were failed, then payment ($7 for the experiment) would be withheld. Figure 3 gives an example of an attention check. Two of the 136 participants’ submissions were rejected due to failed attention checks. As in ? (?), as part of the initial instructions participants were given a simple example problem to complete and then were shown the solution.

We evaluated the performance of three LLMs—GPT-3 (text-davinci-003), GPT-3.5 (gpt-3.5-turbo-0613), and GPT-4 (gpt-4-turbo-0613)—on the same problems given to humans. Following ? (?), all GPT experiments were done with temperature set to zero. GPT-3.5 and GPT-4 require a slightly different input to GPT-3. GPT-3 takes in a single prompt, whereas GPT-3.5 and GPT-4 take in a list of messages that define the role of the system, input from a ‘user’ role, and optionally some dialogue with simulated responses from the model given under the role ‘assistant’.

A crucial aspect of letter-string analogy problems is that they do not necessarily have a “correct” answer, although, as we mentioned above, humans generally agree on what are the “best” rules describing letter-string transformations in this domain. However, there are other rules that can be inferred from a given pair of letter strings. We therefore examined the “incorrect” answers of humans and of GPT-3 and 4 to ascertain whether the kinds of errors made are similar.

For both GPT-3 and GPT-4, we randomly selected five incorrect answers from each problem type and alphabet, giving a sample of approximately 160 incorrect responses per GPT model. This number can be lower if there were fewer than 5 incorrect responses for a problem type and alphabet. For humans, we selected 184 incorrect answers.

By manually examining these selections, we identified four broad categories of errors: 1) Alternate rule formation, where the answer given is consistent with an alternative rule. For example, if we have source transformation [a b c d] [a b c e] with target [i j k l], then according to the Successor rule the answer [i j k m] is correct. However, the answer [i j k e] is consistent with the rule “Replace the last letter with ‘e”’. 2) Incorrect rule use, in which the answer given is clearly related to the target letter string, and some kind of rule has been applied, but the rule is inconsistent with the example pair. For example, for [a b c d] [a b c e] with [i j k l], the response [i j k l m] is given. 3) Wrong, in which the answer given is inconsistent with the expected answer, but related to the target letter string. We could not discern any clear misunderstanding or alternate rule use. For example, for [a b c d] [a b c e] with [i j k l], the response [i j k q] is given. 4) Completely Wrong, in which the answer given is inconsistent with the expected answer, and unrelated to the target letter string. Again, we could not discern any clear misunderstanding or alternate rule use. For example, for [a b c d] [a b c e] with [i j k l], the response [z y x b] is given. Table 4 gives percentages for each error type for humans and for each model. We see that in humans a large percentage (38.59%) of errors stem from using alternate rules. This is also seen in GPT-4 to a lesser extent (22%), but much less in GPT-3 (5.81%). We also see a difference in the percentage of incorrect rules applied, with GPT 3 and 4 both having over 30% of errors in this category and humans having around 15% of errors in this category. GPT models also have a higher percentage in the Wrong category, and for each of the models this category is the largest across the errors they made. Humans have a larger percentage of errors in the Completely Wrong category than do GPT-3 and 4 however. Across these four broad categories GPT-3 and 4 make different patterns of errors than humans.

We can further look at the kinds of alternative rules that are used by humans and by GPT. One key type of alternative rule is where a ‘literal’ interpretation of a rule is applied, illustrated in Table 3. As well as literal rules, humans found alternative rules for the Fix Alphabet problem type: they would interpret the changed letter as being moved a certain number of steps in the alphabet, and would move an equivalent letter in the prompt the same way. Usually “equivalent” means position; sometimes it means the identity of the letter. We find that GPT-4 gives the same kind of literal responses that humans do, but does not use alternative rules other than literal responses. GPT-3 has a limited number of errors in this category, and almost all are literal responses to Remove Redundant. In summary, within the “Alternative Rule” category, the GPT models found literal rules in the same way humans did, but did not find more inventive alternative rules.

Breaking down the Incorrect Rule category, we see more differences between human and GPT behavior. Human responses in this category are mostly where one of the rules has been applied in an incorrect situation, for example Add Letter has been applied instead of Successor. GPT-3 errors include adding two letters instead of one; continuing the alphabet; reversing the target; shifting the target; using an unpermuted alphabet instead of the one given; and repeating the target. GPT-4 made these mistakes and also generated responses that were too long. Very few humans made any of these mistakes. Out of the incorrect responses, the types of response made by humans and GPT models are very different.

Our aim was to assess the performance of LLMs in “counterfactual” situations unlikely to resemble those seen in training data. We have shown that while humans are able to maintain a strong level of performance in letter-string analogy problems over unfamiliar alphabets, the performance of GPT models is not only weaker than humans on the Roman alphabet in its usual order, but that performance drops further when the alphabet is presented in an unfamiliar order or with non-letter symbols. This implies that the ability of GPT to solve this kind of analogy problem zero-shot, as claimed by ? (?), may be more due to the presence of similar kinds of sequence examples in the training data, rather than an ability to reason by abstract analogy when solving these problems.

We further see that the models GPT-3.5 and GPT-4 are no better than GPT-3 at solving these analogy problems, and in the case of GPT-3.5 are worse. GPT-3.5 and 4 have been trained to chat like a human, and this training objective may have reduced the ability to solve letter-string analogies.

We have shown that, contra ? (?), GPT models perform worse than humans, on average, in solving letter-string analogy tasks using the normal alphabet. Moreoever, when such tasks are presented with counterfactual alphabets, these models display drops in accuracy that are not seen in humans, and the kinds of mistakes that these models make are different from the kinds of mistakes that humans make. These results imply that GPT models are still lacking the kind of abstract reasoning needed for human-like fluid intelligence. In future work we hope to extend our investigation to both the letter-string generalizations and the other analogical reasoning domains studied by ? (?).

This work does not probe into how either humans or GPT form responses to these problems. Future work in this area could be to interrogate both humans and LLMs on their justifications for a particular answer. Another avenue for exploration is to investigate performance in a few-shot setting, as here, newer models may come into their own.