Can Stories Help LLMs Reason? Curating Information Space Through Narrative

In the first step, we use the LLM’s ability to explore and clarify the problem. Starting with a specialized prompt, the LLM breaks down the question into its core components, identifying relevant subtopics and areas. This detailed analysis is crucial for generating a coherent narrative that thoroughly addresses the question.

The second step involves generating detailed narratives based on the breakdown and clarification performed in Step 1 (question clarification). These narratives provide a structured context for the questions to enhance the LLM’s understanding, reasoning, and problem-solving abilities. Sadiri Javadi et al. (2024) discuss different narrative techniques required in conversational information seeking systems. We integrate the below subset of these techniques into our prompt and task LLMs to generate a narrative, based on the information from Step 1:

1. 1.

   Progressive Disclosure: Reveals information gradually, guiding the LLM step-by-step through the problem-solving process.

2. 2.

   Branching: Explores different paths or approaches to understanding the problem by providing multiple perspectives.

3. 3.

   Analogy: Uses comparisons to familiar concepts or situations to make abstract components more understandable.

4. 4.

   Analogical Reasoning: Facilitates understanding by reasoning through similarities between the problem and known situations.

5. 5.

   Metaphor: Simplifies complex ideas through metaphorical representation.

In the final step, the LLM uses the narrative generated in Step 2 to solve the original QA task. The structured and contextual understanding provided by the narrative supports LLM in accessing relevant aspects of the task.

We focus our evaluation on reasoning-intensive tasks spanning multiple domains, including physics, biology, and chemistry problem-solving. In particular, we utilize the GPQA (Diamond set) Rein et al. (2024) and JEEBench Arora et al. (2023). GPQA is a Graduate-level Problem-solving QA dataset which comprises expert-crafted multiple-choice questions. It contains 448 multiple-choice questions written by domain experts in biology, physics, and chemistry of high quality and difficulty. We use the Diamond subset, which contains 198 questions on which both expert annotators agree. JEEBench contains 515 challenging pre-engineering mathematics, physics and chemistry problems from the highly competitive IIT JEE-Advanced exam.

These problems span the different sciences and are extremely challenging, requiring in-depth reasoning and domain knowledge, making them well-suited for assessing our approach’s ability to understand complex tasks and contextualize salient information within the problem space.

To evaluate the performance of our approach across a wide range of Large Language Models, we experiment with the following LLM families:

1. 1.

   Meta: Llama 3 8B & Llama 3 70B

2. 2.

   Mistral: Mistral 7B & Mixtral 8x7B

3. 3.

   OpenAI: GPT-3.5-turbo & GPT 4

4. 4.

   Microsoft: Phi 3 Medium & Phi 3 Mini

These models were selected to cover a wide spectrum of capabilities, sizes and families, enabling a comprehensive evaluation of their strengths and limitations. More details on the implementation can be found in Appendix B.

We compared our proposed approach against several prompting techniques:

• Zero-shot Prompting: This method, similar to our approach (SoT), does not rely on labeled examples. Instead, LLMs are prompted to solve tasks based solely on their pre-trained knowledge without any context provided. This approach serves as a baseline, demonstrating the LLMs’ ability to solve problems without explicit guidance.

• Zero-shot CoT Wei et al. (2022): This technique prompts the LLM to explicitly reason through the steps required to arrive at an answer. By prompting the model to generate a chain of thought, this method aims to improve the model’s ability to solve complex problems by breaking them down into smaller, more manageable steps.

• Tree of Thoughts Yao et al. (2023): This method systematically explores multiple reasoning paths instead of a single linear progression. In ToT, a tree-structured solution to a problem is generated by breaking it down into sub-problems. This approach enables the model to consider a broader set of potential solutions by evaluating each branch for correctness before proceeding further.

• Graph of Thoughts Besta et al. (2024): This technique extends the Tree of Thoughts (ToT) approach by allowing for a more flexible and non-hierarchical representation of problem-solving steps. In this method, the reasoning steps are treated as nodes, and the connections between them are edges that represent logical relationships or dependencies. In our experiments, we adopt the same setup described in their original work.

• Analogical Reasoning Yasunaga et al. (2023): This approach leverages analogies to help the model draw parallels between known concepts and the task at hand. By providing analogical examples, the model is guided to understand and apply similar reasoning patterns to new problems. In our experiment, we allow the LLMs to self-generate three exemplars for each question (akin to the prompt described in their paper). This enables them to identify relevant examples and adapt their reasoning accordingly.

• Ours: Knowledge Identification: To measure the effectiveness of our proposed approach, namely utilizing narrative in solving tasks, we prompt LLMs to solve the task based solely on the generated conceptual knowledge from Step 1 (described in Section 3.1). This allows us to compare the model capability in solving tasks using only the identified relevant knowledge versus leveraging this knowledge to structure a coherent narrative.

• Ours: Story of Thought (SoT): This approach represents the core of our proposed method, where we leverage the generated narratives from Step 2 (described in Section 3.2) to solve the given tasks.

We present the results of our experiments on GPQA (Diamond) are presented in Table 1. For this task, SoT is the best method to use with six of eight models. The open-source Llama 3 70B model records the highest accuracy using the SoT method, achieving a score of 51.01%. This is the highest accuracy observed among all models and methods tested in the study. Furthermore, the GPT-4 model shows the most notable improvement in accuracy when the SoT method is employed, compared to its zero-shot baseline. Specifically, the accuracy for GPT-4 increased from 34.7% under zero-shot conditions to 48.98% with SoT (i.e., an absolute increase of 14.28%, or a relative increase of 41% respectively).¹¹1We also find that Llama 3 70B with SoT outperforms zero-shot o1-preview which uses CoT style reasoning internally. https://openai.com/index/learning-to-reason-with-llms/.

Interestingly, all reasoning strategies lead to an accuracy drop for the comparably smaller Phi-3 Mini model, and all CoT strategies except Analogical Reasoning also lead to the accuracy drop of the Phi-3 Medium model compared to its zero-shot baseline. We hypothesize that this is due to the low quality of the generated explanations (whether CoT steps or SoT narrative), as further studied in §6.1.

Figure 2 presents the performance of different models when using SoT across the different problem domains in GPQA. We note that, on average, models improve the most on Biology problems when using SoT. We hypothesize that this is because it is easier to simplify information for graduate level biology problems that can be used by models to come up with a solution.

Table 2 presents detailed experimental results on JEEBench. Our proposed Story of Thought (SoT) method consistently improves the performance of seven out of the eight LLMs. Using SoT, Llama 3 70B performance surpasses even the GPT models. It obtains the highest scores in all subjects and question types (Except Single-Correct), with an overall aggregate score of 0.453. This is a significant improvement on the previous SOTA, which was a strong GPT4 model used with both CoT and Self-Consistency.

Across models, the results highlight the effectiveness of Story of Thought (SoT) in enhancing model performance on complex, multi-disciplinary benchmarks like JEEBench, setting new SOTA results in several categories. The improvements are particularly notable in the subject categories and question types where the other methods struggle.

In Figure 2, we present subject-wise performance of different models on JEEBench. On average, model performance is highest on Chemistry problems when using SoT. This is in contrast to findings on GPQA and could occur due to the difference in degree of difficulty of problems in the two datasets (graduate level vs high school level). Regardless, improvements on Biology problems are not far behind those for Chemistry.

The choice of narrator model (i.e., the model that generates narratives) can impact the problem-solving resuls. In the following experiments, we apply the narratives generated by other large and small open-source LLMs to the Phi-3 Mini and Phi-3 Medium models. The results of these experiments are presented in Table 3.

We observe that the narratives generated by most models consistently improve the accuracy of both Microsoft models compared to the baseline (i.e., when both models use their own generated narratives in Step 2 to solve the tasks, shown in Table 1). The absolute improvements range from 1.0% to 2.8%, with the Llama 3 70B model generating the most effective narratives. A slight decrease in accuracy is observed with the mixture-of-experts Mixtral 8x7B narratives for the Phi-3 Medium model, highlighting the need for careful selection and evaluation of narrator models to ensure compatibility and optimal performance. Larger models generate narratives that break down problems to make them more easily solvable. Unsurprisingly, there is larger room for improving the problem solving abilities of smaller models.

To measure the impact of each of the narrative techniques we jointly prompted on the performance of open-source Meta models, we ablate the designed prompt in Step 2 (of Section 3.2) to apply each of the techniques separately. The results in Table 5 indicate that employing any single narrative technique at a time is notably less effective at boosting QA accuracy than utilizing a combination of these simultaneously.

For both models (Llama 3 8B and 70B), the decrease in accuracy is comparably smaller (-3.0% to -5.6%) when using only the analogical components of the narrative (Analogy and Analogical Reasoning) than when using only the structural instructions (Progressive Disclosure or Branching) which leads to larger (-6.0% to -9.1%) accuracy loss. However, reasoning alone does not perform on par with the full narrative generation listing all the techniques. Prompting for Metaphor usage only leads to a larger accuracy loss in the 70B model (-6.6%) compared to the smaller one (-2.0%). This makes it difficult to determine both how the narrative techniques relate to each other and whether the model truly comprehends the prompts it receives. We disentangle and study the two going forward.

To gain deeper insights into the generated narratives, we designed a prompt (shown below) that utilizes our best-performing model (Llama 3 70B) to annotate the number of occurrences of each narrative technique for each generated narrative by all models used in our experiments. We can better interpret how the model executed the narrative technique prompt, by asking it to label if and where the mentioned techniques are used in the text generated. Less frequently labeled techniques might be the ones where LLM doesn’t have a clear understanding of what it is asked to do. A proportion of the techniques and their correlation can provide us with a better picture of LLM’s interpretation of the instruction as well. We detail the instruction given to the LLM in Appendix C.

We aim to uncover patterns and variations in the use of narrative techniques across different LLMs. Table 4 compares the total number of occurrences for each narrative technique across various LLMs.

Table 6 compares the similarity of Story of Thought (SoT) and Chain of Thought (CoT) reasoning outputs to human explanations for different language models on the GPQA (Diamond set) dataset, using BertScore, ROUGE-L, and BLEU.

The differences between ROUGE-L values are insignificant and do not display any clear trends. However, according to BLEU scores, using SoT results in explanations closer to humans and the differences are more pronounced.

As per BertScore (an embedding-based similarity metric), Llama 3 models (8B and 70B) explanations are more similar to human ones when using SoT reasoning across all three metrics. However, Mistral models (7B and 8x7B), GPT-4, and Phi-3 Mini generate explanations more similar to human explanations when using CoT reasoning across all metrics. The semantic similarity of narratives generated by Llama 3 70B to human explanations combined with their effect of improving smaller models indicates that these narratives present information about the problems in a simplified manner.