Dr3: Ask Large Language Models Not to Give Off-Topic Answers in Open Domain Multi-Hop Question Answering

Vanilla ReAct. To solve ODMHQA using LLMs, Yao et al. (2022) proposed ReAct that uses interleaved steps of reasoning and acting. Specifically, ReAct verbally maintains high-level plans for acting (reason to act), while interacting with external environments such as Wikipedia to incorporate additional information into reasoning (act to reason). Formally, ReAct interacts with the environment in $N$ steps before concluding the final answer. At step $i$, based on the previous step’s observation, $o_{i-1}\in\mathcal{O}$ (where $\mathcal{O}$ is the set of all possible observations from the environment), ReAct develops a thought, $\tau_{i}\in\mathcal{T}$ (where $\mathcal{T}$ is the set of all possible LLM-generated thoughts), by reasoning over the available information, and planning the next step hence deliver an action, $a_{i}\in\mathcal{A}$ (where $\mathcal{A}$ is the set of optional actions like search[query] to retrieve passages or finish[answer] to conclude the question), leading to a new observation, $o_{i}\in\mathcal{O}$, from the environment. Specifically, the action is selected by ReAct’s policy, $\pi:H_{i}\rightarrow a_{i}$, where the context $H_{i}\equiv(o_{0},\tau_{1},a_{1},o_{1},\ldots,\tau_{i-1},a_{i-1},o_{i-1},\tau_{i})$ represents the interaction history between ReAct and the environment.

Modified ReAct. It is noteworthy that ReAct’s reasoning and planning steps are intertwined within its thought component, denoted as $\tau_{i}$ at step $i$. This entanglement presents challenges in identifying potential issues when solving ODMHQA, which hinders further optimization. Accordingly, we develop ReAct+ (shown in Figure 1), an equivalent variant of ReAct better suited for multi-hop reasoning chains in ODMHQA. ReAct+ decomposes a complex question into a set of clear-cut Sub-Questions, inspired by Press et al. (2022). In ReAct+, the Sub-Question at step $i$ is denoted by $S_{i}\equiv(t_{i},a_{i},o_{i},c_{i})$, where $t_{i}$ signifies the planned task, $a_{i}$ represents the selected action, $o_{i}$ represents the observation from the environment, and $c_{i}$ symbolizes the intermediate conclusion. As Figure 1 illustrates, ReAct’s $\tau_{i}$ corresponds to ReAct+’s $(c_{i-1},t_{i})$ for intermediate Sub-Questions. Regarding the first and last Sub-Questions, $\tau_{1}$ corresponds to $(D,t_{1})$ while $\tau_{N+1}$ corresponds to $(c_{N},C)$, where $D$ denotes Decomposition and $C$ represents Composition. By implementing these modifications, we observe that ReAct+ marginally outperforms ReAct on ODMHQA (see Section 5.1), indicating that decomposing a complex question into Sub-Questions does not impair performance, aligning with the findings of Mishra et al. (2022).

We observe that LLMs encounter the problem of generating off-topic answers when attempting to solve ODMHQA. To elaborate, off-topic answers refer to cases where the generated answers are not relevant to the original questions, which is similar to the concepts of off-topic responses in dialogue systems (Malinin et al., 2016) and off-topic essays in high-stakes tests (Louis and Higgins, 2010). Notably, off-topic answers are necessarily incorrect answers. We show two cases of off-topic answers in the 3rd and 4th examples in Table 1. In the 3rd example, on-topic answers should be either “The Secret Life of Pets 2” or “Love Me Deadly”, as indicated in the original question. However, the generated answer “Lover 3” does not match either of these expected on-topic answers. In the 4th example, an on-topic answer should specify the language(s), while the generated answer “Edinburgh” is a city name instead of specifying a language.

To determine the significance of the off-topic issue, we randomly sample 500 cases from the HotpotQA (Yang et al., 2018) and 2WikiMultiHopQA (Ho et al., 2020) datasets and manually label whether the LLM-generated answers are off-topic. As shown in Figure 2, off-topic answers are a prevalent issue observed across different datasets, LLMs, prompts, and methods. We notice that approximately 1/3 of the incorrect answers are identified as off-topic answers, which directly impairs the user experience of employing LLMs for solving ODMHQA.

To further investigate the issue of off-topic answers, we analyze 112 off-topic answers out of 500 cases randomly sampled from the HotpotQA dataset (Yang et al., 2018) using ReAct+. We thoroughly review the full solving history and then locate and classify the causes behind these off-topic answers. For these off-topic cases, the distribution of causes identified through this analysis is shown in Figure 3. Here are the key findings from our analysis:

• •

  The Decomposition step accounts for 31% of the off-topic answers, where LLMs misunderstand the original question or lose the keyword during the process of decomposing the question.

• •

  The Sub-Question steps contribute to the largest portion at 62% of the off-topic answers. We further classify these as planning errors, passage errors, and reasoning errors. Notably, passage errors (Yao et al., 2022) cover 41% of the cases, occurring when the relevant passage was not successfully retrieved or does not exist in the external database. Reasoning errors stem from two main sources: comprehension failures, where the model fails to generate the correct intermediate answer by faithfully following the reasoning chain, known as unfaithful reasoning (Lyu et al., 2023); and selection failures, where the model mistakenly chooses another answer candidate instead of the required one.

• •

  The Composition step accounts for a relatively small portion at 7% of the off-topic answers. In these cases, we observe that the LLM becomes trapped in hallucination (Zhang et al., 2023), missing the key information from the original question or overlooking key evidence from intermediate conclusions, ultimately causing off-topic answers.

In the Discriminator, we leverage the intrinsic capabilities of LLMs to determine whether the generated answer is on-topic. This is accomplished by feeding both the original question and the LLM-generated answer into the Discriminator, which yields a binary judgment of either YES or NO, indicating whether the generated answer is on-topic or off-topic. Specifically, we design an instruction that prompts the LLM to first conceptualize candidate answers and subsequently assess whether the generated answer falls within the range of available options. If the generated answer is among the candidate answers, it should be considered on-topic; otherwise, it should be considered off-topic.

When the generated answer, $\texttt{Ans}_{\texttt{old}}$, is identified as off-topic by the Discriminator, the Corrector revises the reasoning chain in three stages. These three stages follow the order, Re-Compose$\rightarrow$Re-Solve$\rightarrow$Re-Decompose, which mirrors ReAct+’s original order of reasoning, Composition$\leftarrow$Sub-Question$\leftarrow$Decomposi- tion. At each revision stage, the Corrector generates a new answer, $\texttt{Ans}_{\texttt{new}}$, which is evaluated by the Discriminator for on-topic or off-topic. This iterative process continues either until the Discriminator approves $\texttt{Ans}_{\texttt{new}}$ as an on-topic answer, or until the entire reasoning chain has been exhausted. We introduce the Corrector’s three revision stages as follows.

We present the results obtained from two popular datasets: HotpotQA and 2WikiMultiHopQA. HotpotQA (Yang et al., 2018) is a 2-hop QA dataset built from Wikipedia passages where reasoning chains are formed between passage pairs. We use the same data following Yao et al. (2022) as our test dataset, which randomly samples 500 cases from Dev split. For each case, we only provide the question without the paired passages according to the open domain setting. 2WikiMultiHopQA (Ho et al., 2020) constructs multi-hop QA cases by combining Wikipedia articles and Wikidata knowledge. Compared to HotpotQA, 2WikiMultiHopQA is more challenging because it includes four question types: comparison, inference, compositional, and bridge-comparison. Similar to Yao et al. (2022) and Khattab et al. (2022), we randomly sample 500 question-answer pairs from the Dev split as the testing dataset. Following previous work in QA (Yang et al., 2018; Xu et al., 2023), we evaluate using the metrics of token F1 score, exact match (EM), and cover exact match (Cover EM).

The baseline methods used for comparison are briefly described as follows.

• •

  Zero-Shot (Kojima et al., 2022): The zero-shot approach requires the LLM to output the answer to the question without any relevant example.

• •

  Few-Shot (Brown et al., 2020): The Few-Shot approach prompts the LLM with a few relevant question–answer examples, demonstrating the procedure for solving this type of task.

• •

  CoT (Wei et al., 2022): The Chain-of-Thought (CoT) approach facilitates the LLM to generate coherent intermediate reasoning steps that mimic a step-by-step thought process by prompting “Let’s think step-by-step”.

• •

  ReAct (Yao et al., 2022): The Reasoning-Acting (ReAct) approach enhances reasoning-only LLMs by incorporating acting capabilities through the use of external tools, which defines the reasoning pattern as a sequence of interleaved Thought-Action-Observation steps (see Section 2.1).

• •

  ReAct+ (Ours): The ReAct+ approach modifies the reasoning pattern of ReAct to fit multi-hop question answering, which involves interleaved Task-Action-Observation-Conclusion steps (see Section 2.1).

For the Re-Solve step, we set the maximum number of replaced passages $T_{D}$ to 3 for each Sub-Question. We use the off-the-shelf ColBERTv2 (Santhanam et al., 2022) following Khattab et al. (2022) as the IR system, which encodes the Wikipedia corpus by passage. For the Re-Decompose step, we use 6 question-answer examples in prompts. Within each step, we set the maximum number of continuous Sub-Question to 7, following Yao et al. (2022). All methods are implemented using text-davinci-002 as the LLM, in line with Yao et al. (2022) in September 2023. The detailed prompts can be found in Appendix A, B, and C.

In this subsection, we evaluate the overall performance of our Dr3 mechanism on the HotpotQA and 2WikiMultiHopQA datasets, compared to five baseline methods described in Section 4.2. As shown in Table 2, our Dr3 approach outperforms existing methods across evaluation metrics of EM, Cover EM, and F1 score. Specifically, compared to the best baseline ReAct+, our Dr3 approach exhibits significant improvements: on HotpotQA, we observe absolute enhancements of 2.80% in EM, 3.40% in Cover EM, and 4.32% in F1 score. Meanwhile, on 2WikiMultiHopQA, we achieve even larger gains of 3.20% in EM, 3.00% in Cover EM, and 2.97% in F1 score. These compelling results clearly demonstrate the effectiveness of our Dr3 mechanism for improve the performance on ODMHQA.

In this subsection, we examine the performance of the Discriminator in detecting off-topic answers. Discriminator is comparable to human judgment in detecting off-topic answers. As shown in Table 3, the Discriminator achieves an accuracy of 92.77% on the HotpotQA dataset, which manifests its remarkable capability in identifying off-topic answers, closely resembling human judgment. Moreover, the high F1 score of 84.82% further emphasizes the Discriminator’s capability to recall a majority of the off-topic answers while maintaining a high precision.

Discriminator identifies 1/3 off-topic answers and rarely misjudges correct answers. Figure 5 illustrates the confusion matrix of the Discriminator in judging QA correctness on the HotpotQA dataset. The Discriminator has a sensitivity of 97.4%($\frac{151}{151+4}$), indicating that the Discriminator rarely misjudges correct answers as incorrect. Meanwhile, it has a specificity of 33.2%($\frac{114}{114+229}$), signifying that it successfully identifies a substantial portion of incorrect answers. Therefore, the benefit of identifying incorrect answers outweighs the misjudgments.

In this subsection, we conduct ablation studies on three individual Corrector modules to investigate their effectiveness in improving question answering and alleviating the off-topic issue. The results are shown in Table 4. Re-Solve is the most effective module, exhibiting a 1.4% EM improvement. This significant EM improvement can be attributed to Re-Solve identifying passage errors as the primary cause of off-topic answers (see Section 2.3). Re-Compose (+1.0% EM) appears to be more effective than Re-Decompose (+0.4% EM). This is unexpected that Re-Compose is so effective given Composition errors comprise the smallest ratio (7%) as shown in Section 2.3. We conjecture this occurs because Re-Compose not only fixes Composition errors but also encourages the LLM to re-examine the full solving history. By thoroughly reassessing the question and evidence, Re-Compose can implicitly fix reading comprehension and Decomposition errors as well. To demonstrate this phenomenon, we present 2 cases in Appendix D.

In this subsection, we discuss the relationship between off-topic answers and the number of tasks in Sub-Questions. As the line chart in Figure 6 illustrates, the off-topic ratio increases monotonically as the number of tasks rises. Ideally, the reasoning chain should contain exactly 2 Sub-Questions, since the questions from the HotpotQA dataset are designed to be 2-hop. Accordingly, depending on whether the number of tasks is less than, equal to, or greater than 2, we organize the same data into three groups in the bar chart of Figure 6. For the 1-hop reasoning chains, in which off-topic answers account for 21.73% of cases, we notice that ReAct+ tends to conclude the answer prematurely if it is on-topic. For reasoning chains with more than 2 hops, where off-topic answers occur 44.44% of the time, we observe that ReAct+ becomes trapped in self-correction cycles. This unavoidably adds noise to the context, causing the original question to be forgotten and resulting in off-topic answers.

In this subsection, we examine the relationship between off-topic answers and the four question types on the 2WikiMultiHopQA dataset. The results are shown in Figure 7. The Comparison type has the second lowest off-topic ratio, as the answer is already provided within the original question. For example, in a question like “Who is older, A or B?”, the answer can only be either A or B, both options being clearly stated in the question itself. The Bridge-Comparison type has the highest off-topic ratio. For instance, in a question like “Which film has the director born first, A or B?”, ReAct+ is prone to answering the name of the director rather than the name of the film, even though the correct answer must be either A or B. We hypothesize that the Bridge-Comparison type requires a minimum of 4 tasks to be properly solved, as extended reasoning causes the original question to be forgotten. The Inference type exhibits the lowest off-topic ratio but the highest error ratio. We observe that ReAct+ tends to prematurely terminate the reasoning chain when an intermediate answer is on-topic. For example, in a question like “Who is A’s grandfather?”, where the first retrieved passage contains A’s father B, the corresponding conclusion might erroneously regard B as A’s grandfather.