Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems

Current conversational systems encounter numerous problems. Firstly, they depend heavily on user inputs, with the agent merely responding to queries (quac, 1, 2). This method is fundamentally flawed as it fails to actively prompt users to clarify ambiguous or poorly defined questions. As a result, the systems frequently return incorrect responses or resort to apologetic statements when dealing with such inquiries. Secondly, even when the system does provide a correct answer, the user must again formulate an effective follow-up question. This task can be challenging, as users often find it difficult to understand the full scope of the system’s capabilities (tod-cq-clarit, 3). This cycle of inadequate responses forces users to constantly refine their queries, often resulting in more apologies than useful information.

Through this paper, we want to address this change and fill the gap by proposing a suggestion question generator. The generator is designed to generate suggestion questions that the agent can answer, guided by the user’s initial query. Our framework encompasses several key steps. First, it involves identifying pertinent passages (dynamically retrieved contexts) based on the user’s initial query, as the answer to the query hinges on these passages. Secondly, we take the help of relevant triple of (questions, answer, suggestion questions) (dynamic few-shot). Finally, we generate suggestion questions by constituting a dynamic context prompt consisting of the dynamically retrieved contexts, and dynamic few-shot examples. By providing suggestion questions, users are alleviated from the task of question formulation, ensuring a smooth conversational flow (tod-cq-clarit, 3).

Conversational agents can adapt Dynamic Contexts for a variety of tasks, eliminating the need for extensive model training. To generate suggestion questions in the absence of a publicly available dataset, we use in-context learning. In-context learning (few-shot, 4) has been recognized for achieving promising outcomes with minimal training samples.

Our primary goal is to develop a suggestion question generator. We aim to tackle cases where a RAG-based system cannot interpret a user’s query accurately, despite the presence of relevant information in the dataset. Due to this, the system is unable to understand the query’s intent leading to unsuccessful responses. Hence, we generate three suggested questions that the agent is capable of responding based on the user’s initial query.

In Summary, in this paper, we:

• •

  Propose to develop a suggestion question generator to address the gap in conversational systems where users frequently lack awareness of the system’s limitations.

• •

  Formulate a suggestion question generator in low-resource settings which can be applied across various datasets without the need of fine-tuning.

• •

  Propose a dynamic context prompt that constitutes dynamic few-shot examples and dynamic retrieved contexts to generate suggestion questions.

To the best of our knowledge, there is no existing model or literature specifically addressing suggestion question generation. However, this concept is closely related to question generation (sparta, 5) and clarification question generation (tod-cq-clarit, 3, 6).

The concept of clarification questions was formally introduced in (cq1, 7), and since then, research into generating these questions has spanned a wide range of scenarios, including open-domain systems (AmbigQA) (ambigqa, 8), knowledge bases (CLAQUA) (claqua, 6), closed-book systems (CLAM) (clam, 9), information-seeking (ISEEQ) (iseeq, 10), task-oriented dialog systems (CLARIT) (tod-cq-clarit, 3), and conversational search (mixed-initiative, 11). Rahmani et al. (rahmani-etal-2023-survey, 12) surveyed the various methodologies, datasets, and different evaluation strategies used for clarification questions.

The distinct characteristics of RAGs (dpr, 13) make them essential for effective information retrieval and is widely used for generating question-answering tasks (quac, 1) and clarification questions (iseeq, 10, 11).

Advancements in large language models (LLMs) have led researchers to investigate the use of LLMs and prompting techniques for both question generation (sparta, 5) and clarification questions (deng2023prompting, 14).

Dynamic Contexts, the Suggestion Question Generator (Figure 1), is designed to enhance the interaction between users and conversational systems by addressing the gap in users’ understanding of these systems’ capabilities. By aiming to reduce the frequency of apologetic responses (tod-cq-clarit, 3), it provides more informed system suggestions, thereby improving the overall user experience.

Dynamic Contexts generate suggestion questions that are answerable by the agent. This is achieved through two principal components of Dynamic Contexts: dynamic few-shot examples and dynamically retrieved contexts. Dynamic few-shot examples begin with the selection of contextually relevant triplets from the dataset, each consisting of a question, its answer, and the suggestion question (QAS). Dynamic few-shot diverges from traditional few-shot prompting by dynamically choosing each example based on the user’s query, rather than relying on a static set of examples. This dynamic selection is crucial for accommodating the varied formats and structures that different questions may necessitate.

To ensure that the questions generated by the Suggestion Question Generator remain pertinent and answerable, dynamic contexts consist of dynamically retrieved contexts. Dynamically retrieved contexts consist of retrieving 4 dynamic contexts based on the initial user query, similar to RAG-based systems from which suggestion questions should be generated. We use OpenAI embeddings to embed the examples and dynamic contexts and cosine similarity to select dynamic examples and contexts. The complete prompt structure utilized for the approach is detailed in the appendix B.

Due to the absence of a publicly accessible dataset specifically designed for Suggestion Question Generator tasks, we employed 228 blog posts from Pamper’s Baby Sleep Coach as a practical demonstration of the approach. More details of the dataset are present in Appendix A. To test the Dynamic Contexts approach, LLMs- ChatGPT, GPT-4, and Claude2 were used. The evaluation of the suggestion question generator involves a variety of assessment techniques to ensure a thorough analysis of the performance.

Manual Evaluation To assess the effectiveness of the Dynamic Contexts approach in the Suggestion Question Generator, we conducted a manual evaluation using a set of 48 QAS (question, answer, and suggestion question) pairs. During manual assessment, we primarily assessed the correctness, relevance, and soundness of the generated questions. Dynamic Contexts demonstrated its capability by successfully generating relevant suggestion questions (as shown in Table 1). ChatGPT, Claude-2, and GPT-4 each produced 44, 44, and 46 correct samples, respectively. ChatGPT and GPT-4 errors were related to the baby’s age, whereas Claude-2 did not make any age-related errors. However, Claude-2’s primary shortcomings were its failure to formulate questions that aligned with the provided contexts.

Comparative Analysis A comparative analysis was performed to evaluate the improvement of Dynamic Contexts over different prompting strategies, including Zero-Shot, Few-Shot, and Dynamic Few-Shot approaches. This comparison, detailed in Table 1, highlights the distinctions among the methods. The Zero-Shot method employed no examples in its prompts, while the Few-Shot method used a consistent set of three examples for each query. The Dynamic Few-Shot approach, in contrast, dynamically selected its examples. Distinct from other methods, Dynamic Contexts utilizes Retrieval-Augmented Generation (RAG) to dynamically select contexts for generating suggestion questions. This approach marks a significant shift from the static contexts used in the Zero-Shot, Few-Shot, and Dynamic Few-Shot approaches.

The findings indicate that Claude-2 generated only 30 correct questions (compared to 35 for ChatGPT and 43 for GPT-4) in the zero-shot approach out of the 48 samples. While we supplied the answers, the models struggled to generate questions that could be answered from the provided context. Claude-2 notably struggled, often unable to utilize the provided context effectively, and sometimes misconstrued its role, imagining itself as a baby.

The few-shot approach, which involved providing three example questions to the models, showed an overall improvement in performance. However, the questions generated tended to replicate the structure of the examples provided too closely. The dynamic few-shot approach, aimed at introducing a variety of question types and structures, significantly improved the quality of the questions generated but still lagged behind the dynamic contexts approach.

Preference Benchmarking Further analysis was conducted through Preference Benchmarking Evaluation, where GPT-4 and Claude-2 evaluated the suggestion questions generated by the framework. In this blind test, the models, GPT-4 and Claude-2 acted as judges and were prompted to assess the quality of the questions without knowing from which model it was generated. The results showed Claude2 having no preference at all between the models, while GPT-4 favored Claude-2’s output 57% of the time over its own. Human evaluation was also employed to evaluate which output was better which showed a preference for GPT-4’s output in 43% of cases, Claude-2 in 33% of cases, and showing no preference in 24% of cases. This evaluation suggests a close alignment between the judgments of LLMs and human evaluators. Detailed results can be found in Table 3 of the appendix D.

Ablation Study To determine the optimal order of the final prompt, we conducted an ablation study, altering the arrangement of example contexts and queries. Initially, we presumed that presenting the query before the dynamic contexts would enhance comprehension, allowing the model to suggest questions based on the given information. However, upon changing the order, we did not observe any discernible difference in the nature of the generated suggestion questions as seen from the Table 2 in Appendix C.

In this paper, we explore suggestion questions to bridge the gap in users’ comprehension of a conversational system’s capabilities. We present the development of a suggestion question generator that utilizes dynamic context prompting within a RAG-based conversational system. In the future, we plan to personalize the suggestion questions based on the user history.