Planning Like Human: A Dual-process Framework for Dialogue Planning

We propose utilizing a tunable pre-trained language model, e.g., RoBERTa Liu et al. (2019), as the dialogue policy planner to control the dialogue process. Different from previous methods Deng et al. (2023b), we involve not only a policy network but also a Q-network. Our design rationale is based on two points: (1) The CIMA training set comprises only dialogue snippets rather than complete dialogue histories. It is inadequate to only learn a policy network in our proposed Offline RL-based pretraining method. An action-value function, i.e., Q-Network, is required to aid in training the policy network. (2) We utilize the LLM as the reward function, that is we view the critic LLM as part of the environment. Modeling the environment is a typical method to reduce interaction while maintaining performance Luo et al. (2022).

By connecting two different MLP layers, the policy LM planner comprises a policy network $\pi_{\theta}(a|s)$ for action prediction and a Q-network $Q_{\beta}(a|s)$ for state evaluation. At each turn, the subsequent strategy $a_{t}$ is predicted by inputting the dialogue state $s_{t}$ into the policy network.

Following GDPZero Yu et al. (2023), we utilize MCTS Weber (2010); Liebana et al. (2015) for simulating subsequent strategic deliberation, a process typically involving four stages: Selection, Expansion, Evaluation, and Backpropagation. After multiple simulations, the conversational strategy for the next turn of response is determined based on the action most frequently applied during these processes. For further details about MCTS in dialogue planning, please refer to Appendix D.

Different from GDPZero, we apply policy LM to produce prior knowledge for MCTS. To establish the prior action distribution for each Selection step, GDPZero computes action probabilities by repeatedly prompting an LLM. Conversely, we utilize the domain-specific trained policy LM to generate the prior probability. This approach not only enables the application of learned domain-specific knowledge for improved initialization but also diminishes the frequency of calls to the LLM, thus enhancing efficiency and reducing costs. We verify this operation in the following experiments.

Our objective is to synergistically utilize both the policy LM planner and MCTS planner to establish an adaptive dual-process system. Similar to human cognitive processes during proactive dialogue, individuals can rapidly respond to appropriate strategies when encountering familiar conversational states. Conversely, when encountering unfamiliar states, it becomes necessary to select the most suitable strategy by simulating potential reactions in subsequent dialogue turns. In our framework, we give priority to utilizing a policy LM for action selection. If the policy LM detects inadequate confidence regarding the current state, we switch to employing MCTS for action planning.

To reduce reliance on training, we propose a non-parameterized control gate mechanism to control switching. For the action distribution $\pi_{\theta}(a_{t}|s_{t})$ predicted by policy LM, we assess the uncertainty by the probability difference for the top-2 values: $\delta(\pi_{\theta}(a_{t}|s_{t}))=top(1)-top(2)$, where $top(i)$ means the $i$-th largest value in $\pi_{\theta}(a_{t}|s_{t})$. If this difference surpasses a threshold $\eta$, it indicates that the policy LM has high confidence in the current decision-making, prompting the utilization of the policy LM for action selection. Conversely, small differences imply that the policy LM is uncertain, prompting the utilization of MCTS for action selection. By setting appropriate $\eta$, we can roughly control the ratio of MCTS used. For details on how to determine $\eta$, please see Appendix B. Of course, other uncertainty measures like entropy can also be applied in our framework.

Our two-stage training scheme for the Policy LM, depicted in Figure LABEL:b, begins with offline RL-based Pretraining to ensure effective initialization, aiming to reduce interaction time during subsequent online learning. Unlike direct supervised learning, which can introduce biases through suboptimal or noisy data, offline RL Kumar et al. (2020) refines pretraining by using soft rewards to discern valuable strategies from the dataset, avoiding the pitfalls of hard labels. In detail, our approach initially assigns scores to each dialogue turn in the training set using the critic LLM, which serves as rewards for annotated dialogue strategies. As a result, we reconstruct an MDP corpus comprising complete states, actions, and rewards. Utilizing this corpus, we pretrain the policy LM, which consists of the policy network $\pi_{\theta}(a|s)$ and the Q-network $Q_{\beta}(s,a)$.

Specifically, the pretraining details for ESConv Liu et al. (2021) and CIMA Stasaski et al. (2020) datasets differ due to the absence of complete dialogue trajectories on CIMA. We outline the optimization process for ESConv here, while details for CIMA are provided in Appendix A. For the policy network, we optimize it by:

$$\mathcal{L}_{pre,\theta}=-\sum_{t=1}^{T}\hat{Q}(s_{t},a_{t})\log\pi_{\theta}(a_{t}|s_{t}),$$

(2)

where $\hat{Q}(s_{t},a_{t})=\sum_{t=1}^{T}\gamma^{t}R(a_{t}|s_{t})$ represents cumulative rewards, $\gamma$ means the discount factor, and $R(a_{t}|s_{t})$ is the received reward by selecting action $a_{t}$ upon the state $s_{t}$. We use the same strategy as PPDPP, which generates 10 evaluations on the current state, maps each evaluation into a pre-defined score, and finally computes the mean value as the reward Deng et al. (2023b). For ESConv, we map “feel worse”, “feel the same”, “feel better”, “solved” into -1.0, -0.5, 0.1, 1.0. For the Q-network $Q_{\beta}(s,a)$, we optimize it to approximate $\hat{Q}(s,a)$:

$\displaystyle\mathcal{L}_{pre,\beta}=\sum_{t=1}^{T}\text{MSE}(Q_{\beta}(s_{t},a_{t}),\hat{Q}(s_{t},a_{t})).$

(3)

Finally, the overall optimizing loss for the pretraining stage is:

$$\mathcal{L}_{pre}=\mathcal{L}_{pre,\theta}+\lambda_{1}*\mathcal{L}_{pre,\beta},$$

(4)

where $\lambda_{1}$ is a hyperparameter to control loss weight. By doing this, we expect to learn a better initialization compared to direct supervised learning.

Additional interaction with the environment is necessary since static training sets cannot cover the entire state-action space. In interactive online learning, we initiate two LLMs to simulate self-play dialogues between the user and the assistant. Given the current state $s_{t}$, rather than directly utilizing the policy agent to predict the next action, we utilize MCTS for action prediction. The predicted action is then mapped to a pre-defined natural language instruction, $\mathcal{M}_{a}(a_{t})$. Subsequently, the dialogue history $s_{t}$ along with $\mathcal{M}_{a}(a_{t})$ will trigger the LLM to generate the appropriate system response, then prompt the LLM to generate the corresponding user response. Following this, the dialogue process transitions to a new state $s_{t+1}$. We employ an LLM as a critic to calculate the action reward $r_{t}$. The collected transition records $\{s_{t},a_{t},s_{t+1},r_{t}\}$ are used to train the policy model.

We optimize the policy LM using the Actor-Critic algorithm instead of the REINFORCE algorithm Sutton et al. (1999). The optimization loss for the Q-network is as follows:

	$\displaystyle\mathcal{L}_{sp,\beta}$	$\displaystyle=\sum_{t=1}^{T}[Q^{*}(s_{t},a_{t})-Q_{\beta}(s_{t},a_{t})]^{2},$
	$\displaystyle Q^{*}(s_{t},a_{t})$	$\displaystyle=R(a_{t}|s_{t})+\gamma*\max_{a^{\prime}}Q_{\beta}(s_{t+1},a^{\prime}),$

and for the policy network as:

	$\displaystyle\mathcal{L}_{sp,\theta}=\sum_{t=1}^{T}[$	$\displaystyle(Q_{\beta}(s_{t},a_{t})-\hat{Q}(s_{t},a_{t}))$
		$\displaystyle*\log\pi_{\theta}(a_{t}|s_{t})].$

where $Q_{\beta}$ is the Q-network to calculate state-action values. $\hat{Q}(s_{t},a_{t})$ is cumulative rewards defined before. Finally, the overall optimizing loss for the self-play training phase is:

$$\mathcal{L}_{sp}=\mathcal{L}_{sp,\theta}+\lambda_{2}*\mathcal{L}_{sp,\beta},$$

(5)

where $\lambda_{2}$ is also a loss weight.

For CraigslistBargain, firstly, compared to the previous state-of-the-art method, DPDP based on System 1 (Policy LM) significantly improves AT (5.57$\rightarrow$ 5.03), SR (0.6649$\rightarrow$ 0.7447), and SL (0.3376$\rightarrow$ 0.4108), demonstrating that our two-stage training method not only enhances deal success rates but also increases benefits. Secondly, System 2 (MCTS) significantly increases the success rates but reduces benefits. This suggests that the System 2 is more prone to compromise. The reason may be that System 2 doesn’t require training and terminates the dialogue once the benefits exceed the pre-defined threshold; while the optimized Policy LM focuses on higher-benefits dialogues from MCTS, thereby improving benefits compared with previous work while maintaining a high deal success rate. These results also indicate that DPDP needs further improvement for non-collaborative proactive dialogue tasks in the future.

Our framework integrates the capabilities of both policy LM and MCTS planners, enabling seamless dynamic transitions between these methods during operation. To assess the impact of MCTS, we explored variations in Success Rate (SR) and Average Turns (AT) under differing degrees of MCTS engagement. The results are detailed in Table 3.

For the ESConv dataset, our analysis revealed a marked increase in SR and a significant decrease in AT as the involvement of MCTS escalated during inference. This indicates a clear benefit of incorporating MCTS in more complex dialogue scenarios where strategic planning is crucial. Conversely, the results for the CIMA dataset painted a different picture. Here, we observed an improvement in performance metrics up to a 50% MCTS involvement threshold, beyond which the benefits diminished, eventually leading to a decline in performance. This pattern suggests that for tasks requiring specific reactions, such as providing hints or correcting translations, excessive reliance on MCTS for long-term planning is not only unnecessary but can be detrimental. This phenomenon, which we describe as overthinking leads to unforeseen mistakes in simpler tasks, echoes findings from  Ma et al. (2023), highlighting the potential pitfalls of over-reliance on extensive simulations for tasks that demand immediate and straightforward responses.

The optimal performance with balanced MCTS involvement showcases our framework’s ability to blend planning strengths, achieving efficiency and effectiveness tailored to task demands, demonstrating its adaptability and validating its design.

MCTS typically enhances performance but suffers from elevating the frequency of invoking the LLM (i.e., ChatGPT). Selecting an action that will be actually performed with policy LM only needs 3 LLM calls (1 call for generating system utterance, 1 call for generating user utterance, and 1 for critic), whereas MCTS may require up to 3*10 calls, where 10 is the MCTS simulation times for determining one action. Given that the majority of the inference phase is spent awaiting LLM responses, we assess efficiency and cost based on the frequency of LLM invocations. We analyze the evolving trends in the frequency of LLM’s calls across various MCTS participation ratios, as depicted in Figure LABEL:fig:cost_efficiency. The results reveal that as the utilization of MCTS increases, the frequency of LLM calls gradually escalates, resulting in a heightened application cost. Nevertheless, anomalies also occur at x=100% in ESConv and x=50.0% in CIMA. This reason is that while MCTS usage increases, enhancements in system capabilities lead to a reduction in the average turn. Consequently, the used times of MCTS increase while the LLM invocation count may decrease. By selecting a suitable MCTS ratio, we can strike a balance between effectiveness and the frequency of LLM usage.

Note that we utilize the policy LM to produce prior probabilities for MCTS simulations. Hence, we further examine the influence of the policy LM on MCTS planning here. Specifically, we employ policy LMs trained via pretraining (PT), self-play training (SPT), and both pretraining and self-play (PT& SPT), respectively, to supply prior probabilities to MCTS. The whole evaluation solely relies on the MCTS planner. Additionally, we present results obtained by directly utilizing the uniform distribution to initialize prior probabilities and prompting an LLM to calculate the prior probabilities like GDPZero Yu et al. (2023) for comparison. The results are shown in Table LABEL:tab:pn_for_mcts.

The results indicate that the performance of the policy LM indeed impacts MCTS, particularly when the policy LM’s performance is substantially improved. On ESConv, although the improvements are minor, when the policy LM performs optimally, MCTS also demonstrates the best performance. On CIMA, the sharp decline at SR when initializing priors with uniform distribution and the GDPZero method further underscores the significance of effective prior probabilities and the critical role of utilizing the policy LM to furnish prior knowledge. Concurrently, from another perspective, it also illustrates that our self-play training method is an iterative ascending process. As the training progresses, the capability of policy LM improves, providing better prior knowledge and enhancing the planning ability of MCTS, which in turn guides policy LM to further improvement. The performance change significantly varies on ESConv and CIMA when using a uniform distribution and GDPZero. On ESConv, therapists often have multiple valid strategy options for the same dialogue state, i.e., providing suggestions or expressing empathy for the same person faced with job crisis are both acceptable. Conversely, on CIMA, the suitable strategy is often limited based on the student’s translation state (e.g. Hint for student’s question, and Confirmation to confirm the student’s answer). Therefore, it is more challenging to find the unique valid action by MCTS and thus relying on prior knowledge becomes more significant on CIMA.