“Think Before You Speak”: Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs

We regard MADPL as a multi-label classification task, i.e., given the current dialog state that summarizes the dialog history, we want to predict a macro-action that is a set of independent atomic dialog actions that serves as the current system response. Following Li et al. (2020), each atomic dialog action is a concatenation of domain name, action type, and slot name, e.g., “hotel-inform-area”.

The PEDP framework is inspired by the human dialog strategy of “think before you speak” that conceives relevant content to express before deciding the current response. We first discover contextually relevant dialog actions by looking ahead several turns under the single-action dialog dynamics (i.e., Single Action Dialog Planning) and then use the planning result as guidance to predict the atomic dialog actions that should be executed concurrently (i.e., macro-action).

Formally, let $\mathbf{s}_{t}$ denote the current dialog state representing the dialog history up to the current dialog turn. We encode $\mathbf{s}_{t}$ into a dialog state embedding $\mathbf{h}_{t}$. Given the current state embedding $\mathbf{h}_{t}$, we plan $K$ independent single-action dialog paths, and the $k$-th dialog path is represented by a vector $\mathbf{h}^{(k)}$, $k=1,\ldots,K$. Our model then decodes each dialog path to a probability distribution over atomic dialog actions, i.e., $\mathbf{p}^{(k)}$. Finally, these distributions $\{\mathbf{p}^{(k)}\}_{k=1}^{K}$ are aggregated to form a unified distribution, from which atomic dialog actions in the macro-action $\mathbf{A}_{t}$ are sampled.

Figure 2 depicts the architecture of PEDP, which mainly consists of four modules: 1) State Encoding, which encodes the dialog state into a compact vector; 2) Single-Action Dialog Planning, which plans several independent dialog paths with each one looking ahead several steps considering only the single-action dialog dynamics; 3) Path-wise Decoding, which predicts the multi-action probability distribution for each dialog path; and 4) Ensemble Prediction, which aggregates the probabilities and samples the final macro-action. In what follows, we elaborate on each of these modules in PEDP.

The dialog state is a summary of the dialog history. Inspired by Li et al. (2020), we propose to include the following information into dialog state $\mathbf{s}_{t}$ at time step $t$: 1) embedding of returned entities for a query, 2) the last user action, 3) the last system action, and 4) the belief state that contains the requested slots and informed slots of user and agent.

To densify the dialog state, we apply a Fully-connected Feed-forward Network (FFN) to obtain a compact dialog state embedding $\mathbf{h}_{t}$. The FFN consists of two linear transformations with a ReLU activation in the middle layer, i.e.,

In what follows, this dialog state embedding $\mathbf{h}_{t}$ will serve as the initial dialog state embedding for planning.

As discussed in the introduction, the straight-forward multi-label imitation of action co-occurrence suffers from the poor generalization ability issue due to the lack of labeled multi-action training data. To address this issue, we propose a single-action dialog planning module that further leverages single-action dialog samples to model contextual action co-occurrence patterns. It looks ahead several steps under single-action dialog dynamics to predict the next dialog state, which will serve as the additional information to enhance macro-action prediction. In what follows, we describe how to model the dialog state transition in a single-action dialog.

Given the initial dialog state embedding $\mathbf{h}_{t}$ at the current dialog turn $t$, we plan $K$ independent single-action dialogs to discover contextually related atomic dialog actions. In each dialog, a discrete policy model and a world model interact to look ahead several steps. Let $\mathbf{h}_{t,n}^{(k)}$ denote the dialog state embedding at the $n$-th step in the $k$-th dialog for $n=0,\ldots,N_{k}$ where $N_{k}$ is the length of the $k$-th dialog, and $\mathbf{h}_{t,0}^{(k)}=\mathbf{h}_{t}$, $\mathbf{h}_{t,N_{k}}^{(k)}=\mathbf{h}^{(k)}$. The last dialog state embedding $\mathbf{h}^{(k)}$ estimates the hidden vector of the future dialog state $\mathbf{s}_{t+1}$ and summarizes the planned single-action dialog. In what follows, we describe how to plan a single step from $\mathbf{h}^{(k)}_{t,n}$ to obtain $\mathbf{h}^{(k)}_{t,n+1}$. To simplify the notations, we omit the dialog index $k$ and dialog turn $t$.

Given the dialog state embedding $\mathbf{h}_{n}$ at step $n$, we want first to predict a single action $a_{n}$ and then use it to update $\mathbf{h}_{n}$ to obtain $\mathbf{h}_{n+1}$. This can be achieved by cyclical interactions between a discrete policy model (DP) that predicts an atomic dialog action under the current-step dialog state and a world model that simulates user behavior to update the current-step dialog state, i.e.,

Here, DP is implemented as a single linear layer followed by a Gumbel-Softmax function Jang et al. (2016) parameterized by $\tau_{d}$. The Gumbel-Softmax function draws an atomic dialog action sample from a categorical distribution, diversifying the planned dialogs. $\tau_{d}$ is selected to balance the approximation bias and the magnitude of gradient variance. The world model is implemented using a GRU to model dialog state transitions, and $\texttt{Emb}(a_{n})$ denotes the embedding vector of atomic dialog action $a_{n}$.

We observe that both executing a macro-action and sequentially executing the corresponding atomic dialog actions will lead to a similar (or identical) state transition (see Footnote 3). Accordingly, we assume that the planned information is adequate once such a state transition is achieved, and propose to determine if to stop planning by comparing the initial and the next dialog state embeddings, i.e., $\mathbf{h}_{0}$ and $\mathbf{h}_{n+1}$ (see Fig. 2b). This is also modeled using a neural network, i.e.,

where $c_{n}$ is a binary variable, “$:$” denotes vector concatenation, and FFN is a 2-layer fully-connected feed-forward network using the ReLU activation function in the middle layer.

Note that the single-action dialog planning module estimates dialog state transitions in the dialog state embedding space from $\mathbf{h}_{0}$ to $\mathbf{h}_{N}$, and it demands supervision from the labeled structural dialog state transition samples from $\mathbf{s}_{t}$ to $\mathbf{s}_{t+1}$. To enable model training, we incorporate a recovery model for mapping the state embeddings back to the structured state-space. Formally, we have,

Here, Recover is implemented by a 2-layer FFN and is only used during the training stage.

Leveraging the planned single-action dialog, we predict the probability distribution over atomic dialog actions.

Given the initial and the last dialog state embeddings $\mathbf{h}_{0}$ and $\mathbf{h}^{(k)}$, we decode the multi-action probability distribution $\mathbf{p}^{(k)}$ with a neural network applied to each path separately and identically. Since our task formulation posits no dependency among atomic dialog actions, following Li et al. (2020), we perform binary classification for each specific action in the atomic dialog action space independently to decide whether it is selected. Specifically, we instantiate the decoder $\mathbf{p}^{(k)}=[\mathbf{p}^{(k)}_{1}:\ldots:\mathbf{p}^{(k)}_{M}]$, where $k$ refers to the planned path and $M$ is the size of the action space. Each $\mathbf{p}^{(k)}_{m},m=1,\ldots,M$ is a vector computed as:

Future efforts at exploring different decoders may further improve the performance.

The performance of the proposed model is affected by the quality of the planned dialogs. To reduce the impact of low-quality dialogs, we propose to ensemble the multi-action probability distributions with an aggregation function, i.e.,

where $\mathrm{Aggr}(\cdot)$ is the mean average in our case.

Task-oriented dialogs share a critical “one-to-many” nature that different responses may be proper under the same context, making it essential to incorporate stochastic factors to build a multi-action dialog policy. We empirically verified that this stochasticity could be achieved with probability sampling. In our case, we apply the Gumbel-Sigmoid function to sample the macro-action, that is,

Here $\mathrm{GumbelSigmoid}(\cdot)$ is a modification of the Gumbel-Softmax function, regarding sigmoid as a softmax with two logits $p$ and $0$. $\tau$ denotes the temperature factor, $g_{1}$ and $g_{2}$ are Gumbel noises.

We use a multi-task objective to train all the modules jointly.

Evaluation Metrics   We perform the automatic evaluation from two perspectives:

1) Interactive Evaluation, which evaluates task completion quality through dialog interaction with the user simulator. We use Inform F1, Inform Recall, Match, Turn, and Success as evaluation metrics. Inform F1 and Inform Recall evaluate whether all the requested information (e.g., hotel address) has been provided. Match evaluates whether the booked entities match the indicated constraints (e.g., the cheapest restaurant in town). Turn and Success evaluate the efficiency and level of task completion of dialog agents, respectively. Specifically, a dialog is considered successful if all the requested information is provided (i.e., Inform Recall $=1$) and all the entities are correctly booked (i.e., Match $=1$).

2) Standard Evaluation, which measures the generalization ability with the test set of the training corpus. We report the sample-wise Precision, Recall, and $F_{1}$ score of the macro-actions, where each atomic dialog action is assumed to be correct if it is included in the ground truth.

1) MultiWOZ Budzianowski et al. (2018) is a multi-domain dialog dataset with $10,438$ dialogs covering $7$ distinct domains and $24$ slots. We apply the agenda-based user simulator as the interaction environment for interactive evaluation.

2) SGD Rastogi et al. (2020) is published to build scalable multi-domain conversational agents. It has $16,142$ dialogs spanning $16$ domains and $214$ slots. We conduct a standard evaluation on SL-based methods to address the scaling concerns. We do not report interactive evaluation results since the corresponding official user simulator is unavailable.

(SL) DiaMultiClass Li et al. (2020) is a FFN with Sigmoid as the last activation function.

(SL) DiaSeq Li et al. (2020) encodes the multi-hot dialog state vector with a 2-layer FFN and sequentially decodes the macro-action with an RNN.

(SL) DiaMultiDense Li et al. (2020) performs binary classification for each specific action with the same network described in Section 3.7.

(SL) gCAS Shu et al. (2019) is a novel DiaSeq variant that uses tangled RNNs to predicts the macro-action by an auto-regressive generation of intent and slots.

(RL) GP-MBCM Gašić et al. (2015) is a Gaussian process based Multi-domain Bayesian Committe Machine that decomposes the policy into domain-specific policies.

(RL) ACER Wang et al. (2016) is the Actor-Critic RL policy with Experience Replay. ACER is sample efficient and is well adapted for large discrete action spaces.

(RL) PPO Schulman et al. (2017) denotes Proximal Policy Optimization, a simple and stable RL algorithm that applies a constant clipping mechanism.

(AL) ALDM Liu and Lane (2018) is the Adversarial Learning Dialog Model. ALDM estimates a reward at the end of the session to optimize the dialog policy.

(AL) GDPL Takanobu et al. (2019) equips PPO with a “rewarder” for joint reward estimation and policy optimization. GDPL is regarded as the SOTA method.

(AL) DiaAdv Li et al. (2020) further improves the DiaMultiDense model with adversarial fine-tuning.

Interactive Evaluation   We report the performance of each approach that interacts with the user simulator in Table 1.

We observe that our PEDP framework achieves the highest performance in the task success by $3\%$ compared to the second-best model DiaAdv and by $7.4\%$ compared to the SOTA GDPL method on account of the substantial improvement in match rate and inform recall. This demonstrates that PEDP can effectively incorporate more task-relevant information to enrich the response and lead to task success.

We also observe that PEDP performs moderately in terms of average turn. A possible reason is that the model fails to generate denser dialog action combinations. As shown in Table 1, PEDP reaches a comparable F1 score and a higher recall, indicating a lower precision score and implying the incorporation of redundant information that does not contribute to task completion. We report how this affects user experience in the human evaluation.

We observe that our PEDP framework outperforms the others in terms of all evaluation metrics. For the F1 score, PEDP achieves $8.37\sim 32.74\%$ improvements on account of the substantial improvement in precision ($6.03\sim 26.89\%$) and recall ($1.31\sim 36.78\%$) over the baselines.

We also observe that the standard and interactive performance are negatively correlated for the baseline methods (see Table 1). We attribute this to the fact that standard and interactive objectives are fundamentally in conflict. While multiple action combinations can be considered appropriate during dialog interaction, they may not match the ground truth in the test samples. Unlike existing methods, PEDP achieves the best of both worlds, demonstrating our method’s effectiveness on MADPL.

The results show that: 1) The single-action dialog planning module contributes more significantly for performance improvement while the ensemble prediction module effectively stabilizes the model and leads to a lower standard deviation, and please see Appendix C for hyper-parameter sensitivity experiments for the number of paths $K$. 2) Sampling usually leads to a performance gain of match rate and inform recall but reduces the inform precision, indicating the incorporation of redundant information in the response. For GDPL, sampling harms the performance. A possible reason is that GDPL, which learns from the user simulator for evaluation, has adapted to the simulated dialog scenarios. Sampling instead may lead to sub-optimal decisions.

In Table 3, we see that our PEDP framework outperforms the baseline models. As mentioned in robust evaluation (see Section 5.4), our PEDP model shares a lower inform precision ($0.79$) than DiaSeq ($0.82$) and GDPL ($0.87$), which indicates the incorporation of redundant information. However, we find that the performance gap does not severely harm the user experience. A possible reason is that human users tend to focus on goal-related information during task-oriented dialog. Compared to dealing with redundant information, they are more intolerant towards the failure of providing the requested information.