Converse, Focus and Guess - Towards Multi-Document Driven Dialogue

As for encoding large-scale documents along with optimizing dialogue agent is computationally burdening and inviable, we resort to pretrain the documents in advance. We argue that the pre-trained representation should be not only liable to the original meaning but also helpful for constructing a document-driven task-oriented dialogue agent.

To this end, we propose Attribute-aware Document Representation (AaDR). We assume that each document talks about several attributes with index $\{1,...,j,...,L\}$, such as “directed by”, “release year”, “in language” and so on in movie scenario.

Inspired by Hierarchical Attention Networks (HAN) (Yang et al. 2016) and Hierarchical Label-Wise LSTM (Liu, Yuan, and Wang 2020), we introduce an Attribute-aware HAN (AaHAN) to encode each document, which seen attribute as label. Specifically, each attribute is used to index the corresponding parameters in hierarchical attention. This mechanism can capture different information for different attributes. Combined with the contrastive loss (Hadsell, Chopra, and LeCun 2006), an attribute-aware document representation is learned.

We propose Document-aware Language Understanding (DaLU) for NLU. In the previous turn, the agent’s question is $x^{t}$, and the user’s response is $u^{t}$, which are together concatenated into a long sentence and encoded using a BiGRU. Take the last hidden state as output $G^{t}\in\mathbb{R}^{2d}$.

Therefore, the similarity $\hat{S}^{t}\in\mathbb{R}^{M\times L}$ between previous turn $G^{t}\in\mathbb{R}^{2d}$ and candidate documents $Q\in\mathbb{R}^{M\times L\times 4d}$ (concatenation of each candidate $Q_{i}$ etc.) can be calculated directly by a bilinear method. It reflects the matching degree for each candidate $Q$.

where $W^{s}\in\mathbb{R}^{2d\times 4d}$ is a trainable parameter.

In addition, we further calculate two distributions: (1) the attribute type $\tilde{\pi}^{t}\in\mathbb{R}^{L}$. (2) a flag $\alpha^{t}\in\mathbb{R}^{1}$ indicating whether the response is “unknown”. The closer the value is to 1, the less valid information is included in this turn.

where $W^{attr}\in\mathbb{R}^{L\times 2d}$ and $W^{unk}\in\mathbb{R}^{1\times 2d}$ are trainable parameters.

When previous response is “unknown”, the selected probability of each candidate is equal on turn level. Therefore, we expand the similarity $\hat{S}^{t}$ on attribute dimension, concatenating a vector whose similarity is all $1$, and get $S^{t}\in\mathbb{R}^{M\times(L+1)}$. For the attribute type $\tilde{\pi}^{t}$, we also expand the attribute dimension by fusing the $\tilde{\pi}^{t}$ and $\alpha^{t}$, and get $\beta^{t}\in\mathbb{R}^{(L+1)}$.

Further the Turn-Level Doc ²²2Doc is the abbreviation of document and Attr is attribute. Belief $\hat{p}^{t}\in\mathbb{R}^{M}$ is obtained, which is the probability of each candidate document being selected at current turn, and also the Turn-Level Attr Belief $\hat{\pi}^{t}\in\mathbb{R}^{L}$.

We propose Document-aware State Tracking (DaST) for DST. The dialogue state is defined as the following two parts: (1) Dialog-Level Doc Belief $p^{t}\in\mathbb{R}^{M}$ represents the probability of each document being selected. (2) Dialog-Level Attr Belief $\pi^{t}\in\mathbb{R}^{L}$ is the probability of an attribute being unknown. The lower the value, the higher the attribute belief.

When a document is excluded, it will rarely be selected. But the attribute belief is accumulated for each one, indicating whether the attribute will be asked. This two belief distributions are updated as follows.

where $\operatorname{norm}$ is the L1-Normalization method. The initial value $p^{0}$ is a uniform distribution, and $\pi^{0}$ is initialized to zero vector. At the beginning of each dialogue, agent directly use $p^{0}$ and $\pi^{0}$ into the PM module.

We propose Document-aware Policy Optimizing (DaPO) for PM to minimize the number of dialogue turns and guess the true target.

In order to describe the degree of discreteness of the data, we introduce a similar variance measure to calculate the differentiated representation $Q^{diff}_{i}\in\mathbb{R}^{L\times 4d}$ for each document $Q_{i}$. It’s used to describe the degree of attribute discreteness.

where $N$ is the size of whole dataset.

Note that the Dialog-Level Doc Belief $p^{t}$ is the confidence of each document. Therefore, a weighted sum is used on the differentiated representation $Q^{diff}\in\mathbb{R}^{M\times(L\times 4d)}$ (concatenation of each candidate $Q_{i}^{diff}$ etc.) to obtain $v^{t}\in\mathbb{R}^{L\times 4d}$, which is used to describe attribute uncertainty $\gamma^{t}\in\mathbb{R}^{L}$ over all document candidates.

where $W^{diff}\in\mathbb{R}^{4d\times 1}$ is a trainable parameter.

Normally, the agent can directly ask the attribute with the largest uncertainty $\gamma^{t}$, expected to successfully end the dialogue in a minimum number of turns. However, some highly uncertain attributes may have low belief, therefore, the “ask” action of time step $t+1$ should be calculated as:

It is also interesting to note that $a^{t+1}$ is equivalent to the probability of each attribute being asked at timestep $t+1$.

In NLG module, the agent generates natural language response respectively for “ask” and “guess” action. For “ask” action, it produces a response of asking the most probable attribute greedily according to $a^{t+1}$. For “guess” action, the agent guesses the movie corresponding with the most probable document greedily according to $p^{t}$. We use predefined natural language templates to converse with the user. The termination includes two cases.

1. 1.

   Positive termination: when the maximum value of Dialog-Level Doc Belief $p^{t}$ exceeds a certain threshold $K$.

2. 2.

   Passive termination: when the set maximum number of dialogue turns is reached.

We divide GuessMovie into two disjoint parts. The 70% part is used for pre-training document representation and NLU module, and the remaining 30% is used for training MD3 with 50k simulations using REINFORCE (Williams 1992) algorithm. The discount rate is 0.9. After training, we run a further 5k simulations to test the performance.

We construct a user simulator (Schatzmann et al. 2007; Li et al. 2016) with handcrafted rules because the user only need to answer the agent’s questions passively. It randomly selects the target at the beginning. During the dialogue, the relevant value is indexed from the current structured KB and filled into a natural language template to response. Otherwise, if the user doesn’t have knowledge, the answer is unknown. Specifically, in order to increase the dialogue diversity, we randomly mask some values for the 6 attributes with proportion of 0.1 at the beginning of each dialogue. However, all documents remain unchanged in the agent.

The reward function is similar to Dhingra et al. (2017). If the rank of target document is in the top $R=3$ results, the reward is $max(0,2(1-(r-1)/R))$, where $r$ is the actual rank of the target. Otherwise, if the dialogue failes, the reward is $-1$. In addition, a reward of $-0.1$ will be given in each turns, making the dialogue tend to be finshed in a smaller number of turns.

We use Adam(Kingma and Ba 2014) optimizer with learning rate 0.001 and GloVe (Pennington, Socher, and Manning 2014) word embedding. The number of candidate documents for document representation and dialogue is 32 by default. The maximum number of turns is 5. The probability threshold $K$ for whether performing a Guess action is 0.5.

The following introduces several different modules to compare with our method, including the machine reading comprehension (MRC) for NLU, random and fixed policy for PM.

We simulated 5k dialogues randomly for testing the performance. The complete results are shown in the Table 2. Various combinations of NLU and PM module are selected, and different size of candidate documents (32, 64, and 128) are setted. We calculated the dialogue success rate of the target in the candidates top-1 or top-3. In addition, similar to the retrieval task, the Mean Reciprocal Rank (MRR) is also calculated based on the position of the target in the ranking result. At the same time, the average number of dialogue turns and average rewards are also obtained.

As shown in the Table 2, the DaLU-NLU module has a higher dialogue success rate and smaller number of turns compared to the MRC-NLU module with the same PM, because it’s difficult for the MRC model to accurately extract the attributes and there may be implied attributes value in the document. Therefore more interaction is needed to improve the select probability of the target.

In the combinations of DaLU-NLU, DaPO-PM is significantly superior to the random or fixed policy which is impossible to generate appropriate actions for specific candidates, while DaPO-PM can ask attribute with higher uncertainty and belief.

Although MD3 has significant advantages compared with others and is very close to human’s performance, it’s performance reduced quickly when increasing the size of candidate documents to 64 or 128. Also, there is still a lot of room for improvement on top-1 dialogue success rate and larger size of candidate documents.

At the end of each dialogue, a sorting results can be obtained according to the Dialog-Level Doc Belief $p^{t}$. For all 5k simulated dialogs, we visualize the dynamic change of the target document rank (TDR), as shown in 3-a. We can see that from top to bottom, as dialogue goes on, the color of the block is gradually lighter, which means TDR is gradually higher. In addition, the candidate documents information entropy (CDIE) calculated by the Dialog-Level Doc Belief $p^{t}$ can also be visualized, indicating uncertainty change of the selected document, as shown in 3-b. From top to bottom, as dialogue goes on, the CDIE gradually decreases, which means the uncertainty of the guessed document become smaller. It illustrates that our model is interpretable.

In the comparison of several combinations, we can find that MD3 has significant advantages regardless of TDR and CDIE.

Figure 4 shows three dialogue samples between different models with the same candidates, and dynamic changes of the target rank and probability.

At the beginning of the dialogue, each document has the same probability to be guessed as the target. In the MD3 sample, the “release year” and “directed by” attributes are asked, so that the rank of the target document quickly rises to the first place, and the probability value increases to 0.6. As for the MRC-Rand sample, it doesn’t ask the director, which has the greatest difference for candidates. And the probability of selecting the target is always low. This is due to the inaccuracy of the MRC model for attributes extraction and implied attribute information. It can be seen that MD3 is better than the other methods and more robust.