IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems

The research problem of response ranking in information-seeking conversations is defined as follows. We are given an information-seeking conversation data set $\mathcal{D}=\{(\mathcal{U}_{i},\mathcal{R}_{i},\mathcal{Y}_{i})\}_{i=1}^{N}$, where $\mathcal{U}_{i}=\{u_{i}^{1},u_{i}^{2},\dots,u_{i}^{t-1},u_{i}^{t}\}$ in which $u_{i}^{t}$ is the utterance in the $t$-th turn of the $i$-th dialog. $\mathcal{R}_{i}$ and $\mathcal{Y}_{i}$ are a set of response candidates $\{r_{i}^{1},r_{i}^{2},\dots,r_{i}^{k}\}_{k=1}^{M}$ and the corresponding labels $\{y_{i}^{1},y_{i}^{2},\dots,y_{i}^{k}\}$, where $y_{i}^{k}=1$ denotes $r_{i}^{k}$ is a true response for $\mathcal{U}_{i}$. Otherwise $y_{i}^{k}=0$. For user intent information, there are sequence level user intent labels for both dialog context utterances and response candidates $\mathcal{E}=\{(\mathcal{I}_{i}^{u},\mathcal{I}_{i}^{r})\}_{i=1}^{N}$, where $\mathcal{I}_{i}^{u}$ and $\mathcal{I}_{i}^{r}$ are user intent labels for context utterances and response candidates for the $i$-th dialog respectively. Our task is to learn a ranking model $f(\cdot)$ with $\mathcal{D}$ and $\mathcal{E}$. For any given $\mathcal{U}_{i}$, the model should be able to generate a ranking list for the candidate responses $\mathcal{R}_{i}$ with $f(\cdot)$. Note that in practice, $\mathcal{E}$ can come from predicted results of user intent classifiers to reduce human annotation costs. In our paper, $\mathcal{E}$ are predicted results of the user intent classifier (Qu et al., 2019a) for MSDialog and Ubuntu Dialog Corpus. For AliMe data, $\mathcal{E}$ is the output of the intention classifier which is a probabilistic distribution over 40 intention scenarios (Li et al., 2017).

In following sections, we describe the proposed method for intent-aware response ranking in information-seeking conversations. The model incorporates intent-aware utterance attention to derive the importance weighting scheme of different context utterances. Given input context utterances and response candidates, we first generate representations from two different perspectives: user intent representations with a trained neural classifier and semantic information encoding with Transformers. Then self-attention and cross-attention matching will be performed over encoded representations from Transformers to extract matching features. These matching features will be weighted by the intent-aware attention mechanism and aggregated into a matching tensor. Finally a two-layer 3D convolutional neural network will distill final representations over the matching tensor and generate the ranking score for the conversation context/ response candidate pair.

We use the MSDialog data that consists of technical support dialogs for Microsoft products developed by Qu et al. (2018). Over $2,000$ dialogs with $10,020$ utterances were sampled for user intent annotation on Amazon Mechanical Turk.⁷⁷7https://www.mturk.com/ A taxonomy of 12 labels presented in Table 2 were developed to characterize the user intent in information-seeking conversations. The user intent labels include question related labels (e.g., Original Questions, Clarifying Question, etc.), answer related labels (e.g., Potential Answer, Further Details, etc.), feedback related labels (e.g., Positive Feedback, Negative Feedback) and greeting related labels (e.g., Greetings/ Gratitude), which cover most of the user intent types in information-seeking conversations. In addition to MSDialog, we also consider the Ubuntu Dialog Corpus (UDC) (Lowe et al., 2015). User intent annotation is also performed for randomly sampled $4,063$ UDC utterances. More details can be found in Qu et al. (2018).

Given a response candidate $r_{i}^{k}$ and an utterance $u_{i}^{t}$ in the context $\mathcal{U}_{i}$, we represent the utterance/ response pair from two different perspectives: 1) user intent representation with intent classifiers (Section 3.4.1); 2) utterance/ response semantic information encoding with Transformers (Section 3.4.2).

Given the self-attention/ cross-attention interaction matching matrices for different utterances/ response pairs from a dialog, we first stack them to aggregate them as a 4D matching tensor as follows:

where $l_{c},l_{u},l_{r},L$ are the number of utterance turns in conversation context, number of words in the context utterance, number of words in the response candidate and number of stacked layers in TransformerEncoder. $t,p,q,l$ are indexes along these $4$ dimensions of the matching tensor.

We propose an intent-aware attention mechanism to weight matching representations of different utterance turns in a conversation context, so that the model can learn to attend to different utterance turns in context. The motivation is to incorporate a more flexible way to weight and aggregate matching features of different turns with intent-aware attention. Specifically, let $\mathbf{I}_{u}^{t}\in\mathbb{R}^{l_{t}\times 1},\mathbf{I}_{r}^{k}\in\mathbb{R}^{l_{t}\times 1}$ denote the intent representation vectors defined in Section 3.4.1 for context utterances and response candidates, we design three different types of intent-aware attention as follows:

Dot Product. We concatenate the two intent representation vectors of the utterance/ response pair, and compute the dot product between the parameter $\mathbf{w}$ and the concatenated vector: $\mathcal{A}_{t}=\text{softmax}(\exp(\mathbf{w}^{T}[\mathbf{I}_{u}^{t},\mathbf{I}_{r}^{k}]))$, where $\mathbf{w}\in\mathbb{R}^{2l_{t}\times 1}$ is a model parameter.

Bilinear. We compute the bilinear interaction between $\mathbf{I}_{u}^{t}$ and $\mathbf{I}_{r}^{k}$ and then normalize the result: $\mathcal{A}_{t}=\text{softmax}(\exp({\mathbf{I}_{u}^{t}}^{T}\mathbf{w}\mathbf{I}_{r}^{k}))$, where $\mathbf{w}\in\mathbb{R}^{l_{t}\times l_{t}}$ is the bilinear interaction matrix to be learned.

Outer Product. We compute the outer product between $\mathbf{I}_{u}^{t}$ and $\mathbf{I}_{r}^{k}$ and then flatten the result matrix to a feature vector. Finally we project this feature vector into an attention score with a fully connected layer and a softmax function: $\mathcal{A}_{t}=\text{softmax}(\exp(\mathbf{w}^{T}\cdot\text{flat}(\mathbf{I}_{u}^{t}\otimes{\mathbf{I}_{r}^{k}}^{T})))$, where flat and $\otimes$ denote the flatten layer which transforms a matrix with shape $(l_{t}\times l_{t})$ into a vector with shape $(l_{t}^{2}\times 1)$ and outer product operation. $\mathbf{w}\in\mathbb{R}^{l_{t}^{2}\times 1}$ is a model parameter.

Note that the normalization in the softmax function is performed over all utterance turns within a conversation context. Thus the result $\mathcal{A}_{t}$ is the attention weight corresponding to the $t$-th utterance turn in a conversation context. We also add masks over the padded utterance turns to avoid introducing noise matching feature representations. With the computed attention weights over context utterance turns, we can scale the 4D matching tensor to generate a weighted matching tensor:

Finally IART adopts a two layer 3D convolution neural network (CNN)⁹⁹9https://www.tensorflow.org/api_docs/python/tf/nn/conv3d to extract important matching features from this weighted matching tensor $\widehat{\mathcal{B}}$. A 3D CNN requires 5D input and filter tensors, as we can add one more input dimension corresponding to the batched training examples over the 4D weighted matching tensor. We compute the final matching score $f(\mathcal{U}_{i},r_{i}^{k})$ with a MLP over the flattened output of the 3D CNN. For model training, we compute the cross-entropy loss between the predicted matching scores $f(\mathcal{U}_{i},r_{i}^{k})$ and the ground truth matching labels. The parameters of IART are optimized using back-propagation with Adam algorithm (Kingma and Ba, 2014).

We evaluated our method with three data sets: Ubuntu Dialog Corpus (UDC), MSDialog, and a commercial data collected from the AliMe assistant at Alibaba group. The statistics of different experimental data sets are shown in Table 3. The Ubuntu Dialog Corpus (UDC) (Lowe et al., 2015) contains multi-turn technical support conversation chat logs on the Ubuntu system. We used the data copy shared by Xu et al. (2016). It is also used in several previous related works (Wu et al., 2017; Zhou et al., 2018; Yang et al., 2018).¹⁰¹⁰10The data can be downloaded from https://www.dropbox.com/s/2fdn26rj6h9bpvl/ubuntu%20data.zip?dl=0 MSDialog is released from previous related work by Qu et al. (2018). It contains QA dialogs on various Microsoft products crawled from the Microsoft Answer community. For the AliMe dataset, it contains the chat logs between customers and the AliMe assistant bot at Alibaba. For each query of the dataset, it contains several response candidates from the chatbot engine which are labeled by a business analyst. The details about these data sets are in Yang et al. (2018). Note that the proposed model is more on response re-ranking instead of response retrieval in one step.

We present evaluation results over different methods in Table 4. We summarize our observations as follows: (1) On MSDialog, all three variations of IART with dot, outer product and bilinear based intent-aware attention mechanism show significant improvements over all baseline methods, including the recently proposed strong baseline method DAM. On UDC, IART with three different intent-aware attention mechanisms also show improvements under all metrics except for R10@5. With the comparison between the results of DAM and IART, we can find that incorporating user intent modeling and intent-aware attention weighting scheme can help improve the response ranking performance. (2) If we compare three variations of IART, we can find that the bilinear based intent-aware attention mechanism works better for MSDialog and outer product based intent-aware attention mechanism works better for UDC. The overall performances of these three model variations are close to each other. Overall our proposed model IART shows larger performance improvements on MSDialog. One possible reason is that the intent classifier on MSDialog is more accurate due to the larger annotated training data of MSDialog for user intent prediction and more formal language used in MSDialog, as shown in evaluation results by Qu et al. (2019a). (3) On AliMe data, all three variations of IART also show comparable or better results than all baseline methods including the strong baseline DAM. These results on real product data further verify the effectiveness of our proposed methods.

We perform a case study in Table 5 on the top ranked responses by different methods including the best baseline DAM and our proposed model IART with bilinear based intent-aware attention mechanism. We show the conversation context utterances and top-1 ranked response by each method. In this example, IART produced the correct top ranked response. We visualized the learned user intent representation of context utterances and returned top-1 ranked response by DAM and IART in Figure 2. The predicted user intent of conversation utterances is [OQ] $\rightarrow$ [IR] $\rightarrow$ [PA] $\rightarrow$ [IR] $\rightarrow$ [FD/ OQ]. The agent performed “Information Request (IR)” to confirm whether it is the Outlook Web app or the Windows desktop app. The user confirmed “Further Details (FD)” that the problem was related to the Outlook Web app (Outlook.com). Given such a user intent pattern in the conversation context, a reasonable response can be with intent “Potential Answers (PA)” on providing potential solutions to the user’s question, which is captured by IART due to the integration of user intent modeling. The DAM model, without user intent modeling, failed in such cases and selected a response candidate with “Positive Feedback (PF)” intent. The response returned by DAM assumed that “the issue is resolved”, but actually the user was expecting an answer to her unsolved technical problem. This gives an example and interpretation of why user intent modeling can be helpful for response ranking in conversations.