Content Selection Network for Document-grounded Retrieval-based Chatbots

Suppose that we have a dataset $\mathcal{D}$, in which each sample is represented as $(c,d,r,y)$, where $c=\{u_{1},\ldots,u_{n}\}$ represents a conversation context with $\{u_{i}\}_{i=1}^{n}$ as utterances; $d=\{s_{1},\ldots,s_{m}\}$ represents a document with $\{s_{i}\}_{i=1}^{m}$ as sentences; $r$ is a response candidate; $y\in\{0,1\}$ is a binary label, indicating whether $r$ is a proper response. Our goal is to learn a matching model $g$ from $\mathcal{D}$, such that for a new context-document-response triplet $(c,d,r)$, $g(c,d,r)$ measures the degree of suitability of a response $r$ to the given context $c$ and the document $d$.

We propose a content selection network (CSN) to model $g(\cdot,\cdot,\cdot)$, which is shown in Fig. 2. Different from the previous work [25, 21, 4] which uses the whole document contents, we propose a selection module with a gate mechanism to select the relevant parts of document content based on the context. Then, the context-response matching and the document-response matching are modeled based on the sequential, self-attention, and cross-attention representations. Finally, CNNs and RNNs are applied to extract, distill, and aggregate the matching features, based on which the response matching score is calculated.

Consider the $i$-th utterance $u_{i}=(w^{u}_{1},\cdots,w^{u}_{L})$ in the context, the $j$-th sentence $s_{j}=(w^{s}_{1},\cdots,w^{s}_{L})$ in the document, and the response $r=(w^{r}_{1},\cdots,w^{r}_{L})$, where $L$ is the number of words¹¹1To simplify the notation, we assume their lengths are the same.. CSN first uses a pre-trained embedding table to map each word $w$ to a $d_{e}$-dimension embedding $\mathbf{e}$, i.e., $w\Rightarrow\mathbf{e}$. Thus the utterance $u_{i}$, the sentence $s_{j}$, and the response $r$ are represented by matrices $\mathbf{E}^{u_{i}}=(\mathbf{e}^{u_{i}}_{1},\cdots,\mathbf{e}^{u_{i}}_{L})$, $\mathbf{E}^{s_{j}}=(\mathbf{e}^{s_{j}}_{1},\cdots,\mathbf{e}^{s_{j}}_{L})$, and $\mathbf{E}^{r}=(\mathbf{e}^{r}_{1},\cdots,\mathbf{e}^{r}_{L})$, respectively. Then, CSN encodes the utterances, sentences and responses by bi-directional long short-term memories (BiLSTM) [5] to obtain their sequential representations: $\mathbf{u}_{i}=\text{BiLSTM}(\mathbf{E}^{u_{i}})$, $\mathbf{s}_{j}=\text{BiLSTM}(\mathbf{E}^{s_{j}})$, $\mathbf{r}=\text{BiLSTM}(\mathbf{E}^{r})$. Note that the parameters of these BiLSTMs are shared in our implementation. The whole context is thus represented as $\mathbf{C}=[\mathbf{u}_{1},\cdots,\mathbf{u}_{n}]$. With the BiLSTM, the sequential relationship and dependency among words in both directions are expected to be encoded into hidden vectors.

In document-grounded conversation, the document usually contains a large amount of diverse information, but only a part of it is related to the current step of the conversation. To select the relevant part of document contents, we propose a content selection phase by a gate mechanism, which is based on the relevance between the document and the context. We design the gate mechanism at two different levels, i.e., sentence-level and word-level, to capture relevant information at different granularities. If the sentences/words in the document are irrelevant to the current conversation, they will be filtered out. This is an important difference from the traditional gating mechanism, in which elements are assigned different attention weights, but no element is filtered out. We use the conversation context to control the gate, which contains several previous turns of conversation. Along the turns, the conversation topic gradually changes. The most important topic is that of the most recent turn, while more distant turns are less important. To reflect this fact, we design a decay mechanism on the history to assign a higher importance to the recent context than to the more distant ones. The selection process is automatically trained with the whole model in an end-to-end manner.

The next problem is to select the appropriate response by leveraging the selected document parts. Following a recent study [4], CSN uses a dually interactive matching structure (as shown in Fig. 2) to determine context-response matching and document-response matching, where the two kinds of matching features are modeled by the same structure.

Based on the recent work [25, 31, 27] that constructs different matching feature maps, in addition to using the sequential representations of the sentences, CSN also uses matching on both self-attention and cross-attention representations. Given the sequential representations of the context $\mathbf{C}=[\mathbf{u}_{1},\cdots,\mathbf{u}_{n}]$, the document $\mathbf{D}=[\mathbf{s}^{\prime}_{1},\cdots,\mathbf{s}^{\prime}_{m}]$, and the response candidate $\mathbf{r}$, CSN first constructs a word-word similarity matrix $\mathbf{M}_{1}$ by dot product and cosine similarity:

where $\mathbf{H}_{1}\in\mathbb{R}^{2d\times 2d}$ is a parameter, and $\oplus$ is the concatenation operation.

To better handle the gap in words between two word sequences, CSN applies the attentive module, which is similar to that used in Transformer [23]. The input of an attentive module consists of three sequences, namely query ($\mathbf{Q}$), key ($\mathbf{K}$), and value ($\mathbf{V}$). The output is a new representation of the query and is denoted as $f_{\text{ATT}}(\mathbf{Q},\mathbf{K},\mathbf{V})$ in the remaining description.

At first, CSN uses the attentive module over the word dimension to construct multi-grained representations, which is formulated as:

The second similarity matrix $\mathbf{M}_{2}$ is computed based on these self-attention representations:

Then, another group of attentive modules (cross-attention) is also applied to represent semantic dependency between the context, the document, and the response candidate:

Next, CSN also constructs a similarity matrix $\mathbf{M}_{3}$ as:

The above matching matrices are concatenated into two matching cubes:

Then CSN applies a CNN with max-pooling operation to extract matching features from $\mathbf{M}^{cr}$ and $\mathbf{M}^{dr}$. The output feature maps are flattened as matching vectors. As a result, we obtain two series of matching vectors: (1) between the context and the response $\mathbf{v}^{cr}=[\mathbf{v}^{u_{1}},\cdots,\mathbf{v}^{u_{n}}]$; and (2) between the selected document and the response $\mathbf{v}^{dr}=[\mathbf{v}^{s_{1}},\cdots,\mathbf{v}^{s_{m}}]$.

Finally, CSN applies LSTMs to aggregate these two series of matching vectors into two hidden vectors (the last hidden states of the LSTMs):

These vectors are concatenated together and used to compute the final matching score by an MLP with a Sigmoid activation function:

CSN learns $g(c,d,r)$ by minimizing the following cross-entropy loss with $\mathcal{D}$:

We conduct experiments on two public datasets.

Following previous work [29], we employ recall at position $k$ as evaluation metrics (R$@k$), where $k=\{1,2,5\}$. For a single sample, if the only positive candidate is ranked within top $k$ positions, then R$@k=1$, otherwise, R$@k=0$. The final value is the average over all test samples. Note that $R@1$ is equivalent to hits$@1$ that is used in related work [28, 4].

We compare CSN using sentence-level and word-level selection (denoted as CSN-sent and CSN-word respectively) with the following models:

(1) Starspace [24] concatenates the document with the context as a long sentence and learns its similarity with the response candidate by optimizing the embeddings using the margin ranking loss and $k$-negative sampling. Matching is done by cosine similarity of the sum of word embeddings.

(2) Profile Memory Network [28] uses a memory network with the context as input, then performs attention over the document to find relevant sentences. The combined representation is used to select the response. This model relies on the attention mechanism to weigh document contents.

(3) Key-value (KV) Profile Memory Network [28] uses dialogue histories as keys and the next dialogue utterances as values. In addition to the memory of the document, this model has a memory of past dialogues that can influence the response selection.

(4) Transformer [23] is used in [11] as an encoder for the context, document, and response. The obtained representations are input to a memory network to conduct matching in the same way as in Profile Memory Network.

(5) DGMN [29] is the state-of-the-art model on the CMUDoG dataset. It employs a cross attention mechanism between the context and document and obtains a context-aware document representation and a document-aware context representation. The two representations and the original context representation are all matched with the response representation. The three matching features are finally combined to output the matching score.

(6) DIM [4] is the state-of-the-art model on the PersonaChat dataset. It applies a dually interactive matching structure to model the context-response matching and document-response matching respectively. DIM conducts representation, matching, and aggregation by multiple BiLSTMs, and the final matching features are used to compute the matching score by an MLP.

We use PyTorch [13] to implement the model. A 300-dimensional GloVe embedding [14] is used on all datasets. On PersonaChat, another 100-dimensional Word2Vec [12] embedding provided by [4] is used. Dropout [20] with a rate of 0.2 is applied to the word embeddings. All hidden sizes of the RNNs are set as 300. Two convolutional layers have 32 and 64 filters with the kernel sizes as [3, 3] and [2, 2]. AdamW [9] is employed for optimization with a batch size of 100. The initial learning rate is 0.001 and is decayed by 0.5 when the performance on the validation set is not increasing.

The experimental results are shown in Table 1. The results on all three datasets indicate that our CSN outperforms all baselines, including DGMN and DIM, which are two state-of-the-art models. On the PersonaChat dataset, both CSN-word and CSN-sent achieve statistically significant improvements ($p$-value $\leq$ 0.05) compared with DIM, which is the best model on this dataset. In general, CSN-word performs better than CSN-sent, indicating the word-level selection is more able to select fine-grained document contents than the sentence-level selection. This comparison also confirms our intuition that it is advantageous for document-grounded conversation to rely on fine-grained information from the document. On CMUDoG, the two document content selection strategies work equally well. We explain this by the fact that the grounding document is longer in this dataset, and there is no obvious reason that one level of selection can determine more relevant parts than another. Nevertheless, both selection strategies show clear advantages over the baseline methods without selection.

Compared with other baselines that represent the whole document as a single vector, DGMN, DIM and our CSN consider fine-grained matching between parts of the document and response. We can see that these models achieve clearly better performances, confirming the necessity to use parts of the document rather than the whole document. However, DGMN and DIM only assign attention weights to sentences according to the context, without eliminating low-weighted ones. In contrast, our CSN model filters out all the irrelevant parts. In so doing, we expect the model not to be influenced by clearly irrelevant parts. As we can see in the experimental results, CSN achieves significantly higher performance than DGMN and DIM on all the datasets, confirming the usefulness of explicit selection (and filtering) of document contents.