Response-Anticipated Memory for
On-Demand Knowledge Integration in Response Generation

The CbR task provides a conversation context $X$ and a document $D$ as inputs, requiring the model to generate a response $R$ to $X$ by referring to $D$. In the rest of the paper, we use $|X|$, $|D|$, and $|R|$ to denote the number of tokens in $X$, $D$, and $R$ respectively. To pinpoint accurate document information for response generation, we design a teacher-student framework to construct document memory as follows:

• •

  The teacher model learns a response-aware document memory $\mathbf{M}$ used in our base conversation model. Specifically, we construct a response-aware weight matrix $\mathbf{G}\in\mathbb{R}^{|D|\times|D|}$, which considers the correlation between context-aware document representations and response representations, and then impose $\mathbf{G}$ on the memory matrix $\mathbf{M}$. The teacher model is optimized to reconstruct the response with the use of response-aware memory $\mathbf{M}$.

• •

  The student model learns to construct a response-anticipated weight matrix to estimate $\mathbf{G}$ used in the teacher model but without access to the response. It is a feed-forward neural network with document and context as its input.

The teacher model and the student model are jointly optimized with training data, while only the student model is applied to testing data.

Following Qin et al. (2019), we use SAN Liu et al. (2018) as our base model, which mainly consists of three components:

• •

  Input encoder: We use two bi-directional LSTM encoders to extract token-level representations of the document $D$ and the context $X$.

• •

  Memory construction: We build the document memory $\mathbf{M}\in\mathbb{R}^{|D|\times k}$ ($k$ is the hidden size of the memory) which will be used in the decoder. A cross-attention layer is first applied to the outputs of the two encoders to integrate information from the context to the document. Then, we obtain a set of context-aware document representation $\mathbf{D}=[\mathbf{d}_{1},\ldots,\mathbf{d}_{|D|}]$. Since each $\mathbf{d}_{i}$ corresponds to a document token, we treat it as the context-aware token representation of the $i$-th token. Next, a self-attention layer is employed to ingest salient information of the context-aware document representations:

  $$\mathbf{M}=\mbox{SelfAttn}(\mathbf{D})=\mathbf{A}\mathbf{D}^{T},\mathbf{A}=\mbox{softmax}(\mathbf{D}^{T}\mathbf{D})$$

  (1)

  where the softmax conducts the normalization over each row of the matrix.

• •

  Output decoder: We use an attentional recurrent decoder to generate response tokens by attending to the memory $\mathbf{M}$. The initial hidden state is set as the summation of token-level context representations. For each decoding step $t$, we get a hidden state $\mathbf{h}_{t}$:

  	$\displaystyle\mathbf{z}_{t}$	$\displaystyle=$	$\displaystyle\mbox{GRU}(\mathbf{e}_{t-1},\mathbf{h}_{t-1}),$		(2)
  	$\displaystyle\mathbf{h}_{t}$	$\displaystyle=$	$\displaystyle\mathbf{W_{1}}[\mathbf{z}_{t};\mbox{CrossAttn}(\mathbf{z}_{t},\mathbf{M})]$		(3)

  where $[;]$ indicates concatenation, and the cross-attention layer here integrates information from the memory to the recurrent outputs. $\mathbf{e}_{t-1}$ is the word-embedding at step $t-1$. Finally, we generate a token $y_{t}$ by a softmax on $\mathbf{h}_{t}$.

Our model modifies the memory construction by refining its self-attention layer so that the memory represents more accurate and on-demand knowledge that helps generating the response.

To ingest accurate memory information for response generation under the aforementioned base model, our teacher model builds a response-aware weight matrix $\mathbf{G}\in\mathbb{R}^{|D|\times|D|}$ given the context-aware document representation $\mathbf{D}$ and the response $R$, then refines the document memory $\mathbf{M}$ with $\mathbf{G}$. Elements in $\mathbf{G}$’s indicate the importance of tokens or token pairs in the document, with consideration of the response information.

First, we describe how to modify the memory matrix $\mathbf{M}$ when $\mathbf{G}$ is given. The original memory $\mathbf{M}$ is constructed by a self-attention operation as Eq. 1. To facilitate response awareness, we update the attention weight matrix $\mathbf{A}$ by element-wise multiplying $\mathbf{G}$, and then get the refined memory $\mathbf{\widetilde{M}}$ as

In the following, we describe two methods to construct the response-aware weight matrix $\mathbf{G}$: (1) We measure the response-aware token importance (RTI) considering the ground-truth response to construct $\mathbf{G}$. (2) We measure the response-aware pairwise importance (RPI) of each token pair $(i,j)$, which can be directly assigned to the element $G_{ij}$ in $\mathbf{G}$. For both methods, matrix elements can be either continuous or binary.

where $\mathbf{1}\in\mathbb{R}^{|D|}$ represents an identity vector with all elements as $1$. By plugging the above $\mathbf{G}$ in Eq. 5, we can construct a memory matrix with plagiarized signals from the response. In this way, the self-attention distributions can adjust to emphasize important tokens, and their corresponding context-aware document token representations become more important in the memory matrix.

Notice that, the aforementioned binary processing with each $\beta_{i}$ can also be applied on each $\mathbf{B}_{ij}$ here and the resulting $\mathbf{G}$ is binary. By using a binary $\mathbf{G}$ in our model, the memory construction can be considered as passing through a Graph Attention Network (GAT) Veličković et al. (2018), which also constructs a graph and updates its representations relying on the information from itself and neighbors on the graph. However, our neighborhood matrix (i.e. $\mathbf{G}$ in our model) is not pre-defined as in GAT but dependant on the inputs $\mathbf{d}_{i}$’s and $\mathbf{r}_{j}$’s, which involve parameters to be estimated.

The objective of the teacher model for RPI can be modified from Eq. 10 by replacing $\bm{\beta}$ with $\mathbf{B}$ obtained in Eq. 11.

The student model learns to construct a response-anticipated weight matrix to estimate the weight matrix $\mathbf{G}$ in the teacher model without access to the ground-truth $R$. If we employ RTI, the estimated target of the student model is $\bm{\beta}$ in Eq. 5. For RPI, the estimated target is $\mathbf{B}$ in Eq. 11.

We first train the teacher model until it converges, and then train the student model with the use of $\bm{\beta}$ or $\mathbf{B}$ from the converged teacher model. Next, we repeat the above processes iteratively. In the training of the teacher model, we fix parameters in $\theta_{s}$ (except parameters shared with $\theta_{t}$) and train the model subject to $\mathcal{J}_{t}$; for the student model, we fix $\phi$ and $\theta_{t}$ (except parameters shared with $\theta_{s}$) and train the model subject to $\mathcal{J}_{s}$. For inference, only the student model will be used to infer the response-anticipated weight matrix and the decoder applies it for generating the output response.

As stated in RPI, it has better model capacity by allowing a larger space of $\mathbf{G}$ with the use of the weight matrix $\mathbf{B}$ instead of the token importance vector $\bm{\beta}$ in RTI. In terms of optimization, we need to estimate more parameters by using RPI, which requires higher training difficulty.

We use the dataset for the CbR task released by Qin et al. (2019). The dataset contains crawled articles and discussions about these articles from Reddit. The articles act as the documents, while the discussions serve as conversational contexts and responses. In total, we have 2.3M/13k/1.5k samples for training/testing/validation.

For all methods, we set word embedding dimension to 300 with the pre-trained GloVe Pennington et al. (2014). Following Qin et al. (2019), our vocabulary contains top 30k frequent tokens. We use bi-LSTMs with the hidden dimensions of 512 and the dropout rate of 0.4 in our encoders. We optimize models by Adam with an initial learning rate of 0.0005 and the batch size of 32. All conversation contexts/responses/documents are truncated to have the maximum length of 30/30/500. For training, we set $\lambda$ as 1 in the loss of student models after tuning. For inference, we apply a top-$k$ random sampling decoding Edunov et al. (2018) with $k$=20. The validation set is for early stopping. Aforementioned implementation details can be found in our codes ¹¹1https://github.com/tianzhiliang/RAM4CbR.

1. 1.

   Seq2Seq Sutskever et al. (2014). The standard Seq2Seq model that leverages only the conversational context for response generation.

2. 2.

   MemNet Ghazvininejad et al. (2018). A knowledge-grounded conversation model that uses a memory network to store knowledge facts.

3. 3.

   GLKS Ren et al. (2020). It applies a global knowledge selector in encoding and a local selector on every decoding step.

4. 4.

   Conversation with Machine Reading (CMR) Qin et al. (2019). The state-of-the-art model on the CbR task, which is also our base model (Sec 3.2). Here, we use the full model of CMR (called CMR+w in Qin et al. (2019)), since the full model outperforms other CMR’s variants on most metrics. We further apply the copy mechanism See et al. (2017) to this base model (CMR+Copy).

5. 5.

   Four variants of our proposed models: RAM_T denotes our Response-Anticipated Memory-based model with RTI, and RAM_T+Copy denotes its copy version. RAM_P and RAM_P+Copy denote our model with RPI and its copy variant .

Following all metrics in Qin et al. (2019), we evaluate all methods by both automatic and human evaluations. For automatic evaluations, we evaluate the responses in three aspects:

1. 1.

   Appropriateness.

   We use three metrics to evaluate the overall quality of a response: BLEU-4 Papineni et al. (2002), METEOR Banerjee and Lavie (2005), and NIST Doddington (2002). NIST is a variant of BLEU that measures n-gram precision weighted by the informativeness of n-grams.

2. 2.

   Grounding. We measure the relevance between documents and generated responses to reveal the effectiveness of responses exploiting the document information. We define $\#\mbox{overlap}$ as the number of non-stopword tokens in both the document $D$ and the generated response $\hat{R}$ but not in contexts $X$. We calculate the precision P and recall R as

   	$\displaystyle\qquad\#\mbox{overlap}=|(D\cap\hat{R})\backslash X\backslash S|,$		(17)
   	$\displaystyle\qquad\mbox{P}=\frac{\#\mbox{overlap}}{|\hat{R}\backslash S|},\mbox{R}=\frac{\#\mbox{overlap}}{|D\backslash S|},$		(18)

   where $S$ denotes the stopword list. F1 is the harmonic mean of precision P and recall R.

   We further propose to measure the effectiveness of exploiting the document information considering the ground-truth. In this way, we evaluate how many ground-truth information models can exploit from the document. We define $\#\mbox{overlap}_{GT}$ as the number of non-stopword tokens in the document $D$, the generated response $\hat{R}$ and the ground-truth $R$ but not in contexts $X$. The precision and recall are as following,

   	$\displaystyle\qquad\#\mbox{overlap}_{GT}=|(D\cap\hat{R}\cap R)\backslash X\backslash S|,$		(19)
   	$\displaystyle\qquad\mbox{P}_{GT}=\frac{\#\mbox{overlap}_{GT}}{|\hat{R}\backslash S|},\mbox{R}_{GT}=\frac{\#\mbox{overlap}_{GT}}{|D\backslash S|},$		(20)

   where $F1_{GT}$ is the harmonic mean of precision $P_{GT}$ and recall $R_{GT}$.

3. 3.

   Informativeness. Ent-n Mou et al. (2016) measures responses’ informativeness with the entropy of the n-gram count distribution. Dist-n Li et al. (2016a) evaluates the diversity of responses via the proportion of unique n-grams among all responses.

Results of all models on automatic and human evaluations are shown in Table 1 and Table 3. MemNet outperforms Seq2Seq on most metrics, which validates that it is important to utilize document information in CbR. However, MemNet only slightly improves on Grounding. Both GLKS and CMR outperform MemNet on most metrics, indicating that it matters how to construct the document memory used in conversation models for CbR. Compared with CMR, CMR+Copy is more competitive on Grounding but weaker on other metrics.

Our proposed models outperform other competing methods on all metrics, including automatic and human evaluations. For models without the copy mechanism, RAM_T performs the best. For models with copy, RAM_T+Copy and RAM_P+Copy excel CMR+Copy on most metrics. Overall, our proposed strategy works well on both the model with and without copy mechanism. We will compare RAM_T and RAM_P in details in Sec 5.3.

In this section, we investigate whether anticipating response contributes to building a better document memory. We first calculate the semantic similarity between each document token and the response using their Glove embeddings, and select top $K$ document tokens. Next, we accumulate the attention weights of each token in all attention distributions in the self-attention weights $\mathbf{A}$ in Eq. 1, i.e. summation over each column of $\mathbf{A}$. Then we select the top $K$ tokens according to their accumulated attention weights. Here, we set $K=10,20$. We apply metrics in  Liu et al. (2016) to calculate the similarity of two token sets extracted above, including maximal tokens-tokens embedding similarity (Emb-M) and bag-of-word embedding similarity (Emb-B). A higher similarity score indicates more response information anticipated by the model. Table 4 shows the results of our two models RAM_T and RAM_P as well as CMR (We use the original self-attention matrix $\mathbf{A}$ for the above calculation for CMR). Results demonstrate that our model is able to output more response-anticipated self-attention distributions, which benefits generating a response close to the ground truth.

Table 3 shows two test samples with generated responses of all models. For Case 1, Seq2Seq and MemNet cannot generate responses relevant to either the document or context. CMR catches the topic “sports”, while GLKS and CMR+Copy use “first person” and “first one” to reflect “only two” mentioned in the document. The response of RAM_T contains information related to both document (“num teams” and “premier league”) and context (“europa”). RAM_T+Copy is also highly relevant to the document and the context, and copies “player” from the document. For Case 2, the first four methods have little relation to the document or the context. CMR+Copy mentions “played”. Our models mention “played” and “num years”. By examining the cases, our method shows promising improvements over existing methods. However, generation on the CbR task is very challenging and there is still a huge space to improve.

We plot the accumulated attention weights of RAM_T and CMR as in Sec 5.2 of the document tokens on Case 1. Fig. 4 shows that RAM_T’s attention highlights “num” and “premier”, and thus it generates the above words in its response.