PoKE: Prior Knowledge Enhanced Emotional Support Conversation with Latent Variable

Before the task ESC is proposed, there are two relevant well researched dialogue systems, i.e. emotional chatting (Zhou et al., 2018; Wei et al., 2019; Song et al., 2019) and empathetic responding (Rashkin et al., 2019; Lin et al., 2020, 2019; Majumder et al., 2020). Emotional chatting needs to respond in appropriate emotion or the given emotion, such as happy or angry (Zhou et al., 2018). Empathetic responding needs to understand and feel what user is experiencing, and respond with empathy (Rashkin et al., 2019). Compared with them, the emerging task of ESC aims at reducing help-seeker’s emotional stress and help them explore and overcome the problem the face. The first work on ESC task, called BlenderBot-Joint, adopts a chitchat bot BlenderBot (Roller et al., 2021) as backbone and takes emotional support into account in conversation (Liu et al., 2021). Specifically, they encode the context history and predict a strategy token. Then, they concatenate the predicted strategy token to the head of generation to guide the desired response. Meanwhile, they construct an Emotional Support Conversation dataset (ESConv) annotated with support strategies for the ESC task. Based on ESConv, MISC (Tu et al., 2022) uses an off-the-shelf commonsense model COMET (Bosselut et al., 2019) to infer an instant mental state of seeker and encodes them additionally. When predicting strategy, they take the probability of each predicted strategy as weight to get a weighted average representation of strategy, and utilize it for guiding a skillful generation. GLHG (DBLP:conf/ijcai/00080XXSL22) considers the hierarchical relationship between the seeker’s global situation (summarizing the condition of the seeker) and the local intention (inferred by COMET in each dialogue round) in conversation, and uses a graph neural network to encode their relationship for guiding generation. Note that both MISC and GLHG are constrained by the external knowledge in COMET, which may not be applicable to some specific domain. The external knowledge base like COMET also requires significant human effort to develop. Meanwhile, all of them are limited to the scope of current conversation but ignore abundant prior knowledge existing in the dataset. In contrast, we focus on exploring the existing knowledge and the characteristics of the ESC task without using any external knowledge.

There are lots of works for retrieve-based generation. We will detail some classical studies since our main aim is not to compare with them. Some generative models, like GPT2 (Radford et al., 2019), perform well on many tasks such as machine translation and question answer (Yang et al., 2020; Lewis and Fan, 2018; Guo et al., 2018). However, recent some works have pointed out that in dialogue system, the generation model just relied on the input context suffers from some issues, such as dull generation (e.g. “I don’t know”) and hallucination (Chen et al., 2022; Li et al., 2016a; Shuster et al., 2021). To prompt model to generate more engaging response, RetNRef (Weston et al., 2018) proposes a simple but effective retrieve-and-refine strategy. RetNRef appends the retrieved context-relevant responses to context to guide the generation. Similar to this approach, Cai et al. (Cai et al., 2020) retrieves both literally-similar and topic-related exemplars to guide dialogue generation. Majumder et al. (Majumder et al., 2022) employs dense passage retrieval and introduce three communication mechanisms of empathy to facilitate the generation towards empathy. For the ESC task, the abundant prior knowledge in historical conversations has great reference value for reducing seeker’s emotional stress. Besides, the responses with the same strategy are similar in sentence pattern. Thus, we introduce exemplars into generation model and denoise exemplars according to the strategy distribution to emphasize those more relevant exemplars.

It is well known that dialogue systems have a one-to-many mapping problem that given a single context, there exist multiple valid responses (Chen et al., 2022). To model this one-to-many feature and improve the diversity of generations, many works introduce latent variable to model a probability distribution over the potential responses (Zeng et al., 2019; Zhao et al., 2017; Gu et al., 2018; Fang et al., 2019). DialogVED (Chen et al., 2022) combines continuous latent variable into the encoder-decoder pre-training framework to generate more relevant and diverse responses. Except for continuous representation of latent variables, some works utilize discrete categorical variables to promote the interpretability of generation (Bao et al., 2020, 2021). For ESC, there also exist several reasonable support strategies and the corresponding responses at a certain stage. Therefore, it is required to additionally consider the one-to-many mapping relationship of strategies. In our work, we introduce a continuous latent variable to model the distribution over strategy. Furthermore, we employ this strategy distribution to denoise the exemplars at the sequence-level to focus on strategy-relevant exemplars.

Humans tend to use prior knowledge to bias decisions (Hansen et al., 2012), and there is abundant prior knowledge in historical conversation for ESC task. Due to the characteristics of ESC, we consider the prior knowledge of context-related exemplars and the general selection order of support strategies in our work.

Given the target context $c_{q}$ with situation $s_{q}$, we concatenate them as the query input $q=[c_{q},s_{q}]$. For each candidate response $r_{p}$, we get its situation $s_{p}$ and do the same concatenation operation to get the candidate input $p=[r_{p},s_{p}]$. Then, DPR calculates the similarity between the query and candidate input using the dot product of their embeddings:

where $E_{Q}(\cdot)$ and $E_{P}(\cdot)$ are the encoders of query and candidate input respectively. In the end, we select top $k$ candidate responses with the highest similarity as exemplar set $\mathcal{E}=\{e_{1},e_{2},\cdots,e_{k}\}$, where $e_{i}$ denote an exemplar response. Meanwhile, we can get the corresponding strategy set $\mathcal{Y}=\{y_{1},y_{2},\cdots,y_{k}\}$, where $y_{i}$ denotes the strategy label of $e_{i}$. As for inference, we use the candidate encoder $E_{P}(\cdot)$ to pre-compute embeddings of all responses in training set, thus to save the retrieval time of inference. To adapt DPR to the ESC task, we fine-tune DPR on the dataset of ESC.

We use a multi-layer Transformer-based encoder of BlenderBot (Roller et al., 2021) to encode multiple information source, including dialogue context, seeker’s post, situation and the retrieved exemplars. Note that there are multiple source sequences to consider, so building a parameter-isolating encoder for each source will increase parameters and make training time-consuming. To solve this issue, we design a unified encoder, which is parameter-sharing but prepends a unique source token to each input. Source token can act as prompt to distinguish different input. There are four Source tokens including [CTX], [POST], [ST] and [EXEM] representing context, post, situation and exemplar, respectively.

Firstly, we reconstruct the dialogue context by concatenating them with a special token $[SEP]$ and prepending the source token $[CTX]$, i.e. $c=[[CTX],w^{1}_{1},w^{1}_{2},\cdots,[SEP],w^{2}_{1},w^{2}_{2},\cdots,w^{N}_{M}]$. Then, we feed this sequence into the encoder to get its contextualized hidden states:

where $\operatorname{Enc}(\cdot)$ denotes the encoder, and $\mathbf{H}^{c}\in\mathbb{R}^{l\times d_{h}}$ is the hidden states of context sequence with $l$ tokens and hidden size of $d_{h}$. To obtain a single sentence-level representation of context, we take the first one hidden state of sequence, i.e. the output hidden state of source token, as the context representation:

Similarly, for the given situation $s$, seeker’s post $p$, and each exemplar sequence $e_{i}$ in exemplars set $\mathcal{E}=\{e_{i}\}^{k}_{i=1}$, we prepend them with the corresponding source token in the same way, and use the encoder to obtain their sequence representations:

and we use $\mathbf{H}^{\mathcal{E}}=[\mathbf{h}^{e_{1}},...,\mathbf{h}^{e_{k}}]$ to express the representation of the entire exemplars set $\mathcal{E}$. These representations of multiple source are used to model the latent variable in Section 3.3 and fed into the decoder for generation in Section 3.4.

In this section, we introduce the workflow of modeling latent variable and how to build strategy distribution to obtain representations of mixed strategy and denoised exemplars.

CVAE is trained by maximizing a variational lower bound $\mathcal{L}_{ELBO}$, consisting of two terms: negative likelihood loss of decoder and K-L regularization:

where $q_{\phi}(\mathbf{z}|r,x)$ and $p_{\theta}(\mathbf{z}|x)$ are called recognition network and prior network respectively (with parameters $\phi$ and $\theta$), and $p_{\theta}(r|\mathbf{z},x)$ is the decoder for generation, which will be illustrated in Section 3.4. Then we can sample latent variable $\mathbf{z}$ from the well-learned Gaussian distribution (for more detail please see Appendix D).

In order to regularize the latent space and model the one-to-many mapping relationship of strategy, we design an extra optimizing objective of strategy, $\mathcal{L}_{y}$. A strategy prediction network $p_{\theta}(y|\mathbf{z})$ is used to recover the strategy label $y$ by latent variable $\mathbf{z}$:

where $\mathbf{p}$ is denoted as the distribution of strategy. We calculate $\mathbf{p}$ by a fully connected layer and based on the transition matrix $\mathbf{T}$ obtained in Section 3.1:

where $\mathbf{W}_{y}\in\mathbb{R}^{m\times d_{z}}$ and $\mathbf{b}_{y}\in\mathbb{R}^{m}$ are the learnable parameters, $y^{\prime}$ is the previous strategy taken by the supporter, which is provided in dataset, and $\mathbf{T}_{y^{\prime}}\in\mathbb{R}^{m}$ is the transition probability of $y^{\prime}$.

The representations of the latent variable, mixed strategy, and denoised exemplars will be incorporated into the decoder to guide generation, which is illustrated in the following section.

After getting the above-mentioned representations of the latent variable $\mathbf{z}$, mixed strategy $\mathbf{s}$, and denoised exemplars $\mathbf{e}$, the consequent problem is how to effectively incorporate them into decoder¹¹1We apply the decoder in BlenderBot (Roller et al., 2021) to model the distribution $p_{\theta}(r|\mathbf{z},x)$ and optimize the negative likelihood loss $\mathcal{L}_{nll}$ in Eq. (5). for generation. Inspired by (Chen et al., 2022; Li et al., 2020), we apply a memory schema to inject these encoded knowledge. The memory schema regards the representations of the encoded knowledge as additional memory vectors $\mathbf{m}$ for each self-attention layer to attend, as illustrated in Figure 3. We first project the vector of latent variable $\mathbf{z}$ into the $d_{h}$-dimensional space:

where $\mathbf{W}_{z}\in\mathbb{R}^{d_{h}\times d_{z}}$ is the projection matrix. Thus, we can obtain the memory vectors $\mathbf{m}=[\mathbf{z}_{h},\mathbf{s},\mathbf{e}]\in\mathbb{R}^{3\times d_{h}}$ by stacking $\mathbf{z}_{h},\mathbf{s},\mathbf{e}$. Then, we modify the computation of key vector $K$ and value vector $V$ in each self-attention layer by incorporating the memory vectors. Concretely, memory vectors $\mathbf{m}$ are prepended to the hidden states $\mathbf{H}^{l}$, denoted as $[\mathbf{m},\mathbf{H}^{l}]$, to calculate the key vector $K$ and value vector $V$ in each self-attention layer:

where $\mathbf{W}^{K},\mathbf{W}^{V}\in\mathbb{R}^{d_{h}\times d_{h}}$ are parameter matrices of key and value, respectively. The memory schema is equivalent to adding some virtual tokens to the response sequence at each layer and enables the decoder to attend all knowledge directly. Besides, we perform multi-head attention over the encoded context $\mathbf{H}^{c}$ and post $\mathbf{H}^{p}$ for each layer’s cross attention inspired by (Tu et al., 2022). In this way, the knowledge is injected into the decoder to guide the generation at each step.

The final learning objective is defined as the combination of CVAE loss in Eq. (5) and strategy prediction loss in Eq. (6)

where $\varphi$ denotes the parameters of PoKE, and $\lambda$ controls the degree of regularizing latent space by strategy. However, directly training this objective may suffer two optimizing challenges, i.e. KL-vanishing and strategy-unstablity. To alleviate them, we adopt two annealing methods including KL-annealing and Strategy-annealing.

We use the emotional support conversation dataset ESConv (Liu et al., 2021) to evaluate our method. ESConv contains a total of 1,053 dialogues and 31,410 utterances. Each conversation contains a seeker’s situation and a dialog context, and each utterance of supporter is annotated by a support strategy that is taken by the supporter. There are 8 different support strategies roughly uniformly distributed across the dataset. Due to the long turns in ESC, we cut each conversation into several pieces with 10 utterances and the last utterance is supporter’s response. For training and validation, we split the ESConv into the sets of training/validation/test with the proportions of 7:1.5:1.5. The statistics of original ESConv is shown in Table 8 and the split ESConv is shown in Table 2.

Following existing methods, we adopt automatic and human evaluation to evaluate our model and compare with strong baselines.

Since our main purpose is to explore the ESC task under the setting of no external knowledge, we place emphasis on those baselines that do not require any external knowledge. We compare our model with the following baselines, also including a model using external knowledge:

1. (1)

   Transformer (Vaswani et al., 2017). We use a standard Transformer model, which is trained from scratch by a negative likelihood objective.

2. (2)

   Multi-TRS (Rashkin et al., 2018). Multi-TRS is a multitask Transformer trained with an additional learning objective of predicting the target emotion.

3. (3)

   MoEL (Lin et al., 2019). MoEL models the distribution of emotion and assigns it to multiple Transformer decoders to softly combine their output.

4. (4)

   BlenderBot-Joint (Liu et al., 2021). BlenderBot-Joint is built on a pre-trained dialogue model, BlenderBot (Roller et al., 2021). It generates a strategy token and attaches it to the head of response to guide the desired response.

5. (5)

   MISC (Tu et al., 2022). MISC is also built on BlenderBot but requires external knowledge. It injects external knowledge by inferring the user’s fine-grained emotional status using COMET (Bosselut et al., 2019). When generating, they first predict a probability distribution of strategy and use it to obtain a weighted average representation of strategy for guiding generation.

Note that Multi-TRS and MoEL require the emotion label of seeker for training, so we use the conversation-level emotion label provided in ESConv dataset to train them. For a fair comparison, we apply the same hyperparameters for all baselines. The detail of implementation is illustrated in Appendix B

Specifically, the Transformer-based models, i.e. Transformer, Multi-TRS, and MoEL, do not perform well on ESConv. This is because these models are initialized with random parameters and trained on ESConv from scratch. Besides, their training objectives are irrelevant to the support strategy and the characteristics of emotional support, so they are hard to handle the challenging ESC task. As for the BlenderBot-based model, i.e. BlenderBot-Joint and MISC, they gain an improvement by a large margin compared to the previous baselines. It is due to the pre-trained dialogue model BlenderBot, which is trained on a large conversation dataset containing multiple conversation skills (Smith et al., 2020). For MISC, its D-1 and D-2 are comparatively higher, indicating that it tends to generate more diverse responses. This is because MISC incorporates varied information about seeker’s mental state from external knowledge in COMET and merges mixed strategies into one response. However, due to the issue of the local scope of conversation and the one-to-many relationship of strategy, there is still room for improvement.

Compared to those baselines without external knowledge, our proposed model PoKE improves significantly on the majority of metrics. This demonstrates that by effectively exploiting global prior knowledge from historical conversations, PoKE can get more clues to focus on seeker’s problem and generate more relevant responses. For the MISC that uses additional external knowledge, PoKE still can obtain a slight improvement in some metrics except diversity. However, PoKE almost achieves the same diversity performance with MISC. This benefits from using latent variable to model the one-to-many mapping relationship between context and support strategy, and latent variable makes it easier to sample infrequent strategies. Moreover, the technique of mixed strategy facilitates expressing diverse strategies in one response. As for PPL, both MISC and PoKE perform worse than BlenderBot-Joint. A recent work proves that PPL is not so reliable for evaluating text quality (Wang et al., 2022), and due to the insignificant difference of PPL (PoKE only drops by 0.13), we do not further refine the model.

Overall speaking, under the setting of no external knowledge, our proposed PoKE is superior to baselines on both automatic evaluation and human evaluation, which proves the superiority and effectiveness of PoKE. Besides, abundant prior knowledge and latent variable help provide better and diverse emotional support in dialogue system.

In this section, we explore the effect of exemplars in terms of denoising and quantity. To verify the denoised exemplars in Eq. (10), we implement another variant of PoKE without denoising exemplars, i.e. the representation of exemplars is calculated by averaging, i.e. $\mathbf{e}=\frac{1}{k}\sum^{k}_{i=1}\mathbf{h}^{e_{i}}$. The result is displayed in Table 4. All metrics drop when not denoising the exemplars. This demonstrates that the strategies of some retrieved exemplars are irrelevant to the current context, and need to be used selectively.

Figure 4 shows that as the number of exemplars increases, the overall performance tends to improve first and then decrease. This is because when exemplars are insufficient, PoKE lacks adequate reference information. When exemplars are too many, there is a lot of redundant and noisy information to distract the generation. Although PoKE ($k$ = 15) can utilize plentiful information to improve quality (higher B-2 and R-L), it pays the price of decreased fluency and diversity (very low PPL and D-1/2). In the end, we decide to retrieve 10 exemplars for each sample considering both the overall effect and training efficiency.

In this section, we adjust the structure of CVAE to explore the reasonable manner of utilizing strategy. For normal CVAE, namely the PoKE, strategy is only used as the output to regularize the latent space (Eq. (6)). Here, we consider a variant CVAE that strategy is merely as input condition to model latent variable, i.e. the recognition network becomes $q_{\phi}(\mathbf{z}|x,r,y)$ and $\mathcal{L}_{y}$ is ignored. We conduct quantitative and visualization experiments to compare these two structures of CVAE.

Table 5 shows that the overall performance of variant CVAE drops a lot, especially in diversity. Meanwhile, the visualization in Figure 5(b) exhibits that the latent space is independent of strategy, so strategy information is vanished from the latent variable. This demonstrates that only taking strategy as input is inadequate to model an informative latent space. In contrast, PoKE has a better diversity (Table 5) and can learn a meaningful latent space highly correlated with the support strategy (Figure 5(a)). This demonstrates that PoKE effectively regularizes the latent space and incorporates the informative latent variable into decoder to generate diverse responses.

To understand the importance of prior knowledge and latent variable for providing better emotional support, we conduct an ablation study to investigate the effect of the key components in PoKE. We design several variants of PoKE by removing some specific parts:

Table 6 shows the results of ablation studies. We can find that almost all variants perform worse than the PoKE, which verifies each component in PoKE. The results of PoKE w/o $\mathbf{e}$ and w/o $\mathbf{T}$ show that both generation quality and diversity get worse after removing prior knowledge. This suggests that explicitly using prior knowledge in historical conversations benefits more relevant responses, and plenty of various exemplars help generate responses with higher diversity. However, compared to PoKE, the PPL of PoKE w/o $\mathbf{e}$ improves slightly. We speculate that exemplars contain some token-level noise, thus impairing fluency. We leave the research of denoising exemplars at the token-level as future work. Regarding the PoKE w/o $\mathbf{e}$, D-1 and D-2 drop by a large margin. This result is as expected because the latent variable models the one-to-many mapping relationship of strategy. By sampling latent variable, randomness is introduced to strategy distribution and enables infrequent strategies to be considered.