Back to the Future: Bidirectional Information Decoupling Network
for Multi-turn Dialogue Modeling

Our model is implemented based on pre-trained language models (PrLMs), which have achieved remarkable results on many natural language understanding (NLU) tasks and are widely used as a text encoder by many researchers Wu et al. (2022); Li et al. (2022). Based on self-attention mechanism and Transformer Vaswani et al. (2017), together with pre-training on large corpora, PrLMs have a strong capability of encoding natural language texts into contextualized representations. To name a few, BERT Devlin et al. (2019), ALBERT Lan et al. (2020) and ELECTRA Clark et al. (2020) are the most prominent ones for NLU; GPT Radford et al. (2019), T5 Raffel et al. (2020) and BART Lewis et al. (2020) are the most representative ones for natural language generation. In our work, we select BERT, ELECTRA, and BART as the encoder backbones of our model.

There are several previous studies on multi-turn dialogue modeling for different downstream tasks. Li et al. (2021b) propose DialoFlow, which utilizes three novel pre-training objectives to capture the information dynamics across dialogue utterances for response generation. Zhang et al. (2021) design a Pivot-oriented Deep Selection mode (PoDS) to explicitly capture salient utterances and incorporate common sense knowledge for response selection. Liu et al. (2021a) propose a Mask-based Decoupling-Fusing Network (MDFN), which adopts a mask mechanism to explicitly model speaker and utterance information for two-party dialogues. Liu et al. (2021b) propose a Graph Reasoning Network (GRN) to explicitly model the reasoning process on multi-turn dialogue response selection. Different from all these detailed works focusing on specific tasks, in this work, we devote ourselves to a universal dialogue modeling enhancement by effectively capturing the long-term ignored temporal features of dialogue data.

Given a set of input tokens $\mathbb{X}=\{w_{1},w_{2},...,w_{n}\}$, we first embed them into a high dimensional embedding space using an embedding look-up table $\phi$: $E_{T}=\phi(\mathbb{X})=\{\bm{e_{1}},\bm{e_{2}},...,\bm{e_{n}}\}\in\mathcal{R}^{n\times d}$, where $d$ is the hidden size defined by the PrLM. After that, positional embedding $E_{P}$ and segment embedding $E_{S}$ will be added to $E_{T}$ to model the positional and segment information: $E=E_{T}+E_{P}+E_{S}$. $E$ is later fed into the Transformer layers to obtain the contextualized representations $H$. We first introduce the multi-head self-attention (MHSA) mechanism:

$$\begin{split}&\operatorname{Attn}(\bm{Q},\bm{K},\bm{V})=\operatorname{softmax}(\frac{\bm{QK^{T}}}{\sqrt{d_{k}}})\bm{V}\\ &\operatorname{head_{i}}=\operatorname{Attn}(E\bm{W_{i}^{Q}},E\bm{W_{i}^{K}},E\bm{W_{i}^{V}})\\ &\operatorname{MultiHead}(H)=[\operatorname{head}_{1},\dots,\operatorname{head}_{h}]\bm{W^{O}}\end{split}$$

(1)

where $\bm{W_{i}^{Q}}\in\mathcal{R}^{d\times d_{q}}$, $\bm{W_{i}^{K}}\in\mathcal{R}^{d\times d_{k}}$, $\bm{W_{i}^{V}}\in\mathcal{R}^{d\times d_{v}}$, $\bm{W^{O}}\in\mathcal{R}^{hd_{v}\times d}$ are transformation matrices with trainable weights, $h$ is the number of attention heads, and $[;]$ denotes the concatenation operation. $d_{q}$, $d_{k}$, $d_{v}$ are the hidden sizes of the query vector, key vector and value vector, respectively. MHSA is the foundation of Transformer, which is easier to train and can model long distance dependencies. Given the input embeddings $E$, the Transformer layers $\operatorname{Trans}(E)$ is formulated as follows:

		$\displaystyle H^{0}=E\in\mathcal{R}^{n\times d}$		(2)
		$\displaystyle H^{i}_{tmp}={\rm LN}({\rm MultiHead}(H^{i-1})+H^{i-1})$
		$\displaystyle H^{i}={\rm LN}({\rm FFN}(H^{i}_{tmp})+H^{i}_{tmp})$
		$\displaystyle{\rm FFN}(x)={\rm ReLU}(x\bm{W_{1}}+\bm{b_{1}})\bm{W_{2}}+\bm{b_{2}}$

where ${\rm LN}$ is layer normalization, ${\rm ReLU}$ is a non-linear activation function and $\bm{W_{1}}$, $\bm{W_{2}}$, $\bm{b_{1}}$, $\bm{b_{2}}$ are trainable linear transformation matrices and bias vectors, respectively.

We denote the stack of $L$ Transformer layers as $\operatorname{Trans-L}$, the final representation output by the PrLM encoder is:

$$H=\operatorname{Trans-L}(E)\in\mathcal{R}^{n\times d}$$

(3)

Given the token representations output by the PrLM encoder, the Bidirectional Information Decoupling Module will decouple them into three channels in a back-and-forth way. We first introduce a masked Transformer layer $\operatorname{MTrans}(E,M)$ by modifying the first equation on Eq. (1) to:

$$\operatorname{Attn}(\bm{Q},\bm{K},\bm{V})=\operatorname{softmax}(\frac{\bm{QK^{T}}}{\sqrt{d_{k}}}+M)\bm{V}$$

(4)

where $M$ is an $n\times n$ attention mask matrix. The function of $M$ is to convert the original fully-connected attention graphs to partially-connected ones, so that each token will be forced to only focus on part of the input sequence. Here we introduce three kinds of attention masks, which guide the decoupling process of the future-to-current channel, current-to-current channel, and past-to-current channel, respectively. Specifically, suppose $I(i)$ means the index of the utterance that the $i_{th}$ token belongs to, the three kinds of masks are obtained by:

		$\displaystyle M_{f2c}[i,j]=\left\{\begin{array}[]{cl}0,&\text{if}\ I(i)<I(j)\\ -\infty,&\text{otherwise}\end{array}\right.$		(5)
		$\displaystyle M_{c2c}[i,j]=\left\{\begin{array}[]{cl}0,&\text{if}\ I(i)=I(j)\\ -\infty,&\text{otherwise}\end{array}\right.$
		$\displaystyle M_{p2c}[i,j]=\left\{\begin{array}[]{cl}0,&\text{if}\ I(i)>I(j)\\ -\infty,&\text{otherwise}\end{array}\right.$

where $M_{f2c}$, $M_{c2c}$ and $M_{p2c}$ are future-to-current mask, current-to-current mask and past-to-current mask, respectively. After obtaining these masks, three parameter-independent $\operatorname{MTrans-1}(H,M)$ are applied to decouple the original representation $H$ as follows:

		$\displaystyle H_{f2c}=\operatorname{MTrans-1^{f2c}}(H,M_{f2c})$		(6)
		$\displaystyle H_{c2c}=\operatorname{MTrans-1^{c2c}}(H,M_{c2c})$
		$\displaystyle H_{p2c}=\operatorname{MTrans-1^{p2c}}(H,M_{p2c})$

Note that there are tokens who has no connections to any tokens in certain channels, e.g. the tokens of the first utterance has no connections to other tokens in past-to-future channel since there are no previous utterances. To handle this case, we simply ignore the invalid representations (gray squares in Figure 2) by adding a fusion mask during the fusion process, which will be introduced in Section 3.3.

After the decoupling process, $H_{p2c}$ contains the information of the influence that the past dialogue history brings about to the current utterance, or in other words, it reflects what the current utterance should be like given the past dialogue history. $H_{f2c}$ contains the information of the influence that the current utterance brings about to future dialogue contexts, or put it another way, it reflects what kind of current utterance will lead to the development of the future dialogue. Finally, $H_{c2c}$ contains the information of the original semantic meaning resides in the current utterance. By explicitly incorporating past and future information into each utterance, our BIDM is equipped with the ability to capture temporal features in dialogue contexts.

We first introduce the Mixture of Experts (MoE) proposed by Jacobs et al. (1991). Specifically, $m$ experts $\{f_{i}(x)\}_{i=1}^{m}$ are learned to handle different input cases. Then a gating function $G=\{g_{i}(x)\}_{i=1}^{m}$ are applied to determine the importance of each expert dynamically by assigning weights to them. The final output of MoE is the linear combination of each expert:

$$MoE(x)=\sum_{i=1}^{m}\ g_{i}(x)\cdot f_{i}(x)$$

(7)

In this work, $\operatorname{MTrans^{f2c}}$, $\operatorname{MTrans^{c2c}}$ and $\operatorname{MTrans^{p2c}}$ are treated as three experts. We design the gating function similar as Liu et al. (2021a) that utilizes the original output $H$ to guide the calculation of expert weights. In detail, we first calculate a heuristic matching representation between $H$ and the three outputs of Section 3.2, respectively, then obtain the expert weights $G$ by considering all three matching representations and calculate the final fused representation $H_{e}$ as follows:

		$\displaystyle\operatorname{Heuristic}(X,Y)=[X;Y;X-Y;X\odot Y]$		(8)
		$\displaystyle S_{f}=\operatorname{ReLU}(\operatorname{Heuristic}(H,H_{f2c})\bm{W_{f}}+\bm{b_{f}})$
		$\displaystyle S_{c}=\operatorname{ReLU}(\operatorname{Heuristic}(H,H_{c2c})\bm{W_{c}}+\bm{b_{c}})$
		$\displaystyle S_{p}=\operatorname{ReLU}(\operatorname{Heuristic}(H,H_{p2c})\bm{W_{p}}+\bm{b_{p}})$
		$\displaystyle G=\operatorname{Softmax}([S_{f};S_{c};S_{p}]\bm{W_{g}}+M_{g})\in\mathcal{R}^{n\times d\times 3}$
		$\displaystyle H_{e}=\operatorname{Sum}(\operatorname{Stack}(H_{f2c};H_{c2c};H_{p2c})\odot G)$

Here $H_{e}\in\mathcal{R}^{n\times d}$, $\odot$ represents element-wise multiplication, $\bm{W_{f}}$, $\bm{W_{c}}$, $\bm{W_{p}}\in\mathcal{R}^{4d\times d}$ and $\bm{b_{f}}$, $\bm{b_{c}}$, $\bm{b_{p}}\in\mathcal{R}^{d}$ are trainable transformation matrices and bias vectors, respectively. $\bm{W_{g}}\in\mathcal{R}^{3d\times d\times 3}$ is a trainable gating matrix that generates feature-wise expert scores by considering all three kinds of information. $M_{g}$ is a fusion mask added for ignoring invalid tokens, which is introduced in Section 3.2.

After incorporating future-to-current, past-to-current and current-to-current information, we obtain temporal-aware representation $H_{e}$, which can be used for various dialogue-related tasks described in Section 4.

We adopt Multi-Turn Dialogue Reasoning (Mutual, Cui et al. 2020) for response selection, Molweni Li et al. (2020a) for extractive QA over multi-turn multi-party dialogues, and DIALOGSUM Chen et al. (2021) for dialogue summarization.

MuTual is proposed to boost the research of the reasoning process in retrieval-based dialogue systems. It consists of 8,860 manually annotated two-party dialogues based on Chinese student English listening comprehension exams. For each dialogue, four response candidates are provided and only one of them is correct. A plus version of this dataset is also annotated by randomly replacing a candidate response with safe response (e.g. I didn’t hear you clearly), in order to test whether a model is able to select a safe response when the other candidates are all inappropriate. This dataset is more challenging than other datasets for response selection since it requires some reasoning to select the correct candidate. This is why we choose it as our benchmark for the response selection task.

Molweni is a dataset for extractive QA over multi-party dialogues. It is derived from the large-scale multi-party dialogue dataset — Ubuntu Chat Corpus Lowe et al. (2015), whose main theme is technical discussions about problems on the Ubuntu system. In total, it contains 10,000 dialogues annotated with questions and answers. Given a dialogue, several questions will be asked and the answer is guaranteed to be a continuous span in the dialogue context. The reason we choose this dataset as a benchmark for retrieval style task is that we want to test whether our model still holds on multi-party dialogue contexts.

DIALOGSUM is a large-scale real-life dialogue summarization dataset. It contains 13,460 daily conversations collected from different datasets or websites. For each dialogue context, annotators are asked to write a concise summary that conveys the most salient information of the dialogue from an observer’s perspective. This dataset is designed to be highly abstractive, which means a generative model should be adopted to generate the summaries.

On the MuTual dataset, ELECTRA is adopted as the PrLM encoder for a fair comparison with previous works. We follow Liu et al. (2021a) to get the dialogue-level representation $H_{d}$ from $H_{e}$. We first obtain the utterance-level representations by applying a max-pooling over the tokens of each utterance, then use a Bidirectional Gated Recurrent Unit (Bi-GRU) to summarize the utterance-level representations into a single dialogue-level vector. For one dialogue history with four candidate responses, we concatenate them to form four dialogue contexts and encode them to obtain $H_{D}=\{H_{d}^{i}\}_{i=1}^{4}\in\mathcal{R}^{d\times 4}$. Given the index of the correct answer $i^{target}$, we compute the candidate distribution and classification loss by:

		$\displaystyle P_{D}=\operatorname{Softmax}(\bm{w_{d}}^{T}H_{D})\in\mathcal{R}^{4}$		(9)
		$\displaystyle\mathcal{L}_{D}=-log(P_{D}[i^{target}])$

where $\bm{w_{d}}\in\mathcal{R}^{d}$ is a trainable linear classifier and $\mathcal{L}_{D}$ is the cross entropy loss.

On the Molweni dataset, BERT is adopted as the PrLM encoder for a fair comparison with previous works. We simply regard the question text as a special utterance and concatenate it to the end of the dialogue history to form the input sequence. After obtaining $H_{e}$, we add two linear classifiers to compute the start and end distributions over all tokens. Given the start and end positions of the answer span $[a_{s},a_{e}]$, cross entropy loss is adopted to train our model:

		$\displaystyle P_{start}=\operatorname{Softmax}(H_{e}\bm{w_{s}}^{T})\in\mathcal{R}^{n}$		(10)
		$\displaystyle P_{end}=\operatorname{Softmax}(H_{e}\bm{w_{e}}^{T})\in\mathcal{R}^{n}$
		$\displaystyle\mathcal{L}_{SE}=-(\text{log}(P_{start}[a_{s}])+\text{log}(P_{end}[a_{e}]))$

where $\bm{w_{s}}$ and $\bm{w_{e}}\in\mathcal{R}^{d}$ are two trainable linear classifiers.

On the DIALOGSUM dataset, BART is chosen as our backbone since it is one of the strongest generative PrLMs. Different from the previous two PrLMs, BART adopts an encoder-decoder architecture where the encoder is in charge of encoding the input texts and the decoder is responsible for generating outputs. Therefore, we add our BIDM after the encoder of BART. Note that BART is pre-trained on large corpora using self-supervised text denoising tasks, hence there is a strong coupling on the pre-trained parameter weights between the encoder and decoder. Under this circumstance, simply adding our BIDM after the encoder will destroy the coupling between encoder and decoder, resulting in the decline of model performance. To tackle this problem, we propose novel a copy-and-reuse way to maintain the parameter-wise coupling between the encoder and decoder. Specifically, instead of using randomly initialized decoupling layers, we reuse the last layer of BART encoder and load the corresponding pre-trained weights to initialize the future-to-current, current-to-current, and past-to-current decoupling layers, respectively. We train this model by an auto-regressive language model loss:

$$\mathcal{L}_{G}=-\sum_{t=1}^{N}\log p\left(w_{t}\mid\bm{\theta},w_{<t}\right)$$

(11)

where $\bm{\theta}$ is the model parameters, $N$ is the total number of words in the target summary and $w_{t}$ is the token at time step $t$. We also conduct experiments on the SAMSum Gliwa et al. (2019) dataset, and the results are presented in Appendix B.

For hyper-parameter settings and more details about our experiments, please refer to Appendix A.

In this section, we will briefly introduce the baseline models and evaluation metrics, then present the experimental results on different datasets.

To investigate the effectiveness of temporal modeling, we remove BIDM to see how it affects the performance. A sharp performance drop of $2.3\%$ is observed on R@1, demonstrating the necessity and significance of explicit temporal modeling. In order to probe into whether the performance gain comes from the increment of model parameters, we conduct experiments by simply replacing the three kinds of masks defined in Eq. (5) with all-zero masks (fully-connected attention graphs). We see from the table that the increment of parameters does add to the performance. Nevertheless, it is sub-optimal compared with explicitly modeling the temporal features by our BIDM.

We also remove MoE to see whether the dynamic fusion mechanism helps. Specifically, we replace this module with a simple mean pooling over the three decoupled representations. Result shows that MoE makes a huge contribution to the final result. To explore the effect that the task-specific design, Bi-GRU, brings about to our model, we remove the Bi-GRU and simply average the utterance representations to get the dialogue-level vector. We see from the table that Bi-GRU does have positive effects on the final performance, yet only to a slight extent compared with other modules.

When it comes to bidirectional temporal modeling, the simplest way is to use Bidirectional Recurrent Neural Networks (Bi-RNNs). To investigate whether BiDeN can be replaced by these naive temporal modeling methods, we conduct experiments by adding Bi-LSTM or Bi-GRU on top of PrLMs instead of BiDeN. We see from Table 6 that utilizing Bi-RNNs can improve the performance slightly, but they are far behind BiDeN. This is because RNNs model the bidirectional information only at token-level, while BiDeN models them by explicitly modeling the utterance boundary with attention masks, which is more consistent with the data characteristics of dialogue texts.

Intuitively, with longer dialogue contexts comes more complicated temporal features. Based on this point, we analyze the model performance with regard to the number of utterances in a dialogue. As illustrated in Figure 3, the scores first increase from short dialogues to medium-length dialogues. This is because medium-length dialogues contain more information for response matching than short ones. For long dialogues, the baseline model suffers a huge performance drop (see the blue and green lines), while our BiDeN keeps bringing performance improvement, demonstrating a strong ability of it to capture complicated temporal features.

To intuitively investigate how BiDeN works, we visualize the attention weights of both current-to-past and current-to-future attentions. Figure 4 (a) shows the current-to-past attention weights. We see that the utterance My boss told me not to go to work again tends to focus on not in a good mood of the previous utterance, which is a causal discovery. Similarly, the last utterance I am so sorry that you lost your job focuses more on not in a good mood and not to go to work. Figure 4 (b) shows an example of current-to-future attention, which is an incorrect response example taken from MuTual dataset. We see that the current utterance pays great attention on the name Jane, which is supposed to be Joe. This observation indicates that BiDeN is capable of detecting the logical errors in the future responses that contradict previous utterances. For more visualizations, please refer to Appendix C.