Conversational Speech Recognition by Learning
Conversation-level Characteristics

The input of our model consists of three parts at the $k$-th sentence in the conversation: the speech feature of current sentence $X_{k}$, the target text $Y_{k}$ and the contextual input feature $\{C_{\text{role}},C_{\text{dia}}\}$. Here, $C_{\text{role}}$ is transcripts of the current speaker, and $C_{\text{dia}}$ is all historical transcript in the current conversation. For example, $C_{role}=(Y_{1},Y_{3},Y_{5},...,Y_{k-2})$ and $C_{dia}=(Y_{1},Y_{2},Y_{3},...,Y_{k-1})$. The text data is processed in a format in which each person speaks one sentence in turn, so role preference can also be expressed as $C_{role}=(Y_{2},Y_{4},Y_{6},...,Y_{k-1})$ for another speaker. In order to distinguish different sentences, we add start symbol $<sos>$ and end symbol $<eos>$ at the beginning and ending of each sentence. Then, all text inputs will be represented as word embedding vectors.

Recently, Gulati et al. combined transformers and convolutional neural networks as Conformer [13] to simultaneously capture local and global contextual information in ASR tasks, leading to superior performance. Here, we stack $N_{\text{con}}$ conformer blocks as our speech encoder. Specifically, each conformer block includes a multi-head self-attention module (MHSA), a convolution module (CONV) and a feed-forward module (FFN). Assuming the input of the i-th block is $\textbf{h}_{i}$, operations in this block can be expressed as:

The latent variational module consists of a text encoder and two specific VAEs: role VAE and topic VAE. Each VAE consists of multiple transformer layers, a posterior network and a prior network, as shown in the left half of Figure 1. These two VAEs characterize role preference and topical coherence in conversations by learning role-specific latent vectors $Z_{R}$ and topic-specific latent vectors $Z_{D}$.

At training, the posterior distribution conditions on sentences related to the current role. Through KL divergence, the prior network can learn a role-specific distribution by approximating the posterior network [9]. The variable distribution of the posterior network is extracted as

where

here $\textbf{h}_{\text{y}}$ is the vectors calculated by the transformer layers of posterior network. The conditional probability $q_{\phi}$ is learned by the posterior network when training. However, we fit this distribution through a prior network at inference, so as to avoid the dependence on the recognition result of current sentence and the deviation of the recognition result from the real result.

We use an attention decoder in the proposed model. As shown in Figure 1, we get the latent variables$\{\textbf{z}_{\text{role}},\textbf{z}_{\text{dia}}\}$ either from the posterior networks or the prior networks. A transformer layer $\textbf{M}_{\text{trans}}$ is used to merge these intermediate vectors into the decoded states:

where $\textbf{h}_{s}$ is the hidden state of decoder. Then, we send $\textbf{g}_{t}$ to a softmax layer to get the probability of target chars.

We adopt a two-stage training strategy. We first train a sentence-level speech recognition model with the cross-entropy objective:

Then, we finetune the model by minimizing $L_{\text{ce}}$ and the follow objective:

We rescore the output of the ASR model in the process of attention rescoring [28]. Specifically, we classify conversations in the training set into $m$ topics by the topic model CombinedTM. Each topic contains $j$ words like $(v_{1}^{1},v_{2}^{1},...,v_{j}^{1},...,v_{j}^{m})$. Keywords in all topics do not overlap each other. For the top-n sentences $(S_{1},...,S_{n})$ generated by the speech recognition model, we send them to the topic model trained by the transcripts of speech dataset, and get the probability vectors of the sentence attribution to each topic$(\textbf{d}_{1},\textbf{d}_{2},...,\textbf{d}_{m})$, each vector has $m$ dimensions. For the $t$-th word $w_{n}^{t}$ in $S_{n}$, if $w_{n}^{t}$ in the keywords of $b$-th topic we generated, the score of word $w_{n}^{t}$ $s_{\text{old}}(w)$ is recalculated as

At attention rescoring, we add $s_{\text{new}}(w)$ to the attention:

where $s_{\text{attn}}(w)$ is the score calculated by the rescoring decoder.

Then we use the new score to reorder the output sentences, calculate the final sentence score with $s_{\text{sen}}=\begin{matrix}\sum_{i=1}^{t}{s_{\text{final}}({w}_{i}})\end{matrix}$. The sentence with the highest score is considered as the final recognition result.

We conduct our experiments on two Mandarin conversation datasets – HKUST [11] and DATATANG (DDT). The HKUST dataset contains 200 hours of speech data. The dev set is used to verify the recognition results. The DDT dataset contains 350 hours of speech data. The dev and test sets are used to verify the recognition results.

We obtain the topic boundary information and speaker information of each round of dialogues through the additional tags of the data to distinguish speakers and judge the conversion of the topic. For each corpus, the detail configurations of our acoustic features and Conformer model are same as the ESPnet Conformer recipes [29] $(\text{Enc}=12,\text{Dec}=6,d^{\text{ff}}=2048,\text{H}=4,d^{\text{att}}=256)$. We use 3653 and 3126 characters as output units for HKUST and DDT respectively.

We train our models with ESPnet [29]. Speed perturbation at ratio 0.9, 1.0, 1.1 with SpecAugment [30] is used for data augmentation. The baseline results are trained on independent sentence level, without speaker and context information.

In our experiment, we use a 2-layer text encoder and a 2-layer transformer as the feature extractor of the VAEs as shown in the left of Figure 1. Latent variables with 100 dimensions are used to represent speaker information and topic information respectively. In the rescoring experiments, we divide the conversations in HKUST into 50 topics. For DDT, we only divide the conversations into 3 topics as the topics are highly coterminous. In addition, a session-level RNNLM [2] based on the training set text is applied.

We reproduce a comparative model, Hierarchical Transformer [7] (H-Transformer), which consists of a text encoder with 4 transformer layers and a Conformer ASR module in our setups.

Table 1 shows the results of our experiments. We can find that both the role VAE and topic VAE show superior results on the final recognition accuracy. By combining them together, we can even achieve further improvement.

Since the data set contains open-domain topics, the session-level language model makes the final recognition result worse on HKUST. In addition, we also find that the topic-based rescoring operation has a positive effect on both data sets. Meanwhile, on the open-domain data set HKUST, the topic model rescoring is worse than that of the DDT data set with more obvious topic consistency.

In general, we can find that after adding the variational module and the rescoring module, the recognition performance has been greatly improved, resulting a relative 12% improvement on DDT set.

In a conversation, even on the same topic, as the number of conversation rounds increases, the speaker’s speaking habits and the topics they are talking about will also change. At the same time, more recent texts may contain historical information that is more helpful for the recognition of the current sentence. Therefore, we explore the role context length $j$ and topic context length $k$ of the input for the VAEs respectively on the HKUST dataset.

As shown in Table 2 and 3, we design experiments to explore the impact of context length on model performance, and find that when $k=3$ or $j=2$, the proposed model reaches the lowest CERs.