t-SOT FNT: streaming multi-talker ASR with text-only domain adaptation capability

The basic idea of the t-SOT is to generate the tokens of all the speakers in a single sequence with a streaming fashion, according to the order of the token emission time in the audio. To achieve this, the serialized transcriptions are generated as the supervision of the t-SOT model. In the case of up to 2 concurrent speakers, a special token $\langle\text{cc}\rangle$ is inserted between the consecutive two tokens once they are spoken by different speakers, indicating the change of the virtual channel, i.e. active speaker. During inference, a post-processing step can be employed to form the transcriptions of two channels from the t-SOT decoding result by switching the output channel index when $\langle\text{cc}\rangle$ is emitted. For example, a decoding sequence of “hello how are $\langle\text{cc}\rangle$ i am $\langle\text{cc}\rangle$ you $\langle\text{cc}\rangle$ fine thank $\langle\text{cc}\rangle$ good $\langle\text{cc}\rangle$ you” can be reformatted to “hello how are you good” and “I am fine thank you” for the further processing. As the t-SOT only changes the supervised label of the ASR model, with the network structure, training loss function consistent as in conventional ASR. Various improvements from one speaker ASR can be easily integrated into t-SOT framework for further performance improvement, such as better network architecture, training scheduler etc. Meanwhile, we can generalize the t-SOT to support the mixed audio of up to more than 2 concurrent speakers by adding more special “channel change” tokens. However, given the overlaps of two speakers are the most common case in the real environment, we continue to prioritize our efforts on the scenario with up to 2 concurrent speakers.

FNT decomposes the posterior prediction of the output token set $\mathcal{O}$ into two parts, i.e., $\mathcal{O}=\{\mathcal{V}\cup\mathcal{S}\}$, where $\mathcal{V}$ and $\mathcal{S}$ refer to regular vocabulary tokens and special non-vocabulary token $\langle\text{blank}\rangle$, respectively. As shown in Figure.1, given the $t$-th frame output from the acoustic encoder $x_{t}^{\text{enc}}$ and the predicted token $y_{u-1}$ in the previous step, the prediction of the $\langle\text{blank}\rangle$ token follows the standard transducer framework by estimating $z_{t,u}^{s}$ as

where $g_{u}^{s}$ is the output of the special prediction network with a input of $y_{u-1}$. The regular vocabulary prediction $z_{t,u}^{v}$ is computed from $x_{t}^{\text{enc}}$ and the vocabulary predictor output $h_{u}^{v}$ as followings.

Finally, $z_{t,u}^{s}$ and $z_{t,u}^{v}$ are concatenated to form the distribution over $\mathcal{O}$ for the training with the transducer loss $\mathcal{L}_{\text{RNNT}}$. Besides, a negative log likelihood (NLL) loss is also applied on $z_{u}^{v}$ to enforce the vocabulary predictor to act as a standalone LM ¹¹1It’s equivalent to the way adopted in previous FNT work [26, 27] that applied cross entropy (CE) loss on vocabulary predictor’s output $h_{u}^{v}$.. As a result, the final objective function for FNT is defined as

where $\lambda$ is a hyper-parameter to control the weight of the LM loss. During the text only adaptation, the vocabulary predictor is adapted to a target domain based on the available text data. As prior work [27] has shown that adding Kullback-Leibler (KL) divergence loss between the outputs of the adapted vocabulary predictor and original ones can help to avoid the performance degradation on the general domain, the final objective function for text-only adaptation of the vocabulary predictor is

where $\omega$ is the weight of the KL divergence loss.

We used the same encoder structure for all the ASR models in this work, which contained 2 convolution layers and a 18-layer Conformer [6] encoder with the chunk-wise streaming mask, resulting in a latency of 160 msec. The attention dimension of the multi-head self-attention (MHSA) layer in each Conformer block was set to 512 with 8 heads and the 2048-dim feed-forward network (FFN) layer was adopted with the Gaussian error linear unit (GELU).

The prediction network of the single-talker ASR model (referred as “CT”) and t-SOT model (referred as “t-SOT CT”) consisted of a 2-layer 1024 dimensional long short-term memory (LSTM). In both naive and integrated t-SOT FNT models, the non-vocabulary prediction network was a 2-layer 512-dim LSTM and vocabulary predictor was a 2-layer 1536-dim LSTM. We applied a dropout rate of 0.1 to those prediction networks. The dimension of the joint network were all set to 512. The regular vocabulary contained 4003 word pieces thus the output size of single talker model and t-SOT models were 4004 and 4005, respectively.

We used 30 thousand (K) hours Microsoft in-house data, with the personally identifiable information removed, for the training of the single-talker ASR model. For t-SOT CT and t-SOT FNT model training, we used the combination of the multi-talker simulation data based on the 30K data and real meeting corpus from the training set of AMI [18] and ICSI [28] as well as the Microsoft internal meeting recordings. In the multi-talker simulation using the 30K data, we randomly mixed two utterances on-the-fly with a probability of 67%. For the rest of 33%, the original single-talker utterance was used. Finally, we used LibriSpeech text data (18 million (M) words), which was not included in the transcriptions of 30K training data, for the text-only adaptation experiments.

We evaluated our models on several datasets, including the general single-talker ASR test set, single-distant microphone audio from AMI and ICSI, and LibriSpeech-style datasets (LibriSpeech [29], LibriSpeechMix [30], LibriCSS [2]). Our general single-talker ASR test set covers various different application scenarios and consists of a total of 9.9M words. It is used to evaluate the single-speaker ASR accuracy before domain adaptation. On the other hand, AMI and ICSI were used to evaluate the multi-talker ASR accuracy before domain adaptation. We applied a causal logarithmic-loop-based automatic gain control (AGC) on AMI and ICSI to normalize audio volume. Finally, the LibriSpeech-style datasets were used to evaluate the effect of text-only adaptation in both single-talker and multi-talker ASR scenarios. We measured WER as an evaluation metric. For multi-talker test sets, we computed WER based on the algorithm proposed in [31, 19].

The 80-dim log mel-filterbank using 25 msec window and 10 msec hop size was extracted as the input feature for ASR models. We applied global mean and variance normalization. All the ASR models were trained on 16 NVIDIA V100 GPUs with AdamW optimizer. For CT model, we performed 500K-step training with a linear decay learning rate scheduler. 50K warm-up steps were used and the peak learning rate was set to $1.5e^{-3}$. t-SOT CT model was trained for 275K steps, using the CT model as the seed. Warm-up steps were removed and the peak learning rate was set to $2e^{-4}$.

To simplify the training scheme of t-SOT FNT, the encoder parameters were initialized from the well-trained CT or t-SOT CT models, and the peak learning rate was set to $4e^{-4}$ and $2e^{-4}$, respectively. Other configurations were kept same as t-SOT CT, including the training steps and batch size. Following the previous FNT work [27], in order to utilize more text data, we can also initialized vocabulary predictor of the FNT from a pre-trained LM with same architecture but was trained independently on a much larger text corpus. In this work, the text corpus for LM training contains 29 Billion words. When it comes with text-only adaptation, we adapt the vocabulary predictor of the integrated t-SOT FNT model for 10K steps on 4 V100 GPUs with a learning rate decayed from $1e^{-4}$ to 0. $\omega$ was set to 1 to keep the performance on general ASR test set. For ASR decoding, we used a beam size of 16.

The WERs on the general single-talker ASR set, AMI and ICSI are reported in Table 1, where four model structures are listed and compared. With the same training configuration, our proposed integrated t-SOT FNT model outperformed the naive t-SOT FNT on ICSI dataset while achieving comparable WERs on AMI and general single-talker ASR test sets. This result illustrates that the proposed multi-state vocabulary predictor of the integrated t-SOT FNT works on par with the vocabulary predictor of the conventional FNT, while the former provides a way to naturally integrating external LM into the vocabulary predictor.

Among the integrated t-SOT FNT variants, we observed that starting training from t-SOT CT encoder leads better performance than that from CT. This is expected as the former one has been optimized on multi-talker data. On the other hand, initializing the vocabulary predictor with a pre-trained LM on the larger scale text data resulted better WER on general ASR set, but the improvement was limited on AMI and ICSI. The reason might be that the training corpus of the LM covers the scenarios of the general ASR set but lacks sufficient meeting conversation data.

On general single-talker ASR test set, the best t-SOT FNT model (last row) achieved better result than t-SOT CT, closing the WER gap from the single-talker CT from 0.7% (=12.6%-11.9%) to only 0.3% (=12.2%-11.9%). On AMI data, t-SOT FNT achieved similar performance with t-SOT CT while on ICSI, it outperformed t-SOT CT model by a 0.4% and 0.3% on development and evaluation sets, respectively. These results demonstrated the capability to convert an existing t-SOT CT model to a t-SOT FNT model by keeping the accuracy of the original t-SOT CT model.

We picked the top two integrated t-SOT FNT models from Table 1 and performed text-only adaptation using text data from LibriSpeech training set. The results were reported in Table 2, where three data sets, LibriSpeech, LibriSpeechMix, and LibriCSS, were evaluated. The performance of the t-SOT FNT were improved after adaptation, not only on single-talker audio, but also on the multi-talker data regardless of if the data is simulated mixture (LibriSpeechMix) or real mixture (LibriCSS). Overall, the t-SOT FNT with LM initialization achieved best performance. Compared with t-SOT CT, t-SOT FNT brought a relative WER reduction of 8.4% and 7.5% on LibriSpeechMix and LibriCSS, respectively. In addition, compared with CT that achieved 5.9% and 11.9% on LibriSpeech, t-SOT FNT achieved significantly better WERs of 4.7% and 10.4% by enjoying the text-only adaptation capability.

Among the t-SOT FNT models, we observed that the text-only adaptation closed the gap between the LM initialization on all the three test sets. For example, the relative averaged WER difference on LibriSpeechMix was reduced from 8.3% to 2.5%, and that on LibriSpeech was reduced from 14.6% to 4.0%. Overall, our results demonstrated that the proposed t-SOT FNT enjoyed the advantage of the vocabulary predictor where the general single-talker ASR accuracy was improved by utilizing a powerful LM, and the accuracy was further improved by the text-only domain adaptation.