VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition

VarArray is a recently proposed neural network-based CSS front-end that converts long-form continuous audio input from a microphone array into $K$ streams of overlap-free audio signals, where $K$ is usually set to two [15]. VarArray can cope with the input from an arbitrary microphone array without retraining the model parameters. This “array-geometry-agnostic” property is enabled by interleaving temporal-modeling layers and geometry-agnostic cross-channel layers. The model takes a multi-channel short-time Fourier transform as an input to output the time-frequency masks for $K$ speech sources and two noise sources, corresponding to stationary and transient noise sources. The estimated masks are used to form minimum variance distortion-less response (MVDR) beamformers to produce $K$ overlap-free signals. Streaming processing is realized based on a sliding window, where $T^{s}$ sec of $K$ overlap-free signals are generated for every $T^{s}$ sec by using a $T^{f}$-sec look-ahead. In our experiment, both $T^{s}$ and $T^{f}$ were set to 0.4, thereby causing the total algorithmic latency to be 0.8 sec. As a pre-processing step, we apply real-time dereverberation based on the weighted prediction error method [29]. We refer the readers to [15] for more details.

The t-SOT method was proposed to enable multi-talker overlapping speech recognition with streaming processing [24]. With t-SOT, only up to $M$ speakers can be active simultaneously. The following description assumes $M=2$ for simplicity. With t-SOT, the transcriptions for multiple speakers are serialized into a single sequence of recognition tokens (e.g., words or subwords) by sorting them in a chronological order. A special token $\langle\mathrm{cc}\rangle$, which indicates a change of ‘virtual’ output channels, is inserted between two adjacent words spoken by different speakers (such examples can be found in the middle of Fig. 1). A streaming end-to-end ASR model [30] is trained based on pairs of such serialized transcriptions and the corresponding audio samples. During inference, an output sequence including $\langle\mathrm{cc}\rangle$ is generated by the ASR model in a streaming fashion, which is then ‘deserialized’ to generate separate transcriptions by switching between the two virtual output channels at each encounter with the $\langle\mathrm{cc}\rangle$ token (see the top of Fig. 1). The t-SOT model outperformed prior multi-talker ASR models even with streaming processing. See [24] for more details.

Fig. 1 shows the overview of the proposed framework. The input multi-channel audio signals are first fed into the VarArray front-end, which generates two audio streams in a streaming fashion. These audio streams are then provided to the two-channel-input streaming end-to-end ASR model to generate the serialized transcription based on the t-SOT approach. Finally, it is deserialized to produce multi-talker transcriptions.

The proposed combination of VarArray and t-SOT ASR has multiple advantages. Firstly, thanks to the array-geometry-agnostic property of VarArray, the system is applicable to various microphone arrays without retraining. Secondly, by executing the VarArray processing on the microphone array device, we can limit the device-to-ASR-server data transmission needs to only two audio signals, which provides a large practical benefit. Thirdly, training our ASR model is much simpler than other multi-channel multi-talker ASR models as the latter ones require simulating multi-channel training data with realistic phase information [19, 26, 27] while our model does not. Finally, our ASR model can be easily fine-tuned by using the VarArray outputs for real multi-channel recordings and the corresponding time-annotated reference transcriptions. This is not the case with conventional systems that apply a single-talker ASR model to each separated signal. This is because, in order to fine-tune the single-talker model based on the separated signals, the reference transcriptions must be generated so that they match the separated signals, which can be tricky because the front-end may even split one utterance into different streams.

Fig. 2 shows the proposed architecture of the two-channel end-to-end ASR model. Our model is a modified version of the transformer transducer (TT) [31] and is obtained by splitting the transformer encoder to take in two audio signals. First, the audio input of each channel is converted to the log mel-filterbank, followed by two convolution layers. The output from the last convolution layer is then fed into a stack of transformer layers with cross-channel transformation. Let $x_{l,c}$ denote the output of the $l$-th transformer layer in the $c$-th channel, where $c\in\{0,1\}$. The cross-channel transformation output, $\hat{x}_{l,c}$, is calculated in the following two different ways:

	$\displaystyle\text{Scaling:}\;\;\;\;\;\;\;\;\;\;\;\;\>\,\hat{x}_{l,c}=x_{l,c}+s_{l}\cdot x_{l,(1-c)};~{}\text{or}$		(1)
	$\displaystyle\text{Projection:}\;\;\;\;\;\hat{x}_{l,c}=x_{l,c}+\phi(W_{l}\cdot x_{l,(1-c)}+b_{l}),$		(2)

where $s_{l}$, $W_{l}$, and $b_{l}$ are a learnable scalar, a weight matrix, and a bias vector, respectively, for the $l$-th layer while $\phi()$ denotes a rectified linear unit activation function. The outputs of the $N$-th transformer layer are summed across the channels, which is further processed by additional $(L-N)$ layers of transformer, with $L$ being the total number of the transformer encoders. On top of the encoder, a recurrent neural network transducer (RNN-T)-based decoder is applied to generate the serialized transcription described earlier.

Note that the parameters for the two-branched encoders, i.e., the parameters of the convolution layers, the first $N$ transformer layers, and the cross-channel transformations, are shared among the two channels. Therefore, the parameter count of the two-channel TT is almost the same as that of the conventional single-channel TT, where the difference comes only from the cross-channel transformations.

Although VarArray is supposed to produce clean overlap-free signals, the actual outputs contain different kinds of processing errors and artifacts. In particular, two types of errors, called leakage and utterance split, are unique to CSS and VarArray. The leakage refers to generating the same speech content from both output channels. The utterance split refers to splitting one utterance into two or more pieces and generating them from different output channels. While these errors have a detrimental effect on existing systems [15], we can have the two-channel TT learn to handle such erroneous input. For this purpose, we propose a three-stage training scheme as follows, which considers the fact that we have a limited amount of real multi-channel training data while monaural single-talker ASR training data are available in large quantities. Below, let $S_{0}$ denote the large-scale single-talker ASR training set.

In the first stage, a conventional (i.e. single-channel) t-SOT multi-talker TT is trained by using a large-scale monaural multi-talker training set, $S_{1}$, which is derived from $S_{0}$ by simulation. By following [25], each multi-talker training sample of $S_{1}$ can be generated by mixing randomly chosen two single-talker audio samples from $S_{0}$ with a random delay. During the model training, the samples are randomly picked from $S_{0}$ and $S_{1}$ at a 50:50 ratio.

In the second stage, a two-channel TT is initialized by copying the parameters of the t-SOT TT trained in the first stage. The parameters of the cross-channel transformation are randomly initialized. The two-channel TT is then trained by using a large-scale two-channel training dataset, $S_{2}$, which is obtained from $S_{0}$ by the following simulation method. (i) Randomly sample one utterance $u_{1}$ from $S_{0}$, and put it in the $c$-th channel with $c$ being randomly drawn from $\{0,1\}$. (ii) With a 50% chance, randomly sample another utterance $u_{2}$ from $S_{0}$, and put it in the $(1-c)$-th channel after prepending a random delay of $d\sim\mathcal{U}(0,\mathrm{len}(u_{1}))$, where $\mathcal{U}$ is the uniform distribution. (iii) Randomly split the generated two-channel audio into $p$-sec chunks, where $p\sim\mathcal{U}(5,50)$, and swap the signals of the even-numbered chunks. This simulates the utterance split phenomenon. (iv) For each channel, randomly chop the audio into $q$-sec chunks, where $q\sim\mathcal{U}(1,5)$. Then, for each channel of the even-numbered chunks, mix the audio signal of the other channel after scaling the volume by $v\sim\mathcal{U}(0,0.2)$. This step simulates the leakage phenomenon.

In the third stage, the two-channel TT is further fine-tuned by using real multi-channel data. VarArray is first applied to the multi-channel training samples to generate two separated audio signals. The two-channel TT from the second stage is then fine-tuned by using the two separated-signals and the corresponding transcription. During the fine-tuning, the VarArray parameters are frozen.

We used the AMI meeting corpus [2] for the fine-tuning and evaluation of the proposed system. The corpus contains approximately 100 hours of meeting recordings, each containing three to five participants. The audio was recorded with an 8-ch microphone array. We adopted the text formatting and data split of the Kaldi toolkit [32]. There are training, development and evaluation sets, each of which contains 80.2 hr, 9.7 hr, and 9.1 hr of recordings. The training set was used for the ASR model fine-tuning. The development and evaluation sets were used for the WER calculation. We applied causal logarithmic-loop-based automatic gain control (AGC) to normalize the significant volume differences among different recordings. AGC was applied after the front-end processing.

In addition to the AMI corpus, we used 64 million anonymized and transcribed English utterances, totaling 75K hours [25], as $S_{0}$ for the first- and second-stage pre-training. The data consist of monaural audio signals from various domains, such as voice command and dictation, and their reference transcriptions. Each training sample is supposed to contain single-talker speech while it could occasionally contain untranscribed background human speech.

In the evaluation, we followed the utterance and utterance-group segmentations proposed in [25]. The utterance group is defined as a set of adjacent utterances that are connected by speaker overlap regions. By following [25], the utterance segmentation was used for the the single-talker ASR model evaluation while the utterance-group segmentation was used for the multi-talker ASR model evaluation. For the WER calculation, we used the multiple dimension Levenshtein edit distance calculation implemented in ASCLITE [33].¹¹1To reduce the computational complexity, unlike the original way of using ASCLITE where the multiple references are aligned with a single sequence of time-ordered hypothesis, we calculated the WER by aligning the two hypothesis streams with a single sequence of time-ordered reference words obtained based on the official time stamps of the corpus. This procedure allowed the WER to be calculated for all utterance-group regions within reasonable computation time.

Table 1 shows the WERs for various combinations of the VarArray-based front-end and back-end ASR configurations. The systems with IDs staring with B were built to clarify the contributions of individual configurations in a baseline setting. Systems B1 to B3 were based on single-talker ASR²²2In addition to B1-B3, we also evaluated a system that generated two separated signals with VarArray and then performed ASR for each separated signal with a single-talker model. However, due to the difficulty in fine-tuning the single-talker ASR model based on the separated signals (see Section 3.1), this system produced 25.1% and 29.2% for the development and evaluation sets, respectively, which were on par or even worse than the other baselines.. System B4 to B6 used t-SOT for multi-talker ASR. First, we can see that the AMI-based fine-tuning yielded significant WER improvements (B1$\rightarrow$B2, B4$\rightarrow$B5). Especially, the t-SOT TT18 obtained noticeably larger WER reductions, resulting in lower WERs than the single-talker TT18 (B2 vs. B5), which was consistent with prior reports [25]. Applying a single-output (i.e., traditional) multi-channel front-end, the WER was further improved from 25.3% to 23.0% for the evaluation set (B5$\rightarrow$B6).

We then evaluated the systems based on the proposed t-SOT-VA framework (P1–P7). By comparing B6 and P1, we can see that the proposed two-channel TT achieved a relative WER reduction of 5.7–6.8% compared with the conventional single-input model. Introducing the 2nd-stage pre-training further reduced the WERs by relative 2.8-3.6% (P1$\rightarrow$P2). Finally, the cross-channel transform (P3 and P4) further produced relative WER gains of 1.6–2.4%. Overall, compared with the baseline B6, the proposed system achieved relative WER improvements of 10.4–11.6%.

Systems P5–P7 were evaluated to investigate the WER impact of the model size and latency. As expected, the WERs were further improved by increasing the model size and the ASR latency. The largest system, P7, achieved the WERs of 13.7% and 15.5% for the development and evaluation sets, respectively. To the best of our knowledge, these results represent the SOTA WERs for the AMI distant microphone setting by significantly outperforming previously reported results [25, 10, 36] while retaining the streaming inference capability.

Finally, Table 2 shows the WERs of P7 for different microphone numbers. We used the microphones indexed as [0], [0,1], [0,1,2], [0,2,4,6], [0,2,4,6,7], [0-3,5,6], [0-6], or [0-7] for each microphone number setting. The experimental result confirms that the proposed system can make use of different numbers and shapes of microphones without retraining.