Large-Scale Pre-Training of End-to-End Multi-Talker ASR for
Meeting Transcription with Single Distant Microphone

Given input $X\in\mathbb{R}^{f^{a}\times l^{a}}$, where $f^{a}$ and $l^{a}$ are the feature dimension and the sequence length, respectively. The goal of ASR system is to estimate transcription $Y=(y_{n}\in\{1,...,|\mathcal{V}|\}|n=1,...,N)$, where $|\mathcal{V}|$ is the size of the vocabulary $\mathcal{V}$, and $N$ is the number of estimated tokens.

In this paper, we use the attention-based encoder-decoder (AED) [33, 34] as the backbone of the ASR system, which is represented as follows:

	$\displaystyle H$	$\displaystyle={\rm Encoder}(X),$		(1)
	$\displaystyle o_{n}$	$\displaystyle={\rm Decoder}(y_{[1:n-1]},H).$		(2)

The Encoder module first converts $X$ into a sequence of hidden embeddings $H\in\mathbb{R}^{f^{h}\times l^{h}}$ for ASR (Eq. (1)), where $f^{h}$ and $l^{h}$ are the embedding dimension and the sequence length, respectively. At each decoder step $n$, the Decoder module calculates the output distribution $o_{n}\in\mathbb{R}^{|\mathcal{V}|}$ given previous token estimates $y_{[1:n-1]}$ and $H$ (Eq. (2)). The posterior probability of token $i$ (i.e. the $i$-th token in the vocabulary $\mathcal{V}$) at the $n$-th decoder step is represented as

$\displaystyle Pr(y_{n}=i|y_{[1:n-1]},X)=o_{n,i},$

(3)

where $o_{n,i}$ represents the $i$-th element of $o_{n}$. The posterior probability of token $Y$ given input $X$ is represented as,

$\displaystyle Pr(Y|X)=$

$\displaystyle\prod_{n=1}^{N}Pr(y_{n}|y_{[1:n-1]},X).$

(4)

In this paper, we use a modified version of Conformer network [35] for Encoder module, and a conventional Transformer-based decoder [36] for Decoder module. The modifications we have made to the Conformer network are as follows: (i) we insert a squeeze and excitation module [37] just before the dropout of the convolution module; (ii) we do not use batch normalization in the convolution module; and (iii) we add one more point-wise convolution after depth-wise convolution. These changes have been made based on our preliminary test.

SOT was proposed for the AED to recognize a variable number of speakers from possibly overlapped audio [24]. In the SOT framework, the multiple utterances are concatenated to form a single token sequence by inserting a special symbol $\langle sc\rangle$ representing a speaker change. For example, for the three-speaker case, a reference token sequence is given as $R=\{r^{1}_{1},..,r^{1}_{N^{1}},\langle sc\rangle,r^{2}_{1},..,r^{2}_{N^{2}},\langle sc\rangle,r^{3}_{1},..,r^{3}_{N^{3}},\langle eos\rangle\}$, where $r^{j}_{i}$ represents the $i$-th token of the $j$-th speaker. Here, $\langle eos\rangle$, a token for sequence end, is used only at the end of the entire sequence. In the inference, the decoding process is iterated until $\langle eos\rangle$ is detected so that the SOT-based model can theoretically transcribe utterances of any number of speakers while automatically estimating the number of speakers.

There are multiple ways to determine the speaker order to form reference label sequence $R$. One simple yet effective approach proposed in [24] is sorting the reference labels by their start times, which is called “first-in, first-out” (FIFO) training. FIFO training works with complexity of $O(S)$ with respect to the number of speakers $S$ while showing superior accuracy than a scheme that exhaustively considers all possible permutations [24]. In this paper, we always use this FIFO training scheme.

In this paper, we introduce a notion of “utterance group” to appropriately evaluate multi-talker ASR models. The relationship between the utterance and utterance group is illustrated in Fig. 1 top. The utterance group is defined as a set of utterances that are connected by speaker overlap regions. In other words, the utterance groups can be formed by segmenting the recording at silence positions or non-overlapping utterance boundaries.

The utterance-based evaluation is a standard way to evaluate single-talker ASR models for AMI. As shown in the middle of Fig. 1, ASR is applied to each speech segment obtained with reference utterance boundaries. In the MDM setting, beamforming is used before the signals are fed into the single-talker ASR system. Mis-recognized words are counted for each utterance, and the errors are summed up to calculate a final WER.

The utterance group-based evaluation is newly introduced in this paper to evaluate the multi-talker ASR models. As shown in Fig. 1 bottom, multi-talker ASR is applied to each utterance group segment without information about the utterance boundaries inside the segment. The multi-talker ASR system generates hypotheses consisting of one or more utterances from the utterance group audio. We calculate the number of mis-recognized words based on the concatenated minimum-permutation word error rate [6]. Specifically, we first concatenate the reference transcriptions of the same speaker in the utterance group (i.e., in the example of Fig. 1, utterance #1 and #4 in utterance group #1). Then, the best alignment between the hypotheses and the concatenated references is selected among all possible permutations²²2If the number of hypotheses are larger than that of references, all hypotheses that don’t have corresponding references are counted as insertion errors. Similarly, if the number of references are larger than that of hypotheses, all unmatched references are counted as deletion errors. for error counting. Finally, the errors from each utterance group are summed up and divided by the number of total reference words to calculate the WER. Note that the denominator to calculate the WER, i.e. the number of the total reference words, is the same for the utterance-based evaluation and the utterance group-based evaluation.

In our investigation, we trained single-talker ASR models as well as SOT-based multi-talker ASR models. We used the same encoder-decoder architecture for all cases. Specifically, the encoder consisted of 2 layers of convolution layers that subsample the time frames by a factor of 4, followed by 18 conformer layers. Each conformer layer consisted of two 1024-dim feed forward layers in a sandwitch structure, a multi-head attention with 8 heads, a depth-wise convolution with kernel size 3, and a squeeze-and-excitation network with reduction factor 8 [37]. The embedding dimension was set to 512. The decoder consisted of 6 layers, each of which had a multi-head attention with 8 heads and a 2048-dim feed forward layer. 4K subwords [40] were used as a recognition unit. We used a 80-dim log mel filterbank extracted every 10 msec for the input feature.

The single-talker model was pre-trained by the 75K-hour data while the multi-talker model was pre-trained by the 75K-hour-based simulated multi-talker data. For both models, we performed 425k training iterations with 32 GPUs, each of which consumed mini-batches of 24,000 frames (roughly corresponding to 900K hours of data simulation and consumption). We used Adam optimizer with a linear decay learning rate schedule with a peak learning rate of 1e-3 after 25k warm up iterations. In the fine-tuning stage, the pre-trained single-talker model was fine-tuned by the AMI-SDM utterance segments while the pre-trained multi-talker model was fine-tuned by the AMI-SDM utterance group segments with the speaker-based FIFO training scheme [26]. For both cases, we used speed perturbation and SpecAugment [41], and conducted 25k training iterations with 16 GPUs, each of which consumed mini-batches of 6,000 frames. A linear decay learning rate schedule starting at a learning rate of 1e-4 was used. Besides the pre-training-based two-stage approach, we also trained models on the AMI-SDM data without pre-training. In this case, we used mini-batches of 6,000 frames and trained the models for 110k iterations with 16 GPUs, with a linear decay learning rate schedule with a peak learning rate of 1e-4 after 10k warm up iterations.

Table 2 shows our main results comparing single-talker ASR and SOT-based multi-talker ASR models. The two-stage approach of performing both pre-training and fine-tuning significantly improved the WER for both single-talker (Config. 3) and multi-talker models (Config. 7). The two-stage models outperformed both the 75K-hour-only models and AMI-only models with significant margins. Interestingly, although the single-talker model outperformed the multi-talker model before fine-tuning (Configs. 2 and 6), the multi-talker model showed superior performance after fine-tuning (Configs. 3 and 7). Table 3 shows that fine-tuning significantly improved the speaker counting accuracy of the multi-talker model. This improvement in speaker counting led to the larger WER improvement of the multi-talker model especially for segments with many speakers as shown in Table 4 (Config. 6 vs. Config. 7). Table 4 also shows the improvement provided by the pre-training (Config. 5 vs. Config. 7), where large gains were observed for segments with more speakers. This is because such segments were more scarce in the real data as shown in Table 1 (b).

While our main focus in this paper is SDM-based recognition, we also evaluated the fine-tuned single-talker ASR model for the MDM setting using BeamFormIt beamformer [42]. The result is shown in Config. 4 of Table 2. We observed that the proposed SOT multi-talker AED model (Config. 7) outperformed the combination of the single-talker AED and MDM-based beamforming while the SOT multi-talker AED model even does not require the utterance-level boundary information. This result shows the effectiveness of the end-to-end multi-talker ASR model that jointly performs speaker counting, speech separation, and speech recognition.

To better understand why fine-tuning worked so effectively, we conducted an experiment using the AMI-IHM training data for fine-tuning. Table 5 shows the results. Here, we prepared two new datasets for fine-tuning: “IHM real-mix” and “IHM rand-mix”. “IHM real-mix” was generated by first mixing all IHM recordings in the same meeting and then segmenting the mixed audio by the utterance groups. The difference from “SDM” to “IHM real-mix” lies only in the noise and reverberation characteristics. On the other hand, “IHM rand-mix” was generated by randomly mixing the AMI IHM utterances for up to 4 utterances. The difference between “IHM real-mix” and “IHM rand-mix” resides in the speaker overlap pattern. The biggest WER difference was observed between the no-fine-tuning (1st row) and the fine-tuning by “IHM rand-mix” (2nd raw), which suggested the fine-tuning of the language model (LM), English accent, and the accurate overlap of speech region³³3 In pre-training, the simulated mixed audio may not have the overlap of speech region due to the silence of the original audio. Also, the audio used in the simulation could contain interference speech without transcription. We think these factors may be also refined by fine-tuning. had the biggest impact. The second largest WER difference was observed between “IHM rand-mix”-based fine-tuning and “IHM real-mix”-based fine-tuning, showing the importance of including the real overlapping pattern. The difference from “IHM real-mix” to “SDM” was relatively small presumably because the pre-trained representation was already robust to noisy samples.

We further conducted the evaluation to understand the gap between SDM-based recognition and IHM-based recognition, the results of which are shown in Table 6. We prepared two additional evaluation sets, “IHM” and “IHM-mix”. “IHM-mix” was generated by first mixing all IHM recordings in the same session and then segmenting the mixed audio based on the utterance group. The difference between “SDM” and “IHM-mix” is whether the audio is recorded by SDM or IHM. On the other hand, the difference between “IHM-mix” and “IHM” includes whether the speech is overlapped and whether we use additional utterance boundary information. While we observed a significant gap between SDM and IHM-mix, the WER difference between IHM-mix and IHM was relatively small. This indicates that the SOT AED already worked well in terms of speaker counting and multi-talker transcription and that the refinement of the model capacity or the data simulation is rather necessary to better cope with noisier data.