Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels

In order to investigate the impact of the size of training data on the performance of audio-only, visual-only and audio-visual models, we scale the training sources by including publicly-available audio-visual clips into the training set. An overview of our label generation pipeline can be found at the top of Fig. 1. To be specific, audio waveforms from the unlabelled audio-visual datasets are fed into a pre-trained ASR model to produce automatic transcriptions. For the purpose of this study, we use two unlabelled datasets: VoxCeleb2 [17] and AVSpeech [18]. In particular, AVSpeech [18] contains 4 700 hours of YouTube video segments in multiple languages, and VoxCeleb2 [17] consists of 2 300 hours of video segments from more than 6 000 people. However, we are interested in training models in English. Thus, we use the VoxLingua107 language classifier [19] to filter the AVSpeech dataset, resulting in a total of 1 323 hours; the list of English data we use for VoxCeleb2 is obtained from [7], and comprises 1 307 data hours. Next, we leverage publicly-available ASR models to produce automatically generated transcriptions. It is worth pointing out that our work facilitates reproduction and comparison since all datasets and models used are publicly accessible.

We investigate the impact of the automatic transcriptions given by four different ASR models on the performance of audio-only and visual-only models, i.e. Whisper [11], wav2vec2.0 [5], Hidden unit BERT (HuBERT) [8] and Conformer-Transducer [12, 20]. In particular, wav2vec 2.0 [5] and HuBERT [8] are self-supervised learning methods for learning speech representations from unlabelled audio. In contrast, Conformer-Transducer [12] is a conformer-based model trained with recurrent neural network transducers (RNN-T) loss that uses the NeMo ASRSET dataset, consisting of 12 000 hours of English speech. Finally, Whisper [11] is a transformer-based model [21] trained with a total of 680 000 hours of labelled audio. In this work, we access the ASR models from the Hugging Face community, “nvidia/stt_en_conformer_transducer_xlarge”, “facebook/hubert-large-ls960-ft”, “facebook/wav2vec2-base-960h”, and “openai/whisper-medium.en”, respectively.

We adopt the off-the-shelf architecture presented in [22], which has achieved state-of-the-art performance on the LRS2 and LRS3 datasets without the use of external data. The architecture is shown at the bottom of Fig. 1. In particular, the VSR front-end is based on a modified ResNet-18 [23, 24], where the first layer is a spatio-temporal convolutional layer with a kernel size of 5$\times$7$\times$7 and a stride of 1$\times$2$\times$2. The temporal back-end, which follows the front-end, is a Conformer [20]. Similarly, the ASR encoder consists of a 1D ResNet-18 [25] followed by a Conformer. The ASR and VSR encoder outputs are fused via a multi-layer perceptron (MLP). The rest of the network consists of a projection layer and a Transformer decoder for joint CTC/attention training [26].

We conduct experiments on LRS2 [27] and LRS3 [28], which are the two largest publicly available datasets for audio-visual speech recognition in English. LRS2, collected from BBC programs, contains 144 482 video clips with a total of 225 hours. Specifically, the pre-training, training, validation and test set contains 96 318 (195 hours), 45 839 (28 hours), 1 082 (0.6 hours) and 1 243 (0.5 hours) video clips, respectively. LRS3 consists of 151 819 video clips from TED talks with a total of 439 hours. It contains 118 516 (408 hours), 31 982 (30 hours) and 1 321 clips (0.9 hours) in the pre-training, training-validation, and test set, respectively. For training, we also use the English-speaking videos from AVSpeech (1 323 hours) and VoxCeleb2 (1 307 hours) as the additional training data together with automatically-generated transcriptions.

For the visual stream, we follow previous work [22] to pre-process the datasets. We crop the mouth region of interests (ROIs) using a bounding box of 96 $\times$ 96. Each frame is normalised by subtracting the mean and dividing by the standard deviation of the training set. For audio streams, we only perform $z$-normalisation per utterance.

For our audio- and visual-only ASR models, we use a ResNet-based front-end module pre-trained on LRW [29], followed by a Conformer encoder with 12 layers, 768 input dimensions, 3 072 feed-forward dimensions, and 16 attention heads. The decoder is a 6-layer Transformer with the same dimensions and number of heads as the encoder, resulting in a total of 243.1 M and 250.4 M parameters for the audio- and visual-only models, respectively. More specifically, the ASR front-end, VSR front-end, Conformer back-end, Transformer decoder and the projection layer of the CTC have 3.9 M, 11.2 M, 170.9 M, 64.5 M and 3.9 M parameters, respectively. For the audio-visual models, we concatenate the audio-visual encoder outputs and feed them to a 2-layer multi-layer perceptron (MLP) with hidden and output sizes of 8 192 and 768, respectively.

For data augmentation, we apply horizontal flipping, random cropping, and adaptive time masking [30] to the visual inputs, while we only use adaptive time masking for the audio stream. For both streams, we choose a number of masks that is proportional to the utterance length and a maximum masking length of up to 0.4 seconds. For the target vocabulary, we use SentencePiece [31] subword units with a vocabulary size of 5 000. We train the model for 75 epochs with the AdamW [32] optimizer, a cosine learning rate scheduler, and a warm-up of 5 epochs. The peak learning rate is 1e-3. The maximal number of frames in each batch is 1 800 frames. Following [30], our visual-only models are incorporated with a transformer-based language model trained on a corpus of 166 million characters ^†††† The corpus consits of the training sets of LibriSpeech ($960$ h) [33], pre-training and training sets of LRS2 [27] and LRS3 [28], TED-LIUM 3 [34], Voxforge (English) and Common Voice (English) [35] whereas language models are not included for ASR and AV-ASR models since no further improvements are observed.

Given that several publicly-available ASR models are available, we use performance on Librispeech as a criterion for model selection. We use models that have achieved state-of-the-art performance on the test-clean set of Librispeech, i.e., Conformer-Transducer [12] and HuBERT [8]. We also use ASR models that are widely used in the speech community, wav2vec 2.0 [5] and Whisper [11]. Performance of ASR models on Librispeech clean-test set is shown in the first column of Table 1. Results of the ASR and VSR models trained with the automatically-generated transcriptions on the LRS3 dataset are shown in the third and fourth columns, respectively, of Table 1. We observe that overall the WER on Librispeech is not highly correlated with the performance of the ASR and VSR models trained with the automatically-generated transcriptions from the corresponding pre-trained ASR models. The same conclusion is also true when we measure the WER on the LRS3 test. We show that using the transcriptions from most ASR models (i.e., wav2vec 2.0 [5], Whisper [11], and Conformer-Transducer [12]) results in very similar WER for both audio-only and visual-only models. The only exception is the use of automatically-generated transcriptions from HuBERT [8] which results in slightly worse performance despite being one of the best performing models on Librispeech. In this work, we rely on the automatically-generated transcriptions from the Conformer-Transducer [12] since on average it leads to the best performance for both ASR and VSR models.

Table 2 shows the impact of varying the numbers of hours of unlabelled data on the performance of ASR and VSR models on LRS3. An absolute improvement of 1.7 % in WER is observed for VSR by using only labelled data from LRS2 and LRS3 (818 hours) compared to [30]. This gain is likely due to the increase in model capacity. When including 20 % (526 hours) of AVSpeech [18] and VoxCeleb2 [17], the performance of audio- and visual-only models can be further improved to 1.3 % and 26.6 % WER, respectively. Increasing further the number of training hours leads to a further reduction of the WER especially for the VSR model. This is in line with the recent trend observed in the literature [30], where using larger training sets substantially improves performance. In this experiment, we also show that the WER can be improved even by adding data that have been automatically transcribed and inevitably have noisy labels. We also notice that the improvement for the ASR model is marginal when using more than 1 578 hours of unlabelled training data, indicating that the ASR performance may have saturated.

Results on LRS2 and LRS3 are presented in Tables 3 and 4, respectively. For LRS2, it is clear that our visual-only, audio-only and audio-visual models further push the state-of-the-art performance to a WER of 14.6 %, 1.5 % and 1.5 % respectively. For LRS3, the best visual-only model has a WER of 19.1 %, which is outperformed only by [16] (17.0 % WER) which uses 26$\times$ more training data. Similarly, our audio-only model establishes a new state-of-the-art [7] by achieving a WER of 1.0 % when using 1 921 hours of training data from LRW, LRS3 and VoxCeleb2 datasets. However, when further introducing AVSpeech for training, no further improvement is observed, suggesting that the ASR performance may have reached saturation. State-of-the-art performance is also achieved for AV-ASR with a WER of 0.9 %.

Results of ASR and AV-ASR models, when tested with different acoustic noise levels, are shown in Table. 5. During training we use the babble noise from the NOISEX dataset [39], while the SNR level is selected from [-5 dB, 0 dB, 5 dB, 10 dB, 15 dB, 20 dB, $\infty$ dB] with a uniform distribution. For evaluation, we test three types of noise: babble noise [39], pink and white noise from the Speech Commands dataset [40]. We show that, overall, the results are consistent with those presented in [1, 22, 3, 4], i.e. the performance of audio-only models is closer to the audio-visual counterpart in the presence of low levels of noise, whereas the performance gap becomes larger as the noise levels increase. We notice that when using babble noise for evaluation, the performance of either audio-only or audio-visual models has a WER lower than 10 % at -7.5 dB. This is likely mainly a consequence of the overlapping noise type in the training and testing phases (despite mismatched levels of noise).