Audio-Visual Efficient Conformer for Robust Speech Recognition

Audio front-end. The audio front-end network first transforms raw audio wave-forms into mel-spectrograms using a short-time Fourier transform computed over windows of 20ms with a step size of 10ms. 80-dimensional mel-scale log filter banks are applied to the resulting frequency features. The mel-spectrograms are processed by a 2D convolution stem to extract local temporal-frequency features, resulting in a 20ms frame rate signal. The audio front-end architecture is shown in Table 1.

Visual front-end. The visual front-end network [29] transforms input video frames into temporal sequences. A 3D convolution stem with kernel size $5\times 7\times 7$ is first applied to the video. Each video frame is then processed independently using a 2D ResNet-18 [20] with an output spatial average pooling. Temporal features are then projected to the back-end network input dimension using a linear layer. The visual front-end architecture is shown in Table 2.

Back-end networks. The back-end networks use an Efficient Conformer architecture. The Efficient Conformer encoder was proposed in [7], it is composed of several stages where each stage comprises a number of Conformer blocks [16] using grouped attention with relative positional encodings. The temporal sequence is progressively downsampled using strided convolutions and projected to wider feature dimensions, lowering the amount of computation while achieving better performance. We use 3 stages in the audio back-end network to downsample the audio signal to a 80 milliseconds frame rate. Only 2 stages are necessary to downsample the visual signal to the same frame rate. Table 6 shows the hyper-parameter of each back-end network.

Audio-visual fusion module. Similar to [36, 29], we use an early fusion strategy to learn audio-visual features and reduce model complexity. The acoustic and visual features from the back-end networks are concatenated and fed into a joint feed-forward network. The concatenated features of size $2\times d_{model}$ are first expanded using a linear layer with output size $d_{ff}=4\times d_{model}$, passed through a Swish activation function [38] and projected back to the original feature dimension $d_{model}$.

Audio-visual encoder. The audio-visual encoder is a single stage back-end network composed of 5 Conformer blocks without downsampling. The encoder outputs are then projected to a CTC layer to maximize the sum of probabilities of correct target alignments.

The Efficient Conformer [7] proposed to replace Multi-Head Self-Attention (MHSA) [44] in earlier encoder layers with grouped attention. Grouped MHSA reduce attention complexity by grouping neighbouring temporal elements along the feature dimension before applying scaled dot-product attention. Attention having a quadratic computational complexity with respect to the sequence length, this caused the network to have an asymmetric complexity with earlier attention layers requiring more flops than latter layers with shorter sequence length. In this work, we propose to replace grouped attention with a simpler and more efficient attention mechanism that we call patch attention (Figure 2).

Similar to the pooling attention proposed by the Multiscale Vision Transformer (MViT) [13] for video and image recognition, the patch attention proceed to an average pooling on the input sequence before projection the query, key and values.

	$\displaystyle X=AvgPooling1d(X_{in})$		(1)
	$\displaystyle\text{with}\ Q,K,V=XW^{Q},XW^{K},XW^{V}$		(2)

Where $W^{Q}$, $W^{K}$, $W^{V}\in\mathbb{R}^{d\times d}$ are query, key and value linear projections parameter matrices. MHSA with relative sinusoidal positional encoding is then performed at lower resolution as:

	$\displaystyle MHSA(X)=Concat\left(O_{1},...,O_{H}\right)W^{O}$		(3)
	$\displaystyle\text{with}\ O_{h}=softmax\left(\frac{Q_{h}K_{h}^{T}+S^{rel}_{h}}{\sqrt{d_{h}}}\right)V_{h}$		(4)

Where $S^{rel}\in\mathbb{R}^{n\times n}$ is a relative position score matrix that satisfy $S^{rel}[i,j]=Q_{i}E_{j-i}^{T}$. $E$ is the linear projection of a standard sinusoidal positional encoding matrix with positions ranging from $-(n_{max}-1)$ to $(n_{max}-1)$. The attention output sequence is then projected and up-sampled back to the initial resolution using nearest neighbor up-sampling.

$\displaystyle X_{out}=UpsampleNearest1d(MHSA(X))$

(5)

In consequence, each temporal element of the same patch produce the same attention output. Local temporal relationships are only modeled in the convolution modules while global relationships are modeled by patch attention. We use 1-dimensional patches in this work but patch attention could also be generalized to image and video data using 2D and 3D patches. We leave this to future works. The computational complexity of each attention variant is shown in Table 4. Path attention further reduce complexity compared to grouped attention by decreasing the amount of computation needed by Query, Key, Value and Output fully connected layers while keeping the feature dimension unchanged. Similar to previous work [7], we only use patch attention in the first audio back-end stage to reduce complexity while maintaining model recognition performance. Figure 3 shows the amount of FLOPs for each attention module variant with respect to encoded sequence length $n$ and model feature dimension $d$. Using patch or grouped attention variants instead of regular MHSA greatly reduce the amount of FLOPs in the first audio back-end stage.

Inspired by [27] and [33] who proposed to add intermediate CTC losses between encoder blocks to improve CTC-based speech recognition performance, we add Inter CTC residual modules (Figure 4) in encoder networks. We condition intermediate block features of both audio, visual and audio-visual encoders on early predictions to relax the conditional independence assumption of CTC models. During both training and inference, each intermediate prediction is summed to the input of the next layer to help recognition. We use the same method proposed in [33] except that we do not share layer parameters between losses. The $l^{th}$ block output $X^{out}_{l}$ is passed through a feed-forward network with residual connection and a softmax activation function:

	$\displaystyle Z_{l}=Softmax(Linear(X^{out}_{l}))$		(6)
	$\displaystyle X^{in}_{l+1}=X^{out}_{l}+Linear(Z_{l})$		(7)

Where $Z_{l}\in\mathbb{R}^{T\times V}$ is a probability distribution over the output vocabulary. The intermediate CTC loss is then computed using the target sequence $y$ as:

	$\displaystyle L^{inter}_{l}=-log(P(y|Z_{l}))$		(8)
	$\displaystyle\text{with}\ P(y|Z_{l})=\sum_{\pi\in\mathcal{B}_{CTC}^{-1}(y)}\prod_{t=1}^{T}Z_{t,\pi_{t}}$		(9)

Where $\pi\in V^{T}$ are paths of tokens and $\mathcal{B}_{CTC}$ is a many-to-one map that simply removes all blanks and repeated labels from the paths. The total training objective is defined as follows:

	$\displaystyle L=(1-\lambda)L^{CTC}+\lambda L^{inter}$		(10)
	$\displaystyle\text{with}\ L^{inter}=\frac{1}{K}\sum_{k\in\textit{interblocks}}{L^{inter}_{k}}$		(11)

Where interblocks is the set of blocks having a post Inter CTC residual module (Figure 4). Similar to [33], we use Inter CTC residual modules every 3 Conformer blocks with $\lambda$ set to 0.5 in every experiments.

We use 3 publicly available AVSR datasets in this work. The Lip Reading in the Wild (LRW) [8] dataset is used for visual pre-training and the Lip Reading Sentences 2 (LRS2) [1] and Lip Reading Sentences 3 (LRS3) [2] datasets are used for training and evaluation.

LRW dataset. LRW is an audio-visual word recognition dataset consisting of short video segments containing a single word out of a vocabulary of 500. The dataset comprise 488,766 training samples with at least 800 utterances per class and a validation and test sets of 25,000 samples containing 50 utterances per class.

LRS2 & LRS3 datasets. The LRS2 dataset is composed of 224.1 hours with 144,482 videos clips from the BBC television whereas the LRS3 dataset consists of 438.9 hours with 151,819 video clips extracted from TED and TEDx talks. Both datasets include corresponding subtitles with word alignment boundaries and are composed of a pre-train split, train-val split and test split. LRS2 has 96,318 utterances for pre-training (195 hours), 45,839 for training (28 hours), 1,082 for validation (0.6 hours), and 1,243 for testing (0.5 hours). Whereas LRS3 has 118,516 utterances in the pre-training set (408 hours), 31,982 utterances in the training-validation set (30 hours) and 1,321 utterances in the test set (0.9 hours). All videos contain a single speaker, have a $224\times 224$ pixels resolution and are sampled at 25 fps with 16kHz audio.

Pre-processing Similar to [29], we remove differences related to rotation and scale by cropping the lip regions using bounding boxes of $96\times 96$ pixels to facilitate recognition. The RetinaFace [11] face detector and Face Alignment Network (FAN) [6] are used to detect 68 facial landmarks. The cropped images are then converted to gray-scale and normalised between $-1$ and $1$. Facial landmarks of the LRW, LRS2 and LRS3 datasets are obtained from previous work [30] and reused for pre-processing to get a clean comparison of the methods. A byte-pair encoding tokenizer is built from LRS2&3 pre-train and trainval splits transcripts using sentencepiece [26]. We use a vocabulary size of 256 including the CTC blank token following preceding works on CTC-based speech recognition [31, 7].

Data augmentation Spec-Augment [35] is applied on the audio mel-spectrograms during training to prevent over-fitting with two frequency masks with mask size parameter $F=27$ and five time masks with adaptive size $pS=0.05$. Similarly to [30], we mask videos on the time axis using one mask per second with the maximum mask duration set to 0.4 seconds. Random cropping with size $88\times 88$ and horizontal flipping are also performed for each video during training. We also follow Prajwal et al. [37] using central crop with horizontal flipping at test time for visual-only experiments.

Training Setup We first pre-train the visual encoder on the LRW dataset [8] using cross-entropy loss to recognize words being spoken. The visual encoder is pre-trained for 30 epochs and front-end weights are then used as initialization for training. Audio and visual encoders are trained on the LRS2&3 datasets using a Noam schedule [44] with 10k warmup steps and a peak learning rate of 1e-3. We use the Adam optimizer [24] with $\beta_{1}=0.9$, $\beta_{2}=0.98$. L2 regularization with a 1e-6 weight is also added to all the trainable weights of the model. We train all models with a global batch size of 256 on 4 GPUs, using a batch size of 16 per GPU with 4 accumulated steps. Nvidia A100 40GB GPUs are used for visual-only and audio-visual experiments while RTX 2080 Ti are used for audio-only experiments. The audio-only models are trained for 200 epochs while visual-only and audio-visual models are trained for 100 and 70 epochs respectively. Note that we only keep videos shorter than 400 frames (16 seconds) during training. Finally, we average models weights over the last 10 epoch checkpoints using Stochastic Weight Averaging [22] before evaluation.

Language Models. Similarly to [28], we experiment with a N-gram [21] statistical language model (LM) and a Transformer neural language model. A 6-gram LM is used to generate a list of hypotheses using beam search and an external Transformer LM is used to rescore the final list. The 6-gram LM is trained on the LRS2&3 pre-train and train-val transcriptions. Concerning the neural LM, we pretrain a 12 layer GPT-3 Small [5] on the LibriSpeech LM corpus for 0.5M steps using a batch size of 0.1M tokens and finetune it for 10 epochs on the LRS2&3 transcriptions.

Table 5 compares WERs of our Audio-Visual Efficient Conformer with state-of-the-art methods on the LRS2 and LRS3 test sets. Our Audio-Visual Efficient Conformer achieves state-of-the-art performances with WER of 2.3%/1.8%. On the visual-only track, our CTC model competes with most recent autoregressive methods using S2S criterion. We were able to recover similar results but still lack behind Ma et al. [30] which uses auxiliary losses with pre-trained audio-only and visual-only networks. We found our audio-visual network to converge faster than audio-only experiments, reaching better performance using 4 times less training steps. The intermediate CTC losses of the visual encoder could reach lower levels than in visual-only experiments showing that optimizing audio-visual layers can help pre-fusion layers to learn better representations.

We propose a detailed ablation study to better understand the improvements in terms of complexity and WER brought by each architectural modification. We report the number of operations measured in FLOPs (number of multiply-and-add operations) for the network to process a ten second audio/video clip. Inverse Real Time Factor (Inv RTF) is also measured on the LRS3 test set by decoding with a batch size 1 on a single Intel Core i7-12700 CPU thread. All ablations were performed by training audio-only models for 200 epochs and visual-only / audio-visual models for 50 epochs.

Efficient Conformer Visual Back-end. We improve the recently proposed visual Conformer encoder [29] using an Efficient Conformer back-end network. The use of byte pair encodings for tokenization instead of characters allows us to further downsample temporal sequences without impacting the computation of CTC loss. Table 6 shows that using an Efficient Conformer back-end network for our visual-only model leads to better performances while reducing model complexity and training time. The number of model parameters is also slightly decreased.

Inter CTC residual modules. Similar to [33], we experiment adding Inter CTC residual modules between blocks to relax the conditional independence assumption of CTC. Table 7 shows that using intermediate CTC losses every 3 Conformer blocks greatly helps to reduce WER, except for the audio-only setting where this does not improve performance. Figure 5 gives an example of intermediate block predictions decoded using greedy search without an external language model on the test set of LRS3. We can see that the output is being refined in the encoder layers by conditioning on the intermediate predictions of previous layers. Since our model refines the output over the frame-level predictions, it can correct insertion and deletion errors in addition to substitution errors. We further study the impact of Inter CTC on multi-modal learning by measuring the performance of our audio-visual model when one of the two modalities is masked. As pointed out by preceding works [8, 1, 32], networks with multi-modal inputs can often be dominated by one of the modes. In our case speech recognition is a significantly easier problem than lip reading which can cause the model to ignore visual information. Table 8 shows that Inter CTC can help to counter this problem by forcing pre-fusion layers to transcribe the input signal.

Patch multi-head self-attention. We experiment replacing grouped attention by patch attention in the first audio encoder stage. Our objective being to increase the model efficiency and simplicity without harming performance. Grouped attention was proposed in [7] to reduce attention complexity for long sequences in the first encoder stage. Table 9 shows the impact of each attention variant on our audio-only model performance and complexity. We start with an Efficient Conformer (M) [7] and replace the attention mechanism. We find that grouped attention can be replaced by patch attention without a loss of performance using a patch size of 3 in the first back-end stage.

We measure model noise robustness using various types of noise and compare our Audio-Visual Efficient Conformer with recently published methods. Figure 6 shows the WER evolution of audio-only (AO), visual-only (VO) and audio-visual (AV) models with respect to multiple Signal to Noise Ratio (SNR) using white noise and babble noise from the NoiseX corpus [43]. We find that processing both audio and visual modalities can help to significantly improve speech recognition robustness with respect to babble noise. Moreover, we also experiment adding babble noise during training as done in previous works [36, 29] and find that it can further improve noise robustness at test time.

Robustness to various types of noise. We gather various types of recorded audio noise including sounds and music. In Table 10, we observe that the Audio-Visual Efficient Conformer consistently achieves better performance than its audio-only counterpart in the presence of various noise types. This confirm our hypothesis that the audio-visual model is able to use the visual modality to aid speech recognition when audio noise is present in the input.

Comparison with other methods. We compare our method with results provided by Ma et al. [29] and Petridis et al. [36] on the LRS2 test set. Table 11 shows that our audio-visual model achieves lower WER in the presence of babble noise, reaching WER of 9.7% at -5 dB SNR against 16.3% for Ma et al. [29].