Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement

Instead of using the visual information of the entire human face, we use mouth Region-Of-Interest (ROI). These lip movements enable our visual model to focus on the most relevant visual information on the human face and improve the accuracy of AVSE. To extract the feature of visual modality, we utilize a modified ResNet-18 [40] visual encoder pre-trained on the Lip Reading in the Wild dataset (LRW), where the mouth ROI encoder consists of a 3D convolutional layer that processes consecutive frames, followed by a 2D ResNet-18. The output features of the visual encoder will be sent into the following Predictive and Generative models as visual embeddings. Note that since we freeze this visual encoder during our training stage, we can extract the visual embeddings of all the datasets beforehand. It means that we don’t need to include the visual model during our entire training stage, which reduces parameters and memory usage.

Where ${Encoder}$ is the lip movement model, ${\bf v}$ is the visual embeddings, and ${\bf I}$ is the image of ROI.

Since the initialization of the diffusion process significantly affects the enhanced speech of the Generative model, we utilize a Predictive denoiser to provide an initial estimation of noise. The noisy speech is initially converted into a spectrogram using the short-time Fourier transform (STFT) and serves as an input to this model, and the output of this model will then be sent into the Generative model. As a guide for the diffusion process, we assume that this initial estimation will improve the later diffusion process and provide a higher-quality enhancement result. In this study, we use the same model architecture as our Predictive denoiser, which will be further described later.

Where ${Denoiser}$ is the Predictive denoiser, ${\bf y}$ is the noisy speech spectrogram and ${\bf\hat{y}}$ is the preliminary-denoised speech spectrogram.
To train this denoiser, we apply a Mean square error (MSE) Loss between the denoised speech spectrogram and the clean speech spectrogram.

Where ${\bf L_{denoiser}}$ is the loss of the Predictive denoiser, ${\bf x}$ is the clean speech spectrogram.

In the Generative part of the acoustic modality, we adopt a score-based diffusion model “SGMSE” proposed by Richter et al. [26]. “SGMSE” defines the diffusion process in the complex short-time Fourier transform (STFT) domain, which uses Stochastic Differential Equations (SDE) to operate speech spectrograms. In the forward process, noise is gradually added to the clean speech spectrogram, which is described by the Stochastic Differential Equation (SDE):

Where ${\bf{x}_{t}}$ is the spectrogram of the current process state, ${\bf t}\in{[0,T]}$ is a continuous time-step variable describing the progress of the process. Note that since we use the predictive model before the diffusion process, preliminary-denoised speech spectrogram ${\bf\hat{y}}$ is used here, and ${\bf w}$ denotes a standard Wiener process.
Functions ${\bf f}\left({\bf{x}_{t}},{\bf\hat{y}}\right)$ and ${\bf g(t)}$ represent “drift coefficient” and “diffusion coefficient” respectively. The definitions of them are given below:

Where ${\bf\gamma}$ is a constant called “stiffness”, which controls the transition from x0 to y. ${\bf{\sigma}}_{\bf min}$ and ${\bf{\sigma}}_{\bf max}$ are parameters defining the noise schedule of the Wiener process.
While the reverse process learns to generate a clean speech spectrogram in an iterative way starting from the preliminary-denoised speech. This process is described by another SDE:

where ${{\nabla}_{{\bf x}_{t}}}{\bf log}\>{{\rm p}_{t}}\left({\bf x_{t}|\hat{y}}\right)$ is “score” approximated by a DNN. This DNN is therefore called the score model. We can then denote the score model as ${{\bf s}_{\theta}\left({\bf x_{t}},{\bf\hat{y}},{\bf t}\right)}$, which is determined by model parameters in DNN and receives the current process state ${{\bf x}_{t}}$, the preliminary-denoised speech spectrogram ${\bf\hat{y}}$, and the current time-step ${\bf t}$ as inputs. After plugging in ${{\bf s}_{\theta}\left({\bf x_{t}},{\bf\hat{y}},{\bf t}\right)}$, our SDE of the reverse process can be rewritten as:

After training the score model, we can use the output of the score model to plug in reverse SDE, and this equation can be solved by various discrete solver procedures.
At the inference stage, we can use the well-trained model to iterate through the reverse process from ${t}={T}$ to ${t}={0}$. The initial condition of the reverse process at ${t}={T}$ is given below:

Where initial condition ${\bf x_{T}}$ is sampled from ${N_{C}}$, which denotes the circularly symmetric complex normal distribution, and ${\bf I}$ denotes the identity matrix. Following the denoising process, the enhanced spectrogram is converted back to a waveform using the inverse short-time Fourier transform (ISTFT).

At each training step, we sample and pass $({\bf x_{t}},{\bf\hat{y}},{\bf t})$ into the score model ${{\bf s}_{\theta}}$. The sampling procedure can be summarized as: (1) Sample timestep ${\bf t}\sim{u[0,T]}$ (2) Sample ${\bf({x}_{0},\hat{y})}$ from the dataset (3) Sample ${\bf\phi}\sim{N_{C}}\left({\bf\phi};{\bf 0},{\bf I}\right)$ (4) Sample ${\bf x}_{t}$ by the equation:

Where ${\bf{\mu}}\left({\bf x_{0}},{\bf\hat{y}},{\bf t}\right)$ and ${\bf\sigma(t)}$ denote the mean and variance of Gaussian process. Since the initial conditions are known, we can derive the equations for them:

Where ${\bf x_{0}}$ is the clean speech. To explicitly define the objective function at the training stage, we use denoising score-matching to rewrite the approximated score ${{\nabla}_{{\bf x}_{t}}}{\bf log}\>{{\rm p}_{t}}\left({\bf x_{t}|\hat{y}}\right)$ as:

By substituting Eq. (10) into Eq. (13), our overall training objective can be defined as an L2 loss:

Inspired by Stable Diffusion [42], which demonstrates that cross-attention is effective in injecting the conditions from various input modalities for conditional image generation, we’re interested in using cross-attention blocks to integrate visual-acoustic information. For the self-attention blocks in our Predictive and Generative acoustic models, we replace the acoustic key and value with the visual embedding encoded by our visual encoder. We assume that complementing the acoustic input with visual lip movement information leads to more accurate speech reconstruction. The overall performance of the speech enhancement system can be boosted without additional cost.

Where ${\bf\tilde{h}}$ is the visual-acoustic feature, and ${\bf h}$ is the original acoustic feature in the Predictive or Generative model.

To enable the training of linguistic knowledge transfer, we utilize the speech recognition model Whisper [43] to generate the text for each clean speech. Note that we use the large model for higher recognition accuracy.

In the right branch of Fig. 1, the context-dependent linguistic representation is explored from a pre-trained BERT model. The output of this BERT model is termed linguistic guidance, which helps visual-acoustic modality learn the context-dependent representation. The process of extracting linguistic guidance can be formulated as:

Where “${\bf t}$” is the text sequence generated by Whisper [43], “${\bf t}_{{\rm token}}$” is word piece-based tokens “${BERT}$” is a pre-trained BERT model, token symbols “CLS” and “SEP” represent the start and end of an text sequence, $\bf Z\in\mathbb{R}^{T_{t}\times d_{s}}$ is the linguistic guidance which encodes context-dependent linguistic information, $T_{t}$ denotes the text sequence length, and $d_{s}$ denotes the token-based vocabulary size.

By using Cross-modal knowledge transfer (CMKT), we can force the visual-acoustic model to encode context-dependent linguistic information. This is achieved by introducing additional losses from the linguistic modality, which closes the distance between visual-acoustic features from the left branch and linguistic guidance from BERT. However, the dimension and distribution are quite different between visual-acoustic feature $\bf H\in\mathbb{R}^{T_{a}\times d_{a}}$ and linguistic guidance $\bf Z\in\mathbb{R}^{T_{t}\times d_{t}}$. To solve the problem of this huge domain gap, we adopt the concept of “Cross-Modal Alignment” and “Cross-Modal Neural Adapter” proposed by Lu et al.[27] in our CMKT system.

Cross-Modal Alignment consists of a fully connected layer $\bf FC1$ and one of the feature fusion layers, which aims to transfer linguistic knowledge from BERT to our visual-acoustic modality. To achieve this, we force the fusion of the encoded-text embedding and visual-acoustic latent features to behave like our linguistic guidance from BERT. However, since the feature dimension is different between visual-acoustic features and encoded text embedding, a fully connected layer $\bf FC1$ is used to project visual-acoustic features into the visual-acoustic latent features as:

Where ${\bf H}$ is the visual-acoustic feature from the encoder output of the Generative model, ${\bf\tilde{H}}$ is the visual-acoustic latent feature, which has the same dimension as encoded text embedding, $T_{a}$ denotes the length of the visual-acoustic feature, and $d_{t}$ represents feature dimension of text representation.
In this study, we implement two types of fusion layers models, termed “Multi-Head Cross-Attention (MHCA)” and “Optimal Transport (OT)” respectively. Both of them aim to fuse the information from visual-acoustic and information from linguistic modality.

For Multi-Head Cross-Attention (MHCA), we fuse the feature by multiple cross-attention layers [21]. The extraction of encoded-text embedding is:

Where ${\bf Z}_{0}$ is text embedding from the tokenizer, $EMB$ is a token embedding layer, and ${\bf PE}_{T}$ is position encoding of text embedding. In each cross-attention layer, the transform is formulated as:

Where “MHCA” denotes the multi-head cross-attention layer, and “${i}$” is the index of the layer with values from 1 to ${M}_{t}$. After ${M}_{t}$ cross-attention layers, we apply a layer norm and a linear transform, the context-dependent visual-acoustic feature (Cross-Modal Embedding)${\bf\tilde{Z}}_{f}$ can be obtained from:

Where “LN” denotes layer norm, “$\rm{FC}_{H}$” is a full-connected linear transform and “${d}_{s}$” denotes the token-based vocabulary size.

Optimal Transport (OT) is another fusion process. The original OT is proposed by Villani et al.[22], which transports a distribution to another with the minimum amount of cost. Lu et al.[27] use this concept for CMKT and prove that OT is successful in Speech Recognition. We adopt a similar concept, which finds the optimal transport matching between visual-acoustic modality and linguistic modality. This projection matrix will then be used for efficient projection between two feature spaces. In the training process, this projection matrix is obtained by an OT Loss defined below:

Where $L_{{\rm OPT}}$ is the OT Loss for finding an optimal transport matrix, $\gamma\in\mathbb{R}^{T_{t}\times T_{a}}$ is a transport plan matrix, $\prod{U({{\bf Z},{\bf H}})}$ is a set of transport plan between two distributions, $C({{\bf z}_{i},{\bf h}_{j}})$ is a transport cost between ${\bf z}_{i}$ and ${\bf h}_{j}$, where ${\bf z}_{i}$ and ${\bf h}_{j}$ are column vectors in ${\bf Z}$ and ${\bf H}$ respectively. Note that the inputs of this loss function are text embedding from the tokenizer and visual-acoustic latent features. For simplification, we denote ${\bf Z}_{0}$ as ${\bf Z}$, ${\bf\tilde{H}}$ as ${\bf H}$ in this equation.
The transport cost $C({{\bf z}_{i},{\bf h}_{j}})$ is explicitly defined as a cosine distance function below:

The optimal transport plan matrix $\gamma^{*}$ can be obtained by minimizing OT Loss:

To solve this equation, we adopt the “Sinkhorn Attention” proposed by Tay et al.[29] and C. Villani et al. [22]. Minimizing OT Loss by Sinkhorn Attention proves to be successful on Speech Recognition tasks by Lu et al.[28]. We can obtain the final optimal transport plan matrix by the transport cost and projection process, which can be implemented as:

Where $\beta$ and ${\rm C}$ in Eq. (22) define the initial condition $\gamma^{0}$ of transport coupling. $\gamma^{k}$ is the transport coupling at the current iteration process, ${\rm F_{c}(.)}$ and ${\rm F_{r}(.)}$ are column-wise and row-wise normalization respectively. The normalization processes are formulated as:

After several iteration processes, we obtain the optimal transport plan matrix, and the context-dependent visual-acoustic feature (Cross-Modal Embedding) ${\bf\tilde{Z}}_{f}$ can be estimated from:

To further refine the feature representation in our Generative model, Cross-Modal Neural Adapter is attached to the Generative model in the left branch, which is represented by another fully connected layer. The linear transform can be defined as:

Where ${\bf\hat{H}}$ is the visual-acoustic projected feature, $\bf FC2$ is a fully-connected linear transform and $d_{a}$ denotes the dimension of visual-acoustic feature. As the output of $\bf FC1$, ${\bf\tilde{H}}$ will be sent into Cross-Modal Alignment for knowledge transfer, and we assume that after the training stage, it contains context-dependent information from the linguistic modality. After projecting ${\bf\tilde{H}}$ back to the visual-acoustic feature space, ${\bf\hat{H}}$ helps visual-acoustic features to encode context-dependent information. This feature refinement further boosts the feature representation in the Generative model. The process can be formulated as:

Where “$\beta$” is a weighting coefficient of Cross-Modal Neural Adapter, which controls how much linguistic knowledge from the linguistic modality can affect the visual-acoustic representation in the Generative model. The new representation of the visual-acoustic feature ${\bf H}_{t}$ will then be sent into the Up-sampling layers of the Generative UNet for the later diffusion process. (subscript ‘_t’ denotes visual-acoustic features incorporating linguistic modality).

In the training stage, we force the visual-acoustic modality encoding context-dependent information by an additional Cross-Modal Alignment Loss. This loss can be estimated by the cosine distance function given below:

Where “$L_{{\rm align}}$” denotes the Cross-Modal Alignment loss. By minimizing the cross-modal transfer loss defined in Eq. (29), we suppose that feature representation in visual-acoustic modality encodes rich linguistic information.

For AVSE with MHCA, given speech spectrograms (including a noisy one and a clean one), lip movement, and corresponding text token sequence, we can train an AVSE system with Linguistic Cross-Modal Knowledge transfer with the total loss defined as:

Where $L_{{\rm denoiser}}$ is the loss of Predictive model, $L_{{\rm score}}$ is the loss of Generative score model, $L_{{\rm align}}$ is Cross-Modal Alignment loss defined in Eq. (29), $\omega$ weights Generative model loss and Predictive model loss, $\alpha$ weights Cross-Modal Alignment. For AVSE with OT, we aim to find an optimal transport matrix between two distributions. To achieve this, we introduce an additional OT Loss in our total loss, which is defined below:

Where $L_{{\rm OPT}}$ is OT Loss defined in Eq. (21). After the model is trained, only the left branch of our AVSE system (see Fig. 1) is kept for inference, which saves the parameters and computing costs.

Our experiments are conducted on Taiwan Mandarin speech with video (TMSV) dataset and 3rd AVSE Challenge [44] dataset extracted from The Lip Reading Sentences 3 (LRS3). TMSV dataset includes speech recordings from 18 native Mandarin speakers (13 males and 5 females). Each speaker recorded 320 Mandarin sentences with each sentence consisting of 10 Chinese characters. The total duration of each utterance ranges from approximately 2 to 4 seconds. To ensure methodological congruence and experiment reproducibility, we followed the procedures detailed in [30] for the introduction of noise interference into the dataset and train-test split. The 3rd AVSE Challenge dataset contains the speech from 37823 speakers, and each speaker with a 2-10 seconds video. For simplicity, we abbreviate this dataset as “LRS3” in the later sections.

For visual feature extraction, we utilize the Mediapipe [45] face tracker to identify facial landmarks. The final visual crop is a mouth ROI region with size of 96 × 96. Note that each frame is also normalized by the mean and standard deviation of the training set.

For acoustic feature-preprocessing, we follow the experiment settings in “SGMSE”[26]. We use a sampling rate of 16 kHz to convert audio recordings into a complex-valued STFT representation using a window size of 510, resulting in F = 256, and a periodic Hann window, hop length is 128 for TMSV and 160 for LRS3. Moreover, the length of each spectrogram is trimmed to T = 256 STFT time frames to enable multiple examples for batch training with randomly selected start and end times.

For the Predictive model, we adopt and modify NCSN++ in [23]. Since the network is originally used for generative purposes, the timestep is set to 1 and the output score of the model serves as the preliminary-denoised speech spectrogram. For the model architecture of the UNet, please refer to the original paper.

For the Generative model, we also adopt and modify NCSN++ in [26] as our backbone model. For the Stochastic Process, we set $\bf{\sigma}_{min}$ = 0.05, $\bf{\sigma}_{max}$ = 0.5, and $\bf{\gamma}$ = 1.5. We track an exponential moving average of the DNN weights with a decay of 0.999 to be used for sampling. Note that the input spectrograms $({\bf x_{t}},{\bf\hat{y}})$ are all represented in the complex STFT domain. Since NCSN++ only works with real value, the real parts and imaginary parts of the complex inputs are separated into different channels before being sent into the model. After that, the real value channels and imaginary value channels will be concatenated back as complex outputs before calculating loss.

For the sampler used in the inference stage, we use a Predictor-Corrector (PC) Sampler with reverse steps 30, which includes a Reverse Diffusion Predictor (RDP) and an Annealed Langevin Dynamics (ALD) corrector with corrector steps 1.

In the linguistic modality, we use the pre-trained checkpoints “bert-base-chinese” for TMSV, and “bert-base-cased” for LRS3. Both models from huggingface [46] are fine-tuned in our task. In our BERT model, there are $M=12$ transformer encoders. Token-based vocabulary size is 21128 for TMSV and 28996 for LRS3, and the dimension of the linguistic feature representation is $d_{t}=768$. Since the dimension of visual-acoustic features is $(B,C,F,T)=(B,256,4,4)$ after downsampling in our Generative model, the fully-connected transform $\bf FC1$ in Fig. 1 is a $768*4$ weight matrix in order to match feature dimensions between visual-acoustic and linguistic modalities. Similarly, $\bf FC2$ is a $4*768$ weight matrix that transforms the visual-acoustic latent features back to the feature size of visual-acoustic space. For MHCA, we use attention layers with ${M}_{t}$ = 6, and each layer is a 4-head multi-head cross-attention layer. $\omega$ is 0.5, $\alpha$ is 0.2 in Eq. (30) For OT, $\omega$ is 0.5, $\alpha$ is 0.01 in Eq. (31), $\beta$ is 0.5 in (24). The weighting coefficient of cross-modal neural adapter $\beta$ is 0.1 in Eq. (28). For training configuration, we use the Adam optimizer with a learning rate of $10^{-4}$ and batch size of 1.

After training the entire AVSE system, only the left branch (visual-acoustic modality) in Fig. 1 will be used for inference. It means that we don’t need any text inputs to do inference. Moreover, compared to the system without CMKT, we only need two additional fully connected layers to enhance speech.

We first examine the effectiveness of using visual modality and linguistic modality to benefit speech regression tasks. We expect that the visual and linguistic information serves as the additional hint for the acoustic model and thus has better performance. We first compare the results of three types of models on TMSV dataset. All of them are based on our proposed method. The results of them are shown in Table I.

Where “Noisy” represents the original noisy input, “A” represents that we don’t integrate visual embeddings and Cross-Modal Knowledge Transfer into our AVSE system and train the model with speech ground truth only (pure acoustic model). “A+L” represents that we don’t integrate visual embeddings but enable Cross-Modal Knowledge Transfer with acoustic loss and linguistic Loss. “A+V+L” represents our proposed full AVSE framework with Cross-Modal Knowledge Transfer. In this table, we can first observe that incorporating linguistic modality can significantly benefit the original acoustic model. That’s why “A+L” surpasses “A” on TMSV. Moreover, we can include visual modality to further improve the performance of the model, and therefore “A+V+L” yields the best result.

We then compare the results our AVSE system with other benchmark models on TMSV and LRS3. The results of them are shown in Table II and Table III respectively.

In these tables, the model performance was mainly evaluated based on the Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), and Scale-invariant Signal-to-Distortion Ratio (SI-SDR). Moreover, we also examine the modalities used in each model, where “A” represents acoustic modality, “V” represents visual modality, and “L” represents linguistic modality.

For our method, “Ours” represents our AVSE system without Cross-Modal Knowledge Transfer, “Ours (+MHCA)” represents our AVSE system with MHCA, and “Ours (+OT)” represents our AVSE system with OT. “Baseline” represents the baseline model in the 3rd AVSE Challenge [44]. Table II shows that the AVSE system with MHCA surpasses all the other SOTA on TMSV datasets. On the other hand, Table III shows that AVSE system with OT surpasses the 3rd AVSE Challenge baseline model and several benchmark models on LRS3 datasets. These tables not only justify the effectiveness of our AVSE system but also show that Cross-Modal Knowledge Transfer enables the visual-acoustic model to learn linguistic knowledge during the training stage and therefore contributes to the improvement of the AVSE system.

To further investigate the effect of adding linguistic modality, we also performed spectrogram analysis on noisy speech, clean speech, and enhanced speech generated by our proposed model. We visualize an example in the development set on TMSV. The visualized spectrograms can be seen in Fig.3. Where “Acoustic”, “Acoustic + Linguistic”, “Acoustic + Visual”, and “Acoustic + Visual + Linguistic” is the modality model used by the model. At the end of the clean speech spectrogram, the power of the low-frequency component is much smaller. This is because the speaker stops to utter at this moment. The visualization results show that “Acoustic + Linguistic” has better noisy compression capability than “Acoustic” at the end of the speech. The reason is probably that word-based tokens help the model build the correspondence between the frequency frames and the word sequence. This additional token information serves as a hint for the model to identify the difference between pure noise and noisy word utterance. It enables the model to generate enhanced frames with lower power at the end of the speech, which is closer to the clean ground truth. Therefore, “Acoustic + Linguistic” gives better results.

Moreover, we can see that the high-frequency component is much smaller in the clean speech spectrogram. This is because the human voice mainly concentrates on the low-frequency components. The visualization results show that “Acoustic + Visual + Linguistic” has better noise compression than “Acoustic + Visual” on the high-frequency components. The reason is probably that word-based tokens help the model build the correspondence between the frequency frames and the word sequence. This additional token-frame correspondence serves as a hint for the clean word utterance at each moment. It enables the model to generate enhanced frames closer to the clean ground truth. Therefore, “Acoustic + Visual + Linguistic” gives a better result. The visualization results also show that “Acoustic + Visual + Linguistic” is the best among all settings. Our proposed method effectively suppresses the noise components present in the noisy speech spectrogram, which justifies the noise reduction capability of our model.

In this study, we adopt two models in our Cross-Modal Knowledge Transfer. One is the Cross-Modal Neural Adapter, which corresponds to the gray part in the Generative UNet in Fig. 1 and the other is Cross-Modal Alignment, which corresponds to the green parts in Fig. 1. To examine the effectiveness of these two methods, we conduct two additional experiments. In the first experiment, we attach the Cross-Modal Neural Adapter to the acoustic model and train this model with acoustic loss only (without CMKT). In the second experiment, we use an additional Cross-Modal Alignment model to train the ASE system with acoustic loss and linguistic loss but without attaching the Cross-Modal Neural Adapter to the acoustic model. Note that to further clarify the effectiveness of our CMKT model, we don’t integrate the visual embedding and use only the generative stage in our proposed model. It means that the Acoustic Generative model is used in this section. The results are shown in Table IV.

In this table, the entry labeled “AGen” indicates the Acoustic Generative model in our system, “AGen + Adpt” indicates the Acoustic Generative model with Cross-Modal Neural Adapter (without right branch in Fig. 1), and “AGen + Attn” indicates that we apply MHCA Cross-Modal Alignment in model learning, but the adapter does not connect to the Acoustic Generative model (without $\bf FC2$ and feedback addition). “AGen + MHCA” doesn’t incorporate visual information into our proposed Generative model, but enables the entire MHCA Cross-Modal Knowledge Transfer.

The results show that the “AGen + Attn” is better than the original model “AGen”. However, since it’s without attaching Cross-Modal Neural Adapter, the feature refinement in the visual-acoustic modality is limited, and that’s probably the reason why it cannot surpass the performance of “AGen + MHCA”. On the other hand, we can observe that in “AGen + Adpt”, without the context-dependent information of Cross-Modal Alignment, the linear layer $\bf FC2$ with feedback addition in the UNet can not only enable knowledge transfer capability but also lead to overfitting. Therefore, although the PESQ of “AGen + Adpt” is slightly better than the original acoustic model “AGen”, the SI-SDR is significantly decreased. The result of our ablation study explains the effectiveness of using Cross-Modal Alignment and Cross-Modal Neural Adapter in our linguistic modality.

We also conduct additional experiments on the effect of selected features for CMKT. In this study, we use the visual-acoustic feature after the Cross-Attention block in UNet for CMKT. To clarify the effect of the selected feature, we conduct the following experiments in Table V.

In this table, “Down-Sample Layer” represents that we use the feature after the second 2D Down-Sample layer, “Cross-Attention Layer” represents that we use the feature after the Cross-Attention block, and ‘Up-Sample Layer” represents that we use the feature after the last 2D Up-Sample layer in Acoustic Generative UNet. Note that we evaluate the performance of our model on TMSV Validation set. The results can also be attributed to domain gaps. In the ”Down-Sample Layer,” only a few layers are encoded with linguistic information, and this refinement is too weak to effectively bridge the domain gap. In contrast, in the ”Up-Sample Layer,” nearly all layers are linguistically encoded, causing significant information loss in the visual-acoustic domain. As a result, the ”Cross-Attention Layer” proves to be the optimal setting, explaining why we utilize the features from the Cross-Attention block for Cross-Modal Knowledge Transfer (CMKT).

Throughout our experiments, we observed that the values of hyperparameters in Cross-Modal Knowledge Transfer have a great impact on performance improvement, especially the weight controlling loss function. To examine the importance of the parameters chosen, we plan to conduct ablation studies on the weight of Cross-Modal Alignment $\alpha$ and the weight of Cross-Modal Neural Adapter $\beta$. These two hyperparameters control the extent of domain gap bridging and significantly impact the effect of linguistic knowledge transfer. Moreover, we plan to conduct experiments on other benchmarks such as VoxCeleb2 and Lip Reading in the Wild (LRW), which further examine the generalizability of our approach. Furthermore, since it needs more parameters and computations for the linguistic guided training, we also plan to find a more compact implementation of our linguistic modality in our future study.