Deep Complex U-Net with Conformer for Audio-Visual Speech Enhancement

Effective communication through spoken language is the cornerstone of human interaction. In various fields such as telecommunications, voice assistants, hearing aids, and video conferencing, ensuring the quality and intelligibility of speech is crucial. However, the challenge lies in the ability to consistently achieve good quality and intelligibility, especially in adverse acoustic environments characterized by background noise, reverberation, or limited audio quality. In the pursuit of addressing these challenges and enhancing speech quality and intelligibility, speech enhancement (SE) has emerged as a critical area of research and development [1]. The field of SE has witnessed significant advancements through the integration of deep learning methods. While deep learning-based SE techniques [2, 3] have demonstrated remarkable success by primarily focusing on audio signals alone, it is notable that incorporating the visual modality can bring substantial benefits to improving the performance of SE systems in challenging acoustic environments [4, 5, 6].

Depending on the input type, audio-only SE methods can be roughly divided into two categories: time-domain methods and time-frequency (TF) domain methods. Conventional TF domain methods typically rely on amplitude spectral features; yet, studies have shown that SE performance can be limited because phase information is not adequately considered  [3]. To address this issue, approaches that employ complex-valued features, such as complex spectral mapping (CSM) [7] and complex ratio masking (CRM) [8], as SE input have recently been proposed. Many CSM and CRM techniques are built upon real-valued neural networks, while others employ complex-valued neural networks to process the complex input. Deep Complex U-NET (DCUNET) [9] and Deep Complex Convolution Recurrent Network (DCCRN) [10] are notable complex-valued neural networks for the SE task. In this study, we also use a complex-valued neural network to process audio data.

The core idea behind audio-visual speech enhancement (AVSE) is to integrate visual input as supplementary data into an audio-only SE system, aiming to improve SE performance with the help of supplementary information. Several earlier studies have demonstrated the efficacy of incorporating visual input to enhance the performance of SE systems [4, 11, 12]. Most previous AVSE systems focused primarily on processing audio in the TF domain [13, 6]. Some previous studies performed audio-visual speech separation tasks in the time domain [14]. Recently, self-supervised learning (SSL) embeddings have been used to improve the performance of AVSE. For example, Richard et al. [15] introduced the SSL-AVSE method, which integrates visual cues with audio signals. These combined audio-visual data are fed into a Transformer-based SSL AV-HuBERT model to extract features, which are subsequently processed using a BLSTM-based SE model.

In this paper, we propose a novel AVSE approach, termed DCUC-Net (deep complex U-Net with conformer network). DCUC-Net employs a deep complex U-Net architecture to incorporate phase information into AVSE. The audio stream is processed by a complex encoder to create a complex representation of the audio data and then combined with visual features. To further enhance the combined audio-visual features, DCUC-Net integrates conformer blocks into the deep complex U-Net architecture. These conformer blocks enable DCUC-Net to effectively capture long-range dependencies, both local and global, and fine-grained contextual information between audio and visual modalities. The resulting features are then decoded by a complex decoder to estimate a complex mask. By multiplying the complex spectra from the noisy input with the estimated complex mask, we can obtain enhanced complex spectra, which are then converted to enhanced waveforms as the final output. We evaluate the performance of DCUC-Net through experiments on the dataset used in the COG-MHEAR AVSE Challenge 2023, which is built on the LRS3 dataset [16], and the Taiwan Mandarin speech with video (TMSV) dataset. The results demonstrate the efficacy of DCUC-Net, as it surpasses existing baselines and demonstrates robust denoising capabilities.

The proposed DCUC-Net aims to obtain enhanced speech signals with improved intelligibility and quality via the integration of audio and visual information. Figure 1 shows the overall architecture of DCUC-Net. In DCUC-Net, we utilize ResNet-18 to process lip movement cues, thereby preparing the visual information. Additionally, we employ a complex encoder to transform the audio input into complex-valued features. The complex encoder of DCUC-Net comprises five Conv2d blocks, which are designed to extract encoded features from the audio. These extracted audio features are then combined with visual information, and we leverage conformer blocks [17] to effectively process this combined audio-visual information. During decoding, the symmetric complex-valued encoder-decoder structure is considered and used to reconstruct the lower-dimensional representation to the original size of the input, facilitating the restoration of the enhanced speech waveforms. The complex encoder and decoder block in DCUC-Net is constructed based on the implementation details described in [18]. It comprises three main components: complex Conv2d [10], complex batch normalization [18], and real-valued PReLU [19]. The complex Conv2d operation enables the manipulation of complex-valued features. In this operation, the complex-valued convolutional filter, denoted as $W$, is defined as $W=W_{r}+jW_{i}$, where $W_{r}$ represents the real part, and $W_{i}$ represents the imaginary part. Similarly, the input complex spectra, denoted as $X$, is defined as $X=X_{r}+jX_{i}$, where $X_{r}$ and $X_{i}$ denote the real and imaginary parts, respectively. The complex convolution operation, denoted as $X\bigotimes W$, is performed to obtain the complex output. The output feature of each complex layer, represented as $F_{out}$, is calculated as:

Equation (1) within DCUC-Net enables the manipulation of both the real and imaginary parts of the complex-valued features, facilitating effective processing of the audio part.

Expanding on this capability, we integrate the extracted visual features into DCUC-Net by concatenating them with the real values from the complex audio feature. However, this concatenation process introduces a sequential misalignment between the visual and audio features. To address this discrepancy, we employ a temporal upsampling technique to align the temporal sequences of the visual features with the temporal resolution of the audio features. This alignment process ensures proper fusion of the two modalities. Subsequently, the aligned audio and visual features are input into the Conformer blocks.

The Conformer block comprises two Feed-Forward (FFN) modules positioned around the Multi-Headed Self-Attention (MHSA) module and the Convolution (Conv) module. In contrast to the original Transformer block, which incorporates a single feed-forward layer, the Conformer block features two half-step feed-forward layers: one preceding the attention layer and one succeeding it. To put it simply, when the $i$ Conformer block receives an input, denoted as $x_{i}$ the resulting output $y_{i}$ is determined through the following mathematical expression:

The conformer blocks play a critical role in capturing both global and local audio-visual dependencies within our DCUC-Net. These blocks facilitate the effective combination of the audio and visual modalities by leveraging self-attention mechanisms and convolutional operations. The self-attention mechanisms establish relationships between different parts of the audio and visual features, enabling the model to capture long-range dependencies and effectively leverage global audio-visual cues. Simultaneously, the convolutional operations extract local contextual information, ensuring the model’s robustness in capturing fine-grained audio-visual details.

With the incorporation of the audio and visual information, the output of the conformer blocks is passed to the complex decoder component in DCUC-Net. The role of this decoder is to estimate the complex ratio mask and obtain enhanced complex spectra by multiplying them with the noisy complex spectra. Subsequently, the complex spectra are transformed into an enhanced speech waveform by applying the Convolutional inverse Short-Time Fourier Transform (ConviSTFT). It’s worth noting that the decoder benefits from a skip-connection mechanism, which provides access to encode features via the U-Net architecture.

This paper introduces a novel AVSE framework in which we incorporate lip movement cues as visual features into the audio stream, and we emphasize the importance of phase information by leveraging complex features in a deep complex U-Net architecture. The visual features are combined with the output of the complex encoder for the audio stream, and the resulting concatenated features are processed by conformer blocks. The proposed framework effectively captures global and local audio-visual dependencies. Experimental results on the COG-MHEAR dataset demonstrate the superior performance of the proposed DCUC-Net AVSE framework over two baselines. Furthermore, on the TMSV dataset, DCUC-Net performs comparably to a state-of-the-art model that uses the pre-trained self-supervised learning AV-HuBERT model for feature extraction, and outperforms all other compared models.