Extended U-Net for Speaker Verification in Noisy Environments

Recently, deep learning has been applied in various fields with tremendous success, and in speaker verification (SV) tasks, deep neural network (DNN)-based embedding learning methods [1, 2] exhibit satisfactory results compared to traditional schemes [3, 4]. Most embedding extractors perform reliably in clean scenarios but suffer from performance degradation in noisy environments. It is challenging to delicately and accurately extract speaker information from noise-contaminated utterances, because the background noise undermines the intelligibility and quality of speech [5, 6].

Several studies have used a pretrained enhancement model as a front-end module for an embedding extractor (Figure 1 (a)) to address this problem [7, 8, 9]. A denoising autoencoder, a traditional enhancement method, is trained to map noise speech to a clean counterpart using a loss function, such as the mean squared error (MSE) [7, 10]. The denoising autoencoder can improve the auditory quality of noise speech, leading to better results for noisy SV scenarios. Nonetheless, the performance improvement for clean input is marginal, and even independent compensation processes not customized to downstream tasks may distort speaker information [11, 9].

VoiceID loss [11] was proposed to construct an enhancement model specialized for SV tasks. As presented in Figure 1 (b), the mask output from the enhancement model is multiplied by the input feature element-wise to filter out redundant information polluted by noise. Then, the refined feature is fed to the pretrained SV model, and only the weights of the enhancement model are updated to reduce the speaker identification (SID) loss. This framework can generate an enhancement model suitable for SV but cannot jointly optimize both models. Therefore, the fixed SV model cannot continuously learn the modified distributions of enhanced features, which may lead to information mismatch [12].

To prevent the distortion of speaker information and mitigate information discrepancy caused by independent enhancement, we argue that the embedding extractor and enhancement model should be trained in a fully joint, rather than exclusive, manner. The U-Net [13] is an image-to-image translation network proposed in the biomedical image segmentation field based on the encoder-decoder structure, which delivers a processed output with the identical (or analogous) size of the input data. Due to the network’s structural nature, the U-Net has been widely used in speech processing tasks, such as voice activity detection [14] or speech enhancement [15]. Inspired by the architectural characteristics of the U-Net and its successful adaptation for the speech domain, we propose to exploit the U‑Net framework to optimize SID and feature enhancement (FE) losses simultaneously. The proposed U-Net-based system is depicted in Figure 1 (c). This model is jointly trained to classify the speaker identity of the embeddings transformed from the intermediate features while reducing the Euclidean distance between the enhanced features from the decoder and its clean versions. Thus, the U-Net-based system can directly derive a noise-robust speaker embedding through this training process.

Although we devised a joint-trainable framework, the proposed U-Net-based system has several structural limitations. First, this system did not use enhanced features from the decoder to extract speaker embeddings. Furthermore, noise compensation is performed only with the encoder in the evaluation phase without using a decoder. Therefore, we hypothesize that the vanilla U-Net is not an ideal fully joint training structure for noise SV tasks and further propose an extended U-Net (ExU-Net) to improve these shortcomings. The ExU-Net (Figure 1 (d)) is jointly trained by combining an additional embedding extractor with the U-Net framework. This extended structure can extract embeddings using the enhanced features and entirely exploit the jointly trained model for the evaluation. Moreover, we applied a metric learning loss for direct embedding enhancement (EE).

The models were trained using VoxCeleb 1 [16] training data and the MUSAN corpus [17]. For evaluation in noisy environments, we used the VoxCeleb 1 test set synthesized from MUSAN and the VOiCES [18] development dataset containing various noise sources. The experiment results demonstrated that the U-Net-based system achieved improved performance compared to the baseline and that the ExU-Net exhibited state-of-the-art results for noisy scenarios.

The SV studies for noisy environments have been conducted from various perspectives. Several researchers have focused on algorithmic methods that compensate for noisy utterances directly at the signal level [19, 20]. At the feature level, data augmentation [6] and feature normalization [21] techniques were used to construct noise-robust SV systems. The SV research for noisy utterances has also been studied at the model level. Using a separate enhancement model as a front-end or back-end system for the embedding extractor can prevent performance deterioration due to noise [11, 22]. Furthermore, Cai et al. [6] proposed specialized loss functions that induce noise-mitigated speaker embeddings, and Wu et al. [12] introduced asynchronous subregion optimization to avoid collision between losses. The present study also explored improving the performance of the noisy SV task at the model level and focused on a fully joint training scheme to alleviate the distortion of speaker information that may occur during noise compensation.

To mitigate the adverse effects caused by independent noise compensation, we speculate that the SV system should be trained by simultaneously optimizing the SID and FE losses. This section explicates the proposed fully joint training framework. As illustrated in Figure 2, we describe the models according to the scope of the framework in the following order: baseline, U-Net, and ExU-Net.

In Table 2, we compare the models with the recently proposed SV systems for noisy environments on the VoxCeleb test set. For the convenience of description, we calculated the average of the EER and $C_{det}^{min}$ derived from all evaluation conditions. The baseline system trained only on the original clean data $D$ has difficulty coping with noise utterances, resulting in significant degradation under noisy scenarios. When comparing the results of the baselines under the two training dataset conditions, simply using augmented training data $D^{N}$ could enhance the robustness to noise utterances (6.12% vs. 10.85%). In addition, the proposed U-Net-based system (U-Net) showed a relative error reduction (RER) of 8.5% based on the average EER compared to the baseline trained using $D+D^{N}$ (5.6% vs. 6.12%). The ExU-Net proposed in this study outperformed other models based on the original test trial and achieved state-of-the-art performance in all noise scenarios, with a 25.7% of RER vs. the baseline (4.55% vs. 6.12%). Furthermore, to demonstrate the superiority of the ExU-Net structure, we additionally constructed a lightweight variant of the ExU-Net (ExU-Net-L) by adjusting the number of parameters similar to the baseline. The ExU-Net-L demonstrated better results than the baseline and even the U-Net and exhibited tolerable performance compared to the recently proposed models. Based on these results, we interpreted that the fully joint training scheme is effective for noisy SV tasks, and the proposed ExU-Net is further robust to noisy environments, including clean scenarios.

Table 3 presents the results of the ablation experiments to analyze the effectiveness and appropriateness of each ExU-Net loss function. The model was evaluated based on the EER and $C_{det}^{min}$ of the original test set and the mean of all noise-type evaluations. Systems 1 and 3 exhibited notable performance degradation compared to the ExU-Net (System 5), whereas System 2 had comparable results even though FE learning was excluded. These results indicate that the SID loss is crucial to the generalization of the proposed model, and noise compensation at the embedding level is more effective than at the feature level. System 4 exhibited greater deterioration in the original evaluation conditions than the noise test sets compared to System 5. These results imply that the MSE used for EE learning can compensate for noise utterances by reducing the Euclidean distance between pairs of clean-noise embeddings, but it cannot consider the inter-class variance between speakers as opposed to APN loss.

Table 4 delivers the results of the VOiCES development set for each model. All models were trained using the VoxCeleb1 training set ($D+D^{N}$). Similar to the previous results, the U-Net displayed improved performance compared with the baseline. In addition, ExU-Net achieved the best performance among all models and exhibited an RER of 15.3% vs. the baseline. Through the results for the evaluation dataset in different domains from the training set, the models demonstrated superior generalization.

Focusing on the drawbacks of the independent enhancement module for noise utterance in SV tasks, we emphasized the necessity of a fully joint training scheme between noise compensation and SV models. This paper proposed models that simultaneously optimize feature (and embedding) enhancement and SID losses using the U-Net or extending it (ExU-Net). Due to the joint learning of two or three tasks, the proposed models can extract noise-robust speaker embeddings customized for SV tasks. The evaluation of the noise-synthesized VoxCeleb1 test set demonstrated that the U-Net-based system exhibited improved performance compared to the baseline, and ExU-Net achieved state-of-the-art performance. In addition, we proved the effectiveness and validity of the ExU-Net loss functions through ablation experiments. Finally, the proposed models demonstrated outstanding generalization performance in the evaluation of the VOiCES dataset. As a future work, we will evaluate the proposed models in various real-world noise environments.

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (2020R1A2C1007081).