Effective Noise-aware Data Simulation for Domain-adaptive Speech Enhancement Leveraging Dynamic Stochastic Perturbation

The generator $G$ transforms a clean spectrogram from the source domain into its simulated counterpart. Specifically, the clean spectrogram is first encoded through two 2D downsampling convolutional layers (kernel size: $3\times 3$, stride: $2\times 2$), followed by nine residual blocks for capturing deep representations. Each residual block consists of two convolutional layers (kernel size: $3\times 3$, stride: $1\times 1$) and a dropout layer. Finally, two transposed convolutional layers (kernel size: $3\times 3$, stride: $2\times 2$) act as a decoder, upsampling the representations into the corresponding simulated spectrogram.

The discriminator $D$ differentiates between the simulated spectrogram and the real target noisy spectrogram. It consists of five 2D convolutional layers (kernel size: $4\times 4$) followed by Leaky ReLU activation functions. The stride is set to $2\times 2$ for the first three layers and $1\times 1$ for the last two. We employ the adversarial loss during training:

$$\begin{split}\mathcal{L}_{adv}(G,&D,\mathbf{X}^{T},\mathbf{Y}^{S},\mathbf{N}^{T})=\mathbb{E}_{\mathbf{x}\sim\mathbf{X}^{T}}\left[\log D(\mathbf{x})\right]\\ &+\mathbb{E}_{\mathbf{y}\sim\mathbf{Y}^{S},\mathbf{n}\sim\mathbf{N}^{T}}\left[\log(1-D(G(\mathbf{y},\mathbf{n})))\right].\end{split}$$

(1)

This loss function encourages the generator to produce a spectrogram similar to the target noisy spectrogram. Although the speech content might differ, the discriminator here focuses exclusively on the background noise for discrimination.

Drawing inspiration from recent advancements in noise-aware speech processing [18, 19], we propose the incorporation of a dedicated noise encoder, denoted by $B$ for our purpose. This encoder is tasked with extracting salient noise embeddings of the target noisy spectrogram, denoted by $\mathbf{N}^{T}$, from the outputs of its penultimate layer. By explicitly providing these noise representations to the generator module, our model gains a deeper understanding of the specific noise characteristics inherent to the target domain, enabling more informed and tailored noise synthesis.

To extract noise embeddings richly imbued with noise-specific information, we leverage a pre-trained audio model known as BEATs [20]. Pre-trained on the extensive AudioSet dataset [21], BEATs possesses a strong foundation in generating audio feature representations. To further enhance its noise discriminative capabilities, we fine-tune BEATs using a two-pronged approach: 1) We train BEATs to classify noise types on the VoiceBank-DEMAND dataset [22]. This dataset provides a diverse set of 10 distinct noise types, empowering the encoder to discern between a variety of noise characteristics. 2) We treat each utterance within the target noisy dataset used for GAN training as a unique noise type. This fine-tuning step further reinforces the noise encoder’s ability to capture the specific noise nuances present in the data it will encounter during the speech enhancement process.

To ensure faithful noise information reconstruction, we introduce the noise reconstruction loss:

$$\mathcal{L}_{nse}(G,\mathbf{Y}^{S},\mathbf{N}^{T})=\mathbb{E}_{\mathbf{y}\sim\mathbf{Y}^{S},\mathbf{n}\sim\mathbf{N}^{T}}\left[\left\|\mathbf{n}-B(G(\mathbf{y},\mathbf{n}))\right\|_{1}\right].$$

(2)

This loss function ensures that the generated samples retain the noise details present in the target domain.

We adopt Feature-wise Linear Modulation (FiLM) [23] to incorporate noise embeddings into the generator module. At the outset, the noise embedding $\mathbf{N}^{T}$ undergoes separate linear transformations to produce weight $\mathbf{W}$ and bias $\mathbf{b}$:

$$\mathbf{W}=Linear(\mathbf{N}^{T}),\>\mathbf{b}=Linear(\mathbf{N}^{T}).$$

(3)

Following this, the output feature $\mathbf{F}$ from a specific layer within the generator is multiplied by $\mathbf{W}$ and biased by $\mathbf{b}$:

$$\mathbf{F}^{\prime}=\mathbf{W}\times\mathbf{F}+\mathbf{b}.$$

(4)

Finally, the adjusted feature $\mathbf{F}^{\prime}$ propagates through the remaining layers of the generator. This approach is applied to the output features of the encoder layer and all nine ResNet blocks within the generator.

To preserve linguistic consistency between the generated noisy speech and the original clean audio, we employ patch-wise contrastive learning, as introduced by Park et al. [24], which homes in maximizing the mutual information, specifically the shared speech content information, between the source clean and the simulated spectrograms.

We utilize the generator to extract feature representations for both the clean and simulated spectrograms. Within these representation spaces, a small patch from the simulated representation is selected as the “query”. The corresponding patch from the clean representation serves as the “positive” sample. To provide contrasting examples, we randomly sample 256 patches from the clean representation to act as “negative” samples. These patches are then projected into a lower-dimensional embedding space using two linear layers with 256 units each, followed by ReLU activation. The contrastive loss, calculated across five layers of the generator’s feature hierarchy, measures the cross-entropy loss between the “query” patch and both the positive and negative patches. This formulation encourages the model to learn representations where corresponding patches in the clean and noisy domains exhibit high similarity, while simultaneously being easily distinguishable from randomly sampled patches. Formally, the patch-wise contrastive learning loss is defined by:

$\mathcal{L}_{pcl}(G,\mathbf{Y}^{S})=\sum\limits_{l=1}^{L}\sum\limits_{i=1}^{I}-\log\left[\frac{e^{\left(\hat{z}_{l}^{i}\cdot z_{l}^{i}/\tau\right)}}{e^{\left(\hat{z}_{l}^{i}\cdot z_{l}^{i}/\tau\right)}+\sum_{j=1}^{J}e^{\left(\hat{z}_{l}^{i}\cdot z_{l}^{j}/\tau\right)}}\right]$,

(5)

where $z_{l}^{i}$ represents the $i^{th}$ positive patch from clean representations at the $l^{th}$ layer of the generator, $\hat{z}_{l}^{i}$ denotes the corresponding patch from simulated representations, and $z_{l}^{j}$ refers to the $j^{th}$ negative patch from simulated representations at the same layer. The temperature parameter $\tau$ controls the contrastive learning process. We apply the loss function to both source clean ($\mathcal{L}_{pcl}(G,\mathbf{Y}^{S})$) and target noisy ($\mathcal{L}_{pcl}(G,\mathbf{X}^{T})$) spectrograms to allow for consistent speech content and minimize unnecessary alterations.

During the training phase, the main objective is to optimize the GAN model using a comprehensive loss function that integrates multiple components. The total loss function combines adversarial loss, patch-wise contrastive learning losses for both source clean and target noisy spectrograms, and the noise reconstruction loss, which is formulated by:

$$\begin{split}\mathcal{L}_{total}=&\mathcal{L}_{adv}(G,D,\mathbf{X}^{T},\mathbf{Y}^{S},\mathbf{N}^{T})+\mathcal{L}_{pcl}(G,\mathbf{Y}^{S})\\ &+\mathcal{L}_{pcl}(G,\mathbf{X}^{T})+\lambda_{nse}\mathcal{L}_{nse}(G,\mathbf{Y}^{S},\mathbf{N}^{T}),\end{split}$$

(6)

where $\lambda_{nse}$ is a hyperparameter that regulates the influence of the noise reconstruction loss.

Given limited noisy data from the target domain, the same amount of clean speech are randomly sampled from the source domain. Subsequently, our model is trained using the unpaired dataset and optimized with $\mathcal{L}_{total}$ in Eq. (6).

After training, our well-trained generator serves as a domain converter $F^{S\sim T}$ from $\mathbf{Y}^{S}$ to $\mathbf{X}^{T}$, working alongside the fine-tuned noise encoder to simulate noisy speech. The simulation utilizes abundant clean speech data and randomly selected target noisy speech from the training phase, providing sufficient paired data of clean and simulated speech. This augmented data can be further utilized to fine-tune any downstream speech enhancement models for domain adaptation.

Dynamic stochastic perturbation is a technique used during the inference stage to improve the adaptability of speech enhancement models. It involves adding Gaussian noise, with adjustable standard deviation, to noise embeddings extracted from target noisy spectrograms. The standard deviation represents the different degrees of spread used for generating the stochastic perturbation. The dynamic noise injection allows for fine-tuning of noise levels, assisting in preventing overfitting to specific noise patterns encountered during training. Moreover, it enhances the resilience of speech enhancement models to unseen noise conditions, thereby bolstering their generalization capabilities.

We evaluated our proposed NADA-GAN method on the VoiceBank-DEMAND dataset [22], a widely used benchmark for speech enhancement. This dataset consists of noisy speech recordings generated by artificially mixing clean speech samples from the VoiceBank corpus [22] with noise samples from the DEMAND database [25]. It comprises recordings from 30 speakers, with 28 used for training and 2 reserved for testing. The training set (source domain) contains 11,572 utterances created by mixing clean speech with 10 noise types at four signal-to-noise ratio (SNR) levels (0, 5, 10, and 15 dB). Conversely, the test set (target domain) consists of 824 utterances featuring five unseen noise types at four SNR levels (2.5, 7.5, 12.5, and 17.5 dB).

To facilitate efficient training with limited target data, we randomly sampled 40 unpaired utterances: 40 clean speech utterances from the source domain and 40 noisy speech utterances from the target domain (8 utterances per noise type). Furthermore, we excluded the portion of the original test set used for training the data simulation models. Consequently, the final test set was reduced from 824 to 784 utterances.

We employed DEMUCS [26] as our downstream evaluation model. DEMUCS is a real-time SE model that operates directly in the waveform domain. It utilizes a causal encoder-decoder architecture with U-Net style skip connections and convolutional layers, enabling real-time processing. DEMUCS directly processes the raw waveform to suppress stationary and non-stationary noise, as well as room reverberations. The model optimizes for both time and frequency domain losses by incorporating an L1 loss on the waveform and a multi-resolution short-time Fourier transform (STFT) loss on the spectrogram. To further enhance robustness, data augmentation techniques, including frequency band masking and signal reverberation, are applied directly to the raw waveform during training. For comparison, we trained four variants of speech enhancement models:

1. Vanilla-SE: DEMUCS trained solely on source domain data without any domain adaptation.

2. UNA-GAN [16]: Vanilla-SE fine-tuned with a dataset generated by the baseline domain adaptation approach.

3. NADA-GAN: Vanilla-SE fine-tuned with a dataset generated by our proposed method.

4. Oracle: Vanilla-SE fine-tuned with a mixture of clean speech and the real target noise used for GAN training.

To better capture the short-time characteristics of noise, we segmented the magnitudes into frames with dimensions of $129\times 128$. This segmentation allows the noise encoder to learn more detailed noise-specific information. We trained our method for 400 epochs to learn a robust mapping between source clean speech and target noisy speech. The hyperparameter $\lambda_{nse}$ in Eq. (6) was set to 10.0 based on empirical results, striking a balance between noise reconstruction accuracy and overall generative quality. We used the Adam optimizer [27] with an initial learning rate of 0.0002 for training.

After training, we utilized the generator and the fine-tuned noise encoder to create a simulated dataset. Following the setup in [16], we used 11,572 clean speech samples from the source domain and the same 40 target noisy samples used during training. For each clean sample, we generated a corresponding noisy sample using the generator. We applied dynamic stochastic perturbation with a given standard deviation during generation, resulting in a set of 11,572 paired clean and simulated utterances for SE model training.

We fine-tuned the Vanilla-SE model on the simulated dataset for two epochs. This fine-tuning process adapted the model to the specific noise characteristics present in the simulated data, improving its ability to generalize to unseen noise conditions.

Table 1 presents the main results on the VoiceBank-DEMAND dataset. Our proposed method outperforms the baseline approach in terms of perceptual evaluation of speech quality (PESQ) [28] and short-time objective intelligibility (STOI) [29]. These results highlight the effectiveness of integrating the noise encoder into the domain adaptation process.

Interestingly, utilizing pre-trained BEATs within our proposed method leads to decreased performance. This finding underscores the importance of fine-tuning the noise encoder for the specific target domain. Most notably, our method surpasses the performance achieved by training solely on real target noisy data. This result demonstrates the superior generalization capabilities of our method compared to simply adding real noise to clean speech during training.

Figure 2 illustrates the impact of varying the standard deviation of the dynamic stochastic perturbation on our model’s performance. The results clearly demonstrate that perturbation, preventing model from overfitting with only 40 target noisy utterances, can improve performance, but the optimal standard deviation is crucial. Specifically, increasing the standard deviation initially enhanced PESQ scores, peaking at 3.14 with a standard deviation of 2.0. However, further increases beyond this point led to a decrease in PESQ scores. This suggests that excessive perturbation can degrade the perceived quality of the enhanced speech. Conversely, STOI scores remained relatively stable across different standard deviations, indicating that speech intelligibility is less sensitive to changes in perturbation intensity.

To assess the contribution of each component in our model, we conducted ablation studies, with results presented in Table 2. Removing the noise reconstruction loss (- $\mathcal{L}_{nse}$) led to a slight decrease in performance. This suggests that while the noise reconstruction loss contributes to maintaining noise fidelity during the domain adaptation process, its overall impact on the final enhancement performance is relatively small.

However, eliminating the noise embeddings (- Embeddings) resulted in a more substantial performance drop. This finding underscores the crucial role of noise embeddings in capturing and transferring noise characteristics from the target domain, ultimately leading to improved generalization to unseen noise conditions.

To visualize the learned noise representations, we employed t-distributed stochastic neighbor embedding (t-SNE) [30] on the noise embeddings extracted from the five unseen noise types in the VoiceBank-DEMAND test set. Figure 3 (a) and (b) display the embeddings obtained from the pre-trained BEATs and fine-tuned BEATs, respectively.

A clear separation between different noise categories is evident in the embeddings from the fine-tuned BEATs, even for non-stationary noise types. This improves discriminability, compared to the pre-trained embeddings, highlighting the effectiveness of fine-tuning in adapting the noise encoder to the target domain. The ability to distinguish between diverse noise profiles contributes significantly to the enhanced performance of our proposed method.

To further evaluate the perceptual similarity of the generated noisy speech to the target domain, we conducted a MOS evaluation focusing on the non-stationary noise types. Table 3 presents the results. Ten participants, representative of typical listeners, participated in the evaluation. Each participant listened to 10 audio samples per model through headphones in a quiet room, using the target noisy speech as a reference. They were asked to rate the similarity of the background noise in each generated sample to the reference on a scale from 1 to 5, with 5 indicating the highest similarity. The MOS for each model was then calculated by averaging the scores across all participants and all samples.

The results demonstrate that our proposed model achieves a significantly higher MOS compared to the baseline model. This indicates that our method generates noisy speech with background characteristics that more closely resemble the target domain data, as perceived by human listeners.

To evaluate the robustness of our method under different noise conditions, we assessed the performance of all models across various SNR levels. Table 4 presents the results.

As expected, the baseline model outperforms Vanilla-SE, particularly at lower SNR levels, highlighting the effectiveness of its noise reduction capabilities. However, our proposed method consistently achieves the highest performance across all SNR levels, demonstrating significant and consistent improvements over the baseline. These results demonstrate that our proposed method is the most effective for speech enhancement under a range of noise conditions.

Figure 4 presents the SNR distributions of the VoiceBank-DEMAND dataset. Notably, the SNR distribution of the target noisy speech closely resembles that of the test speech, with peaks around 0 dB and 15 dB SNR levels. This alignment between the target noisy speech and test speech corresponds to our data manipulation described in Section 3.1.

In contrast, the simulated speech for the SE models exhibits a peak predominantly between 0 dB to 5 dB SNR. This concentrated distribution does not adequately reflect the broader SNR spectrum present in the test speech or real-world conditions. This observation highlights an area for improvement in future work: developing methods to generate simulated speech that encompasses and simulates a wider range of real-world SNR conditions would likely lead to further performance gains.