DIFFRENT: A DIFFUSION MODEL FOR RECORDING ENVIRONMENT TRANSFER OF SPEECH

The role of the recording environment encoder $E_{r}(\,\cdot\,;\psi)$ is to extract environmental acoustic features other than the speech content from the reference audio. We assume that the recording environment remains unchanged over time. From input mel-spectrograms $R$, $E_{r}(\,\cdot\,;\psi)$ produces embedding $z_{r}\in\mathbb{R}^{C}$, the disentangled recording environment representation. The diffusion decoder $f(\,\cdot\,;\theta)$ takes $z_{r}$ as an input condition. Without any other objectives, $E_{r}(\,\cdot\,;\psi)$ and $f(\,\cdot\,;\theta)$ are trained jointly with a denoising objective. We employ the ECAPA-TDNN [3] architecture for the recording environment encoder. ECAPA-TDNN was originally designed to extract speaker embedding for speaker verification. Given its strong capability in capturing static features, we deduce its suitability for our task. We removed the AAM-softmax in ECAPA-TDNN and modified the number of nodes in the final fully-connected layer to align with the condition channel size of the diffusion decoder.

In order to preserve the speech content from the input audio and filter out the recording environment features, the content enhancer $T_{c}(\,\cdot\,;\Phi)$ is trained by minimizing the following mean absolute error (MAE) loss:

where $X_{clean}$ is the log mel-spectrogram of clean speech having the same content as $x$. Note that the goal of $T_{c}(\,\cdot\,;\Phi)$ is to facilitate improved content capturing for the diffusion decoder rather than speech enhancement. There are some specific artifacts or distortion in $X_{c}\in\mathbb{R}^{F\times T_{1}}$ as it is trained with the simple L1 loss. However, they are addressed in the diffusion decoder and so their influence on the overall model performance remains negligible. We employed a ResUNet from [11] and slightly modified it to operate on mel-spectrogram as in [8].

To generate a target mel-spectrogram $Y_{0}\sim p(Y|X_{c},z_{r})$, from the two input conditions $X_{c}$ and $z_{r}$, we apply a denoising diffusion probabilistic model [7]. A diffusion probabilistic model belongs to the category of generative models that learn a model distribution $p_{\theta}(Y_{0})$ which approximates the data distribution $q(Y_{0})$. A diffusion model, with the total number of diffusion steps $T$, is a latent variable model having the form $p_{\theta}(Y_{0})=\int p_{\theta}(Y_{0:T})dY_{1:T}$, where $Y_{1}$, …, $Y_{T}$ are latent variables of the same dimensionality as $Y_{0}$. It comprises two processes as described below.

We evaluated our proposed model on the DDS dataset [14] which includes 12 hours of clean speech data with 48 speakers (24 female and 24 male) and corresponding paired data re-recorded with different combinations of 3 microphones, 9 spaces, and 6 microphone positions. For the test set, we selected four speakers (two males and two females) and one recording environment setup (Uber Microphone, livingroom1, and F position).

To enhance the robustness of content capturing, we implemented data augmentation by convolving clean speech with diverse impulse responses and adding noise. The noise samples were sourced from the REVERB Challenge database [9], and a total of 1207 impulse responses are obtained from the DetmoldSRIR dataset [1] and the MIT Impulse Response Survey Dataset [29]. Note that data augmentation is exclusively applied to content speech, because the goal of the diffusion decoder is to learn the distribution of realistic environments rather than synthetic ones.

For our task, it is important that the neural vocoder operates in a speaker-independent manner and acquires the capability to synthesize speech in diverse recording environments. To achieve this, we trained the HiFi-GAN vocoder [10] with the DDS dataset [14] and the LibriSpeech corpus [20]. The LibriSpeech corpus [20] initially designed for automatic speech recognition (ASR) research contains 982 hours of speech data from 2,484 speakers. Despite some considerable noisy audio files within this dataset, they align with the requirements of our task, as the vocoder is also required to generate noise effectively.

All audio sample rates were resampled to 16 kHz. Speech data were randomly segmented or padded to match the length of 4 seconds. The spectrogram was extracted from the audio by short-time Fourier transform (STFT) with a Hann window size of 1024 samples and a hop size of 256 samples. The log mel-spectrogram was computed through an 80-channel mel filterbank and the log magnitude compression. For the diffusion decoder, it was normalized to the range between -1 and 1 by a min-max normalization.

Diffusion Decoder: We conducted experiments with two diffusion decoders: WaveNet [30] architecture in [16] and U-Net architecture in [22]. The embedding $z_{r}$ channel size, contingent on the conditioning method of the diffusion decoder, was set to 256 for WaveNet and 80 for U-Net. We empirically found this difference in channel size does not yield a significant variance in the performance of the recording environment encoder. To align the time frame size of $z_{r}$ with $X_{c}$, $z_{r}$ is repeated along the time axis. In the case of U-Net, the final condition is established by concatenating $z_{r}$ with $X_{c}$. For the WaveNet architecture, the channel size of $X_{c}$ is transformed to 256 via a linear layer, followed by the summation with $z_{r}$ to make the input condition.

We compare DiffRENT with the acoustic matching model [27]. Due to the absence of source code, we reproduced the model based on the description provided in [27]. To evaluate the speech enhancement capabilities of our model, we also compare our method with CDiffuSE[17] and VoiceFixer [15], the state-of-the-art level speech enhancement model. The official implementations of them were employed. Comparison models were trained with the identical data configuration used in the proposed model.

We conducted an ablation study to evaluate the effectiveness of each component in DiffRENT. For the recording environment encoder, we used a simple encoder composed of 1D convolution with a kernel size of 1, followed by the attentive statistics pooling [19] and batch normalization as a baseline model. Furthermore, we evaluated the effect of the content enhancer with the models using the original input $X$ instead of the enhanced speech $X_{c}$.

Table 1 shows the result of the objective evaluation. In all test cases, all proposed models except those using the WaveNet decoder without the content enhancer outperform the acoustic matching model in all metrics. In Env-to-Clean, three proposed models have better PESQ values than VoiceFixer, which performs best among the comparison models. Model U-R2-C achieves comparable performance compared to VoiceFixer across all metrics. In the majority of cases, the model employing the U-Net architecture in the diffusion decoder demonstrates superior performance compared to the model with the WaveNet decoder. Additionally, the models with R2 encoder consistently outperform the models with R1 encoder. It implies that enhanced disentanglement features contribute to the improved ability of the decoder to generate sound within the target environment. There is only a single case where the model with R1 outperforms the model with R2 (W-R1 and W-R2). This might be attributed to the limited capacity of the decoder, which could lead to degraded performance with diverse input conditions. The results also show the effectiveness of the content enhancer.

Table 2 shows the result of the subjective evaluation. While no significant differences were observed in objective evaluation, our model notably surpasses VoiceFixer in subjective evaluation. This result implies that our model generates sound with greater perceptual quality in comparison to VoiceFixer. Across all metrics, our model exhibits superior performance to the acoustic matching network. We found that performance of the acoustic matching network deteriorates when there is a substantial difference between the environments of the target and the content speech, while our model maintains consistent performance.

We investigated the recording environment encoder to understand the behavior better by visualizing latent spaces with t-SNE. We used 400 audio files composed of 4 unseen speakers, 10 excerpts, and 10 environments. Test environments include a clean environment, environments with one unseen condition, and a totally unseen environment. Fig. 2 shows the t-SNE plots of each embedding from the acoustic embedding network [27] and two recording environment encoders of DiffRENT. Points sharing the same color represent audio within the identical recording environment. The point locations show that two encoders in DiffRENT disentangle the recording environment better than the acoustic embedding network. Furthermore, the encoder adapting ECAPA-TDNN exhibits more vivid isolation among different recording environments than the baseline encoder.