Preserving background sound in noise-robust voice conversion via multi-task learning

Due to the large parameters and complex structure, most current source separation models are usually too bulky to be jointly trained with the VC task. Therefore, we adopt DCCRN as our SS module [19], which only has 3.7M parameters and achieved the best performance for Deep Noise Suppression (DNS) challenge in the real-time track [22].

The original DCCRN estimates complex ratio mask (CRM) by the real and imaginary parts of the noisy complex spectrogram. In this paper, DCCRN is used as a source separation model by adding additional CRM of clean speech, as shown in Figure 1 (b). Meanwhile, we use power-law compressed phase-aware asymmetric (PLCPA-ASYM) loss to replace the original scale-invariant signal-to-noise ratio (SI-SNR) loss to confine speech distortion and achieve better power control [20]. Suppose $x$ and $\hat{x}$ are the estimated and clean spectrograms, respectively. PLCPA loss is defined as follows:

	$\displaystyle\mathcal{L}_{a}(t,f)$	$\displaystyle=\left.||x(t,f)\right|^{p}-\left.|\hat{x}(t,f)|^{p}\right|^{2}$		(1)
	$\displaystyle\mathcal{L}_{p}(t,f)$	$\displaystyle=\left.||x(t,f)\right|^{p}e^{j\varphi(x(t,f))}-\left.|\hat{x}(t,f)|^{p}e^{j\varphi(\hat{x}(t,f))}\right|^{2}$
	$\displaystyle\mathcal{L}_{\texttt{plcpa}}$	$\displaystyle=\frac{1}{T}\frac{1}{F}\sum_{t}^{T}\sum_{f}^{F}\left(\alpha\mathcal{L}_{a}(t,f)+(1-\alpha)\mathcal{L}_{p}(t,f)\right),$

where $T$ and $F$ are the total time and frequency frames, respectively, while $t$ and $f$ stand for time and frequency index. The spectral compression factor $p$ is set to 0.3 and operator $\varphi$ calculates the argument of a complex number while the weighted coefficient between the phase-aware components and amplitude is $\alpha$. The asymmetric loss [23] is adapted to the amplitude part of the PLCPA loss to alleviate the over-suppression (OS) issue, which is defined as

	$\displaystyle h(x)$	$\displaystyle=\left\{\begin{array}[]{lr}0,\quad if\ \ x\leq 0,&\\ x,\quad if\ \ x>0,&\end{array}\right.$		(2)
	$\displaystyle\mathcal{L}_{\texttt{os}}(t,f)$	$\displaystyle=\Big{|}h(|x(t,f)|^{p}-|\hat{x}(t,f)|^{p})\Big{|}^{2}.$

So that the final PLCPA-ASYM loss can be defined as

$$\mathcal{L}_{\texttt{plcpa-asym}}=\mathcal{L}_{\texttt{plcpa}}+\beta\frac{1}{T}\frac{1}{F}\sum_{t}^{T}\sum_{f}^{F}\mathcal{L}_{\texttt{os}}(t,f),$$

(3)

where $\beta$ is the positive weighting coefficient for $\mathcal{L}_{\texttt{os}}(t,f)$.

After extracting the speech from source waveforms with background sound through the SS module, we hire a VC module to transform the speech into the target speaker. As illustrated in Figure 1(c), the VC module goes through single-stage training for efficient end-to-end learning. The VC module consists of a convolutional long short-term memory (CLSTM) encoder and a HiFiGAN-based decoder [24], which aim at high-level linguistic representation encoding and waveform reconstruction, respectively. CLSTM consists of three stacks of convolution layers followed by the LeakyReLU activation function and a LSTM layer. The speaker embedding from the lookup table is fed to the decoder as conditions for target voice generation. The architecture and objective of the decoder generator follow the same configuration as HiFi-GAN [24].

To reduce the mismatch between the SS module and VC module during voice conversion in the presence of background sound, we adopt the multi-task learning strategy in the following steps: 1) only the VC module is optimized by the VC training loss with the SS module frozen; 2) only the SS module is optimized by PLCPA-ASYM loss with VC module frozen; 3) both the SS and VC modules are jointly optimized in a multi-task learning manner. The bottleneck feature extraction module is frozen at each stage.

Reconstruction loss for the unified task. The unified task aims at optimizing the composed waveform with vocals and background sound by minimizing the reconstruction loss. We denote the Mel-spectrogram of training data as $M=(M_{\texttt{s}},M_{\texttt{b}})$, which is the Mel-spectrogram composed of speech audio $X_{\texttt{s}}$ and background sound $X_{\texttt{b}}$. The reconstruction loss for the unified task is defined as

$$\mathcal{L}_{\texttt{rec}}^{\texttt{uni}}=||M-\hat{M}||_{1},$$

(4)

where $\hat{M}$ is the Mel-spectrogram of the reconstructed waveform containing speech voice and background sound.

SS training loss. The source separation task considers critical phase information and uses PLCPA-ASYM loss to optimize the estimated CRM and confine the separated speech and background distortion. PLCPA-ASYM loss is calculated for the separated speech and background sound, denoted by $\mathcal{L}_{\texttt{s}}^{\texttt{ss}}$ and $\mathcal{L}_{\texttt{b}}^{\texttt{ss}}$, respectively.

VC training loss. The objective of the typical VC task with no background sound follows HiFiGAN [24], which consists of reconstruction loss $\mathcal{L}_{\texttt{rec}}^{\texttt{vc}}$, feature matching loss $\mathcal{L}_{\texttt{fm}}^{\texttt{vc}}$, and adversarial loss $\mathcal{L}_{\texttt{adv}}^{\texttt{vc}}$. We adopt the L1 loss as reconstruction loss to optimize the spectrogram $\hat{M}_{\texttt{s}}$ of the predicted speech $\hat{X_{\texttt{s}}}$ as

$$\mathcal{L}_{\texttt{rec}}^{\texttt{vc}}=||M_{\texttt{s}}-\hat{M_{\texttt{s}}}||_{1}.$$

(5)

To improve the performance of voice conversion, we employ adversarial training for more natural speech. The adversarial generator loss of the single-task VC is calculated as

$$\mathcal{L}_{\texttt{adv}}^{\texttt{gen}}=(D(\hat{X_{\texttt{s}}})-1)^{2},$$

(6)

$$\mathcal{L}_{\texttt{adv}}^{\texttt{dis}}=(D(X_{\texttt{s}})-1)^{2}+D(\hat{X_{\texttt{s}}})^{2},$$

(7)

where $D$ is a discriminator network. For adversarial training stability, feature matching loss is also used as

$$\mathcal{L}_{\texttt{fm}}^{\texttt{vc}}=\sum_{i=1}^{T}\frac{1}{N_{i}}\left\|D_{i}(X_{\texttt{s}})-D_{i}(\hat{X_{\texttt{s}}})\right\|_{1},$$

(8)

where $T$ denotes the total number of layers in the discriminator and $D_{i}$ produces the feature map of the $i$-th layer of the discriminator with $N_{i}$ number of features. The joint training process will combine the SS and VC modules into a single unit if the source separation losses and voice conversion losses are not added, rendering the separated intermediate results useless and unable to control whether the background sound is preserved.

The SS and VC modules share a uniform reconstruction loss constrained by joint training and the SS module can be regarded as data augmentation or a desirable pre-process model for the VC module to improve robustness. The final loss function of our multi-task learning approach is a weighted sum of the following losses:

	$\displaystyle\mathcal{L}_{\texttt{mtl}}=$	$\displaystyle\lambda_{\texttt{uni}}\mathcal{L}_{\texttt{rec}}^{\texttt{uni}}+\lambda_{\texttt{ss}}(\mathcal{L}_{\texttt{s}}^{\texttt{ss}}+\mathcal{L}_{\texttt{b}}^{\texttt{ss}})+$		(9)
		$\displaystyle\lambda_{\texttt{vc}}(\mathcal{L}_{\texttt{rec}}^{\texttt{vc}}+\mathcal{L}_{\texttt{adv}}^{\texttt{vc}}+\mathcal{L}_{\texttt{fm}}^{\texttt{vc}}),$

where $\lambda_{\texttt{uni}}$, $\lambda_{\texttt{ss}}$, and $\lambda_{\texttt{vc}}$ are hyper-parameters balancing loss terms of each task. We set $\lambda_{\texttt{uni}}=45$, $\lambda_{\texttt{ss}}=1$ and $\lambda_{\texttt{vc}}=1$ empirically.

We train the SS module and VC module on two different datasets. To train the SS module, we use the MUSDB18-train dataset [25], which contains 100 full lengths of music tracks of different genres and their isolated drums, bass, vocals, and other items. The VC module is trained on VCTK dataset [26] which contains 110 English speakers and 400 utterances per speaker. For the joint training of the SS and VC module, MUSDB18-train and VCTK datasets are mixed to conduct the evaluation of the background sound VC. Background sound is randomly selected from the dataset and reprocessed to be clipped as the same length as speech data by a certain signal-to-noise ratio (SNR) range of $0$ to $10$ dB. We randomly select 4 source speakers (i.e. p245, p264, p270, and p361) and 2 target speakers (i.e. p294 and p334) from the VCTK dataset mixed with clips from the MUSDB18-test dataset for conducting evaluations. For each source and target speaker, 30 utterances are reserved as the test set. All training and evaluation data are conducted on 16 kHz sampling rate.

For a fair comparison, we conduct evaluations on the following systems with the same VC module: 1) Upper Bound, where the audio is converted from clean source voice and then superimposed with the original background sound; 2) Baseline1 (Sep), consisting of a VC module and a separation model ²²2https://github.com/bytedance/music_source_separation by individual training, the same source separation approach as [18]; 3) Baseline2 (Sep + Denoise), containing a VC module, the same separation model as Baseline1 and the same denoising model as [17], which can be regarded as a more robust source separation way than [17]; 4) Proposed, the multi-task learning framework proposed in this paper.

The effectiveness of the proposed framework compared with the baseline systems is investigated by objective and subjective evaluations. For subjective evaluation, we use comparative mean opinion score (CMOS) to compare the converted speech with background sound to the baseline system and use Upper Bound results as the reference to determine the ensemble quality. To evaluate the quality and similarity more effectively, we also conducted a mean opinion score (MOS) test on the converted speech without superimposing the background sound. Scale-invariant-signal-to-distortion ratio (SI-SDR) [27], and perceptual evaluation of speech quality (PESQ) [28] are employed as the objective metrics to evaluate separated background sound and converted speech. Audio samples are available online ³³3https://yaoxunji.github.io/background_sound_vc/.