Remixed2Remixed: Domain adaptation for speech enhancement
by Noise2Noise learning with Remixing

Let us denote speech and noise signals drawn from corresponding distributions by $\textsf{\boldmath$s$}\sim\mathcal{D}_{\textsf{\boldmath$s$}}$ and $\textsf{\boldmath$n$}\sim\mathcal{D}_{\textsf{\boldmath$n$}}$, respectively. Synthetic noisy speech can be obtained as $\textsf{\boldmath$x$}=\textsf{\boldmath$s$}+\textsf{\boldmath$n$}$. With pair data $(\textsf{\boldmath$x$},\textsf{\boldmath$s$},\textsf{\boldmath$n$})$, a model predicting both speech and noise $\hat{\textsf{\boldmath$s$}},\hat{\textsf{\boldmath$n$}}=\mathcal{F}(\textsf{\boldmath$x$};\theta)$ parameterized by $\theta$ can be trained under full supervision by optimizing the following cost function (i.e., minimizing the reconstruction error of both signals):

$\displaystyle\mathcal{L}_{\rm supervised}={\mathbb{E}}_{(\textsf{\boldmath$x$},\textsf{\boldmath$s$},\textsf{\boldmath$n$})}\big{[}\mathcal{L}(\hat{\textsf{\boldmath$s$}},\textsf{\boldmath$s$})+\mathcal{L}(\hat{\textsf{\boldmath$n$}},\textsf{\boldmath$n$})\big{]}.$

(1)

RemixIT [15] comprises a teacher model $\mathcal{F}_{\mathcal{T}}$ and a student model $\mathcal{F}_{\mathcal{S}}$. Both models are initialized with a supervised pre-trained model using synthetic OOD pair data $(\textsf{\boldmath$x$},\textsf{\boldmath$s$},\textsf{\boldmath$n$})$ and further trained to enhance better real-world recorded data $\textsf{\boldmath$x$}^{\prime}\sim\mathcal{D}_{\textsf{\boldmath$x$}^{\prime}}$ with only the in-domain data accessible. Given a mini-batch of in-domain noisy data $\boldsymbol{\mathbf{x}}^{\prime}=\boldsymbol{\mathbf{s}}^{\prime}+\boldsymbol{\mathbf{n}}^{\prime}\in{\mathbb{R}}^{B\times T}$, the teacher model estimates speech and noise signals as follows

$\displaystyle\tilde{\boldsymbol{\mathbf{s}}}^{\prime},\tilde{\boldsymbol{\mathbf{n}}}^{\prime}=\mathcal{F}_{\mathcal{T}}(\boldsymbol{\mathbf{x}}^{\prime};\theta_{\mathcal{T}}^{(k)}),$

(2)

where the bold roman font represents a batch $\boldsymbol{\mathbf{a}}=[\textsf{\boldmath$a$}_{1},\ldots,\textsf{\boldmath$a$}_{B}]^{\textsf{T}}$ including multiple signals $\textsf{\boldmath$a$}_{b}$ drawn from distribution $\mathcal{D}_{\textsf{\boldmath$a$}}$ and $\theta_{\mathcal{T}}^{(k)}$ denotes the parameters of teacher model at the $k$-th training epoch. ${}^{\textsf{T}}$ denotes transpose operator and $B$ and $T$ denote mini-batch size and signal length, respectively. The estimated signals are then shuffled and remixed to generate bootstrapped mixture $\tilde{\boldsymbol{\mathbf{x}}}^{\prime}$, which is expressed as

$\displaystyle\tilde{\boldsymbol{\mathbf{x}}}^{\prime}=\tilde{\boldsymbol{\mathbf{s}}}^{\prime}+\boldsymbol{\mathbf{P}}\tilde{\boldsymbol{\mathbf{n}}}^{\prime}.$

(3)

Here, $\boldsymbol{\mathbf{P}}\sim\Pi_{B\times B}$ is a permutation matrix. The bootstrapped mixture is then used to generate in-domain pseudo-paired data $(\tilde{\boldsymbol{\mathbf{x}}}^{\prime},\tilde{\boldsymbol{\mathbf{s}}}^{\prime},\tilde{\boldsymbol{\mathbf{n}}}^{\prime})$. The student model $\mathcal{F}_{\mathcal{S}}$ with parameter $\theta_{\mathcal{S}}^{(k)}$ is then trained by minimizing the reconstructed error between the outputs of the model and the pseudo-targets $\tilde{\boldsymbol{\mathbf{s}}}^{\prime}$ and $\tilde{\boldsymbol{\mathbf{n}}}^{\prime}$ as follows:

	$\displaystyle\hat{\boldsymbol{\mathbf{s}}}^{\prime},\hat{\boldsymbol{\mathbf{n}}}^{\prime}$	$\displaystyle=\mathcal{F}_{\mathcal{S}}(\tilde{\boldsymbol{\mathbf{x}}}^{\prime};\theta_{\mathcal{S}}^{(k)}),$		(4)
	$\displaystyle\mathcal{L}_{\rm RemixIT}$	$\displaystyle=\sum_{b=1}^{B}\big{[}\mathcal{L}(\hat{\boldsymbol{\mathbf{s}}}^{\prime}_{b},\tilde{\boldsymbol{\mathbf{s}}}^{\prime}_{b})+\mathcal{L}(\hat{\boldsymbol{\mathbf{n}}}^{\prime}_{b},[\boldsymbol{\mathbf{P}}\tilde{\boldsymbol{\mathbf{n}}}^{\prime}]_{b})\big{]}.$		(5)

To generate more accurate pseudo-targets, the teacher model is continuously updated by taking a weighted moving average (WMA) with the student model’s weights at constant epoch, which is expressed by $\theta_{\mathcal{T}}^{(k+1)}=\gamma\theta_{\mathcal{S}}^{(k)}+(1-\gamma)\theta_{\mathcal{T}}^{(k)}$. Here, $0\leq\gamma\leq 1$ is a weight parameter.

It is noteworthy that the cost function of RemixIT $\mathcal{L}_{\rm RemixIT}$ has convergence properties when Euclidean norm-based metric is used to measure the reconstruction error:

		$\displaystyle\mathcal{L}_{\rm RemixIT}\propto{\mathbb{E}}\big{[}||\hat{\boldsymbol{\mathbf{s}}}^{\prime}-\tilde{\boldsymbol{\mathbf{s}}}^{\prime}||^{2}_{2}\big{]}={\mathbb{E}}\big{[}||(\hat{\boldsymbol{\mathbf{s}}}^{\prime}-\boldsymbol{\mathbf{s}}^{\prime})-(\tilde{\boldsymbol{\mathbf{s}}}^{\prime}-\boldsymbol{\mathbf{s}}^{\prime})||^{2}_{2}\big{]}$
	$\displaystyle=$	$\displaystyle{\mathbb{E}}\big{[}||(\hat{\boldsymbol{\mathbf{s}}}^{\prime}-\boldsymbol{\mathbf{s}}^{\prime})||^{2}_{2}\big{]}+{\mathbb{E}}\big{[}||(\tilde{\boldsymbol{\mathbf{s}}}^{\prime}-\boldsymbol{\mathbf{s}}^{\prime})||^{2}_{2}\big{]}-2{\mathbb{E}}\big{[}(\tilde{\boldsymbol{\mathbf{s}}}^{\prime}-\boldsymbol{\mathbf{s}}^{\prime})^{\textsf{T}}(\hat{\boldsymbol{\mathbf{s}}}^{\prime}-\boldsymbol{\mathbf{s}}^{\prime})\big{]}$
	$\displaystyle\approx$	$\displaystyle\underbrace{{\mathbb{E}}\Big{[}||\boldsymbol{\mathbf{\epsilon}}^{\prime}_{\mathcal{S}}||^{2}_{2}\Big{]}}_{\hbox to0.0pt{\rm\footnotesize supervised\leavevmode\nobreak\ loss\hss}}-\underbrace{{\mathbb{E}}\Big{[}||\boldsymbol{\mathbf{\epsilon}}^{\prime}_{\mathcal{T}}||^{2}_{2}\Big{]}}_{\hbox to0.0pt{\rm\footnotesize constant\leavevmode\nobreak\ w.r.t\leavevmode\nobreak\ $\theta_{\mathcal{S}}$\hss}}-2{\mathbb{E}}\Big{[}\underbrace{(\tilde{\boldsymbol{\mathbf{s}}}^{\prime}-\boldsymbol{\mathbf{s}}^{\prime})^{\textsf{T}}}_{\hbox to0.0pt{\rm\footnotesize teacher\leavevmode\nobreak\ error\hss}}\underbrace{\frac{1}{M}\sum_{m=1}^{M}(\hat{\boldsymbol{\mathbf{s}}}^{\prime}_{m}-\tilde{\boldsymbol{\mathbf{s}}}^{\prime})}_{\hbox to0.0pt{\rm\footnotesize empirical\leavevmode\nobreak\ mean\leavevmode\nobreak\ student\leavevmode\nobreak\ error\hss}}\Big{]},$		(6)

where $\boldsymbol{\mathbf{\epsilon}}^{\prime}_{\mathcal{S}}$ and $\boldsymbol{\mathbf{\epsilon}}^{\prime}_{\mathcal{T}}$ are reconstruction errors between the target signal $\boldsymbol{\mathbf{s}}^{\prime}$ and the output of the student and teacher models, respectively, and $||\cdot||_{2}^{2}$ denotes squared $L2$ norm. (6) shows that when the third term is zero, the RemixIT loss approaches the supervised loss. This can be achieved by reducing either the teacher error to zero with an accurately estimated signal in the teacher model or the empirical mean student error to zero by exposing the student to a wide variety of bootstrapped mixtures $\tilde{\boldsymbol{\mathbf{x}}}^{\prime}_{m}=\tilde{\boldsymbol{\mathbf{s}}}^{\prime}+\tilde{\boldsymbol{\mathbf{n}}}^{\prime}_{m},\leavevmode\nobreak\ m=1,\ldots,M$ involving the same teacher estimate $\tilde{\boldsymbol{\mathbf{s}}}^{\prime}$ so that ${\mathbb{E}}_{m}[\hat{\boldsymbol{\mathbf{s}}}^{\prime}_{m}|\tilde{\boldsymbol{\mathbf{x}}}^{\prime}_{m}]=\tilde{\boldsymbol{\mathbf{s}}}^{\prime}$ when $M\to\infty$. This property is important to ensure that RemixIT can learn models as supervised learning. However, reducing the third term to zero with limited training resources, for example, with $M=1$ is not feasible. As a result, the performance of RemixIT inevitably depends to some extent on the performance of its teacher model, and furthermore, there may remain a gap with supervised learning.

To evaluate the performance of the proposed Re2Re for domain adaptation, we conducted speech enhancement experiments on the CHiME-7 unsupervised domain adaptation for conversational speech enhancement (UDASE) task [23, 24], which consists of three datasets: (1) the LibriMix paired dataset for training OOD supervised SE model and development; (2) the CHiME-5 in-domain unlabeled dataset for adopting domain adaptation, development, and evaluation; (3) the reverberant LibriCHiME-5 close-to-in-domain paired dataset for development and evaluation. All datasets contain three subsets labeled with a maximum number of speakers: 1-spk, 2-spk, and 3-spk. LibriMix [25]: A noisy speech separation benchmark comprises clean speech and noise signals from LibriSpeech [26] and WHAM! [27], respectively. Libri2Mix and Libri3Mix with two or three overlapping speakers in each mixture can be used as subsets of 2-spk and 3-spk, and a subset of 1-spk (Libri1Mix) is obtained by discarding one of the two speakers in the Libri2Mix mixtures. The proportion of 1-spk, 2-spk, and 3-spk mixtures is 0.5, 0.25, and 0.25, respectively. CHiME-5 [28]: A dataset originally consists of noisy multi-speaker speeches of twenty conversation sessions recorded in 4-people dinner parties. CHiME-7 UDASE excerpted the recording channel where participants wearing microphones did not speak (i.e., the maximum number of simultaneously active speakers is three) and divided signals into four subsets, including short segments of at least 3 seconds long labeled by the maximum number of speakers according to the transcript. The subset containing noise-only segments is used to create the reverberant LibriCHiME-5 dataset for objective evaluations. Other subsets are further divided for train ($\approx$83h), development ($\approx$15.5h), and evaluation ($\approx$7h), respectively. Segments for training are cut into chunks of up to 10 seconds, and a voice activity detector (VAD) is applied as post-processing to obtain two versions of the training dataset: CHiME-5 w/o VAD and CHiME-5 w/ VAD. Reverberant LibriCHiME-5: A synthetic dataset consists of reverberant noisy speech labeled with clean speech, where clean speech and noise signals are excerpted from LibriSpeech [26] and the above-mentioned noise-only subset, respectively. The room impulse responses (RIRs) excerpted from the VoiceHome corpus are recorded in the living room, kitchen, and bedroom of 3 real homes with 18 different microphone arrays and loudspeaker settings. The mixtures are generated by adding noise segments into randomly sampled speech utterances convolved with randomly sampled RIRs, where the signal-to-noise ratio (SNR) for each speaker is distributed as a Gaussian with a mean of 5 dB and a standard deviation (std) of 7 dB to match the CHiME-5 dataset. The proportion of 1-spk, 2-spk, and 3-spk subsets was 0.6, 0.35, and 0.05, respectively. Data duration for development and evaluation is about 3 hours each.

We used the recipe provided by CHiME-7 without modification except for the cost function to demonstrate the effectiveness of the cost function. We used Sudo rm-rf [6] architecture for both teacher and student models, whose encoder and decoder consisted of one-dimensional convolution and transpose convolution, respectively, with 512 filters of 41 taps and a hop size of 20 samples, and the separator consisted of 8 U-Conv blocks. The pre-trained teacher model initialized the student model and was continually updated by WMA with a weight of $\gamma=0.01$ every epoch. The batch size was 24. Negative scale-invariant signal-to-distortion ratio (SI-SDR) [29] was used as the cost function for training teacher and student models in RemixIT. We used the mean squared error between the estimated speech signal and bootstrapped mixture as $\mathcal{L}_{\rm Re2Re}$. For Re2Re_reg, we set $\beta=100$ according to the development set. We calculated DNS-MOS [30] scores on the 1-spk subset of the CHiME-5 dataset and SI-SDR [dB] on the reverberant LibriCHiME-5 dataset. More details about the datasets and baseline system are available in [23, 24].

We first compared the proposed Re2Re and Re2Re_reg with the baseline system of CHiME-7. Table 1 shows SI-SDRs [dB] on the reverberant LibriCHiME-5 dataset and DNS-MOS scores in the 1-spk subset of CHiME-5 dataset. All models were trained from the Sudo rm-rf checkpoint provided by CHiME-7. The two proposed methods outperformed the baseline method in terms of SI-SDR, regardless of whether VAD was applied to the training data. Re2Re, using the N2N loss only, achieved SI-SDR about 0.71 dB and 1.08 dB higher than RemixIT. However, no improvement was observed for DNS-MOS. One possible reason is that Re2Re only considered the reconstruction error of the speech signal, resulting in a less accurate estimation of background noise. Table 2 summarizes the SI-SDR[dB] and its std for each subset averaged over ten teacher models. The two proposed methods achieved better and relatively stable performance in all cases. The models trained on data without and with VAD achieved SI-SDR improvements of 0.99 and 1.62 dB on the 2-spk and 0.58 and 0.85 dB on the 3-spk subsets, respectively, while the improvements on the 1-spk subset were limited to 0.29 dB and 0.23 dB. This could be another reason for the lack of improvement in DNS-MOS. The standard deviations were approximately halved when trained on data without VAD and slightly reduced when trained on data with VAD, indicating that the performance of the student model relative to the teacher could be stabilized by N2N loss, even just as a regularization. We then compared our best systems to those submitted to the challenge, whose results are summarized in Table 3. The proposed methods achieved performance comparable to the system ranked second in the challenge regarding SI-SDR and baseline RemixIT regarding DNS-MOS.