Diffusion-based signal refiner for speech separation

DDPM is a deep generative model that aims to learn the model distributions $p_{\theta}(\bm{x})$ of data $\bm{x}\in\mathbb{R}^{n}$ that approximate an unknown data distribution $q(\bm{x})$. It generates a sample $\bm{x}_{0}\sim q(\bm{x})$ by following a Markov chain structure $\bm{x}_{T},\cdots,\bm{x}_{0}$ whose joint distribution is defined by the following reverse process:

To train the diffusion model, the forward process is defined as the factorized variational inference distribution

which is followed by an ELBO objective on the negative log likelihood. In practice, the distributions of $p_{\theta}$ and $q$ are assumed to be conditional Gaussian for all $t<T$, and $q(\bm{x}_{t}|\bm{x}_{0})$ also follows a Gaussian distribution with variance $\sigma_{t}$. This leads to a reduction of the ELBO objective into the denoising autoencoder objective

where $f_{\theta}^{(t)}$ is a neural network to estimate $\bm{x}_{0}$ given $\bm{x}_{t}$ and $\gamma_{1:T}$ is a series of tunable parameters.

DDRM is a diffusion model with a Markov chain structure conditioned on $\bm{y}\in\mathbb{R}^{m}$, where the joint distribution is defined as

The likelihood $p(\bm{y}|\bm{x})$ is characterized by the following linear inverse problem:

where $\bm{H}\in\mathbb{R}^{m\times n}$ is a known degradation matrix and $\bm{z}\sim{\cal N}(\bm{0},\sigma_{\bm{y}}^{2}\bm{I})$ is an i.i.d. additive Gaussian noise with known variance $\sigma_{\bm{y}}^{2}$. The corresponding factorized variational distribution to (4) is defined in the same manner as DDPM, after which the ELBO objective is optimized (see detailed derivations in [24]).

DDRM can handle various linear inverse problem designs by switching a couple of variational distributions. The update of DDRM is conducted in a spectral space of the singular value decomposition (SVD) of $\bm{H}$. The measurements with zero singular value are likely to have no information of the index, so the update formula becomes similar to the regular unconditional generation. In contrast, components with non-zero singular values are further classified by the noise level of measurements in the spectral space $\bm{\sigma}_{y}/s_{i}$ compared to that of the diffusion model $\sigma_{t}$. The unique property of DDRM is that the ELBO optimal value also corresponds to that of DDPM under a couple of assumptions [14]. This enables us to inherit the pretrained unconditional DDPM as a good approximation to DDRM without having to re-train the diffusion model for different linear inverse problems each time.

Given the outputs of the preceding model $\hat{\bm{x}}^{(i)},\ i\in\{1,\ldots,N\}$, DDRM can simply refine each signal one by one with a linear observation, as

where $\bm{z}^{(i)}\sim{\cal N}(\bm{0},\sigma^{(i)}_{\bm{y}})$ is an additive Gaussian noise with known variance $\sigma^{(i)}_{\bm{y}}$. However, this observation design, which we call isolated, gets rid of other speakers’ preceding outputs. This leads to poor refinement of the missing part of the target speech because the preceding models are not always perfect enough to separate out target speech components into a single output channel.

To fully leverage the available information and generate a faithful sample to the reference signal, a shared refiner is contrived with mixture signal $\bm{m}$ as an additional measurement. Here, the observation of the mixture is given by

For example, if $N=2$, the observation equation in (5) becomes

Let the SVD of $\bm{H}$ be $\bm{H}=\bm{U\Sigma V}^{\top}$, where $\bm{U}\in\mathbb{R}^{M\times M},\ \bm{V}\in\mathbb{R}^{N\times N}$ are orthogonal matrices and $\bm{\Sigma}\in\mathbb{R}^{M\times N}$ is a rectangular diagonal matrix containing the singular values $s_{1}\geq s_{2}\geq\cdots\geq s_{N}$ of the degradation matrix $\bm{H}$ in which $M=N+1$. The proposed update is then summarized as Algorithm 1.

Table 1 lists the separation performance results of each method. As we can see, both the isolated and shared refiners exhibited a significant improvement of the reference-free NISQA, while the reference-based metrics were generally degraded as expected, since the output of the generative model did not necessarily match the ground truth. The shared refiner, however, could recover the performance of the reference-based metrics with a small decrease in NISQA, which corresponds to the hypothesis given in Sec.3 that the isolated refiner may cause a less faithful result to the ground truth. The processed examples are shown in Fig. 2. Thanks to using the proposed refiner, the clearness of the harmonics structure was improved and the non-speech ambient noise was sufficiently suppressed compared to the original output of the Sepformer.

Table 2 summarizes the performance change when different amounts of measurement noise were applied. Since the variance of measurement noise is a tunable parameter for each frequency bin and time frame, users can try a realistic scenario when designing the parameter. For example, since speech separation is basically difficult in areas where the interfering speech is loud, we consider a sigmoid design in which the variance of measurement noise is determined by the target-to-interference ratio of the signal. Specifically, let the variance for all frequency and time indices $k,l$ be

Here, we set $\alpha,\beta$, and $\gamma$ to $2.0,2.0$, and $0.8$, respectively, and set the variance for mixture measurement $\hat{\sigma}^{(0)}_{k,l}$ to $0.5$ (same as in the first experiment). All results showed improvement in both reference-based and reference-free metrics, which indicates that choosing an appropriate value for $\hat{\sigma}^{(i)}_{k,l}$ improves the performance. We hypothesize that even larger gains are possible with other choices of $\hat{\sigma}^{(i)}_{k,l}$.

Finally, we can observe that the discriminative and generative outputs each have their own strengths: namely, discriminative outputs are better in the reference-based metrics while generative ones are better in the reference-free ones. This inspires a new hypothesis: i.e., that we can obtain further improved outputs by blending the discriminative output $\hat{\bm{x}}$ with the generative output $\bm{x}_{0}$ by $\tilde{\bm{x}}=\xi\hat{\bm{x}}+(1-\xi)\bm{x}_{0}$, where $\xi$ is a mixing weight. Fig. 3 shows the results when $\xi$ was changed in $[0,1]$ by the factor of $0.1$. Here, we used the separations of Sepformer as a discriminative output and the shared refined outputs as a generative one. The blue and red lines respectively show the scores for SI-SDR and for NISQA. As we can see, there is a clear trade-off with these metrics: the highest SI-SDR was $23.1$ dB achieved at $\xi=0.8$, and the highest NISQA was $3.79$ at $\xi=0.2$. Both are higher than solely using either discriminative or generative output alone, which proves the effectiveness of the blending technique. One of the best blended results, $\xi=0.8$ (shown in Table 1), was superior to the original Sepformer in both reference-based and reference-free metrics.