FastMVAE2: On improving and accelerating
the fast variational autoencoder-based
source separation algorithm for determined mixtures

Let us consider a situation where $I$ source signals are captured by $I$ microphones. We use $x_{i}(f,n)$ and $s_{j}(f,n)$ to denote the short-time Fourier transform (STFT) coefficients of the signal observed at the $i$th microphone and $j$th source signal, where $f$ and $n$ are the frequency and time indices, respectively. If we use

to denote the vectors containing $x_{1}(f,n),\ldots,x_{I}(f,n)$ and $s_{1}(f,n),\ldots,s_{I}(f,n)$, the relationship between the observed signals and source signals can be approximated as

under a determined mixing condition, where $\boldsymbol{\mathbf{W}}^{{\textsf{H}}}(f)$ represents the separation matrix, and $(\cdot)^{{\textsf{T}}}$ and $(\cdot)^{{\textsf{H}}}$ denote the transpose and Hermitian transpose of a matrix or a vector, respectively. The goal of BSS is to determine $\mathcal{W}=\{\boldsymbol{\mathbf{W}}(f)\}_{f}$ solely from the observation $\mathcal{X}=\{\boldsymbol{\mathbf{x}}(f,n)\}_{f,n}$. Here, the notation $\{\textsf{\boldmath$E$}_{b}\}_{b}$ is used as an abbreviation for $\{\textsf{\boldmath$E$}_{b}\mid b\in\mathcal{B}\}$, where $\mathcal{B}$ denotes the set of all possible indices.

In the following, we assume that $s_{j}(f,n)$ independently follows a zero-mean complex proper Gaussian distribution with variance (power spectral density) $v_{j}(f,n)={\mathbb{E}}[|s_{j}(f,n)|^{2}]$:

This assumption is often referred to as the local Gaussian model (LGM) [57, 58]. If $s_{j}(f,n)$ and $s_{j^{\prime}}(f,n)$ are independent for $\forall j\neq j^{\prime}$, the density of $\boldsymbol{\mathbf{s}}(f,n)$ becomes

where $\boldsymbol{\mathbf{V}}(f,n)=\mathop{\mathrm{diag}}\nolimits[v_{1}(f,n),\ldots,v_{I}(f,n)]$. From (3) and (6), the density of $\boldsymbol{\mathbf{x}}(f,n)$ is obtained as

where $|\boldsymbol{\mathbf{W}}^{{\textsf{H}}}(f)|^{2}$ is the Jacobian of the mapping $\boldsymbol{\mathbf{x}}(f,n)\mapsto\boldsymbol{\mathbf{s}}(f,n)$. Therefore, the log-likelihood of $\mathcal{W}=\{\boldsymbol{\mathbf{W}}(f)\}_{f}$ and $\mathcal{V}=\{v_{j}(f,n)\}_{f,n,j}$, given $\mathcal{X}=\{\boldsymbol{\mathbf{x}}(f,n)\}_{f,n}$ is expressed as

where we have used $\overset{c}{=}$ to denote equality up to constant terms and a bold italic font to indicate a set consisting of TF elements, namely $\textsf{\boldmath$S$}_{j}=\{s_{j}(f,n)\}_{f,n}$ and $\textsf{\boldmath$V$}_{j}=\{v_{j}(f,n)\}_{f,n}$. The log-likelihood will be split into $F$ frequency-wise terms if no additional constraint is imposed on $v_{j}(f,n)$ or $\boldsymbol{\mathbf{W}}(f)$, implying that there is a permutation ambiguity in the separated components for each frequency. Thus, the separated components of different frequency bins that originate from the same source need to be grouped together in order to complete source separation. This process is called permutation alignment [59, 60].

Incorporating an appropriate constraint into the power spectrogram $\textsf{\boldmath$V$}_{j}=\{v_{j}(f,n)\}_{f,n}$ not only helps eliminate the permutation ambiguity but also provides a clue for estimating $\mathcal{W}$. In the MVAE method, the complex spectrogram of a single source $\textsf{\boldmath$S$}=\{s(f,n)\}_{f,n}$ is modeled using a CVAE [31] conditioned on a class variable $\boldsymbol{\mathbf{c}}$. Here, $\boldsymbol{\mathbf{c}}$ is a one-hot vector consisting of $C$ elements that indicates to which class the separated signal belongs. For example, speaker IDs can be used as the class category in multispeaker separation tasks, where the entries of $\boldsymbol{\mathbf{c}}$ will be $1$ at the index of a certain speaker and $0$ at all other indices.

Since the following applies to all sources, index $j$ will be omitted throughout this paragraph. A CVAE consists of decoder and encoder networks. The decoder network is designed to produce the parameters of the distribution $p_{\theta}^{*}(\textsf{\boldmath$S$}|\boldsymbol{\mathbf{z}},\boldsymbol{\mathbf{c}})$ of data $S$ given a latent space variable $\boldsymbol{\mathbf{z}}$ and a class variable $\boldsymbol{\mathbf{c}}$. The encoder network is designed to generate the parameters of a conditional distribution $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ that approximates the exact posterior $p_{\theta}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$. The goal of the CVAE training is to find the weight parameters in the encoder and decoder networks, namely $\theta$ and $\phi$, such that the encoder distribution $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ becomes consistent with the posterior $p_{\theta}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})\propto p_{\theta}^{*}(\textsf{\boldmath$S$}|\boldsymbol{\mathbf{z}},\boldsymbol{\mathbf{c}})p(\boldsymbol{\mathbf{z}})$. Note that the KL divergence between $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ and $p_{\theta}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ is shown to be equal to the difference between the log marginal likelihood $p_{\theta}^{*}(\textsf{\boldmath$S$}|\boldsymbol{\mathbf{c}})=\int_{\boldsymbol{\mathbf{z}}}p(\textsf{\boldmath$S$}|\boldsymbol{\mathbf{z}},\boldsymbol{\mathbf{c}})p(\boldsymbol{\mathbf{z}})\,d\boldsymbol{\mathbf{z}}$ and its variational lower bound. Hence, minimizing the KL divergence between $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ and $p_{\theta}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ amounts to maximizing the following variational lower bound [61]:

where we have used ${\mathbb{E}}_{(\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})}[\cdot]$ to denote the sample mean of its argument over the training examples $\{\textsf{\boldmath$S$}_{m},\boldsymbol{\mathbf{c}}_{m}\}_{m=1}^{M}$, and ${\rm{KL}}[\cdot||\cdot]$ to denote the KL divergence. Although it is difficult to obtain an analytical form of the expectation ${\mathbb{E}}_{\boldsymbol{\mathbf{z}}\sim q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})}[\cdot]$ in the first term of $\mathcal{J}$, we can use a reparameterization trick [61] to obtain a form that allows us to compute the gradient with respect to $\phi$ using a Monte Carlo approximation. Now, $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$, $p_{\theta}^{*}(\textsf{\boldmath$S$}|\boldsymbol{\mathbf{z}},\boldsymbol{\mathbf{c}})$, and $p(\boldsymbol{\mathbf{z}})$ are distributions that need to be modeled. In the MVAE method, $p(\boldsymbol{\mathbf{z}})$ and $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ are described as Gaussian distributions as with a regular CVAE:

where $\boldsymbol{\mathbf{\mu}}_{\phi}^{*}(\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ and $\boldsymbol{\mathbf{\sigma}}_{\phi}^{*}{}^{2}(\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ are the encoder network outputs. For stable training, the total energy of each training utterance is normalized to 1. However, the energy of each source in a test mixture does not necessarily equal 1. To fill this gap, a scale factor $g$ is additionally introduced into the decoder distribution as a free parameter to be estimated at test time. Specifically, we use an expression of the decoder distribution with variance scaled by $g$. Hence, the decoder distribution for the complex spectrogram $S$ is expressed as

where $\sigma^{*}_{\theta}{}^{2}(f,n;\boldsymbol{\mathbf{z}},\boldsymbol{\mathbf{c}})$ denotes the $(f,n)$th element of the decoder network output. (12) is called the CVAE source model.

If we use the above CVAE source model to represent the complex spectrogram of the $j$th signal in a mixture signal, $\boldsymbol{\mathbf{z}}_{j}$, $\boldsymbol{\mathbf{c}}_{j}$, and $g_{j}$ are the unknown parameters to be estimated. Since the CVAE source model is given in the same form as the LGM in (5) if we denote $g_{j}\sigma_{\theta}^{*}{}^{2}(f,n;\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j})$ by $v_{j}(f,n)$, using this as the generative model for each source gives the log-likelihood in the same form as (II-A).

The goal of the source separation algorithm in the MVAE method is to maximize the posterior $p(\mathcal{W},\Psi,\mathcal{G}|\mathcal{X})\propto p(\mathcal{X}|\mathcal{W},\Psi,\mathcal{G})p(\boldsymbol{\mathbf{z}})p(\boldsymbol{\mathbf{c}})$ with respect to $\mathcal{W}$, $\Psi=\{\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j}\}_{j}$, and $\mathcal{G}=\{g_{j}\}_{j}$, where $\boldsymbol{\mathbf{z}}$ is assumed to follow ${\mathcal{N}}(\boldsymbol{\mathbf{0}},\boldsymbol{\mathbf{I}})$, and $p(\boldsymbol{\mathbf{c}})$ is the empirical distribution of the training examples $\{\boldsymbol{\mathbf{c}}_{m}\}_{m}$, expressed as a multinomial distribution. Hence, the objective function is $\log p(\mathcal{X}|\mathcal{W},\Psi,\mathcal{G})+\log p(\boldsymbol{\mathbf{z}})+\log p(\boldsymbol{\mathbf{c}})$. A stationary point of this function can be found by iteratively updating $\mathcal{W}$, $\Psi$, and $\mathcal{G}$ so that the function value is guaranteed to be non-decreasing. To update $\mathcal{W}$, we can use the IP method [44]:

where $\boldsymbol{\mathbf{\Sigma}}_{j}(f)=\frac{1}{N}\sum_{n}\boldsymbol{\mathbf{x}}(f,n)\boldsymbol{\mathbf{x}}^{\textsf{H}}(f,n)/v_{j}(f,n)$ and $\boldsymbol{\mathbf{e}}_{j}$ denotes the $j$th column of an $I\times I$ identity matrix. As for $\mathcal{G}$, the update rule

maximizes the objective function with respect to $g_{j}$ when $\mathcal{W}$ and $\Psi$ are fixed. Under fixed $\mathcal{W}$ and $\mathcal{G}$, the optimal $\boldsymbol{\mathbf{z}}_{j}$ and $\boldsymbol{\mathbf{c}}_{j}$ that maximize the objective function can be found using the gradient descent method. Note that $\boldsymbol{\mathbf{c}}_{j}$ can be updated under the sum-to-one constraint by inserting an appropriately designed softmax layer that outputs $\boldsymbol{\mathbf{c}}_{j}$.

One important feature of VAE in general is its generalization capability, namely the ability to learn the distribution of unseen data. Thanks to this feature, we expect that the CVAE source model trained on speech samples of sufficiently many speakers can generalize somewhat well to the spectrograms of unknown speakers, thus allowing the above algorithm to handle speaker-independent scenarios reasonably well. Another advantage is that it is guaranteed to converge to a stationary point, making it easy to handle in practical use. However, the downside is that the backpropagation algorithm required for each iteration can be computationally expensive.

The motivation behind the FastMVAE method is to accelerate the process of updating $\Psi$. Under fixed $\mathcal{W}$ and $\mathcal{G}$, the objective function of the MVAE method is equal to the sum of $\log p(\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j}|\textsf{\boldmath$S$}_{j},g_{j})$ up to a constant, where $\textsf{\boldmath$S$}_{j}$ is the set $\{\boldsymbol{\mathbf{w}}_{j}^{\textsf{H}}(f)\boldsymbol{\mathbf{x}}(f,n)\}_{f,n}$, namely the complex spectrogram of the signal separated from the observed signal using the current estimate of $\mathcal{W}$. The idea of the FastMVAE method is to express this posterior as $p(\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j}|\textsf{\boldmath$S$}_{j},g_{j})=p(\boldsymbol{\mathbf{z}}_{j}|\textsf{\boldmath$S$}_{j},\boldsymbol{\mathbf{c}}_{j},g_{j})p(\boldsymbol{\mathbf{c}}_{j}|\textsf{\boldmath$S$}_{j},g_{j})$ and use two trainable networks to approximate these two conditional distributions. Once these networks have been trained, an approximation of the maximum point of the posterior $p(\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j}|\textsf{\boldmath$S$}_{j},g_{j})$ can be obtained by finding the maximum points of the two approximate distributions.

To obtain approximations of the two conditional distributions, the FastMVAE method employs the idea of ACVAE training [47]. ACVAE is a CVAE variant that incorporates the expectation of the mutual information [62]

into the training criterion with the aim of making the decoder output as correlated as possible with the class variable $\boldsymbol{\mathbf{c}}$. Here, $p_{D}(\boldsymbol{\mathbf{c}})$ is the empirical discrete distribution of the samples of $\boldsymbol{\mathbf{c}}$ in the training set and $H(\boldsymbol{\mathbf{c}})$ represents the entropy of $\boldsymbol{\mathbf{c}}$, which can be regarded as a constant. Since it is difficult to express $I(\boldsymbol{\mathbf{c}},\textsf{\boldmath$S$}|\boldsymbol{\mathbf{z}})$ in analytical form, rather than using it directly, ACVAE uses its variational lower bound

defined using a variational distribution $r_{\psi}^{*}(\boldsymbol{\mathbf{c}}|\textsf{\boldmath$S$},g)=\rm{Mult}(\boldsymbol{\mathbf{c}}|\boldsymbol{\mathbf{\rho}}_{\psi}^{*}(\textsf{\boldmath$S$}/g))$ for optimization, where ${\mathbb{E}}_{(\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}}^{\prime})}[\cdot]$ is equivalent to ${\mathbb{E}}_{(\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})}[\cdot]$, ${\mathbb{E}}_{\boldsymbol{\mathbf{c}}}[\cdot]$ denotes the mean of its argument over all one-hot vectors $\boldsymbol{\mathbf{c}}\sim p_{D}(\boldsymbol{\mathbf{c}})$, which can be approximated by a Monte Carlo approximation, and ${\mathbb{E}}_{\boldsymbol{\mathbf{z}}\sim q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})}[\cdot]$ and ${\mathbb{E}}_{\textsf{\boldmath$S$}\sim p_{\theta}^{*}(\textsf{\boldmath$S$}|\boldsymbol{\mathbf{z}},\boldsymbol{\mathbf{c}}^{\prime})}[\cdot]$ are approached by a Monte Carlo approximation after reparameterization tricks. Here, $\rm{Mult}(\boldsymbol{\mathbf{c}}|\boldsymbol{\mathbf{\rho}})\propto\prod_{i}\rho_{i}^{c_{i}}$ denotes a multinomial distribution, where $\boldsymbol{\mathbf{c}}=[c_{1},\ldots,c_{I}]^{\mathsf{T}}$ and $\boldsymbol{\mathbf{\rho}}=[\rho_{1},\ldots,\rho_{I}]^{\mathsf{T}}$. $\boldsymbol{\mathbf{\rho}}_{\psi}^{*}(\textsf{\boldmath$S$}/g)$ is a neural network that takes $S$ normalized by $g$ as an input and produces a probability vector consisting of $C$ elements that sum to 1. $r_{\psi}^{*}(\boldsymbol{\mathbf{c}}|\textsf{\boldmath$S$},g)$ is an auxiliary classifier. Since the exact bound is obtained when $r_{\psi}^{*}(\boldsymbol{\mathbf{c}}|\textsf{\boldmath$S$},g)=p(\boldsymbol{\mathbf{c}}|\textsf{\boldmath$S$},g)$, the trained auxiliary classifier $r_{\psi}^{*}(\boldsymbol{\mathbf{c}}|\textsf{\boldmath$S$},g)$ is expected to be a good approximation of the distribution $p(\boldsymbol{\mathbf{c}}|\textsf{\boldmath$S$},g)$ of interest. ACVAE also uses the negative cross-entropy

as the training criterion. Therefore, the entire training criterion to be maximized is given by

where $\lambda_{\mathcal{L}},\lambda_{\mathcal{I}}\geq 0$ denote the regularization weights that weight the importance of the regularization terms. The set of the networks trained in this way using the spectrograms of the training utterances is called the ACVAE source model. An illustration of ACVAE is shown on the left of Fig. 1.

After ACVAE training, we achieve $p(\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j}|\textsf{\boldmath$S$}_{j},g_{j})\approx r_{\psi}^{*}(\boldsymbol{\mathbf{c}}_{j}|\textsf{\boldmath$S$}_{j},g_{j})q_{\phi}^{*}(\boldsymbol{\mathbf{z}}_{j}|\textsf{\boldmath$S$}_{j},\boldsymbol{\mathbf{c}}_{j},g_{j})$. Since the maximum points of $r_{\psi}^{*}(\boldsymbol{\mathbf{c}}_{j}|\textsf{\boldmath$S$}_{j},g_{j})$ and $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}_{j}|\textsf{\boldmath$S$}_{j},\boldsymbol{\mathbf{c}}_{j},g_{j})$ can be found through the forward passes of the auxiliary classifier and encoder, respectively, we can quickly find an approximate solution to $(\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j})=\mathop{\rm{argmax}}_{\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j}}p(\boldsymbol{\mathbf{z}}_{j},\boldsymbol{\mathbf{c}}_{j}|\textsf{\boldmath$S$}_{j},g_{j})$ without resorting to gradient descent updates. Specifically, $\boldsymbol{\mathbf{c}}_{j}$ is given as the probability vector produced by the auxiliary classifier network:

and $\boldsymbol{\mathbf{z}}_{j}$ is given as the mean of the encoder distribution:

Here, if the $j$th separated signal corresponds to a speaker unseen in the training set, the elements of (20) can be interpreted as quantities indicating how similar that speaker is to all the speakers in the training set. If the signal of any speaker can be assumed to be expressed as a point in the manifold spanned by all the speakers in the training set, our algorithm is expected to be able to handle even mixtures of unknown speakers.

However, our preliminary experiments revealed that directly using the mean of the encoder distribution tends to degrade source separation performance for speakers not included in the training set. To stabilize the inference for unknown speakers, we previously proposed reapplying the prior $p(\boldsymbol{\mathbf{z}}_{j})$ to the encoder output based on the PoE framework [48]to ensure that $\boldsymbol{\mathbf{z}}_{j}$ will not be updated to an outlier. Namely, the prior $p(\boldsymbol{\mathbf{z}}_{j})$ is redefined as the product of two distributions with respect to $\boldsymbol{\mathbf{z}}_{j}$, namely, $\mathop{\rm{argmax}}_{\boldsymbol{\mathbf{z}}_{j}}p(\boldsymbol{\mathbf{z}}_{j}|\textsf{\boldmath$S$}_{j},\boldsymbol{\mathbf{c}}_{j},g_{j})p(\boldsymbol{\mathbf{z}}_{j})^{\alpha}$. Accordingly, the modified update rule of $\boldsymbol{\mathbf{z}}_{j}$ is given as

Here, $\alpha$ is a parameter that weights the importance of the prior $p(\boldsymbol{\mathbf{z}}_{j})$ in the inference, and $\boldsymbol{\mathbf{\Sigma}}_{\phi,j}=\mathop{\mathrm{diag}}\nolimits(\boldsymbol{\mathbf{\sigma}}_{\phi}^{*}{}^{2}(\textsf{\boldmath$S$}_{j}/g_{j},\boldsymbol{\mathbf{c}}_{j}))$. Note that (22) reduces to the mean of the encoder distribution when $\alpha=0$. The algorithm of the FastMVAE method is summarized in Algorithm 1.

We first describe our motivation and ideas for developing an improved version of the ACVAE source model, which we call the “ChimeraACVAE” source model.

Since the latent variable $\boldsymbol{\mathbf{z}}$ no longer depends on $\boldsymbol{\mathbf{c}}$, we must first rewrite the training loss of ACVAE, i.e., (19), by replacing $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ with $q_{\phi}^{+}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$})$. Note that we omit $g$ in this subsection, assuming that $g$ is set to 1 and normalized spectrograms are used during training. Thus, the reformulated training criteria are given as

Here, ${\mathbb{E}}_{\boldsymbol{\mathbf{S}}^{\prime}}[\cdot]$ in (25) denote the mean of the arguments over all spectrograms $\textsf{\boldmath$S$}^{\prime}\sim p_{D}(\textsf{\boldmath$S$})$ in the training dataset. The superscript ⁺ is used to distinguish the networks in the ChimeraACVAE model from those in the original ACVAE model superscripted with ^∗.

Unlike in the training phase, where the class label $\boldsymbol{\mathbf{c}}$ is known and given, in the separation phase, the spectrogram $S$ needs to be constructed using the estimated $\boldsymbol{\mathbf{z}}$ and $\boldsymbol{\mathbf{c}}$. Therefore, it is reasonable to simulate this situation in the training phase as well. Namely, we consider not only the reconstruction error defined using the given label $\boldsymbol{\mathbf{c}}$ but also the reconstruction error defined using the estimated $\boldsymbol{\mathbf{c}}\sim r_{\psi}^{+}(\boldsymbol{\mathbf{c}}|\textsf{\boldmath$S$})$. Thus, we propose including

in the training objective. Here, it should be noted that both $\mathcal{J}^{\prime}$ and $\mathcal{L}^{\prime}$ involve expectations over $\boldsymbol{\mathbf{c}}\sim r_{\psi}^{+}(\boldsymbol{\mathbf{c}}|\textsf{\boldmath$S$}^{\prime})$. However, there is currently no known reparametrization trick that can be applied to random variables that follow multinomial distributions. Instead, we use the Gumbel-Softmax (GS) distribution as an approximation to the multinomial distribution, which allows the use of the reparameterization trick [66, 67]. The GS distribution of a continuous multivariate variable $\textsf{\boldmath$k$}=[k_{1},\ldots,k_{\rm I}]^{\textsf{T}}$ is defined as

This expression is derived analytically as a distribution that is followed by the variables

where ${\rm g}_{\rm i},\leavevmode\nobreak\ {\rm i}=1,\ldots,{\rm I}$ are Gumbel samples drawn independently and identically from ${\rm Gumbel}(0,1)$, $\rho$ is the class probability vector produced by the classifier, and $\tau$ is called the softmax temperature. Here, it is important to note that (29) is shown to become identical to $r_{\psi}^{+}(\textsf{\boldmath$k$}|\textsf{\boldmath$S$}^{\prime})$ as $\tau$ approaches 0. By replacing $r_{\psi}^{+}(\textsf{\boldmath$k$}|\textsf{\boldmath$S$}^{\prime})$ with (29), (27) and (28) can be approximated as

Unlike the original expressions, these expressions allow the computations of the derivatives with respect to $\psi$ using the reparameterization trick.

With the reduced number of model parameters, the challenge is how to make the ChimeraACVAE model have a high generalization capability. To this end, we further introduce training criteria derived based on the KD [51] using a pretrained CVAE model as the teacher model. KD, also known as teacher-student learning, is a technique to transfer the knowledge from a teacher model to a student model, originally proposed for model compression [51] and later shown to improve the generalization capability of the student model [68]. There are three types of knowledge that can be transferred between models: response-based knowledge, feature-based knowledge, and relation-based knowledge. These refer to the knowledge of the last output layer, the knowledge of each output layer, and the knowledge of the relationship between layers, respectively. Since the networks in both the teacher and student models are reasonably shallow, we consider response-based KD to be sufficient, as it requires a minimal increase in training cost.

Specifically, we transfer the knowledge of the distributions of the latent variable $q_{\phi}^{*}(\boldsymbol{\mathbf{z}}|\textsf{\boldmath$S$},\boldsymbol{\mathbf{c}})$ and the complex spectrograms $p_{\theta}^{*}(\textsf{\boldmath$S$}|\boldsymbol{\mathbf{z}},\boldsymbol{\mathbf{c}})$ learned by the CVAE model into the ChimeraACVAE model by using these distributions as priors. We use the KL divergences to measure the differences between the distributions estimated by a student model and the pretrained teacher model such that

Here, (35) is a criterion that measures the difference between the teacher distribution and decoder distribution computed using the GS distribution. An illustration of KD for training the ChimeraACVAE model is shown in Fig. 3.

The total training criterion of the ChimeraACVAE is a weighted linear combination of the above-mentioned criteria:

Here, $\lambda_{\ast}$ denotes a non-negative parameter that weighs the importance of that term.

With the trained ChimeraACVAE source model, we can use the same procedure as Algorithm 1 to perform source separation. We call it FastMVAE2 to distinguish it from the method using the ACVAE source model. Note that in FastMVAE2, the PoE-based update rule is no longer required thanks to the improved generalization capability, but of course it can be used in addition.

For the speaker-dependent source separation experiment, we used speech utterances of two male speakers (SM1, SM2) and two female speakers (SF1, SF2) excerpted from the Voice Conversion Challenge (VCC) 2018 dataset [69] . The audio files for each speaker were about seven minutes long and manually segmented into 116 short sentences, where 81 and 35 sentences (about five and two minutes long, respectively) served as training and test sets, respectively. We used two-channel mixture signals of two sources as the test data, which were synthesized using simulated room impulse responses (RIRs) generated using the image method [70] and real RIRs measured in an anechoic room (ANE) and an echo room (E2A). The reverberation times ($RT_{60}$) [71] of the simulated RIRs were set at 78 and 351 ms, which were controlled by setting the reflection coefficient of the walls at 0.20 and 0.80, respectively. For the measured RIRs, we used the data included in the RWCP Sound Scene Database in Real Acoustic Environments [72]. The $RT_{60}$ of ANE and E2A were 173 and 225 ms, respectively. The test data included four pairs of speakers, i.e., SF1+SF2, SF1+SM1, SM1+SM2, and SF2+SM2. For each speaker pair, we generated ten mixture signals. Hence, there were a total of 40 test signals for each reverberation condition, each of which was about four to seven seconds long.

The datasets for the speaker-independent experiment were generated in the same way by using the Wall Street Journal (WSJ0) corpus [73]. All the utterances in WSJ0 folder $\tt si\_tr\_s$ (around 25 hours) were used as the training set, which consists of 101 speakers in total. A test set was created by randomly mixing two different speakers selected from the WSJ0 folders $\tt si\_dt\_05$ and $\tt si\_et\_05$, where the number of speakers was 18. We generated test data using simulated RIRs with $RT_{60}=78$ ms and $RT_{60}=351$ ms, where 100 mixture signals were generated under each reverberation condition. All the speech signals were resampled at 16 kHz. The STFT and inverse STFT were calculated by using a Hamming window with a length of 128 ms and half overlap.

We chose ILRMA [7], the MVAE method [26]¹¹1Code: https://github.com/lili-0805/MVAE , and the FastMVAE method [46] as the baseline methods for both the speaker-dependent and speaker-independent cases, and IDLMA [30] as another baseline method only for the speaker-dependent scenario. For all the methods, the parameter optimization algorithms were run for 60 iterations, and the separation matrix $\boldsymbol{\mathbf{W}}(f)$ was initialized with an identity matrix.

We set the basis number of ILRMA at 2, which is the optimal setting for speech separation. For IDLMA, we used the same network architecture and training settings as those in [30] except for the optimization algorithm, where we used Adam [74] instead of Adadelta [75]. Note that unlike other methods where speaker information is estimated, IDLMA requires speaker information in order to properly select the corresponding pre-trained network. The network architectures for the CVAE and ACVAE source models were the same as those used in [46], where the encoder consisted of 2 convolutional layers using GLU following a regular convolutional layer, the decoder consisted of 2 deconvolutional layers using GLU following a regular deconvolutional layer, and the classifier consisted of 3 convolutional layers using GLU following a regular convolutional layer. All the GLU layers used batch normalization to stabilize the training. Adam was used to train the networks and estimate $\boldsymbol{\mathbf{z}}_{j}$ and $\boldsymbol{\mathbf{c}}_{j}$ in the MVAE method. In the training of ChimeraACVAE, the weight parameters were empirically set with the KD criterion $\mathcal{K}_{\boldsymbol{\mathbf{z}}}$ as 10 and the rest as 1. The temperature $\tau$ for the GS distribution was set at 1.

We calculated the source-to-distortions ratio (SDR), source-to-interferences ratio (SIR), and sources-to-artifacts ratio (SAR) [76] to evaluate the source separation performance, and used perceptual evaluation of speech quality (PESQ)²²2Code: https://github.com/vBaiCai/python-pesq [77] and short-time objective intelligibility (STOI) ³³3Code: https://github.com/mpariente/pystoi [78] to ascertain the speech quality and intelligibility of the separated waveforms.

We first investigated the effectiveness of each training criterion proposed in Subsection IV-B in training the proposed ChimeraACVAE source model. The correspondence between the models and their training criteria are shown in Table II. Table III shows the results, which were calculated by averaging over the entire dataset including multiple reverberation conditions. The results show that it is effective to further exploit the reconstruction loss and classification loss of the spectrograms reconstructed with the estimated class label in the speaker-dependent scenario, where small amounts of training data were available. Comparing the models trained without KD (1st and 2nd rows) with that trained with KD (3th to 5th rows), we found an improvement in SDR of about 2.6 dB in speaker-dependent situations and more than 1 dB in speaker-independent ones, which confirmed that KD can significantly improve source separation performance. In particular, knowledge transfer of the distribution of the latent variable $\boldsymbol{\mathbf{z}}$ was effective in stabilizing the inference accuracy even for unseen speakers. A further improvement was achieved in the speaker-independent setting by transferring knowledge between distributions of generated complex spectrograms, but no improvement was seen in the speaker-dependent setting.

In Table IV, we show a comparison of source separation performance between the FastMVAE and FastMVAE2 methods. To demonstrate the effectiveness of the proposed training criterion, we trained an ACVAE with the architecture that respectively replaces BN and GLU with LN and SiLU, which is referred to as “compact FastMVAE”. The results of FastMVAE and compact FastMVAE indicate that the replacement of the normalization method and nonlinear activation did not lead to an improvement of the source separation performance. Therefore, the performance improvement by FastMVAE2 can be attributed mainly to the proposed training criterion. The FastMVAE2 method obtained the highest scores in terms of all the criteria. Particularly, FastMVAE2 achieved an SDR improvement of about 6.6 and 2.6 dB from the FastMVAE without and with PoE, respectively. These results indicated that the ChimeraACVAE source model had good generalization to unseen data, which made the FastMVAE2 method able to handle speaker-independent scenario without the heuristic inference method. Table V shows the average scores achieved by each method with their optimal parameter settings. The proposed method significantly outperformed ILRMA and the FastMVAE method, and narrowed the performance gap with the MVAE method.

In this subsection, we investigate the computational time of each method. We conducted speaker-independent experiments with more sources and channels, and compared the computation time of each method for each update iteration and overall processing time.

As in the above speaker-independent experiment, the simulated RIRs in the {2, 3, 6, 9, 12, 15, 18}-channel cases were generated using the image method [70] with the reflection coefficient of the walls set at 0.20. The details of the room configuration and microphone array are shown in Fig. 4. In each case, more sound sources and microphones were added and placed in the order of increasing numbers. Speech utterances were randomly selected from the WSJ0 folders $\tt si\_dt\_05$ and $\tt si\_et\_05$. We generated 10 samples for each case. The minimum, maximum, and average lengths of the mixture signals are shown in Table VI. The average SDR of the generated mixture signals for each case is shown in the first row of Table VIII. All algorithms were processed using an Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz and a Tesla V100 GPU. Other experimental settings were the same as those in the above speaker-independent experiment.

The inference times of ILRMA, FastMVAE, and FastMVAE2 are shown in Fig. 5, and those of MVAE are shown in Table VII as a reference. The fast algorithms performed extremely fast by using a GPU. Comparing the computation times in the CPU, we found that the FastMVAE2 method achieved runtimes comparable to ILRMA in the 2-source and 3-source cases, and faster than ILRMA in cases with more than 3 sources. This indicates that the proposed method is more efficient in situations with a large number of sources and microphones. The average SDR scores obtained by each method are shown in Table VIII. The proposed FastMVAE2 outperformed ILRMA and the FastMVAE without PoE, and even outperformed the MVAE method in the 2-source case, demonstrating the effectiveness of the proposed ChimeraACVAE source model. Note that although the performance of ILRMA was superior to the proposed method in the cases of 3 and 6 sources, this might change with different initialization of the basis and activation matrices of the NMF. On the other hand, the performance of the proposed method is independent of the initialization. We show an example of the magnitude spectrograms of separated signals obtained by ILRMA, MVAE, and FastMVAE2 with their corresponding ground truth signals in Fig. 6⁴⁴4Audio samples are available at http://www.kecl.ntt.co.jp/people/
kameoka.hirokazu/Demos/mvae-ss/index.html. We found that although the MVAE and FastMVAE2 methods suffered from the phenomenon called block permutation [79, 80], in which the permutations in different frequency blocks are inconsistent, the deep generative model-based source models improved the estimation accuracy in the low-frequency band (0-2 kHz), which resulted in a more remarkable SDR improvement compared with ILRMA.

In this subsection, we evaluate the proposed FastMVAE2 in the spatialized WSJ0-2mix benchmark[18], which is widely used for evaluating the recent DNN-based methods. There are 20,000 ($\sim$30h), 5,000 ($\sim$10h), and 3000 ($\sim$5h) utterances in the training, validation, and test sets. The training and validation mixtures were generated from data in $\tt si\_tr\_s$ folder and the test mixtures were generated from data in $\tt si\_dt\_05$ and $\tt si\_et\_05$ folders. Therefore, the speaker-independent settings mentioned in the above experiments are still valid. RIR used for every utterance was simulated with a random configuration, including room characteristics, speaker locations, and microphone geometry. $RT_{60}$ for the reverberant case was randomly selected from 200 to 600 ms.

We compared FastMVAE2 with (1) oracle ideal binary mask (IBM), (2) oracle ideal ratio mask (IRM), (3) oracle mask-based minimum variance distortionless response (MVDR) beamformer [81], (4) oracle signal-based MVDR, 5) time-domain audio separation network (TasNet) [19], (6) multichannel TasNet [23], and (7) Beam-TasNet [23]. The oracle IBM and IRM were obtained using the first channel of the spatialized clean sources and applied to the first channel of observed mixture signals. The difference between the oracle mask-based and signal-based MVDR was the signal used for computing spatial covariance matrices, where the former used the multichannel IRM of each source and the later directly used the clean reverberant speech of each source. We investigated window lengths of 128 ms and 512 ms. Settings for TasNet, multichannel TasNet, and Beam-TasNet are available in [23] ⁵⁵5We would like to appreciate Dr. Tsubasa Ochiai from NTT Communication Science Laboratories for providing us with the test dataset and evaluation script so that we could compare our methods with results reported in [23].. One important factor here is the window length. Beam-TasNet used a length of 512 ms to meet the instantaneous mixture model for reverberant signals, while the proposed method used that of 128 ms since the motivation of the FastMVAE2 is to bridge the high performance of MVAE and real-time applications with low latency.

We first show the results of the spatialized anechoic WSJ0-2mix dataset in Table IX. With the anechoic setup, beamforming algorithms achieved higher performance than mask-based methods. From these results, we confirmed that the MVAE and proposed FastMVAE2 achieved even better performance than the oracle mask-based MVDR beamformer, indicating the effectiveness of these two methods when the instantaneous mixture model assumption is satisfied. Next, we show the results of the spatialized reverberant WSJ0-2mix dataset in Table X. The performance of the MVAE and FastMVAE2 degraded significantly due to reverberations. The main reason was the instantaneous mixture model assumption, which was not satisfied anymore with heavy reverberation and short window length. We found that even the performance of oracle MVDRs degraded significantly when the window length became shorter. Two promising approaches can be considered to deal with this problem, including using longer window length and performing separation along with dereverberation [82, 83, 40]. It is straightforward that longer window length helps deal with heavy reverberant conditions, which has also been confirmed from the results of oracle MVDRs with longer window length and Beam-TasNet. However, a longer window length is undesirable and should be avoided in real-time applications because it increases algorithmic latency. Therefore, we consider the second approach, performing separation and dereverberation simultaneously, as one direction of our future works to overcome this limitation of the FastMVAE2 method.