Conditional Deep Hierarchical Variational Autoencoder for Voice Conversion

In this paper, mel-spectrogram is used as an acoustic feature. Let $x\in X$ be a segment of mel-spectrogram where $X$ is the ($80\times T$)-dimensional Euclidean Space. Here, $80$ is the dimension of the features. $T$ is the sequence length of a segment, and we set $T=40$, which corresponds to 0.5 seconds, in our experiments. Note that an utterance would be split into segments with a sequence length $T=40$ without overlapping at a conversion step. Also, let $y\in Y$ be the speaker label where $Y$ is the group of all speakers. The training set $D=\{\{x_{1},y_{1}\},...,\{x_{m},y_{m}\}\}$ contains $m$ pairs of $(x_{i},y_{i})\in(X,Y)$, where $x_{i}$ is produced by the speaker $y_{i}$. Given a tuple of a source speech, its speaker label, and target speaker label $(x_{s},y_{s},y_{t})$, the goal of VC is to obtain an acoustic feature $x_{s\to t}$ that contains speaker characteristics of $y_{t}$ while preserving the linguistic content of $x_{s}$.

Here we present a brief overview of VAE-VC [10, 12]. VAE-VC approaches are aimed at obtaining speaker-invariant latent variables that contain only linguistic information by reconstructing speech from the latent variables and speaker labels. Specifically, the methods consider a data generating process where an acoustic feature $x$ is generated from the latent variable $z$ and speaker label $y$, and parameterize the process with a deep neural network (DNN) decoder $p(x|z,y)$. Here, $z$ and $y$ are defined to be independent, i.e., $p(z,y)=p(z)p(y)$. The independence encourages $z$ to model information invariant to the speaker, i.e., linguistic information. The objective of the VAE is given as follows:

	$\displaystyle\max\mathcal{L}=$	$\displaystyle-D_{KL}[q(z|x,y)||p(z)]+\mathbb{E}_{q(z|x,y)}[\log p(x|z,y)]$
	$\displaystyle\eqqcolon$	$\displaystyle-\mathrm{KL}-\mathrm{Rec},$		(1)

where the first term is the Kullback-Leibler (KL) divergence between the encoder $q(z|x,y)$ and the prior $p(z)$, and the second term is a negative reconstruction error.

Using the trained VAE, the conversion step is performed by decoding the speech from the speaker-invariant latent variable and the label of the target speaker. Specifically, first, the encoder $q(z_{s}|x_{s},y_{s})$ is used to extract speaker-invariant latent variables $z_{s}$ from the source speech $x_{s}$. Then, VC can be performed by inputting $z_{s}$ and target speaker $y_{t}$ to the decoder $p(x|z_{s},y_{t})$.

DHVAEs, which are also called as bidirectional-inference autoencoder [21], are the family of VAE variants with high model expressiveness [17, 18, 19, 20]. They share the same data generating process while their architectural details are different for each study. It has three components parameterized by DNNs: an encoder $\Pi_{l=1}^{L}q(z_{l}|x,z_{<l})$, prior $\Pi_{l=1}^{L}p(z_{l}|z_{<l})$, and decoder $p(x|z)$. Here, contrary to standard Gaussian prior for vanilla VAEs, the prior of DHVAEs enhances the expressiveness by hierarchically modeling $L$ latent variables $z=\{z_{1},...,z_{L}\}$, where $L$ is a hyperparameter. The objective is given as follows:

	$\displaystyle\max\mathcal{V}=$	$\displaystyle-\sum_{l=1}^{L}\mathbb{E}_{q(z_{<l}|x)}D_{KL}[q(z_{l}|x,z_{<l})||p(z_{l}|z_{<l})]$
		$\displaystyle+\mathbb{E}_{q(z|x)}[\log p(x|z)],$		(2)

which also consists of the KL and reconstruction terms.

In this section, we adapt the RD analysis on VAE from [22] to voice conversion setting. Based on the analysis, we will point out that it is necessary to use VAE with high model expressiveness for getting both good naturalness and similarity. Speech naturalness and similarity are subjective measures. The former qualifies how natural (human-like) the converted speech $x_{s\to t}$ is. The latter qualifies how similar the converted speech $x_{s\to t}$ is to that of the target speaker $y_{t}$. Here, the rate equals to the KL divergence term ($\mathrm{KL}$), which measures the degree of compression. Also, the distortion equals to the reconstruction error term ($\mathrm{Rec}$).

The RD analysis givin in [22] suggests that there is a trade-off between $\mathrm{KL}$ and $\mathrm{Rec}$, given a model architecture and the accompanied expressiveness. Specifically, the phase diagram in Figure 1 shows the realizable combination of $\mathrm{KL}$ and $\mathrm{Rec}$ for a given model (either large or small). Also, the Pareto frontier, i.e., the optimal combination of $\mathrm{KL}$ and $\mathrm{Rec}$, is called a RD curve. The RD curve consists of a set of models trained by the following $\beta$-VAE objective, in which the weighting parameter $\beta\geq 0$ is introduced into Eq. 1:

$\displaystyle\max\mathcal{L}_{\beta}=$

$\displaystyle-\beta\mathrm{KL}-\mathrm{Rec}.$

(3)

When $\beta=1$, the objective targets a single point on the RD curve with slope 1 because $\mathrm{Rec}$ (y-axis) can be expressed as a linear function $\mathrm{Rec}=-\mathcal{L}-\mathrm{KL}$. Setting $\beta<1$ leads the situation where $\mathrm{Rec}$ is low but $\mathrm{KL}$ is high while $\beta>1$ leads the opposite case (high $\mathrm{Rec}$ and low $\mathrm{KL}$), all without having to change the model architecture.

Next, we summarize the relationship between the RD curve and VC performance. (i) First, low $\mathrm{Rec}$ is a necessary condition for high naturalness. This is because high $\mathrm{Rec}$ disables the decoder to construct natural speech, and degrades the naturalness of the converted speech sampled from the decoder. (ii) Second, low $\mathrm{KL}$ is a necessary condition for high similarity. This is because high $\mathrm{KL}$ suggests that $z$ contains much information about $x$, which contradicts the intention of obtaining speaker-invariant latent variables. On the other hand, low $\mathrm{KL}$ indicates that $z$ is well-compressed so redundant speaker information is lost from $z$ (note that the speaker information is inherently redundant for $z$ because the decoder is conditioned on $y$). Based on (i) and (ii), we conclude that $\mathrm{Rec}$ and $\mathrm{KL}$ affect naturalness and similarity, respectively. Note that, however, low $\mathrm{KL}$ and low $\mathrm{Rec}$ are necessary conditions but not sufficient on their own: to obtain high naturalness and similarity both $\mathrm{KL}$ and $\mathrm{Rec}$ need to be low. For example, even if $\mathrm{Rec}$ is low, the naturalness can be low when $\mathrm{KL}$ is so large that the source and target speaker information is entangled in the inferred latent variables.

Based on the analysis, we can take two approaches to getting good naturalness and similarity. One is using a hyperparameter $\beta$ in Eq. 3 to achieve the best combination of $\mathrm{KL}$ and $\mathrm{Rec}$ for a given model (note that there is no guarantee that $\beta=1$ achieves the best combination for VC). The other is using a VAE with higher model expressivess to expand the realizable combinations, i.e., to move the RD curve to the left.

Based on the analysis in Section 3.1, we propose using a VAE with high model expressiveness, as well as using the $\beta$-VAE objective. Namely, this paper adapts DHVAEs (see, Section 2.3) to VC tasks because they have high model expressiveness thanks to the hierarchical architecture. However, they cannot be naively applied to VC tasks because they are unconditional models and have hierarchical latent variables. Then, we propose CDHVAE, which separates speaker-dependent and invariant latent variables to perform VC.

First, we present a data generating process of CDHVAE, where the conditioning with speaker label $y$ is added to DHVAEs. Namely, as well as DHVAE, CDHVAE has encoder, decoder, and prior. However, the three components are conditioned with $y$ as follows:

	$\displaystyle\mathrm{Encoder}$	$\displaystyle=q(z_{l}|x,z_{<l},y)\;\;\forall l\leq L,$
	$\displaystyle\mathrm{Decoder}$	$\displaystyle=p(x|z,y),$
	$\displaystyle\mathrm{Prior}$	$\displaystyle=\begin{cases}p(z_{l}|z_{<l})&\text{if}\;l\leq K,\\ p(z_{l}|z_{<l},y)&\text{else}.\end{cases}$

The data generating process is illustrated in Figure 2-(a, b). Here, $p(z_{\leq K})$ and $p(z_{>K},y)$ are the speaker-invariant and dependent latent variables, respectively. Also, $K$ is a hyperparameter that controls the expressiveness of the speaker-invariant and dependent latent variables. It is because $p(z_{\leq K})$ becomes more expressive when $K$ becomes large, and vice versa ($p(z_{>K},y)$ becomes more expressive when $K$ becomes small). More specifically, as $K$ increases, the hierarchy of $p(z_{\leq K})$ increases, so the model expressiveness of $p(z_{\leq K})$ increases. Then, a significantly small $K$ makes it difficult for speaker-invariant variables $z_{\leq K}$ to model linguistic information, which contradicts the intention of VAE-VC. Also, a significantly large $K$ makes it difficult for speaker-dependent variables $z_{>K}$ to model speeches, which can degrades output speech quality.

Using the components, the objective of CDHVAE is given as follows:

	$\displaystyle\max$	$\displaystyle\mathcal{J}_{\beta}=-\beta\sum_{l=1}^{K}\mathbb{E}_{q(z_{<l}|x,y)}D_{KL}[q(z_{l}|x,z_{<l},y)||p(z_{l}|z_{<l})]$
		$\displaystyle-\beta\sum_{l=K+1}^{L}\mathbb{E}_{q(z_{<l}|x,y)}D_{KL}[q(z_{l}|x,z_{<l},y)||p(z_{l}|z_{<l},y)]$
		$\displaystyle+\mathbb{E}_{q(z|x)}[\log p(x|z)].$		(4)

Here, we introduce a weighting parameter $\beta$ to balance the KL and reconstruction terms, as was done in Eq. 3.

Using the trained CDHVAE, VC can be performed with the following procedure. First, the speaker-invariant latent variable $z_{\leq K}$ is obtained from the encoder $q(z_{\leq K}|x_{s},y_{s})$, given $x_{s}$ and $y_{s}$. Then, the converted speech $x_{s\to t}$ can be obtained by inputting $z_{\leq K}$ and the target speaker label $y_{t}$ to the prior and the decoder. This data generating process is illustrated in Figure 2-(c), and can be expressed as follows:

$\displaystyle x_{s\to t}\sim\mathbb{E}_{q(z_{\leq K}|x_{s},y_{s})}\mathbb{E}_{\Pi_{l=K+1}^{L}p(z_{l}|z_{<l},y_{t})}p(x|z,y_{t}).$

(5)

In practice, we approximate expectations with the mean of $q(z_{\leq K}|x_{s},y_{s})$ and $\Pi_{l=K+1}^{L}p(z_{l}|z_{<l},y_{t})$. Also, we split an utterance $x_{s}$ into segments with a sequence length $T=40$ and perform segment-wise conversion to adopt the same settings as training, but in principle it is possible to perform utterance-wise conversion.

The remaining challenge is what neural network architecture should be used to parameterize the data generating process of CDHVAE and to incorporate speaker labels. We choose to adapt the model architecture of Nouveau VAE (NVAE) [19] for CDHVAE, which is mainly composed of depthwise convolutions [23, 24] and recently achieved state-of-the-art results in image generation. However, because the original NVAE is an unconditional model, we add a simple modification for enabling conditioning: replacing all Batch Normalization (BN) layers with Conditional Instance Normalization (CIN) [25, 26]. Specifically, the original NVAE have BN layers in components called residual cells which correspond to $r$ in Figure 2. Then, we obtain speaker embedding from one-hot label $y$ using linear transformation, and use them as scale and location parameters in CIN. Also, since NVAE has been proposed in the literature on image generation, the input should be ($C,H,W$)-dimensional tensor, where $C$ is the channel size, $H$ is the image height, and $W$ is the image width. Therefore, we set $C=1$, $H=80$, and $W=T=40$ such that the mel-spectrogram can be used instead of an image.

In this section, we will evaluate CDHVAE on many-to-many VC tasks. We strongly encourage readers to listen to the demos that can be found in https://dena.github.io/CDHVAE/. The evaluation is performed on the VCTK corpus [35]. While the original corpus includes utterances from 109 speakers, we trained the models on 20 speakers (10 females and 10 males), as performed in [29, 30]. The acoustic features were mel-spectrograms extracted from 48kHz audio. As the vocoder, we used MelGAN [36], which was trained on the same corpus.

CDHVAE is trained with a batch size of 8 for 200 epochs. Regarding the model architecture, we adapt CelebA model in Table 6 of [19] with two modifications: replacing BN with CIN for conditioning, and removing normalizing flows for faster training. The other training settings are the same with the official NVAE implementation ¹¹1https://github.com/NVlabs/NVAE. In the CelebA model, the number of latent variables is set to $L=35$. Also, based on our informal preliminary experiments, the hyperparameter in Eq. 3.2 is set to $K=10$. While this paper does not include hyperparameter study regarding $K$ due to computational resource limitation, we recommend not to use very small values nor very large values since $K$ balances the expressiveness of the speaker-invariant and speaker-dependent latent variables. CDHVAE is compared to the recent autoencoder-based method proposed by [29], whose pretrained-model is publicly available ²²2https://github.com/jjery2243542/voice_conversion. Also, CDHVAE was trained with various values of $\beta\in\{1,10,50\}$. Here, because we observed that CDHVAE with $\beta=1$ resulted in high naturalness but low similarity in our preliminary experiments, we did not test $\beta<1$.

We conducted naturalness and similarity tests using MOSs in the same manner as in previous studies [30, 11]. We first selected one sentence for each of randomly selected 10 speakers (5 males and 5 females), and converted it to one randomly selected male speaker and one randomly selected female speaker, resulting in 20 utterances composed of 5 male-to-male (M2M), 5 male-to-female (M2F), 5 F2M, and 5 F2F samples. Then, subjects in Amazon Mechanical Turk were asked to evaluate 80 utterances (20-utterance times 4-method). After applying post-screening [37], the answers for 29 and 45 subjects, who rated more than 70 utterances, were collected for naturalness and similarity tests, respectively.

From the MOS results, we can make the following observations. (i) Using the VAE with high model expressiveness improved naturalness and similarity: the results in Figure 3 shows that CDHVAE with $\beta=10$ achieved higher performance than the baseline method [29]. Moreover, it achieved MOSs higher than 3.5 on both naturalness and similarity in inter-gender settings. To give readers a better idea of what this means, notice that none of the existing autoencoder-based VC methods (e.g., AutoVC [30] nor the methods in VCC 2020) reached 3.5 on naturalness and similarity at the same time. Even though these are not fair comparisons due to the difference in experimental settings (e.g., training data or Hz of speeches), it is encouraging to see that even with the simple $\beta$-VAE objective, the increasing expressiveness of VAEs enabled competitive or higher quality compared to the existing autoencoder-based VC methods.

(ii) $\beta$ is an important factor for balancing the tradeoff between naturalness and similarity: the results in Figure 4 shows that CDHVAE with low $\beta$ value ($\beta=1$) achieved lower similarity scores than those with $\beta=50$ in inter-gender settings. This indicates that a small $\mathrm{KL}$ value is a necessary condition for good similarity. On the other hand, CDHVAE with high $\beta$ value ($\beta=50$) achieved the lowest naturalness scores because the very large $\beta$ makes reconstruction difficult. This indicates that a small $\mathrm{Rec}$ value is a necessary condition for good naturalness. Here, note that small $\mathrm{KL}$ and $\mathrm{Rec}$ values are merely the necessary conditions as noted in Section 3.1, and properly selected $\beta=10$ achieved the highest MOSs for both naturalness and similarity.

Conversion speed: On a 16-GB Tesla T4 GPU, we can convert a segment of mel-spectrogram of the size 80 $\times$ 40 in 0.172 seconds (344 ms / 1-second speech).