Disentangled Speech Representation Learning for
One-Shot Cross-lingual Voice Conversion Using $\beta$-VAE

A basic observation about speech is that the content varies constantly across an utterance while the speaker identity remains the same for the whole utterance. That is, the content is a time-variant feature while speaker identity is a time-invariant one. This informs us about the structures of the content and speaker representations. Suppose that the acoustic feature $\mathbf{x}$ is of the shape $\mathbb{R}^{T\times D_{x}}$, where $T$ and $D_{x}$ denote the number of frames and the dimension of $\mathbf{x}$, respectively. Then $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ should have the corresponding shapes of $\mathbb{R}^{T\times D_{c}}$ and $\mathbb{R}^{D_{s}}$ to separately capture the time-variant content information and time-invariant speaker identity information, where $D_{c}$ and $D_{s}$ are the dimensions of $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$, respectively. A content encoder transforming the input frame-wisely and a speaker encoder aggregating the whole input sequence into a single vector should thus be adopted. Many methods introduced in Section 1 have adopted similar architectures.

However, this architecture alone cannot guarantee the disentanglement of content and speaker representations. The reasons are two-fold: First, the architecture cannot ensure that the two representations learned are complementary to each other. Intuitively, the two representations will strive to capture as much information about the input data as possible to reduce the reconstruction loss. Thus, it is possible that the two representations have overlap in the captured information. Second, even if we can restrict the two representations to be complementary (e.g., via bottleneck), then we can reasonably assume that $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ respectively captures complementary time-variant and time-invariant information of speech, but it is not guaranteed that the learned time-variant and time-invariant elements are exactly content and speaker identity. The reason is that there are many combinations of time-variant and time-invariant information elements other than the targeted one. For example, background noise can (mostly) be considered as a time-invariant element, against the time-variant element of clean speech, but the architecture alone cannot distinguish this pair of elements from the content-speaker identity pair.

Aside from the architectural design to facilitate the separation of time-variant and time-invariant feature into $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$, we also need to ensure that: 1) the information captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ are complementary and 2) the complementary elements captured are content and speaker identity instead of other element pairs – and the key is to restrict the amounts of information of $\mathbf{x}$ that can be captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$. To solve issue 1), we can restrict the overall amount of information captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ together to a low level. With no redundant information encoded, the two representations will be compact and contain complementary information. To further tackle issue 2), it is important to restrict the relative amounts of information captured in the two representations, such that $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ captures exactly the content part and the speaker identity part of information in $\mathbf{x}$, respectively.

The information captured by a latent variable can be quantified as its MI with the data. Following the typical practice by approximating the joint distribution $p(\mathbf{x},\mathbf{z})$ with its variational version $q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})$ [20, 21], we can derive the variational MI between $\mathbf{x}$ and $\mathbf{z}$ as shown in Eqn. (3.2), where $\mathbf{z}$ is a general variable name covering $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$.

	$\displaystyle\mathbb{I}_{v}(\mathbf{x},\mathbf{z})$	$\displaystyle=\mathbb{E}_{q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})}\left[\log{\frac{q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})}{p_{d}(\mathbf{x})q_{\phi}(\mathbf{z})}}\right]$
		$\displaystyle=\mathbb{E}_{q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})}\left[\log{\frac{q_{\phi}(\mathbf{z}|\mathbf{x})}{p(\mathbf{z})}}+\log{\frac{p(\mathbf{z})}{q_{\phi}(\mathbf{z})}}\right]$
		$\displaystyle=\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z}|\mathbf{x})\parallel p(\mathbf{z})\right]\right]$
		$\displaystyle-\text{KL}\left[q_{\phi}(\mathbf{z})\parallel p(\mathbf{z})\right]$		(2)

Here $q_{\phi}(\mathbf{z})$ is the marginal distribution of $\mathbf{z}$ and is defined as $q_{\phi}(\mathbf{z})=\int_{\mathbf{x}}{q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})}d\mathbf{x}$. The results in Eqn. (3.2) is different from the MI derived in [18], which claim that $\mathbb{I}_{v}(\mathbf{x},\mathbf{z})=\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z}|\mathbf{x})\parallel p(\mathbf{z})\right]\right]$. This derivation is resulted from approximating the marginal distribution of $\mathbf{z}$ with its prior $p(\mathbf{z})$, which we consider is generally not applicable.

Eqn. (3.2) denotes that $\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z}|\mathbf{x})\parallel p(\mathbf{z})\right]\right]$ is an upper-bound of $\mathbb{I}_{v}(\mathbf{x},\mathbf{z})$. In this sense, an approximate method to restrict the amount of information of $x$ captured by the latent variable $\mathbf{z}$ is to restrict the KL divergence term $\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z}|\mathbf{x})\parallel p(\mathbf{z})\right]\right]$. Though the term $\text{KL}\left[q_{\phi}(\mathbf{z})\parallel p(\mathbf{z})\right]$ is also penalized as a side-effect, which encourages the marginal distribution of $\mathbf{z}$ to be more like an isotropic Gaussian and does not other harm. Motivated by this, we introduce two weight parameters $\beta_{c}$ and $\beta_{s}$ to the content KL and speaker KL terms, respectively, with the goal to restrict the information that can be captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$. The restricted objective function is shown in Eqn. (3), while Eqn. (4) explicitly denotes the relationship between two KL terms and their corresponding MI terms.

	$\displaystyle\mathcal{L}_{\beta}$	$\displaystyle=-\mathbb{E}_{p_{d}(\mathbf{x})q_{\phi}(\mathbf{z_{c}}|\mathbf{x})q_{\phi}(\mathbf{z_{s}}|\mathbf{x})}\left[\log p_{\theta}(\mathbf{x}|\mathbf{z_{c}},\mathbf{z_{s}})\right]$
		$\displaystyle+\beta_{c}\cdot\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z_{c}}|\mathbf{x})\parallel p(\mathbf{z_{c}})\right]\right]$
		$\displaystyle+\beta_{s}\cdot\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z_{s}}|\mathbf{x})\parallel p(\mathbf{z_{s}})\right]\right]$		(3)
		$\displaystyle=-\mathbb{E}_{p_{d}(\mathbf{x})q_{\phi}(\mathbf{z_{c}}|\mathbf{x})q_{\phi}(\mathbf{z_{s}}|\mathbf{x})}\left[\log p_{\theta}(\mathbf{x}|\mathbf{z_{c}},\mathbf{z_{s}})\right]$
		$\displaystyle+\beta_{c}\cdot\left[\mathbb{I}_{v}(\mathbf{x},\mathbf{z_{c}})+\text{KL}\left[q_{\phi}(\mathbf{z_{c}})\parallel p(\mathbf{z_{c}})\right]\right]$
		$\displaystyle+\beta_{s}\cdot\left[\mathbb{I}_{v}(\mathbf{x},\mathbf{z_{s}})+\text{KL}\left[q_{\phi}(\mathbf{z_{s}})\parallel p(\mathbf{z_{s}})\right]\right]$		(4)

While the reconstruction loss in Eqn. (3) encourages both $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ to capture as much information about $\mathbf{x}$ as possible, the weight parameters $\beta_{c}$ and $\beta_{s}$ act as two gates to control the amounts of information of $\mathbf{x}$ going through $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$, as indicated by Eqn. (4). The learning objective (3) can help solve the problems of the architectural design in the following ways: By choosing the proper absolute values of both $\beta_{c}$ and $\beta_{s}$, the information captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ together will be compact and thus the information contained in each latent variable will be complementary. On the other hand, by tuning relative values of $\beta_{c}$ and $\beta_{s}$, the amounts of information in $\mathbf{x}$ allocated to $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ will change accordingly. In this case, we can find the optimal parameters pair that yields exactly the separation of content and speaker identity.

Note that if we let $\beta_{c}=\beta_{s}$ and combine $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ together, Eqn. (3) will become exactly the learning objective of the standard $\beta$-VAE [11]. In this sense, Eqn. (3) can be considered as a generalized form of $\beta$-VAE.

The dataset used to train the model is also a very important inductive bias to facilitate the learning of disentangled representations. One consideration is that the two information elements to be disentangled should vary independently in the dataset. Besides, other elements of variation that may interfere the interested elements should not appear in the dataset. For example, when disentangling the content and speaker identity elements, too much variation in emotions may disturb the learning of speaker identity element since they are both sequence-level elements. In our case, we combine two monolingual corpora as the training dataset, in which the speaker and general content are the main variations. More details are given in Section 5.1.

We combine two openly available corpora, which are VCTK [27] and AISHELL-3 [28] together for training and evaluation of the proposed model. VCTK contains 110 speakers’ English speech data while AISHELL-3 consists of speech uttered by 218 native Mandarin speakers. 88 speakers’ data from VCTK and 116 speakers’ data from AISHELL-3 are combined as a bilingual training set. Another 20 speakers’ data from VCTK are evenly split for validation and testing, while for AISHELL-3 15 and 16 other speakers’ data are used so. Note that there are no common speakers among different splits of the dataset. We down-sample all speech into 16kHz and extract 80-dimensional logarithm Mel-Spectrograms with 200ms window length and 50ms window shift as the feature.

Following prior works [16, 18, 29], we conduct speaker verification (SV) on the learned content and speaker representations and report the equal error rate (EER) as a metric of the disentanglement. Intuitively, a high EER produced by $\mathbf{z_{c}}$ and a low EER yielded by the $\mathbf{z_{s}}$ can denote a good disentanglement of content and speaker representations. To compute the EER, we randomly select 4 utterances from each speaker in the test set as the enrolled samples, which are used to compute the speaker embedding (by averaging their $\mathbf{z_{c}}$’s or $\mathbf{z_{c}}$’s). The remaining utterances of the speaker are used as the positive trials while all other speakers’ utterances are negative trials. Cosine similarity is computed as the score. The EERs are evaluated separately for English and Mandarin test sets.

As discussed in Section 3.2, the weight parameters $\beta_{c}$ and $\beta_{s}$ restrict the respective amounts of information contained in $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$, and thus are very important for disentanglement. We show EERs with respect to $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ for both English (EN) and Mandarin (CN) test set in Table 1, where $\beta_{c}$ varies among $\{10^{-3},10^{-2}\}$ and $\beta_{s}$ is taken from $\{10^{-5},10^{-4},10^{-3}\}$.

We can first observe the effect of decreasing the absolute values of both $\beta_{c}$ and $\beta_{s}$ by comparing the results yielded by cases $(\beta_{c},\beta_{s})=(10^{-2},10^{-4})$ and $(\beta_{s},\beta_{c})=(10^{-3},10^{-5})$. While the ratios between $\beta_{c}$ and $\beta_{s}$ for both cases are 100, they produce quite different representations in terms of disentanglement. As we can observe from results of English test set, the EERs computed using $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ are $0.369$ and $0.115$ respectively for the former case, while those values become $0.198$ and $0.069$ for the latter case. While the former case shows more desirable disentanglement, the second case yields a $\mathbf{z_{c}}$ with too much speaker information. This is because with restrictions that are too loose on both $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ when setting $(\beta_{s},\beta_{c})=(10^{-3},10^{-5})$, it cannot be ensured that the information captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ are complementary. Thus, different from [18] which claims that a proper ratio $\frac{\beta_{c}}{\beta_{s}}$ can induce the desired disentanglement, we argue that it is also important to first set proper absolute values for $\beta_{c}$ and $\beta_{s}$, which restricts $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ to be more complementary.

Furthermore, setting proper relative values of $\beta_{c}$ and $\beta_{s}$ is also important for disentanglement. As can be observed from Table 1, increasing the value of $\beta_{c}$ causes significant increases of EERs for both EN and CN content representations and all $\beta_{s}$ values, denoting the reduction of the speaker information captured by $\mathbf{z_{c}}$. On the other hand, the increase of $\beta_{s}$ puts more penalization on the speaker representation and thus allows more speaker information to leak into $\mathbf{z_{c}}$, which is indicated by the overall declining trend of EERs for $\mathbf{z_{c}}$ and the opposite trend for $\mathbf{z_{s}}$ horizontally. Meanwhile, larger $\beta_{c}$ ($10^{-2}$) produces a stabler speaker-independent $\mathbf{z_{c}}$ as the changes in EERs caused by the increasing $\beta_{s}$ are limited.

We conduct subjective evaluations on both intra-lingual and cross-lingual VC. We select 4 Mandarin speakers and 4 English speakers, both consisting of 2 male speakers and 2 female speakers. One utterance of each speaker is randomly chosen as the source and also the reference speech. All utterances are converted to all other speakers, thus in total we obtain 56 converted samples, while 24 of them are intra-lingual (EN2EN and CN2CN) and 32 are cross-lingual cases (EN2CN and CN2EN). Twelve subjects are asked to score these converted samples based on their naturalness and speaker similarity. 5-scale mean opinion score (MOS) is applied for both speech naturalness and speaker similarity.

We adopt two competitive unsupervised one-shot VC models AdIN-VC [9] and VQMIVC [8] as our baselines, while Hifi-GAN [30] is used as the vocoder. We denote the proposed model as $\beta$-VAEVC. We train the two baseline models as well as the vocoder on the same acoustic features and training set as ours. The copy-synthesized speech of the source speech is included in the naturalness evaluation, while those of randomly selected speech from the target speakers are scored in the speaker similarity evaluation. Some samples are available on https://beta-vaevc.github.io.

We first show the EERs obtained by content and speaker representations of three compared models in Table 2. For $\beta$-VAEVC the $\beta_{c}$ and $\beta_{s}$ are set as $3\times 10^{-3}$ and $10^{-7}$ respectively. However, as shown in Table 2, this setting does not yield the best disentanglement results in terms of EERs. As $\mathbf{z_{c}}$ of $\beta$-VAEVC seems to contain more speaker information than two baselines. Though we can achieve more speaker-independent $\mathbf{z_{c}}$ for $\beta$-VAEVC via further increasing $\beta_{c}$, we find it can also decrease the generation quality. Thus, we choose the parameter setting that works better on the validation set in terms of speech generation quality.

The naturalness and similarity MOS results are shown in Table 3 and Table 4, respectively. We can observe that the proposed method achieves overall both better naturalness and speaker similarity than the two baselines. While AdIN-VC works comparably well for English intra-lingual conversion, its performance is much worse when it comes to Mandarin intra-lingual conversion and two directions of cross-lingual conversion. While VQMIVC archives overall good conversion naturalness, the speaker similarity performance is not as satisfying as that for speech naturalness, especially for cross-lingual cases.

In addition, we utilize the transcription error obtained by open-source pre-trained ASR models [31] as indicators of the conversion intelligibility. We conduct both intra-lingual and cross-lingual VC on the whole test set, that is, all utterances are converted to all the other speakers. There are in total 42,340 converted utterances for English intra-lingual VC, 77,920 for Mandarin intra-lingual VC, 67,744 for English-to-Mandarin conversion and 48,700 for Mandarin-to-English conversion. Pre-trained English and Mandarin ASR models are used to transcribe the corresponding utterances, then the word error rate (WER) and character error rate (CER) are computed respectively for English and Mandarin. The recognition results on re-synthesized utterances of all samples in the test set by Hifi-GAN are also included as reference. The results are shown in Table 5. The proposed model surpasses baseline models by large margins for two intra-lingual VC cases and the Mandarin-to-English conversion case, and shows comparable performance with other two methods for the English-to-Mandarin conversion. Besides, we notice that VQMIVC achieves better WER / CER than AdIN-VC for cross-lingual conversion, while the latter model is better for intra-lingual cases.

Though all compared models can realize cross-lingual VC, we can see that there are gaps between the performance of intra-lingual VC and cross-lingual VC for all metrics, as shown in Table 3, 4 and 5. This is reasonable since the inputs to the content encoder and speaker encoder are from the different domain for cross-lingual VC cases, which is not the case during training. Besides, the difference in the recording conditions of two corpora VCTK and AISHELL-3 can make the train-inference mismatch issue even worse. We will tackle this problem in our future work.