crank: an open-source software for nonparallel voice conversion based on vector-quantized variational autoencoder

VC is a technique used to convert paralinguistic information such as gender, speaker individuality, and emotions beyond their physical constraints while keeping the linguistic information of a source speech [1]. One of main goals in VC research is to freely control arbitrary factors of a source voice into objective factors depending on the situation in which VC is used. However, control capabilities and the sound quality of the converted voice are usually degraded due to the insufficient modeling accuracy of speech production. If individual speakers could freely control various factors of the speech, it would open up an entirely new speech communication style.

VC research was initially started to develop a speaker individuality conversion technique enabling a source speaker to change his/her speaker individuality to that of another target speaker while preserving the linguistic content. In this technique, a statistical mapping function that converts acoustic features of the source speech into those of the target speech is preliminarily trained using a parallel dataset consisting of source and target speakers’ utterances with the same linguistic contents. To improve the modeling accuracy of the statistical mapping function, several techniques such as the use of the Gaussian mixture model (GMM) [2] and deep neural networks [3] have been proposed.

End-to-end VC [4, 5] is one of the most powerful mapping techniques using a parallel dataset. Unlike conventional statistical mapping techniques, it is not necessary to explicitly calculate alignment functions between source and target utterances. That is, the estimation process of the alignment functions is implicitly included in the model training based on sequence-to-sequence networks and their attention mechanisms. These techniques usually yield considerable improvements of conversion performance compared with the conventional methods using explicit alignment functions. Moreover, it is also possible to convert not only the voice timbre but also prosodic information. On the other hand, it is not straightforward to train the end-to-end mapping function only using a small number of training utterances. Therefore, collecting many parallel utterances usually becomes a burden for users to develop end-to-end VC systems.

To ease the burden of collecting parallel utterances, nonparallel VC has been developed. There are two major nonparallel VC techniques, namely phonetic posteriorgram (PPG)-based VC methods [6, 5] and autoencoder-based VC methods including those using the variational autoencoder (VAE) [7, 8, 9, 10], vector-quantized VAE (VQVAE) [11, 12, 13, 14, 15], and generative adversarial network (GAN) [16, 17, 18, 19]. For the PPG-based VC method, the PPG vector is first estimated using a preliminarily trained automatic speech recognition (ASR) system. Then, the conversion function is trained using the PPG vector and acoustic features of the target speech. The PPG-based methods achieve relatively higher performance than the autoencoder-based VC methods owing to the speaker-independent linguistic feature of the PPG. However, it is necessary to prepare many training utterances, including contextual information, to build the ASR system for extracting a convincing PPG vector. On the other hand, the autoencoder-based VC methods do not rely on any supervised label such as context labels or parallel utterances, excluding speaker labels. Therefore, the autoencoder-based VC methods are straightforward for building the VC systems compared with the VC methods using parallel utterances and PPG-based VC methods.

In this paper, we introduce an open-source nonparallel VC software based on VQVAE named crank. In addition to the VQVAE-based VC method, we have implemented several components such as WaveNet-like [20] encoder/decoder networks, the hierarchical architecture, the cyclic architecture, and generative adversarial networks. Using crank, One may possible to easily 1) reproduce the VQVAE-based nonparallel VC method using preliminarily stored recipes such as Voice Conversion Challenge (VCC) 2018 and VCC 2020, and 2) develop a nonparallel VC system using one’s own speech corpus. In this paper, we describe 1) technical details and usage, 2) brief comparisons with VCC baseline systems, and 3) experimental results of objective measures calculated using the VCC 2018 dataset.

VQVAE-based voice conversion takes training and conversion phases.

In the training phase, the original feature vector $x$ is modeled by the VQVAE consisting of encoder/decoder networks based on the reconstructed loss. The encoder network encodes the original feature vector into the latent vector $h$. The latent vector is quantized into the discrete latent symbol $q$, which minimize the distance between the latent vector $h$ and the codebook $e$.

$$\mbox{\boldmath$q$}=\mbox{\boldmath$e$}_{k}\;\;\text{where}\;\;k=\mathop{\rm arg~{}min}\limits_{j}||\mbox{\boldmath$h$}-\mbox{\boldmath$e$}_{j}||_{2}.$$

(1)

Then, the decoder network generates the reconstructed feature vector $\hat{x}$ conditioned on the discrete latent symbol $q$ and the auxiliary features $\mbox{\boldmath$c$}_{org}$ such as the speaker code and $F_{0}$ of the original speech sample. The objective function of VQVAE is as follows:

$$\mathcal{L}_{obj}=||\mbox{\boldmath$x$}-\mbox{\boldmath$\hat{x}$}||^{2}_{2}+||\text{sg}[\mbox{\boldmath$h$}]-\mbox{\boldmath$e$}||^{2}_{2}+\beta||\mbox{\boldmath$h$}-\text{sg}[\mbox{\boldmath$e$}]||^{2}_{2},$$

(2)

where $||\mbox{\boldmath$x$}-\mbox{\boldmath$\hat{x}$}||^{2}_{2}$, $||\text{sg}[\mbox{\boldmath$h$}]-\mbox{\boldmath$e$}||^{2}_{2}$, and $||\mbox{\boldmath$h$}-\text{sg}[\mbox{\boldmath$e$}]||^{2}_{2}$ are the reconstruction loss, codebook loss, and commitment loss, respectively. $\beta$ and $\text{sg}[\cdot]$ indicate the hyperparameter for the commitment loss and the stop gradient function, respectively. Because there is an $\mathop{\rm arg~{}min}\limits$ function to find the discrete latent symbol, it is not straightforward to optimize this network. To avoid this problem, VQVAE utilizes a straight-through estimator [21] to pass through gradients from the decoder to the encoder via the vector-quantizer.

In the conversion phase, the original feature vector $\mbox{\boldmath$x$}^{\prime}$ is first encoded into the latent vector $\mbox{\boldmath$h$}^{\prime}$ to find the discrete latent symbol $\mbox{\boldmath$q$}^{\prime}$ on the basis of the trained encoder network and codebook. Then, the target auxiliary features $\mbox{\boldmath$c$}_{tar}$ with the codebook of predicted discrete latent symbols $\mbox{\boldmath$q$}^{\prime}$ are fed into the decoder network to generate a converted feature vector $\hat{y}^{\prime}$.

crank is an open-source software that implements nonparallel VC frameworks. The license of crank is linked to the MIT license. The implementation of crank has been continued on a GitHub repository¹¹1https://github.com/k2kobayashi/crank. In this section, we introduce the basic structure of the crank recipe and representative components. Table 1 shows the fundamental differences in features between crank and successive VCC baseline systems.

As brief comparisons between representative functions developed using crank, we evaluated objective measures calculated using VCC 2018 recipes. The sampling rate was set to 22050 Hz. The number of training speakers was 12, and each speaker spoke 80 utterances. We used 75 utterances for training and the remaining five utterances for development. We used the other 35 evaluation utterances in each speaker. The evaluation was performed under the speaker-closed condition (i.e., training speakers were used for the evaluation as well.).

An 80-dimensional mel-filterbank was used as the feature vector. Continuous $F_{0}$, an unvoiced/voice decision symbol, and a speaker code were used as the auxiliary features for the decoder. The ParalellWaveGAN vocoder trained using the same dataset was used as a neural vocoder. We compared mel-cepstrum distortion and predicted naturalness on the basis of MOSNet in this evaluation. The values were averaged among all-speaker pairs. We used a 35-dimensional mel-cepstrum to calculate the distortion. The other settings and resulting voices were described on the website⁴⁴4https://k2kobayashi.github.io/crankSamples/.

The following techniques were compared in this evaluation.

Baseline VQVAE

Three-stacked hierarchical VQVAE

CycleVQVAE

Baseline VQVAE with cyclic architecture

VQVAEGAN

Baseline VQVAE with GAN

CycleVQVAEGAN

Baseline VQVAE with cyclic architecture and GAN

CycleVQVAEGAN w/ STFTLoss

Baseline VQVAE with cyclic architecture and GAN with STFT loss

The STFT loss means that the network utilizes not only L1 loss but also STFT loss for the reconstruction loss.

Table 2 shows the experimental results of the mel-cepstrum distortion and MOSNet predictions. Compared with the baseline VQVAE, the CycleVQVAE method achieves higher performance in terms of Mel-CD and MOSNet. Moreover, integrating GAN-based training, we can see that the CycleVQVAE w/ STFT method yields the highest performance among methods shown in Table 2. On the other hand, the VQVAEGAN method has a lower performance than the baseline VQVAE method. It is considered that it is not straightforward to optimize the VQVAE decoder network based on the GAN framework, and a cyclic architecture may maintain the stability of the training similarly to CycleGAN-VC [19].

In this paper, we introduced an open-source nonparallel VC software named crank. The main objective of developing crank is to build a nonparallel VC system with limited constraints for collecting the speech corpus. In addition to the vector-quantized variational autoencoder-based VC method, several representative methods such as these using the hierarchical architecture, cyclic architecture, generative adversarial network, and speaker adversarial training have been implemented in crank. Moreover, it also supports the ParallelWaveGAN vocoder to decode a converted mel-filterbank and calculate objective measures such as mel-cepstrum distortion and pseudo mean opinion score on the basis of MOSNet. For our future work, we will continue to develop methods to realize high-quality, easy-to-use nonparallel VC software.

This work was partly supported by JSPS KAKENHI Grant-in-Aid for JSPS Research Fellow Number 19K20295, and JST, CREST Grant Number JPMJCR19A3.