Glow-WaveGAN 2: High-quality Zero-shot Text-to-speech Synthesis and Any-to-any Voice Conversion

A robust vocoder for unseen speakers is critical for zero-shot speech generation. To achieve this goal, we build a variational auto-encoder (VAE) [26] based universal WaveGAN model to conduct speech reconstruction, which has shown its strong ability to reconstruct unseen speakers in our previous work [24].

In the universal WaveGAN, the encoder aims at extracting the distribution of acoustic representation $z\sim p(z|w)$ from the waveform $w$, where $z$ also includes the speaker characteristics through a pitch predictor. Meanwhile, the decoder tends to reconstruct the speech $\hat{w}\sim p(w|z)$ from sampled $z$. To ensure the quality of reconstructed speech, we use adversarial training to model the $p(w)$ of the waveform. The training objective of the WaveGAN keeps the same as [24].

The encoder of WaveGAN can be treated as a robust feature extractor, and the extracted speech representation $z$ contains the speaker information. The decoder plays the role of vocoder to reconstruct high-quality speech waveform for both seen and unseen speakers. In this way, the proposed method guarantees the speech quality of zero-shot speech reconstruction.

With the extracted latent speech representation, the skeleton of our acoustic model follows Glow-TTS [22] to model the conditional distribution $p(z|t,s)$ from texts $t$, where $s$ is the conditional constraint to provide speaker identity. In detail, the text encoder transforms the text $t$ into a linguistic prior distribution $p(c|t)$. Considering the frame rate difference between the text and the latent $z$, monotonic alignment search (MAS) is adopted to align the prior $c$ and acoustic feature $z$. With the speaker embedding fed into the flow-based decoder $f$, the target $z$ from WaveGAN is transformed into the aligned prior distribution $p(c)$ through forward pass of $f$ during training. Thus the log-likelihood of the target distribution $p(z)$ can be obtained through the flow-based decoder:

Note that we specifically sample $z$ from the same $p(z)$ of the WaveGAN encoder in each training step, which mitigates the mismatch problem between the acoustic model and the vocoder to achieve high synthesis quality without fine-tuning. During inference, the predicted $\hat{z}$ is generated by the reverse decoding process $f^{-1}$ with the speaker condition from the text prior. Since the text encoder is speaker-independent, it’s easy to conduct voice conversion through the forward $f$ and reverse $f^{-1}$ flow decoding with different speaker constraints [22], as shown in Figure 1 (d).

To achieve the zero-shot TTS and any-to-any VC, a key problem is how to model the speaker characteristics, especially for unseen speakers. In this paper, we investigate two alternative ways, a pre-trained encoder and a jointly-trained encoder, to build the speaker space in the acoustic model to produce speaker-dependent $z$ for unseen speakers.

A straightforward way to represent speakers is to utilize a pre-trained speaker encoder based on the speaker embedding extraction in speaker verification [11, 27, 28, 29], through which we can obtain the speaker representation $s$ by only one utterance of a few seconds to generate speech of unseen speakers. Besides, a jointly-trained encoder based on speaker classification is also adopted in our work to learn $s$ from $p(z)$ within the Glow-TTS model. With the jointly-trained speaker module, we optimize the acoustic model with an extra cross entropy objective. Specifically, we sample two vectors $\{z_{1},z_{2}\}$ from the same distribution $p(z)$, where $z_{1}$ and $z_{2}$ are used to train the acoustic model and speaker classification module respectively.

As shown in Figure 1 (c) and (d), given an utterance of any target speaker during inference, the speaker identity $s_{tgt}$ can be extracted from the speaker encoder, which can be treated as the speaker constraint in our model. At inference time of zero-shot TTS, $s_{tgt}$ is utilized as the condition for the acoustic model to generate the sampled $z$ containing the target speaker information. At inference time of zero-shot VC, the source speaker $s_{src}$ is conditioned for the flow module to generate the speaker-independent linguistic prior distribution$p(c)$, which eliminates the source speaker information. Then $p(c)$ is fed into the reversed flow conditioned on the target speaker $s_{tgt}$ to generate the target sampled $z_{tgt}$. In this way, zero-shot TTS and VC can be achieved for generating speech of any speaker.

We conduct experiments on two different datasets named VCTK [30] and LibriTTS [31] at 24 kHz. The VCTK corpus contains 109 English speakers, where we reserved 3 male and 3 female speakers as unseen speakers. As for the LibriTTS corpus, we use 1,151 speakers from the train-clean-360 and train-clean-100 subsets to train different systems, while the 39 speakers from the test-clean subset are treated as unseen speakers. The jointly-trained speaker encoder consists of two 1d-convolution layers with kernel size 5, followed by the layer normalization and dropout. Finally, another convolution layer with 128 channels is adopted to extract the speaker embedding through mean pooling. As for the pre-trained speaker encoder, we utilize the voice encoder in the Resemblyzer tool [27, 29] to extract a 256-dimensional speaker representation.

To evaluate the capability of the proposed Glow-WaveGAN 2 in zero-shot speech generation ¹¹1Audio samples can be found at https://leiyi420.github.io/glow-wavegan2/, we set up two state-of-the-art models as baselines: (1) GlowTTS-HiFiGAN, which contains a multi-speaker flow-based acoustic model and the GAN-based vocoder to reconstruct mel-spectrogram; and (2) VITS [23], which conducts multi-speaker synthesis in an end-to-end manner. Both of them utilize speaker ID as the speaker condition of the acoustic model. As for the Glow-WaveGAN family, we build different models for evaluation: (1) the basic Glow-WaveGAN with explicit speaker labels like the above baselines, (2) the Glow-WaveGAN2-joint with jointly-trained speaker encoder, and (3) the Glow-WaveGAN2-pre with pre-trained speaker encoder.

We first evaluate the speaker similarity of the synthesized speech for TTS and VC between different models in the objective manner, which is critical to the performance of zero-shot speech generation. Following [10, 32], we treat the Speaker Encoder Cosine Similarity (SECS) between the speaker embeddings extracted from synthesized and ground-truth audios as an objective measure, where SECS scores range from $0$ to $1$ and a higher score means higher speaker similarity.

Table 1 shows the SECS results of different models. Note that the two baselines and the Glow-WaveGAN system adopt the explicit speaker labels to model the speaker identity, so they can only conduct speech synthesis and voice conversion for seen speakers. For the results of both TTS and VC on seen speakers, we find the VITS and the Glow-WaveGAN models achieve higher similarity than the GlowTTS-HiFiGAN model, especially in the LibriTTS dataset. And the Glow-WaveGAN family is slightly better than VITS for seen speakers in general.

For the zero-shot TTS of proposed models, the results show that there exists a gap in speaker similarity between seen and unseen speakers for the Glow-WaveGAN2-joint model. We believe that this is mainly because the jointly-trained speaker module only learns a weak speaker space on the limited training speakers since we can find such a gap is smaller in LibriTTS than that of VCTK, where LibriTTS has more speakers. Based on this assumption, we also evaluate the zero-shot TTS of the system Glow-WaveGAN-pre with the pre-trained speaker encoder, where the gap between seen and unseen speakers becomes obviously alleviated.

As for the any-to-any voice conversion task, we find that Glow-WaveGAN2-pre outperforms Glow-WaveGAN2-joint in general. The results of LibriTTS show that the Glow-WaveGAN2-pre model achieves similar SECS scores on the unseen target speakers to the seen targets. And it is worth noticing that the SECS score declines when the source speaker is seen, from which we argue that the model may still maintain the identity of source training speakers in the VC procedure. While for the VCTK corpus, the proposed methods achieve similar scores for the same target speaker, whether the source speaker is seen or unseen.

To further investigate the speaker similarity of the synthetic speech by the proposed methods individually trained with different corpora, the speaker embeddings of generated speech are visualized through t-SNE [33] as shown in Figure 2. For the seen (green and red dots) and unseen (blue and yellow dots) speakers on both TTS and VC tasks, the generated audios and corresponding reference speech of each speaker can form a distinct cluster in our two proposed methods, which shows that the proposed methods can effectively model and control the speaker identities.

Compared with the indivisible clusters of unseen speakers formed from ground-truth and synthesized speech in the Glow-WaveGAN2-pre model with the pre-trained speaker encoder, there still exists boundaries of generated and ground-truth audios for some speakers in the Glow-WaveGAN2-joint model, especially for the smaller training corpus, i.e. VCTK, where the speaker’s name begins with “p” in Figure 2.

To evaluate speech naturalness and quality of different systems, we conduct Mean Opinion Score (MOS) tests, as shown in Table 2. In each MOS test, there are 20 listeners rating 20 randomly chosen utterances for each model. In general, the MOS scores of VCTK are better than that of LibriTTS due to its better recording quality.

The MOS results of both TTS and VC on seen speakers demonstrate that the Glow-WaveGAN family and the VITS model have obviously higher scores than the GlowTTS-HiFiGAN model, which comes from the mismatch problem mentioned in Section 1. As for the unseen speakers in TTS, both the two proposed models can achieve similar MOS scores to that of the seen speakers, which indicates the effectiveness of the proposed models for zero-shot TTS. In the VC scenarios, the MOS results indicate that there is no significant difference between the Glow-WaveGAN family and the VITS model on seen speakers, where they both achieve satisfactory MOS scores on the VC task. When the source or the target speakers are unseen, the MOS results show that the proposed Glow-WaveGAN 2 models can also produce high-quality converted speech with only one utterance of the target speaker. Therefore, the experimental results demonstrate that our proposed models can generate high-quality speech in both zero-shot TTS and VC.

To further evaluate the generalization ability of our proposed models in zero-shot speech generation, we also calculate the SECS scores and conduct MOS tests across datasets. Specifically, the target speaker is from another dataset different from model training in TTS or the target and source speakers are both from another dataset different from model training in VC. Results are summarized in Table 3. For speaker similarity, we find the model trained on the LibriTTS corpus has high SECS scores for cross-dataset evaluation since it can learn richer speaker space with more speakers. As for the subjective evaluation, it is interesting that we find the quality of generated speech from Glow-WaveGAN2-joint is better than that from Glow-WaveGAN2-pre in cross-dataset evaluation. We conjecture the reason is that the speaker constraint $s$ and the target $z$ are directly from the $p(z)$ of speech, which means that they may contain the channel information of the speech that affects the quality of generated speech in both training and inference.