Voice Conversion by Cascading Automatic Speech Recognition and Text-to-Speech Synthesis with Prosody Transfer

The ASR model is used to decode the text from source utterances, which determines the content of the converted utterances. By definition of voice conversion task, linguistic content should not be changed during this process. Therefore, it’s crucial that the ASR model to recognize the text information as much accuracy as possible. For the ASR model, we compared two methods, including the commercial iFLYTEK ASR engine and an open-source ASR model based on ESPnet [26]. And results were reported in our experiments section.

We extended Transformer TTS to the multi-speaker condition by using speaker x-vectors [27] as additional inputs to its decoder. The x-vectors are projected by linear transformation then added to each frame of encoder outputs. Another difference with the original Transformer TTS paper is that we don’t use the encoder Prenet, since no explicit improvement was found in our preliminary experiments. The amount of target training data is usually small in voice conversion task. Thus the model often suffers from over-fitting effect if no external dataset is exploited. Therefore, in order to improving the generalization capacity of the model, it is first pre-trained on a multi-speaker dataset, and then adapted to the target speaker by finetuning.

We assume that the speech signal contains not only the text and speaker identity information, but also the prosody information. Therefore, a prosody code is further provided to the synthesis model to describe the prosodic ingredients of the speech. In our method, a prosody encoder is employed to extract a global prosody code for each utterance as shown in Figure 3. The Transformer TTS model is further conditioned on the prosody code. Specifically, the code is projected by linear transformation then added to each frame of encoder outputs. During conversion, the source prosody code is transferred and used to synthesize the converted utterance to make it more natural sounded. However, because that the TTS model is pre-trained on the multi-speaker dataset, the desired prosody information can be easily entangled with the speaker identity information. Therefore, we further adopted a speaker adversarial loss to regularize the extracted code to be speaker-independent. To this end, a speaker classifier is adopted to predict the speaker identity from the prosody code. While the prosody encoder is optimized with the opposite objective to lower the classifier’s accuracy of prediction.

In our implementation, a gradient reversal layer is inserted between the prosody code and the classifier. The speaker classifier is updated four times for each update of the prosody encoder. It is critical to keep the classifier close to the optimal so that it can backpropagate useful gradients rather than noises. Our prosody encoder is only updated during the multi-speaker pre-training stage while it’s frozen during fine-tuning on the target speaker.

Dataset from the mono-lingual task of Voice Conversion Challenge 2020 was adopted, which contained 4 source English speakers (SEF1, SEF2, SEM1 and SEM2) and 4 target English speakers (TEF1, TEF2, TEM1 and TEM2). Each speaker contained 70 utterances for training, 20 of which were parallel for source and target speakers and the remaining were non-parallel. Our model only relies on the target utterances for model adaptation. Therefore, the non-parallel part of source utterances were used as our internal evaluation set. For evaluation in Voice Conversion Challenge 2020, 25 test utterances were further provided for each source speaker. And they were converted to each target speaker. LibriTTS corpus¹¹1http://www.openslr.org/60/ was adopted for pretraining our model. We used about 460 hours training data from 1150 speakers.

For ASR model, we compared the iFLYTEK ASR engine and a open-source ASR model²²2https://drive.google.com/file/d/1BtQvAnsFvVi-dp_qsaFP7n4A_5cwnlR6/view?usp=drive_open based on ESPnet [26] and the results on our internal evaluation set were presented in Table 1. We observed that the iFLYTEK ASR engine achieved better performance than the ESPnet based one. Therefore, the former was adopted in our experiments.

The Transformer TTS model contained 6 blocks for both encoder and decoder. The number of attention head was 4 and the layer width was 1536. The architecture of the prosody encoder followed the reference encoder of GST-Tacotron [28], producing a 128-dimensional prosody code for each utterance. The speaker classifier was a $3\times 512$ DNN. 512-dimensional x-vectors were extracted by Kaldi toolkit³³3https://kaldi-asr.org/. For acoustic features, 80-dimensional Mel-spectrograms were extracted with Mel filters spanning from 80 Hz to 7600 Hz. We used Adam optimizer and Noam scheduler [25] for adjusting the learning rate. The batch size was 120, which was later set as 16 during finetuning. At last, WaveNet vocoder was trained to model 24kHz-16bit waveforms for each target speaker.

We first compared our proposed ASR-TTS method without using prosody code to a baseline method, which was an improved version of N10 system in Voice Conversion Challenge 2018, denoted by VCC2018+. Compared to N10 system, this baseline adopted a new auto-regressive synthesis model. Also, a higher quality WaveNet vocoder was adopted. For the details of this model, the reader can refer to another our paper for the cross-lingual task of Voice Conversion Challenge 2020.

We conducted the subjective listening test in terms of both naturalness and similarity. 40 utterances for each method were randomly selected from our internal evaluation set. We used Amazon Mechanical Turk⁴⁴4https://www.mturk.com/ platform and at least 30 native listeners participated in each of our experiments. They were required to used headphones during the test and the stimuli were presented in random order. Table 2 summarized the results. We observed from the table that our proposed method achieved higher naturalness and similarity than the VCC2018+ baseline. And it even obtained similar results to the natural target speech.

In order to analyze the latent space of prosody code, they were extracted from four source speakers, i.e., SEF1, SEF2, SEM1 and SEM2, then projected to 2 dimension by t-SNE for visualization. The 25 test utterances for each speaker were used and the results were presented in Figure 4. We observed from this figure the prosody code from each speaker equally distributed in the same latent space, which demonstrated the speaker-indepentent property of the prosody code.

Based on the preliminary experiments, we heuristically scaled the prosody code with a factor of 1.5 during conversion. And the proposed method with and without prosody code were compared by subjective listening test. 10 sentences from internal evaluation set were randomly selected for each target speaker, resulting a total of 40 sentences for evaluation. The results were grouped by the target speaker as presented in Table 3. As we can see from the table, the proposed method with and without prosody code obtained close naturalness and similarity score. Averagely, w pc method obtained slightly better naturalness but slightly lower similarity than w/o pc method. For the target speaker TEF2, w pc method achieved slightly better results in both naturalness and similarity. It should be notice that our w/o pc method already achieved comparable results with the natural target speech, as shown in previous section. Therefore, there’s no clear advantage of using prosody transfer strategy in our experiments. To explore this strategy with dataset containing expressive utterances will be an interesting research direction in the future.

Different conversion methods were selected according to their performance on the conversion pairs in Voice Conversion Challenge 2020⁵⁵5http://www.vc-challenge.org/. Based on the experimental results in previous section, we adopted the proposed method with prosody transfer strategy when the target speaker was TEF2. For SEM1-to-TEM1 and SEM1-to-TEM2, VCC2018+ method was used since the proposed ASR-TTS achieved similar results to it on those conversion pairs. For remaining conversion pairs, we used the proposed method without prosody transfer strategy.

The subjective listening results, which were collected from 68 native English speakers, were shown in Figure 5. As we can observed from this figure, our method achieved the best performance among the participating teams. Also, it’s noticeable our method achieved similar similarity score with the natural target speech. According to further results provided by the organizer, converted utterances by our method had no statistical significant difference with the natural target speech in terms of similarity.

Unlike text, the content-related features contained the source speaker information, therefore may harm the similarity of converted speech. In order to prove this, we compared between inputting the target content features (i.e., the content features and acoustic model are matched) and the source content features (i.e., the content features and acoustic model are mismatched). And the voice conversion is mismatched condition. From Table 4, we observed that the mismatched condition underperformed the matched one, especially in term of similarity.

The same target utterances were synthesized but with the prosody code transferred from different source speakers. Mel-spectrograms were shown in Figure 6. We observed from the figure that the synthesized Mel-spectrograms were similar to each other while slightly difference appeared, indicating the effectiveness of prosody code.

However, we found the control effect of prosody code was subtle thus hard to identify it. In order to further investigate the relationship of prosody code to the prosody aspect of generated speech, we further conducted the experiments that first scale the prosody code by multiply it with a scale factor, then use it to control the synthesis process. Mel-spectrogram of an utterance converted from SEF1 to TEF1 was presented in Figure 7. From this figure, we observed that as we scaled the prosody code from 3 to -3, the generated Mel-spectrogram got more blurred, the speaker rate got faster and the pitch contour got more flatten.