Content-Dependent Fine-Grained Speaker Embedding for Zero-Shot Speaker Adaptation in Text-to-Speech Synthesis

To model and transfer personal pronunciation characteristics, we first extract the corresponding local content embeddings and speaker embeddings from the reference mel-spectrograms.

As shown in Fig.1, the reference mel-spectrograms are first passed to a pre-net which consists of two 1-D convolutional layers containing 512 filters with shape $5\times 1$. The frame-level features from the pre-net are encoded by a mel content encoder composed of 4 feed-forward Transformer blocks to get frame-level content embeddings. For constraining the mel content encoder to encode content information, a phoneme classifier is introduced to predict the frame-level phoneme labels from the outputs of the mel content encoder. Then the frame-level content embeddings are passed to the downsample content encoder, meanwhile, the frame-level features are passed to the downsample speaker encoder. Both two downsample encoders are made up of 4 1-D convolutional layers and a 256-dim fully-connected output layer. The 4 convolutions contain 128, 256, 512, 512 filters with shape $3\times 1$ respectively, each followed by an average pooling layer with kernel size 2. That is, the temporal resolution is reduced 16 times, which can be regarded as quasi-phoneme level inspired by [19]. All the convolutional layers are followed by ReLU activation and batch normalization [22], while the output layer is followed by Tanh activation. To introduce speaker information, an average pooling layer is used to summarize the local speaker embeddings across time followed by a speaker classifier. Local content embeddings and local speaker embeddings are obtained from two downsample encoders respectively. Due to the same local segment input and the same downsampling scale encoding structure, they are exactly one-to-one correspondence in the speech. Therefore, each local speaker embedding can be considered as carrying fine-grained speaker characteristics related to phoneme content.

The speaker characteristics of a person include not only global timbre information but also some local pronunciation variations. These local variations contain different pronunciation patterns affected by one’s pronunciation habit, which work on a small scale like phoneme level. For example, there is a difference between a person’s pronunciation of “/æ/” and his pronunciation of “/i:/”. Thus, more accurate fine-grained speaker embedding shall be applied to a certain phoneme in text.

The content of the reference speech and input text is different in phoneme permutation and combination during synthesis. To make better use of local speaker embeddings extracted from reference speech, a content-dependent reference attention module is introduced to obtain the appropriate fine-grained speaker embeddings inspired by [19, 21].

We adopt scaled dot-product attention [23] as the reference attention module. The current phoneme encoder output is used as the query, while all the local content embeddings from reference speech are used as keys. The relevance between them is used to guide the selection of fine-grained speaker embeddings, which means the local speaker embeddings are values. In this manner, the fine-grained speaker embedding sequence generated by the reference attention has the same length as the phoneme embedding sequence.

The fine-grained characteristics of a speaker are very diverse, for example, the style and pronunciation details are not exactly the same even if one speaker says a sentence twice. Regarding this, the reference and target utterance had better be consistent in the training stage so that the model can learn correct content relevance and transfer meaningful fine-grained speaker embeddings. However, the reference attention module easily learns the temporal alignment between reference speech and input text in the previous trial [19]. Such fine-grained embedding sequence is more about modeling prosodic trends in time, which is however unsuitable for the input text whose content is different from the reference speech, and will result in strange prosody or poor intelligibility of the synthesized speech in this situation.

To make the model focus more on content relevance rather than simple temporal alignment between reference speech and input text, we introduce some preprocessing operations in the training stage. The mel-spectrogram of a reference utterance is first labeled with frame-level phoneme tags by forced alignment [24] and divided into fragments by phoneme boundaries. These fragments corresponding to phonemes are randomly shuffled and concatenated to form a new reference mel-spectrogram. In this way, the temporal consistency between the paired text and the reference speech is eliminated, and the basic content information of the speech also can be preserved. The shuffled frame-level phoneme tag sequence is sent to the phoneme classifier as the ground truth for calculating the cross-entropy phoneme classification loss that is added to the total loss.

All the models are trained on AISHELL-3 [25], which is an open-source multi-speaker Mandarin speech corpus containing 85 hours of recordings spoken by 218 native Chinese speakers. To evaluate the performance on unseen speakers, 8 speakers (4 male and 4 female) are selected as the test set. For the remaining 210 speakers, 95% of the utterances are used for training and 5% are used for validation. Waveforms are transformed to 80-dim mel-sepctrograms with 22.05kHz sampling rate. The frame size is 1024 and the hop size is 256. Raw text is converted to phoneme sequence composed of Pinyin initials and tonal-finals by a Chinese grapheme-to-phoneme conversion toolkit¹¹1Pypinyin: https://pypi.org/project/pypinyin. We train all the models for 250K iterations with a batch size of 16 on an NVIDIA P40 GPU. The Adam optimizer is adopted with $\beta_{1}=0.9$, $\beta_{2}=0.98$, $\epsilon=10^{-9}$. Warm-up strategy is employed before 4000 iterations. A well-trained HiFi-GAN [26] is used as the neural vocoder to generate waveforms.

We compare the proposed content-dependent fine-grained speaker embedding (CDFSE) approach with two typical fixed-length speaker embedding methods and a variable-length embedding method based on Attentron. These three methods are also implemented based on FastSpeech 2²²2Implemented based on: https://github.com/ming024/FastSpeech2.

GSE Global speaker embedding (GSE) uses a bank of base vectors and multi-head attention to represent the global speaker embedding from reference speech unsupervisedly. The implementation is consistent with the original method [14]. We also try more base vectors but observe no difference in performance.

CLS The speaker classifier (CLS) is a kind of supervised speaker encoder based on multi-task learning or transfer learning [7, 11, 12]. To compare with the proposed, we use the same speaker encoder as shown in Fig.1. The utterance-level speaker embedding generated by the average pooling layer is replicated to phoneme level and added to the phoneme encoder outputs.

Attentron* Attentron proposes an attention-based variable-length embedding method to leverage features near to raw reference speech for better generalization. It is originally implemented based on Tacotron 2 [1], consisted of a coarse-grained encoder and a fine-grained encoder with attention mechanism, which extracts both utterance-level and frame-level embeddings from reference speech. To compare with the proposed, we use Attentron (1-1) mode (details in [21]) and adapt its major implementation to FastSpeech 2 framework, named as Attentron*. The several adjustments are to keep the main structure of the acoustic model unchanged, including: i) The utterance-level embedding from the coarse-grained encoder is added to encoder output rather than concatenated; ii) The outputs of FastSpeech 2 decoder (before the mel linear layer) are directly used as the queries for attention mechanism to generate frame-level embeddings instead of the autoregressive way in Attentron.

By following [21], we employ two mean opinion score (MOS) tests to evaluate the naturalness and speaker similarity of the synthesized speeches³³3Speech samples, detailed figures and open-sourced codes are available at: https://thuhcsi.github.io/interspeech2022-cdfse-tts. 8 unseen speakers from the test set and 6 seen speakers randomly selected from the training set are used as reference voices. The text sentences are from the test set, varying in length and content. For each speaker, only one utterance is used as the reference speech to guide speech synthesis. 15 native Chinese speakers serves as subjects to take part in the evaluation and rate on a scale from 1 to 5 with 1 point interval.

As shown in Table 1, the results demonstrate our proposed CDFSE method outperforms all three baselines in terms of speaker similarity. CDFSE gets the best SMOS of $4.11$ for seen speakers and $3.51$ for unseen speakers, and Attentron* performance is relatively better than the two others. For unseen speakers, the improvement on SMOS of CDFSE is more significant by a gap of over $0.2$, indicating that personal pronunciation characteristics are very helpful to improve the speaker similarity from the sense of listening for zero-shot speaker adaptation. The MOS results on naturalness of these methods are generally comparable. CDFSE has a slight decrease in MOS compared with Attentron*, but is still acceptable in terms of naturalness and intelligibility. This is understandable since frame-level features from reference speech are applied to the TTS decoder output in Attentron*, which helps improve quality and naturalness.

To investigate the impact of local speaker embeddings with different granularity, we adjust the kernel size of the average pooling layer in the downsample encoders. In Table 2, the number after ‘CDFSE-’ represents the overall downsampling times in temporal compared with the reference mel-spectrogram. All the models are trained with the same settings as mentioned above. We find that some synthesized speeches are poor in intelligibility, which will affect the subjective judgment of similarity. Therefore, we employ objective evaluations rather than subjective MOS in this part. To evaluate the intelligibility of synthesized speech, the mispronunciation cases (excluding accents) are marked by listeners and counted. To evaluate speaker similarity, we employ a speaker verification system [27] to extract the utterance-level speaker vector and calculate the cosine similarity between synthesized speech and ground truth.

Table 2 shows the performance comparison among different granularity models, and the results of three baselines are also presented for reference. It is observed there exist several mispronunciation cases in all models, which are more likely caused by FastSpeech 2 itself and the training data. CDFSE-16 gets the lowest mispronunciation rate and the highest speaker vector cosine similarity. With the decrease of downsampling times, the mispronunciation rate of synthesized speech increases significantly. That is, the granularity of local speaker embeddings is crucial to the intelligibility and stability of synthesized speech, rather than finer-grained speaker embeddings being better. This can explain why we use the downsample encoder to extract quasi-phoneme level embedding as stated in 2.1.

Apart from that, we have also employed some ablation studies to demonstrate the effectiveness of each module. We first remove the explicit supervision of local speaker embedding by excluding speaker classification loss, and this model is denoted as ‘CDFSE-16 w/o SC’ shown in Table 2. The decline in both two evaluation metrics indicates that introducing speaker information can improve speaker similarity and synthesis stability. We also remove the explicit supervision of local content embedding by excluding phoneme classification loss, and find it will cause the reference attention module fail.

To clearly present content relevance between reference speech and input text, we plot an alignment example from the reference attention module in CDFSE. As shown in Fig.2, when the phoneme in the input text exists in the reference speech, the reference attention tends to focus mainly on the corresponding segment, like “sh”; when the phoneme does not exist, the model will focus on similar segments, like “er2” in text similar to “ai2” and “a2” in reference speech. For comparison, another case with specific-designed input text is given, presenting alignments from CDFSE and the attention mechanism in Attentron*. As shown in Fig.3, the reference attention module in CDFSE successfully learns the right content alignment (especially, the correct phoneme order within Chinese characters is maintained) between reference speech and text, while Attentron* does not show this ability.

We further visualize the fine-grained speaker embeddings by 2D t-SNE [28]. As shown in Fig.4, the fine-grained speaker embeddings of the same speaker tend to group together while exhibiting certain content dependent diversities that capture the local pronunciation variations as stated in 2.2.