ZMM-TTS: Zero-shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-supervised Discrete Speech Representations

The concept of zero-shot multispeaker TTS was initially introduced in [33] and the main idea is to employ embeddings from an external speaker encoder as a reference signal. Subsequent research by [14] aimed to reduce the quality gap between seen and unseen speakers by utilizing more informative embeddings. [34] further explored the utilization of attentive speaker embeddings for the purpose of encoding general speaking styles, while [35] proposed a speaker-conditional architecture and investigated a flow-based decoder capable of generating unseen speaker voices. Despite these efforts, the task of zero-shot multispeaker TTS has yet to be fully solved. Moreover, capturing a wide range of voice properties requires a substantial amount of high-quality data from numerous varied speakers.

Regarding multispeaker multilingual TTS, a primary idea is to train the decoder by incorporating learnable language and speaker embeddings[6]. However, the majority of speech attributes are interrelated and difficult to disentangle through embedding tables alone. Attempts have been made to mitigate the decline in multi-language multispeaker performance caused by speaker and language entanglement, especially in cross-lingual scenarios. [6] incorporated adversarial domain training, enabling them to transfer different voices between languages. [36] proposed the use of mutual information minimization to keep speaker consistency in cross-lingual synthesis. [37] utilized joint training incorporating a speaker classifier to enhance speaker similarity.

Previous multilingual TTS models [6, 7] are primarily built using Tacotron [1, 3]. When synthesizing speech, Tacotron-based models often repeat or skip words due to an autoregressive approach that uses attention to align input text and target speech. Instead of autoregressive iteration, several non-autoregressive [38, 39, 2, 40] models adopt Transformer encoder and decoder blocks to generate Mel spectrograms in parallel. For example, [41] create a multilingual speech synthesis system including a Transformer-based acoustic predictor and a WaveNet [42] neural vocoder. [43] presented a multilingual, multiaccented, multispeaker speech synthesis model based on RADTTS [44] which is a parallel flow-based generative model. Recently, several fully end-to-end multilingual speech synthesis systems [45, 8] were built on VITS[46], which could produce natural-sounding synthesized speech. Moreover, VITS requires training a single model, so there is no need for an independent vocoder. Among these fully end-to-end systems, YourTTS [8], which aims at zero-shot multispeaker TTS and zero-shot voice conversion, presented promising results for a few language combinations. However, it had limited success in transferring to languages with few speakers and used a curriculum learning approach, making training cumbersome.

Unlike these systems that all use acoustic features such as the Mel spectrograms as intermediate features, we built a multilingual synthesis system that is based on SSL representations to take advantage of better speaker/content disentanglement in discrete SSL representations.

Traditionally, end-to-end TTS models have utilized input representations in the form of characters or phonemes[1, 2]. Employing each character as the default input for end-to-end TTS models requires the model to implicitly learn how to convert graphemes to phonemes as part of the synthesis task. Incorporating phoneme input simplifies the TTS task, eliminating the need for the model to learn complicated grapheme-to-phoneme rules for languages such as English. Furthermore, language-independent pronunciation information can be obtained through language expertise, such as making a unified set of all the languages in a multilingual model, often derived from the International Phonetic Alphabet (IPA) [5, 47].

Over the past decade, self-supervised pre-training on extensive text corpora, employing language model (LM) or MLM objectives, has demonstrated remarkable success in the field of TTS[9, 10, 11]. Toward multilingual tasks, [12] proposed XPhoneBERT which is the first pre-trained multilingual model for phoneme representations for TTS. Based on the monolingual experiment results, using XPhoneBERT as a phoneme encoder for input has been shown to significantly enhance the performance of a powerful neural TTS model in terms of naturalness and prosody. It also helps to produce high-quality speech using low-resource training data. [5, 48] conducted pre-training on a multilingual text-only dataset using an MLM. They then trained this model in a supervised manner on paired text and audio data, while keeping the language-aware embedding layer frozen. This approach enables the model to synthesize languages that are not present in the paired data but exist in the text-only data.

Alternative approaches adopt UTF-8 bytes [49] or phonological features (PFs) [50, 51] as input representations. UTF-8 byte representations encode typographic rather than phonological information. The model does not learn unseen byte combinations, requiring retraining when enrolling in a new language. For PFs, IPA symbols can be converted to PFs in accordance with certain aspects of their articulation, as implemented in Epitran [52]. Although this conversion is based on an explicit database of IPA symbols and diacritics, it can be considered universally applicable to any language for which IPA transcription is available. However, there are no pre-trained phoneme-level encoders similar to XPhoneBERT currently available. The effect of different input representations is still not clear in multilingual speech synthesis with SSL representations. Taking into account the complexity of the model and the support for new languages and low-resource scenarios, we evaluated the effects of using characters, IPA, and XPhoneBERT as input for multilingual TTS.

We attempt to use different input representations including characters, IPA, and pre-trained phoneme representations for multilingual TTS.

1. a)

   Characters: To extend a character-based input vocabulary to multilingual TTS systems, we only need to concatenate character sets in the training corpus for each language.

2. b)

   Phonemes-IPA: To bring together the many languages under the same input space, we also use IPA, mapping all orthographic text into their IPA representations through the use of Epitran[52].

3. c)

   Pre-trained phoneme representations: First, we convert text sentences into a sequence of phonemes using the CharsiuG2P¹¹1https://github.com/lingjzhu/CharsiuG2P toolkit because of the input labels on which XPhoneBERT relies. Then, we convert this sequence of phonemes into a sequence of phoneme representations using a pre-trained XPhoneBERT. XPhoneBERT uses the BERT-base architecture, pre-trained on RoBERTa with 330M phoneme-level sentences across 100+ languages. Here, we use XPhoneBERT as an input phoneme encoder rather than FFT, which is used as the phoneme encoder in FastSpeech.

In summary, when characters or IPA symbols are used as input, FFT is used as the encoder. However, when phonemes from CharsiuG2P are used as input, pre-trained XPhoneBERT is used as the encoder.

To enable control over language and speaker identity, we add language and speaker embeddings as inputs to the decoder and the duration predictor. To extract speaker embeddings, we use a pre-trained ECAPA-TDNN speaker encoder model. For language embeddings, we adopt a trainable language embedding table to extract the language embedding of the target language. In detail, with the target language ID, a 64-dimensional language embedding can be produced by a language look-up table.

In this txt2vec stage, the text encoder operates on the input tokens $x$ and produces hidden states, which, combined with language and speaker embeddings, are used to predict the duration $d$ by the duration predictor. The up-sampled encoder outputs, together with the language and speaker embeddings, are used as the decoder input. The decoder then generates discrete values for both $V$ and $R$.

During training, we first adopt a classification task that is optimized by the cross entropy loss between the $\hat{p}(v)$ (predicted probability distributions of $V$) and the target probability distributions $p(v)$. Given a sequence of decoder output features $Q=\{q_{1},q_{2},...,q_{t}\}$, the $p(v)$ can be obtained through

where $F$ is a fully-connected layer that maps the $q_{t}$ to $G\times D$ dimensions. Subsequently, it further partitions based on the number of codebooks, $G$, to obtain the hidden representation with $D$ dimensions for computing the corresponding classification probability value. The classifications loss $\mathcal{L}_{cla}$ can be represented as

To learn the cross-lingual information from shared quantized latent speech representations in continuous space, we also incorporated predictions for $R$ during training as

Eq. (4) is a weighted sum of the code vectors, where the weights are the predicted probabilities of choosing the codes. In Eq. (4), $p(v_{t}^{g}=i|h_{t}^{g})$ represents the probability of predicting the code index value $v_{t}^{g}$ as $i$ at time $t$. Here, $g\in G$ denotes the g-th codebook and $C_{g,i}$ denotes the vector corresponding to the i-th index in the g-th codebook at $C^{G\times M\times D}$. The regression L2 loss $\mathcal{L}_{MSE}$ between the predicted and targeted $R$ could be represented as

The total loss consists of the sum of the distance between the ground-truth and predicted discrete features and the duration loss:

Given the monotonic alignment assumption between text tokens and discrete representation frames, the $\mathcal{L}_{dur}$ is optimized by maximizing the joint likelihood of each token and frame to find the most likely monotonic path [69, 44].

Note that since $p(v_{t}^{g})$ is a one-hot vector, the $r_{t}^{g}$ is only one entry in the codebook instead of being multiplied by softmax weights as in Eq. (5). Therefore, in inference, we directly select the maximum probability index in the $P(i|h_{t}^{g})$, and look up the corresponding $(\hat{r}_{t}^{g})$ in the code $C^{G\times M\times D}$ as the input of the next-stage model:

In the process of converting discrete features $R$ into audio, we can still use a Mel-based vocoder, which requires us to first convert the discrete features $R$ into Mel spectrograms.

This vec2wav model consists of four parts: up-sampling, down-sampling, decoder, and pre-trained Mel-based vocoder as shown in Figure 3. To learn Mel spectrograms from $R$, the duration model is no longer necessary. Although the sequences $R$ and $Mel$ have different “sampling rates,” they can be treated as being “uniformly aligned.” For example, if we adopt a 12.5 ms frame-shift to extract Mel spectrograms, considering the framerate of XLSR is 20ms, the length ratio of ${\color[rgb]{0,0,0}R:Mel:Speech}$ would be $5:8:1600$. In this work, we adopt an up-sampling factor of 8 and then down-sample by a factor of 5 to map discrete representations to Mel ones. Transposed convolution is used for up-sampling, followed by a Multi-Receptive Field Fusion (MRF) module, similar to the HiFi-GAN generator. With an up-sampling factor of 8, the configuration of the generator was changed for up-sampling rates to a sequence of (4, 2) with corresponding kernel sizes (12,8), while the hyper-parameters of residual blocks are the same as those in HiFi-GAN V1. For the down-sampling, we use average pooling. The decoder part is the same as that of txt2vec, which is an FFT. The decoder takes the output of average pooling and language and speaker embeddings to produce the Mel spectrograms. The optimization objective of this vec2mel is defined as: ${\mathcal{L}_{Mel}={\left\|\hat{Mel}-Mel\right\|}_{2}^{2}}$.

Instead of using an additional vocoder to convert Mel to a waveform, we also proposed a vec2wav model as in [54], which directly maps discrete representations to a waveform. To learn the cross-lingual information from SSL representations at different time resolutions, we include a multi-stage multi-head codebook for the waveform modeling as [71, 72].

The details of the architecture and operation of the multi-stage encoder are provided in [71, 72]. Here, we illustrate the audio waveform generation process with a 2-stage encoder as an example as shown in Figure 4, which consists of the following steps: 1) The first-stage encoder receives discrete sequences $R$ and produces the first hidden representation sequences. These sequences are then downsampled and passed to the second-stage encoder to obtain the second hidden representation, which has a lower time resolution. 2) The second hidden representation sequences are quantized using codebook 2 to obtain $z^{(2)}$. These sequences are then up-sampled and combined with the first hidden representation sequences to obtain $z^{(1)}$ using codebook 1. 3) The residual output of $z^{(1)}$ and $z^{(2)}$ is used to reconstruct the waveform using the frame decoder and HiFi-GAN-based waveform generator.

To up-sample the audio from $R$, the generator’s configuration was changed to a sequence of (5, 4, 4, 2, 2) with corresponding kernel sizes of (11, 8, 8, 4, 4).

During the training process, a UnivNet [73] discriminator is utilized for adversarial training of this model. The final loss is composed of three main components: waveform level loss $\mathcal{L}_{w}$, calculated using the HiFi-GAN loss function to compare the real waveform with the reconstructed waveform; frame-level loss $\mathcal{L}_{d}$, calculated from the mean squared error between two discrete representations; and VQ loss and stabilization loss, which are similar to those described in [71].

We synthesized sentences from test sets in six languages for each TTS system. For each language, we select 6 (3 female, 3 male) seen speakers and 6 (3 female, 3 male) unseen speakers for the test set, and two sentences for each speaker. Ten native listeners per language participated in our listening tests. Listeners evaluated one test utterance at a time, initially assigning a rating on a Likert scale from 1 to 5 for the Mean Opinion Score (MOS) to assess naturalness. Subsequently, they provided ratings for speaker similarity concerning a reference utterance using a Differential MOS (DMOS) scale, ranging from 1 (indicating a different speaker) to 5 (indicating the same speaker). Reference utterances were chosen randomly from the original speech of the target speaker. Due to the high cost of subjective evaluation, only the original YourTTS is used as the YourTTS related baseline to test MOS and DMOS. Our preliminary experimental results also demonstrated that YourTTS achieves better performance than YourTTSE and YourTTSW.

To assess the similarity between the synthesized voice and the original speaker, we determine the Speaker Encoder Cosine Similarity (SECS) by measuring the cosine similarity between the speaker embeddings of two audio samples extracted from the speaker encoder. The SECS score falls within the range of -1 to 1, with a higher value indicating better speaker similarity. In accordance with prior research [35, 8], we compute the SECS using the speaker encoder from the Resemblyzer package⁷⁷7https://github.com/resemble-ai/Resemblyzer, facilitating comparisons with those studies.

We also objectively evaluate performance by measuring the intelligibility of speech content using an ASR algorithm. We synthesized 2,000 sentences (1,000 with seen and unseen speaker embeddings, respectively) for each language. Note that the language of the target speaker and text are always consistent. Although our model also has the ability of cross-lingual synthesis, considering that there is no clear standard for the evaluation of cross-language synthesis, this paper only focuses on the intra-lingual scenarios.

The evaluation texts are from the CMU ARCTIC sentences⁸⁸8http://festvox.org/cmu_arctic/cmuarctic.data for English, test data in MLS for French, German, Portuguese, and Spanish, and test data in NST for Swedish. All synthesized sentences were sent to the Whisper⁹⁹9https://github.com/openai/whisper model for ASR. We computed the character error rate (CER) between the input text and the ASR-produced transcripts.

In terms of speech naturalness, our proposed systems ZMM-TTS1 and ZMM-TTS2 are significantly better, according to a Mann-Whitney U test given $\alpha=0.05$ with Holm-Bonferroni correction, than the FastSpeech-based systems (FSM1 and FSM2) in each language on both seen and unseen speaker conditions. Our training data come mainly from MLS and GLB. The sound quality of these datasets is significantly worse than several monolingual datasets such as LJSpeech [77] and the Chinese Standard Mandarin Speech Corpus [81] that are commonly used in speech synthesis. For example, most recordings of GLB were done in ordinary rooms rather than professional recording studios, and several audio samples contain some noise.

As a currently popular method based on FastSpeech and HiFi-GAN, the baseline FastSpeech-based FSM model has achieved SOTA on many monolingual synthesis tasks while obtaining the worst results in speech naturalness in our multilingual datasets. This result indicates that for FSM models based on Mel spectrograms, the sound quality of synthesized audio may be limited due to the sound quality of the training data itself. On the other hand, since the intermediate features used by our ZMM-TTS model are extracted from SSL models trained on large-scale non-ideal data, the proposed system is considered to be less sensitive to the sound quality of the training data. The reason that the proposed method ZMTTS-2x has higher MOS values than ground-truth audio on seen (Spanish) and unseen speakers (French and Swedish) would also be due to the fact that the sound quality of the ground-truth audio is not high.

In terms of speaker similarity, our proposed ZMM-TTS system achieves better DMOS than the FSM-based system. The speaker encoder used in the ZMM-TTS and FSM-based systems is the same. Also, the difference between the DMOS of the ZMM-TTS and FSM-based systems is smaller than their difference in MOS. This suggests that the improvement in naturalness using features extracted from SSL models is more prominent than the improvement in speaker similarity because discrete features may contain less speaker information.

Another finding is that adding the VQ module in the process of converting the Mel spectrograms or discrete representations to a waveform can improve the naturalness of the audio. The results in Tables V and VI show that FSM2 outperformed FSM1 on MOS. One potential reason is that the discrete representations in vec2wavVQ (Mel) may reduce some redundant information, such as noise, to some extent. Similar findings are observed when comparing ZMM-TTS1 and ZMM-TTS2. ZMM-TTS2 has an additional VQ module and converts discrete representations directly to speech without an independent vocoder. Directly mapping discrete representations to a waveform instead of a Mel spectrogram prevents error propagation caused by inaccurate Mel spectrogram prediction in the vec2mel stage and hence, like the result of FSM1 and FSM2, multi-stage discrete representations learned by additional VQ may also be helpful for improving sound quality.

In terms of speaker similarity, the use of VQ in vec2wav does not always result in an improvement. We also find that MOS is not always correlated with DMOS. For example, as shown in the French results in Table V, the naturalness of ZMM-TTS2c is significantly higher than that of ZMM-TTS1c (4.20 vs 2.90), while their DMOS values are very similar (4.27 vs 4.28).

On speech naturalness, the YourTTS method outperforms the two-stage FSM baseline while remaining worse than two of our proposed systems, ZMM-TTS2c and ZMM-TTS2x. We observed that the YourTTS model exhibits instability in its stochastic duration predictor, resulting in the production of unnatural durations for certain speakers and sentences.

On speaker similarity, ZMM-TTS2c and ZMM-TTS2x has better performance than YourTTS systems for the seen speaker condition in five languages (en, fr, ge, pt, and sw). However, under unseen conditions, YourTTS is better than our method in terms of DMOS in English, French, German and Spanish. This will be analyzed in detail in the next subsection.

Interestingly, determining which is better for speech naturalness, character-based inputs or pre-trained phoneme representation based inputs, depends on the specific language and synthesis systems. FSM1x and FSM2x always have a higher MOS than FSM1c and FSM2c among the FSM-based systems. Utilizing pre-trained phoneme representations as input and XPhoneBERT as the text encoder can enhance the quality of synthesized speech. However, for our proposed ZMM-TTS system, better naturalness is not always achieved with ZMM-TTS1x and ZMM-TTS2x. For example, in Table VI, comparing the MOS of ZMM-TTS2x and ZMM-TTS2c, ZMM-TTS2x has achieved better results in English, French, and Portuguese, while ZMM-TTS2c has achieved better results in other languages. Furthermore, we observed that the MOS of ZMM-TTS2x is significantly better than ZMM-TTS2c in English. This may be because XPhoneBERT was pre-trained on a dataset with more English text than other languages. We will investigate how the amount of pre-training text affects XPhoneBERT’s representation performance in the future.

In terms of speaker similarity, different text representations have little impact. This result aligns with expectations as speaker and semantic information are not closely related.

An interesting result is that the naturalness of the synthesized speech of unseen speakers is not worse than that of seen speakers. However, there was still a gap between seen and unseen speakers in terms of DMOS for most languages. Although the pre-training process for the speaker encoder involves more than 7,000 speakers, the TTS system is only trained on a few hundred speakers. It is still challenging to achieve the ideal zero-shot performance when training with limited multilingual data.

First, we found very strong correlations ($r=0.807$), using the Pearson correlation coefficient (PCC), between the SECS and the DMOS values. Compared with the FSM-based model, ZMM-TTS performs better in both SECS and DMOS metrics for both seen and unseen speakers. Compared with the YourTTS model, ZMM-TTS achieves similar SECS for seen speakers, but there are differences for unseen speakers, particularly English speakers. One possible reason for this difference is that the original implementation of YourTTS utilizes SCL to enable generalization in the characteristics of unseen speakers. The different SECS results of YourTTSE and YourTTSW also demonstrate the impact of SCL.

To investigate whether the difference is related to language and the impact of two speaker encoders, we plotted speaker embeddings of seen and unseen speakers from natural speech via T-SNE[82]. Figure 5 presents the T-SNE visualization and shows the following: Several unseen speakers are not close to the seen speakers in the training set. For example, there are two obvious outliers in the ECAPA-TDNN speaker embeddings of the English unseen speakers. We conducted a study to calculate the SECS values of the six unseen English speakers whose speech was synthesized by the ZMM-TTS1c system. We found two English outlier points in Figure 5(a) with SECS values of 0.766 and 0.644, which were significantly lower than that for the speaker who was closer to the seen English speakers. This explains why the speaker similarity of our proposed ZMM-TTS1c for English unseen speakers is significantly lower than that of YourTTS.

Although there are many different types of SOTA neural speaker embeddings, currently, ZMM-TTS has only attempted using a popular speaker representation from ECAPA-TDNN, and we will explore the performance with different speaker embeddings such as H/ASP and SCL in future work.

Table VII summarizes the obtained CER. The first finding is that the FSM1 and FSM2 baseline models perform better, although their sound quality is worse than the other systems. This demonstrates that the speech generated using Mel spectrograms as intermediate features is intelligible but not as natural sounding as when using SSL representations. Additionally, the speech synthesized through the Mel spectrograms tends to over-smooth the high-frequency region more than through SSL representations.

We also find that the CER is not always correlated with the MOS values. Although YourTTS CER is worse, the sound quality is obviously better compared with the FSM-based model. Furthermore, YourTTS CER in Portuguese and Swedish is significantly worse than in other languages. This also explains why the YourTTS system is significantly worse than our ZMM-TTS on MOS in these two languages. YourTTS only uses the character transcriptions rather than phonemes, which makes it more prone to mispronunciation issues as mentioned in [8].

Additionally, we found that the pre-trained phoneme representation input system performance is the best, and IPA is only better than characters in English. A notable result is that compared with FSM1, the CER performance of FSM2 has dropped significantly. For example, sometimes the pronunciation of the word “advice” in a sentence becomes incorrect and sounds more like the word “addressed,” resulting in ASR recognition errors. This shows that learning a discrete representation from Mel spectrograms to speech conversion in vec2wavVQ (Mel) can improve the quality of speech naturalness, but some fine-grained information related to linguistic content may be lost.

We chose Italian and Polish as two unseen languages for this experiment. In the language family tree [83], Italian is closely related to French, Spanish, and Portuguese, all of which belong to the Romance family. Polish is relatively far away from our six high-resource languages and is from the Slavic language family. We selected two speakers with ID 1595 and 6892 from MLS for our Italian and Polish data, respectively.

Considering the performance on six high-resource languages, we choose three models FSM2, ZMM-TTS2 and YourTTS to implement on new languages. We have attempted to use both characters and pre-trained phonemes as input representations. Low-resource language synthesis is investigated in two scenarios: few-shot and zero-shot. Few-shot means that the model is fine-tuned using the data from of the language before synthesing speech of that language. In contrast, as mentioned in Section I, zero-shot means that our model performs inference on unseen languages without model fine-tuning.

For the few-shot scenario, we created training sets of different sizes (2.5, 5, and 15 mins of audio). We use limited training data to finetune the model trained on six languages in Section IV. For language embeddings, we reserve a free ID for new languages, which will be trained from scratch using fine-tuning data only. Similarly, since different languages may use different characters, symbol embeddings for any characters unseen during pre-training must also be learned from fine-tuning data only. Specifically, in our experiments, we found that Italian and Polish had 3 and 14 unseen symbols, respectively, compared with the six pre-training languages. All few-shot models are fine-tuned for 10,000 steps with a batch size of 16. To compare the performance differences of the same language with high and low resources, we also perform fine-tuning with much larger datasets of 35h for Italian and 21h for Polish.

For the zero-shot scenario, to address the mismatch between language embeddings of seen and unseen languages, we removed the language embedding layer. Therefore, the models tested on the zero-shot scenario are trained as in Section IV on six seen languages but without a language embedding layer. Additionally, we used pre-trained phoneme representations as input, making it easily applicable to many unseen languages.

Objective Evaluation. We kept 100 sentences for the speech naturalness and speaker similarity tests and 1,000 sentences for ASR tests like those mentioned in Section IV-D. Furthermore, we adopted a publicly available automatic MOS (UTMOS) prediction model [84] to assess naturalness as in [5, 32].

Subjective Evaluation. We synthesized eight sentences from test sets for each TTS system. 15 native listeners of each language participated in our listening tests. First, we employed the same evaluation as described in Section IV-D to assess MOS and DMOS. Furthermore, ensuring the intelligibility of generated speech is crucial, particularly in low-resource scenarios. To this aim, we also designed an intelligibility test (IMOS) in which listeners provided ratings for speech regarding the extent to which words were pronounced wrong or were not understandable. The scale of IMOS is also from 1 (indicating very bad) to 5 (indicating very good).

In high-resource baselines, the intelligibility test results (CER/IMOS) of ZMM-TTS2x and ZMM-TTS2c were quite similar. However, when only 15 minutes of data were used for fine-tuning, the synthesized speech produced by ZMM-TTS2x was much more intelligible than that of ZMM-TTS2c in low-resource scenarios. The same phenomenon is also observed for FSM2x and FSM2c. This result demonstrates that the pre-trained phoneme representations from XPhoneBert are more suitable for low-resource scenarios.

On both subjective and objective evaluation, the ZMM-TTS2x model produces significantly more natural speech (MOS/UTMOS) with better speaker similarity (DMOS/SECS) than the FSM2x model, both in high and low resource scenarios on two unseen languages. This result indicates that self-supervised features can better reconstruct sound quality and timbre compared with the Mel spectrograms, even with limited data. Furthermore, on the basis of the subjective IMOS evaluations, in low-resource scenarios, the intelligibility of ZMM-TTS2x surpasses that of FSM2x. This demonstrates the advantage of our model in low-resource scenarios.

We can see that YourTTS cannot be fine-tuned with limited data from an unseen language. Even if one hour of data is used for fine-tuning, YourTTS still cannot synthesize understandable audio. Although promising results are evident in several language combinations with sufficient training data, YourTTS demonstrates limited efficacy when adapting to languages with limited available speakers.

As more training data is added, the synthesized speech quality improves, particularly in terms of speech intelligibility. Furthermore, the results for speaker similarity (SECS/DMOS) of ZMM-TTS2x are similar across different quantities of training data (2.5, 5, and 15 mins). In general, with only 2.5 mins of speech from an unseen speaker in a new language, our fine-tuning of the ZMM-TTS2x still resulted in high speaker similarity in terms of SECS.

We found that for Italian, using pre-trained phoneme representations as input, we were able to synthesize intelligible speech in zero-shot scenarios. In Polish, there is a significant gap between the zero-shot and few-shot scenarios, but both FSM2x and ZMM-TTS2x were able to generate intelligible speech utilizing a few minutes of data with pre-trained phoneme representations as input. The SSL speech-based features do not seem to provide an advantage over the Mel spectrograms, unlike in Italian. Although Italian and Polish are both included in the XLSR training data, the performance of SSL representations may be heavily affected by the similarity between the domains of the ZMM-TTS pre-training data and the fine-tuning data. Future work may include investigating the impact of language similarity on cross-lingual transfer.