Towards Lightweight and Stable Zero-shot TTS with Self-distilled Representation Disentanglement

In the first part, we extract speaker-independent content representation from both text (phoneme sequence) and the corresponding ground truth speech (mel-spectrogram). To enhance the expressive capacity for complex speech content, we integrate a Mel Variational Autoencoder (VAE) with a flow-based model to construct a latent space of speech, transcending the limitations of text-only representations. We first encode both the text and speech at the phoneme level using a linguistic encoder and a mel encoder, generating representations $\bm{e}_{\text{ling}}$ and $\bm{e}_{\text{mel}}$, respectively. We use $\bm{e}_{\text{ling}}$ as a condition for predicting $\bm{e}_{\text{mel}}$ through Volume-Preserving Flow (VP-Flow). After that, we fuse $\bm{e}_{\text{ling}}$ with $\bm{e}_{\text{mel}}$, forming the content representation $\bm{e}_{\text{con}}$, which is then fed into the speaker adaptivation part. Specifically, the text encoder consists of multiple feed-forward Transformer blocks with relative positional encoding[23]. The mel encoder consists of multiple 2D Residential blocks (ResBlock) to downsample and encode the input mel-spectrogram, followed by average pooling to map frame-level features to phoneme-level features. Another multi-level ResNet estimates the mean and variance of the variational latent space. Note that the source speech contains not only content information, but also information related to the target speaker. This content-speaker coupling leads to the leakage of speaker information from $\bm{e}_{\text{con}}$. We address this issue using a self-distillation framework in Section 3.

In the second part, we explicitly model multi-level speaker-specific representations from the prompt speech, i.e., temporal style representations and global timbre representations. These representations serve as conditions for steering the synthesis of the target speech, ensuring that it accurately reflects the desired speaker characteristics.

We adopt a trainable mel encoder to extract the temporally-related style representation, $\bm{e}_{\text{sty}}$. This encoder is similar to that used in HierSpeech++[10] and GPT-SoVITS[17], but it includes positional encoding and omits the final temporal averaging layer to better capture temporal information. We use $\bm{e}_{\text{sty}}$ as a condition for a variation adapter, which consists of three predictors for independent speech attribute predictors, i.e., duration, pitch, and energy. We integrate and align $\bm{e}_{\text{sty}}$ with the input sequence of the predictors through a multi-head cross-attention layer. These predictors are individually trained under supervised conditions using annotated data. To ensure the model’s stability, we separate the gradients of these predictors from the main network.

For timbre extraction, we utilize a speaker recognition model as the timbre encoder, i.e., ERes2NetV2[24], pre-trained on a large-scale multi-domain dataset, Speaker3D[25]. This encoder generates a comprehensive speaker latent space that encompasses various speakers and speaking conditions, facilitating the effective application of unseen speaker features to achieve precise modeling of their vocal styles. Additionally, a trainable linear layer is integrated into this model to yield the speaker representation $\bm{e}_{\text{spk}}$. To transfer the timbre representation $\bm{e}_{spk}$ extracted from arbitrary speech into synthesized speech, we employ a speaker-adaptive mel Decoder based on a multi-level ResNet architecture with Adaptive Instance Normalization (AdaIN) layer.

We begin by training a teacher model $\text{M}_{T}$, which is expected to effectively reconstruct the linguistic content from the given text and preliminarily clone the speaker’s voice from the prompt speech.

Given a single speech sample $x\in\mathbb{R}^{T\times F}$, where $T$ and $F$ represent the temporal and frequency dimensions, respectively, we extract $\bm{e}_{\text{mel}}$ from $x$ and $\bm{e}_{\text{ling}}$ from the corresponding text $c$. These two representations are combined to obtain $\bm{e}_{\text{con}}$. Next, we use the timbre encoder and the style encoder to extract $\bm{e}_{\text{tim}}$ and $\bm{e}_{\text{sty}}$ from the prompt speech $x^{\prime}$. Using $\bm{e}_{\text{sty}}$ as a condition, we predict time-dependent speech attributes and apply these to $\bm{e}_{\text{con}}$. Finally, we input the fused feature into the Mel decoder to reconstruct the speech mel spectrogram, ensuring that the generated speech matches the target speaker’s timbre by conditioning on $\bm{e}_{\text{tim}}$. We employ the following loss functions to guide the training process.

Primary Losses. A reconstructive loss ($L_{\text{rec}}$) based on the $L_{1}$-norm is employed to quantify the discrepancy between the reconstructed and original speech. Additionally, a Kullback-Leibler (KL) divergence loss ($L_{\text{KL}}$) is utilized to constrain the distribution of the speech and linguistic representations. A PatchGAN-structured discriminator is employed to leverage its capability for multi-scale analysis, distinguishing synthesized mel-spectrograms from genuine audio features. This discriminator facilitates an adversarial training paradigm with the generator, utilizing the Least Squares Generative Adversarial Network (LSGAN) loss ($L_{\text{adv}}$).

Style-related Losses. A suite of loss functions refine the accuracy of our speech attribute predictors within the variance adapter. For every speech sample $x$, our model generates predictions for the attributes, i.e., duration, pitch, and energy, denoted as $\hat{y}_{i}$, where $i$ spans {duration, pitch, energy}. The loss function, $L_{\text{pred},i}$, quantifies the mean squared error (MSE) between the model’s predictions and the ground truth values.

Timbre-related Loss. A cyclic contrastive loss function $L_{\text{cyc}}$ is employed to enforce alignment between the output speech and the prompt audio within the speaker space of the timbre encoder. Let $\bm{e}_{i}$ and $\bm{\hat{e}}_{i}$ denote the timbre representations of the original mel-spectrogram and reconstructed mel-spectrogram, respectively, where $i$ indexes the samples within a batch of size $B$. The speaker representation $\bm{\hat{e}}_{i}$, obtained by feeding $\hat{x}_{i}$ into the speaker encoder, should closely match $\bm{e}_{i}$. Conversely, speaker representations of other utterances $\bm{\hat{e}}_{j\neq i}$ within the mini-batch should significantly diverge. The contrastive loss is defined as[26]:

$$L_{\text{cyc}}=-\frac{1}{B}\sum_{i=1}^{B}\log\left(\frac{\exp(\cos(\bm{\hat{e}}_{i},\bm{e}_{i}))}{\sum_{j=1}^{N}\mathbf{1}_{j\neq i}\exp(\cos(\bm{\hat{e}}_{i},\bm{e}_{j}))}\right),$$

(1)

where, $\cos(\cdot,\cdot)$ denotes the cosine similarity between two vectors, and $\mathbf{1}_{j\neq i}\in\{0,1\}$ is an indicator function.

By leveraging the well-trained teacher model, we proceed to generate parallel data pairs for distilling a student model.

Given a speech sample $x$, and the corresponding text, represented as a phoneme sequence $c$, our method begins by employing the teacher model to synthesize a new speech sample $\hat{x}^{\prime}$. The synthetic speech sample preserves the textual content $c$, but is delivered by a distinct speaker. To this end, we begin by randomly selecting a prompt speech sample $x^{\prime}$ from our training dataset. The sample $x^{\prime}$ corresponds to a different speaker $s^{\prime}$ than the speaker $s$ in $x$. Then, we infer the teacher model with $c$ and $x^{\prime}$ for the generation of $\hat{x}^{\prime}=\text{M}_{T}(c,x^{\prime})$. For the strict temporal alignment of the phonemes of the $x$ and $\hat{x}^{\prime}$, we explicitly specify the prediction result of the duration predictor as the annotated phoneme duration of $x$ during the inference process.

After obtaining the time-aligned parallel data pair $\{x,\hat{x}^{\prime}\}$, we proceed to train the student model. The student model mirrors the same model structure as the teacher model but is initialized randomly. We input $\hat{x}^{\prime}$ as the speech content for the student model, accompanied by $c$ as the linguistic content input. Additionally, we randomly select another speech sample $\hat{x}^{\prime\prime}$ as the prompt, which is uttered by the same speaker as $x$. We optimize the student model following the same procedure as detailed in Section 3.1, using $x$ along with its detailed annotation serving as the ground truth.

To regulate the intensity of distillation, we blend real and synthesized samples within each training batch. To this end, we introduce a coefficient $\sigma\in[0,1]$, which modulates the proportion of synthetic speech samples. Specifically, in each batch of speech content input, we replace $\lfloor\sigma\cdot B\rfloor$ randomly selected speech samples with their corresponding synthesized speech samples, while the remaining samples remain as the ground truth speech samples.

The self-distillation framework ensures that the student model’s content extraction avoids speaker-specific details, compelling it to rely solely on prompt speech for speaker adaptation, thereby enhancing the separation of the representations.

We train both the teacher and student models on a diverse dataset, comprising 4,678 speakers, totaling 531 hours of audio. The preprocessing pipeline involve extracting 80-dimensional mel-spectrograms from the audio clips. For audio reconstruction from mel-spectrograms, we employ a NFS-HiFiGAN model[27] as vocoder. The TTS models are trained using an AdamW optimizer and a batch size of 64 for 200,000 training steps. When training the student model, we set $\sigma=0.8$ by default.

Our proposed model is compared against four state-of-the-art zero-shot Text-to-Speech (TTS) models: Vall-E[14], X-TTSv2[28], CosyVoice[16], and GPT-SoVITS[17]. We reproduce Vall-E based on the published methodology[14] and trained it on a speech corpus of approximately 60K hours and 19K speakers. The other systems utilize the open-source code and pre-trained models provided by their respective authors.

Prompt speeches for the test dataset are collected from 20 volunteers (10 males and 10 females) who are not part of the training set, ensuring that the baseline system’s training data does not contain these test cases. For each volunteer, we record 5 audio clips, ensuring high audio quality, clear pronunciation, and varied durations ranging from 5 to 10 seconds. For each prompt speech, we generate 100 synthetic speeches using different sentences, with lengths ranging from approximately 30 to 50 characters.

The metrics employed in the experiments are as follows,

• •

  Speaker Similarity (SIM): We quantified the similarity between generated and prompt speech using cosine similarity of speaker embeddings, extracted via an Ecapa-TDNN model[29] pre-trained on Speaker3D dataset[25].

• •

  Character Error Rate(CER): We use a pre-trained ASR model[30] to infer transcripts, and then derive CERs by calculating the edit distance between the transcripts and the correct text.

• •

  Real-Time Factor(RTF): We measure RTF on a Sever platform equipped with an Intel(R) Xeon(R) Platinum [email protected] CPU and an Nvidia Tesla A100 GPU.

• •

  Mean Opinion Score (MOS): We invite 50 volunteers to score 20 groups of generated speech based on their on their subjective feelings in terms of content consistency ($\text{MOS}_{\text{con}}$) and speaker similarity ($\text{MOS}_{\text{sim}}$) on a scale of 1 to 5. Final MOS scores are calculated with a 95% confidence interval.

Table 2 presents the overall performance comparison of our system with baseline systems regarding content integrity and speaker similarity. In terms of content integrity, our system achieves the lowest CER of 1.8 and the highest $\text{MOS}_{\text{con}}$ of 4.43, surpassing all baseline models. These results highlight our system’s ability to generate accurate and reliable content that meets expectations. In terms of speaker similarity, our system attains a SIM score of 0.73, trailing only behind the state-of-the-art model, CosyVoice, which scores 0.84. $\text{MOS}_{\text{sim}}$ may slightly differ from the SIM results, because the objective scoring model primarily assesses timbre similarity, while human raters consider additional factors such as speaking style and emotional expressiveness. Overall, the experimental results demonstrate that our system outperforms baseline models in content integrity while showing promise in speaker similarity.

Table 2 shows the efficiency comparison of our system with baseline systems in terms of parameter scale, data requirements, and real-time performance metrics. All baseline systems exceed 200M parameters, with CosyVoice and X-TTSv2 even surpassing 400M parameters. While this extensive parameter count enables robust speaker modeling, it necessitates considerable training datasets and computational resources, as illustrated by the significant data hours and RTF involved. In contrast, our system employs a lightweright architecture with only 22.5M parameters, making it both data and computationally efficient. Regarding real-time performance, all baseline systems operate slower than real-time on CPUs, with their RTF values significantly exceeding 1. In comparison, our system achieves RTFs of 0.13 on CPUs and 0.012 on GPUs, indicating at least a tenfold improvement over the baselines.

Figure 3 shows the t-SNE visualization of speaker embeddings from our system with and without self-distillation. We can obvious that, compared with real speech samples(triangles), the non-distillation model shows a clear separation between the point clusters of real speeches and synthesized speeches (circles). In contrast, the self-distilled model exhibits less separability. The distance between the feature points of real speech and synthesized speech is significantly reduced, and in some cases, they even highly overlap. This indicates a high degree of timbre consistency achieved by the self-distilled model, highlighting its effectiveness in enhancing the similarity of model speaker characteristics.

Figure 4 quantitatively illustrates the impact of the self-distillation coefficient $\sigma$ on the SIM and the CER. As $\sigma$ increased, the SIM score steadily improves to a maximum of 0.73 at $\sigma=0.8$. Further increases in $\sigma$ led to a slight decline in the score. This trend emphasizes the importance of selecting an appropriate $\sigma$ value to enhance speaker feature disentanglement and improve the model’s generalization capabilities. Meanwhile, CER scores exhibit stability across varying $\sigma$ values, experiencing minimal fluctuations between $1.78$ and $1.90$. This finding indicates that the self-distillation framework can effectively improve victim similarity without compromising the integrity of the content.