Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech

Conditional flow-matching [15] is an objective for training generative models. Continuous normalizing flows [16] is employed to convert a simple prior distribution $p_{0}$ into a complex distribution $p_{1}$ that aligns with the data. To be specific, for a given data point $x$, a neural network parameterized by $\theta$ models a time-dependent vector field $v_{t}(x;\theta)$. This vector field constructs a flow $\phi_{t}$, which reshapes the prior distribution into the target distribution. Lipman et al. [15] proposed to train such a neural network with the conditional flow-matching objective:

$$\mathcal{L}^{\rm CFM}(\theta)=\mathbb{E}_{t,q(x_{1}),p_{t}(x|x_{1})}||u_{t}(x|x_{1})-v_{t}(x;\theta)||^{2},$$

(1)

where $q$ denotes the training data distribution, $x_{1}$ represents the random variable for the training data, $p_{t}$ denotes the probability path at time step $t$, and $u_{t}$ is the vector field associated with $p_{t}$. Also, Lipman et al. [15] present a conditional flow known as the optimal transport path, defined by the equations $p_{t}(x|x_{1})=\mathcal{N}(x|tx_{1},(1-(1-\sigma_{\rm min})t)^{2}I)$ and $u_{t}(x|x_{1})=(x_{1}-(1-\sigma_{\rm min})x)/(1-(1-\sigma_{\rm min})t)$.

Voicebox [13] is the first to utilize conditional flow matching for training zero-shot TTS. It is designed to perform speech-infilling tasks given audio context and frame-wise phoneme sequence as conditions. Given the promising performance of Voicebox, we also employ conditional flow matching to develop our TTS model.

ELaTE [14] was proposed to generate natural laughing speech with fine-grained controllability. It utilized a frame-level laughter representation derived from the laughter detector [17, 18]¹¹1https://github.com/jrgillick/laughter-detection to condition the flow-matching-based zero-shot TTS, showing significantly higher quality and better controllability in generating laughing speech compared to conventional models. However, ELaTE was only tested with laughing speech, and the effects on other NVs, such as crying, have not been investigated. In our preliminary experiment, we found that ELaTE sometimes generates laughing speech even when the audio prompt contains different NVs, such as crying. Our work can be regarded as an extension of ELaTE, where we aim to achieve better emotion controllability as well as the generation of various NVs.

Figure 1 (a) illustrates the training procedure of EmoCtrl-TTS. Given a training audio sample $s$ with transcription $y$, we extract its mel-filterbank features $\hat{s}\in\mathbb{R}^{F\times T}$, where $F$ denotes the feature dimension and $T$ represents the sequence length. Additionally, we employ force alignment and a phoneme embedding layer to obtain a frame-wise phoneme embedding $a\in\mathbb{R}^{D^{phn}\times T}$, where $D^{phn}$ is the phoneme embedding dimension. The phoneme embedding layer is a part of the audio model and is jointly trained. Furthermore, we extract frame-wise embeddings that represent NV $h\in\mathbb{R}^{D^{NV}\times T}$ and emotion $e\in\mathbb{R}^{D^{emo}\times T}$, where $D^{NV}$ and $D^{emo}$ denote the dimensions of NV and emotion embeddings respectively. The embeddings $h$ and $e$ are extracted by using pre-trained NV and emotion detector, respectively, which are discussed in Section 3.2 and 3.3. We leverage the speech infilling task introduced in [13] to train the audio model, focusing on training a conditional flow-matching model to estimate the distribution $P(m\odot\hat{s}|(1-m)\odot\hat{s},a,h,e)$, where $m\in\{0,1\}^{F\times T}$ represents a binary temporal mask, and $\odot$ is the Hadamard product.

Figure 1 (b) illustrates the inference procedure of EmoCtrl-TTS. During inference, the model takes four inputs: text prompt $y^{text}$, speaker prompt audio $s^{spk}$, NV prompt audio $s^{NV}$, and emotion prompt audio $s^{emo}$. The text prompt represents the content of the generated speech. Meanwhile, the speaker, NV, and emotion prompts control the characteristics of the speaker, NV, and emotion in the generated speech, respectively. In speech-to-speech translation scenario, we use the source audio for $s^{spk}$, $s^{NV}$ and $s^{emo}$, and translated text as $y^{text}$. This results in the translated speech maintaining the source speaker’s voice and emotional characteristics.

The speaker prompt $s^{spk}$ is first converted to the mel-filterbank features $\hat{s}^{spk}$. It is also converted to phoneme embeddings $a^{spk}$ by applying automatic speech recognition (ASR) and then the phoneme embedding layer. The speaker prompt is further converted to NV embeddings $h^{spk}$ and emotion embeddings $e^{spk}$ based on the NV detector and emotion detector, respectively.

Meanwhile, the text prompt $y^{text}$ is converted to text prompt embeddings $a^{text}$ based on the phone duration model [13] followed by the phoneme embedding layer. The NV prompt embedding $h^{NV}$ and the emotion prompt embedding $e^{emo}$ are extracted from the NV detector and emotion detector, respectively. Note if the lengths of $h^{NV}$ and $h^{emo}$ are different from that of $a^{text}$, we apply linear interpolation to $h^{NV}$ and $h^{emo}$ to match their lengths to that of $a^{text}$.

The flow-matching-based audio model will then generate mel-filterbank features $\tilde{s}$ based on the learned distribution of $P(\tilde{s}|[\hat{s}^{spk};z^{text}],[a^{spk};a^{text}],[h^{spk};h^{NV}],[e^{spk};e^{emo}])$, where $z^{text}$ is an all-zero matrix with a shape of ${F\times T^{\rm text}}$, and $[;]$ denotes concatenation operation in the time dimension. The generated part of $\tilde{s}$ is then converted to the speech signal using a vocoder.

We used three training datasets: Libri-light, LAUGH, and IH-EMO. Libri-light was used for pre-training the audio model without NV and emotion embeddings. On the other hand, LAUGH and IH-EMO were used for fine-tuning the audio model with NV and emotion embeddings. During fine-tuning, we also used the Libri-light data with a certain probability as suggested in [14]. The overview of each dataset is as follows.

Libri-light [27]: 60k hours of untranscribed English audiobooks from over 7,000 speakers. We transcribed the data by using a pre-trained Kaldi ASR model5, which was trained on the 960-hour LibriSpeech dataset [28].

LAUGH: 460 hours of laughing speech collected from the AMI meeting corpus [29], Switchboard corpus [30], and Fisher corpus [31]. We gather all the utterances marked with laughter from the transcriptions of each corpus. Note that the dataset still contains a substantial amount of neutral speech, as laughter tends to occur at certain parts of the speech.

IH-EMO: 27k hours of collected emotional speech as described in Section 3.4.

We used four evaluation datasets as presented in Table 2.

JVNV speech-to-speech translation (S2ST): To evaluate the emotion transferability of zero-shot TTS models, we established an experimental setting based on the Japanese-to-English S2ST scenario. In the evaluation, we used a Japanese speech from the JVNV corpus [32]. The JVNV corpus includes speeches from four speakers (two males and two females) with six emotions: anger, disgust, fear, happiness, sadness, and surprise with various NVs. With its intense emotional expressions and various NVs, the JVNV dataset offers comprehensive coverage of expressive emotions, making it ideal for our emotion transfer testing. In the evaluation process, we first applied speech-to-text translation to the Japanese speech to obtain the English translations. We then used a zero-shot TTS model, using the English translations as a text prompt and the Japanese speech as an audio prompt, from which we extracted both NV and emotion embeddings. The generated speech is expected to be English-translated speech with the original speaker’s voice and emotional characteristics. We assessed the similarity of the speaker and emotion between the source Japanese speech and the translated English speech using various metrics described in the next section.

EMO-change: To test the model’s capacity for fine-grained emotional speech generation, we created the EMO-change dataset based on the RAVDESS dataset [33], which contains English emotional speech data with emotions such as calm, happy, sad, angry, fearful, surprised, and disgusted. The RAVDESS dataset includes two transcriptions: “kids are talking by the door” and “dogs are sitting by the door” with intense emotional expressions. To generate the EMO-change dataset, we randomly select two utterances with different emotions with the transcription “kids are talking by the door,” remove the silence from each, and concatenate them to create the audio prompts. During testing, the concatenated emotion change sample serves as the audio prompt, and a repeated sentence “dogs are sitting by the door dogs are sitting by the door” serves as the text prompt for the zero-shot TTS model. This setup evaluates the model’s ability to generate speech that mimics the emotional transitions in the audio prompt while following the given text prompt, and speaker characteristics.

Laughter-test: To evaluate the zero-shot TTS capability in generating laughing speech, we employed a Chinese-to-English S2ST experiment by following the protocol presented in [14]. Specifically, we used 154 Chinese utterances containing laughter⁶⁶6Data can be found at https://aka.ms/elate. from the evaluation subset of the DiariST-AliMeeting test set [34]. We applied zero-shot TTS, where we used the ground-truth English transcription as the text prompt and the Chinese speech as the audio prompt to extract both NV and emotion embeddings. The generated speech is expected to be English speech with the original speaker’s voice and laughter characteristics.

Crying-test: To evaluate the zero-shot TTS capability in generating crying speech, we further conducted a Chinese-to-English S2ST experiment using 33 Chinese crying speech samples collected from publicly available data sources. We followed the same procedure with the Laughter-test, and the generated speech is expected to be English speech with the original speaker’s voice and crying characteristics.

We utilized the following objective metrics. Among them, AutoPCP, EMO SIM, and Aro-Val SIM are closely related to the emotion controllability.⁷⁷7https://github.com/hbwu-ntu/EmoCtrlTTS-Eval

Word error rate (WER): To evaluate the intelligibility of the generated audio, we applied a Whisper-Large [35] to the generated audio and computed the WER. In all our tables, we express the WER in percentage.

Speaker SIM-o: To evaluate the speaker similarity between the generated audio and the audio prompt, we computed a cosine similarity between speaker embeddings of the two audios. We used a WavLM-large-based speaker verification model [36]⁸⁸8https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification, following previous works [37, 13].

AutoPCP: AutoPCP [38] is an utterance-level estimator to quantify the prosody similarity between two speech samples. We computed the score between the generated audio and the audio prompt. We leveraged AutoPCP_multilingual_v2⁹⁹9https://github.com/facebookresearch/seamless_communication.

Emo SIM: To evaluate the similarity of time-varying emotion states, we applied the emotion2vec model [25]3 to extract the emotion embeddings. We performed interpolation to ensure the embeddings of the audio prompt and the generated audio have the same length. We then computed the cosine similarity between these two embedding sequences for each frame and took the average to obtain the EMO SIM score.

Aro-Val SIM: As another metric for the similarity of time-varying emotion states, we computed the arousal-valence values based on [22]2 using a sliding window with a window size of 0.5 sec and a hop size of 0.25 sec. Similar to EMO SIM, we computed the cosine similarity between the audio prompt and the generated audio for every frame and took the average as the Aro-Val SIM.

We used the following subjective evaluation metrics.

SMOS: Speaker similarity mean opinion score, which is the similarity between the speaker prompt and the generated speech from 1 (not at all similar) to 5 (extremely similar).

NMOS: Naturalness MOS, which is the naturalness of the generated speech from 1 (bad) to 5 (excellent).

EMOS: Emotion MOS, which measures the similarity of emotion between the audio prompt and the generated speech from 1 (not at all similar) to 5 (extremely similar).

Table  3 presents the objective evaluation results on the JVNV and EMO-change datasets. During the evaluation, we generated speech with three different random seeds, and the average of the scores was taken to report.

Baseline analysis: For the JVNV S2ST test set, we first observed that the ELaTE (B3), trained with LL and LAUGH data, achieved superior performance over the SeamlessExpressive model [38]¹⁰¹⁰10Our experiment is based on SeamlessExpressive, supported by the Seamless Licensing Agreement. Copyright © Meta Platforms, Inc. All Rights Reserved. (B1) and the reproduced VoiceBox model (B2) across all the metrics, except for WER. The comparison between B2 and B4 highlights the effect of LAUGH data, showing that the latter improved performance in SIM-o and AutoPCP while maintaining WER, EMO SIM, and Aro-Val SIM. By replacing LAUGH with the IH-EMO data (B4 vs. B5), SIM-o and WER were improved from 0.410 to 0.479, and from 2.5% to 2.2%, respectively. However, this came at the cost of a slight degradation of AutoPCP (3.07$\rightarrow$2.96), Emo SIM (0.645$\rightarrow$0.641), and Aro-Val SIM (0.438$\rightarrow$0.397). Combining all training data (B6) gave us a balanced result. However, even after these trials, no Voicebox model achieved better AutoPCP, EmoSIM, and Aro-Val SIM compared to ELaTE. The same trends were also observed for the EMO-change test set. These results suggest that simply adding emotional data does not improve the emotion transferability of the zero-shot TTS models.

Results of EmoCtrl-TTS: From JVNV S2ST results, we first observed (P1) showed a significant improvement in AutoPCP, Emo SIM, and Aro-Val SIM compared to the baselines. By executing a longer fine-tuning (P2), EmoCtrl-TTS achieved further better SIM-o, WER, AutoPCP, and Emo SIM while almost preserving the high Aro-Val SIM. For the EMO-change test set, EmoCtro-TTS achieved the best SIM-o and Aro-Val SIM and the second-best AutoPCP. It is also noteworthy that SIM-o score was significantly improved from (B6) to (P1), which suggests the effectiveness of the use of NV and emotion embeddings not only for the emotion-related metrics but also for the speaker similarity. Note that we have trained more variants of EmoCtrl-TTS to analyze the impact of each data and each embedding, which will be discussed in Section 4.5.3 with Table 5.

We performed the subjective evaluation for selected models. We randomly picked 24 samples from JVNV S2ST data and 28 samples from EMO-change data, and asked native English testers to rate SMOS, NMOS and EMOS. For JVNV S2ST test set, each sample was judged by 9, 11, and 10 testers for SMOS, NMOS, and EMOS, respectively. For EMO-change test set, each sample was judged by 12 testers for all metrics.

The results are presented in Table  4. From the JVNV S2ST result, the Voicebox model (B6), leveraging the IH-EMO data, demonstrated significant improvements of SMOS and EMOS from the vanilla Voicebox model (B2), which illustrates the importance of the training data. Meanwhile, ELaTE (B3) excelled in NMOS, suggesting that the integration of NV features significantly boosts the naturalness of the audio by generating an appropriate NV. Finally, EmoCtrl-TTS (P2) achieved even better NMOS while keeping the high SMOS and EMOS scores.

In the EMO-change dataset, Voicebox (B6) showed significantly worse EMOS compared to ELaTE (B3) and EmoCtlr-TTS (P2). This result demonstrated that the conventional zero-shot TTS cannot mimic the time-varying emotional states presented in the audio prompt. The proposed EmoCtrl-TTS (P2) achieved best SMOS and comparable NMOS and EMOS to other top baselines, again showcasing the efficacy of the proposed approach.

Table 5 presents the results of the JVNV S2ST test set with various training data configurations.

The first four rows compare the performance of models trained by Libri-light and LAUGH data with a 0.5:0.5 ratio. By comparing the first two rows, we observe that the NV embedding $h$ significantly improved SIM-o, AutoPCP, Emo SIM, and Aro-Val SIM with the expense of degradation of WER. By introducing emotion embedding $e$, AutoPCP, EmoSIM, and Aro-Val SIM were further improved. However, it came with a cost of further degradation of WER as well as a significant drop in SIM-o.

In the 5th to 9th rows, we trained models using the Libri-light and IH-EMO data. Similar to the case with LAUGH data, we observed that the emotion embedding $e$ improved all the emotion-related metrics while keeping SIM-o and WER nearly intact. However, in contrast to the results with the LAUGH data, we observed a severe degradation in WER when the NV embedding $h$ was introduced. Upon examination, we found that the NV embedding often resulted in unwanted NV generation, such as laughter from multiple speakers.

Both observations suggest that including both NV and emotion embeddings in model training is not always beneficial. Based on these findings, we opted to use only emotion embeddings for IH-EMO and only NV embeddings for LAUGH when combining all training data.

The last four rows in Table 5 show the impact of data ratio and training steps on model performance with the combination of all training data. We first observed that the data ratio of 0.5:0.4:0.1 for Libri-light, IH-EMO, and LAUGH provided us with a better SIM-o and WER than the ratio of 0.5:0.25:0.25 while keeping the emotion-related metrics similar. By fine-tuning the model longer, further marginal improvement was obtained for all evaluation metrics.

Table  6 and  7 present a comparison between the EmoCtrl-TTS model and selected baselines on the Laughter-test and Crying-test datasets, both of which are composed of real data. Compared to the baselines, the EmoCtrl-TTS consistently achieved the best performance for Emo SIM and Aro-Val SIM while achieving either the best or the second-best Auto PCP and SIM-o. It demonstrated EmoCtrl-TTS’s robustness for the real data. On the other hand, we observe a moderate degradation of WERs.