As shown in the upper part of Figure 1, the SER model consists of a feature extraction encoder and an emotion classifier. The encoder is a CNN-RNN network, which is a popular structure in SER. Firstly, a 4-layer of 2D convolution network takes the 80-dimension log mel spectrum as input feature and outputs a 2D feature map which is flattened into a sequence of feature vectors. Then, a bidirectional GRU layer is followed and outputs the hidden state vector of the last time step as the final feature vector. The emotion classifier is a 2-layer dense network with a softmax activation for the output. The number of output units is 4 for the emotion category classification task and 2 for the arousal and valence polarity classification task.

Since the TTS and SER datasets are quite different in speakers, recording devices and recording environment, some domain adaptation technique is necessary for our SER task to reduce the distribution shift of these two datasets. The Maximum Mean Discrepancy (MMD)[20] is a kernel-based test statistic to judge whether two distributions are equal, which is widely used in domain adaptation [21, 22] for measuring the similarity of two distributions. It has also been proved to be effective for cross-domain SER in [16, 17]. Considering that the training of MMD is stable and our TTS dataset has no available emotion labels for parameter tuning,we choose the MMD method for our cross-domain SER task.

We refer to the SER dataset as the source domain $D_{s}$ and the TTS dataset as the target domain $D_{t}$. As described in [23], we minimize the following MMD loss to reduce the distribution discrepancy between the two domains.

	$\displaystyle L_{MMD}=\frac{1}{m^{2}}\Sigma_{i=1}^{m}$	$\displaystyle\Sigma_{j=1}^{m}k(\bm{s}_{i},\bm{s}_{j})+\frac{1}{n^{2}}\Sigma_{i=1}^{n}\Sigma_{j=1}^{n}k(\bm{t}_{i},\bm{t}_{j})$
		$\displaystyle-\frac{1}{mn}\Sigma_{i=1}^{m}\Sigma_{j=1}^{n}k(\bm{s}_{i},\bm{t}_{j})$		(1)

Where $\bm{s}_{i}$ and $\bm{t}_{i}$ are the output features of the encoder from $D_{s}$ and $D_{t}$ respectively; $m$ and $n$ are the number of samples of $D_{s}$ and $D_{t}$ respectively; $k(.,.)$ is the kernel function that is a linear combination of multiple RBF kernels:$k(\bm{x}_{i},\bm{x}_{j})=\Sigma_{n}\eta_{n}\exp\{-\frac{1}{2\sigma_{n}}\|\bm{x}_{i}-\bm{x}_{j}\|^{2}\}$, where $\sigma_{n}$ is the standard deviation and $\eta_{n}$ is the weight for the $n$-th RBF kernel. As for the samples from $D_{s}$, we use a weighted cross-entropy loss for training the emotion classification task.

$\displaystyle L_{CE}=-\Sigma_{i=1}^{m}{\omega_{argmax(\bm{y}_{i})}\cdot\bm{y}_{i}^{T}\cdot\log(\hat{\bm{y}_{i}})}$

(2)

Where $\omega_{k}$ is the weight of the $k$-th emotion category which is inversely proportional to the number of samples in this emotion category. Therefore, the final loss function for training SER model is:

$\displaystyle L=L_{CE}+\lambda L_{MMD}$

(3)

Where $\lambda$ is the weight of MMD loss.

As shown in the lower part of Figure 1, our emotional TTS model consists of a TTS module, reference encoder and GST module. The TTS module is a standard Tacotron2 model except that we add a post-net as in [1] to convert the mel spectrum to linear spectrum. The reference encoder and GST module are also the same as in [6] except that we add an auxiliary emotion prediction task to explicitly guide the style tokens to learn emotion-related features. For the generation of waveform, we simply use the Griffin-Lim algorithm[23] to convert the predicted linear spectrum to waveform samples, because our goal is mainly to verify the effectiveness of the proposed method rather than to generate high fidelity speech.

The original GST[6] is designed to unsupervisedly learn the styles from a reference audio, where the style learned by each token is random and uncertain, and therefore uncontrollable. In our emotional TTS task, in order to force the GST module pay more attention to learn the emotion-related styles, we explicitly add an emotion prediction task based on the style token weights, which will also been verified in our experiments to be critical for the emotional expressiveness of speech.

In this study, we conduct two emotion control methods corresponding to the two most commonly used emotion descriptions: basic emotion categories and emotion dimensions. For the emotion category, the emotion prediction task is a classifier of a single dense layer that takes the style token weight vector as input and outputs a vector of length 4 for the 4 emotion categories (neutral, happy, sad and angry). For the emotion dimensions, since the used SER dataset is originally annotated as discrete scores for emotion dimensions, we build two binary classifiers for predicting arousal and valence polarity respectively as the emotion prediction task. Meanwhile, in order to independently control these two orthogonal emotion dimensions, we split the style token weight vector into two valves, which are then fed into the arousal and valence classifiers respectively.

There are two common methods for the synthesis of GST model: selecting a reference audio with the desired style or manually specifying the weights of the style tokens. In our experiments, the quality and stability of speech synthesized from a single reference audio heavily depend on the choice of reference audio and the content of the text to be synthesized, which may be due to the fact that the GST module still does not completely disentangle the audio style and text content. Therefore, we manually specify the weight of style tokens by averaging the style token weights of an audio set that belongs to a certain kind of emotion. The audio set can be directly specified as all the utterances with the same emotion label, as used in [7], when the TTS dataset has emotion labels. However, our TTS dataset has no ground-truth emotion labels, and only the soft labels predicted from the cross-domain SER model are available. Because the cross-domain SER model is far less reliable than humans, these predicted emotion labels may contain a great number of mispredictions and we can not use all the utterances with the same soft label as the reference audio set. In order to choose a more reliable reference audio set, we propose that only the K utterances with highest posterior are selected as the reference audio set, rather than the full set of a certain emotion. Choosing these K high-confidence audios can greatly reduce the impact of the prediction error of the SER model and it is a very important choice for generating an emotional speech in our experiments. We set K=50 for all of our experiments.

In this section, we perform mean opinion score (MOS) evaluation experiment in terms of the overall quality of speech (naturalness, intelligibility and speech quality). Table 2 and Table 3 report the MOS results of the above three systems. The average score of base-4cls is higher than that of the our-4cls and our-2d models, but the p-values are much greater than the singificance leval of $\alpha=0.05$ indicating that there is no significant difference between these models. This result suggests that our proposed our-4cls and our-2d models can almost achieve as good speech quality as the baseline system. In addition, it can also be found that the MOS score of happy is greatly lower than the others for both base-4cls and our-4cls. One possible reason is that the SER model itself has lower accuracy for happy in our experiments, which will lead to great differences in the prosody consistency of the top K reference audios selected by the SER model. This also indicates that the performance of the cross-domain SER model is critical for our proposed approach.

In this section, we carry out emotion perception experiments in terms of emotional expressiveness. Specifically, for a given text, we generate a set of audios for all emotion categories and then randomly shuffle these audios. Then, the subjects are asked to choose a most likely emotion category for each audio, according to two reference audios given for each emotion. In order to verify the role of our proposed top-K scheme, we also add a scheme full-4cls that use the same checkpoint with our-4cls but chooses the full set of audios predicted by the SER model as the same emotion category as the reference audio set. Figure 2 and Figure 3 show the confusion matrices of the subjective emotion prediction results.

We can find that the average accuracies of both our-4cls (78.75%) and full-4cls (49.25%) are much higher than the base-4cls’s 36.75%. This result shows that explicitly adding an emotion prediction task based on style token weights can help the GST module to more effectively the extract emotion-related feature. Moreover, a large gap in terms of average accuracy can also be found between our-4cls and full-4cls, which proves that the proposed top-K scheme is fairly effective for the emotional expressiveness. Similarly, as shown in Figure 3, the average accuracy of arousal and valence is 91.0% and 55.5% respectively, which are greater than the random level of 50.0%. This also suggests that our proposed systems can effectively model the emotional expressiveness in speech.

Finally, we also use the t-SNE[26] algorithm to project the style token weights on the reference audio sets into the 2D space for our-4cls and base-4cls. As shown in Figure 4, the style token weights of the 4 emotion categories are clearly clustered into 4 clusters for the our-4cls model, while for the base-4cls model, except for the angry catetory, there is no obvious clustering boundary for the other three categories. This can also be considered as another possible evidience to explain the role of the auxiliary emotion prediction task.