Disentangling Style and Speaker Attributes for
TTS Style Transfer

Skerry-Ryan et al. extend the Tacotron architecture [2] by adding a reference encoder module [18, 35] that compresses the style of a variable-length audio signal into a fixed-length vector, as the reference embedding. Here, the reference encoder is to learn the embedding space of style from the speech data directly in the training process. The learned embedding, when used as a condition in synthesis, can generate speech signals with a style similar to that of the reference signal, even when the reference and target speakers are different.

Another popular single-reference speech style transfer is the GST model [20], which augments the reference encoder by utilizing a multi-head attention [36] based style token layer to extract rich style information in the training data. The extracted information is then used to control the synthesis, such as varying speed and speaking style. Similarly, it can be used for style transfer, replicating the speaking style of a single audio clip across a long-form text corpus.

Deep generative models, such as VAE [25] and GAN [37, 38], are powerful architectures which can learn complicated distributions in an unsupervised manner. Particularly, VAE, which can explicitly model the corresponding latent variables, has become one of the most popular schemes and achieved significant advancement in text generation [39], image generation [40, 41] and speech generation [12, 42]. Zhang et al. [21] introduce the VAE in the neural TTS model, to learn the latent representations of speaking style in an unsupervised manner. The learned style representations are then fed into a TTS network to control the style of the synthesized speech.

Theoretically, the above approaches can model any complex styles in a continuous latent space, so that one can control the transferred style by manipulating the latent variables or conducting a variational inference from a reference audio. However, these methods model all speech styles into one single representation, which is not versatile enough to control specific speech attributes independently. When conducting style transfer, one then has to transfer all the embedded styles, desired or not, which may not fit well with the contexts and may hurt its generalization. In addition, these approaches fail to generalize to a new domain which is unseen in training. For example, to create speech in different speaker identities and speaking style classes by using a single model, a dataset containing audio samples for each speaking style class and speaker identity is needed, and yet the model can still fail to transfer the speaking style from a new speaker with an arbitrary, unknown style.

The single-reference speech style transfer has primarily focused on the transfer of a single-style reference audio sample. Those methods are inadequate for disjoint, multi-style datasets because of their lack of domain adaptation [43] capability.

Recently, Bian et al. [22] introduce a multi-reference encoder to GST [20] and adopt an intercross training scheme, to ensure that each sub-encoder of the multi-reference encoder disentangles and controls a specific style independently. The model shows successful style transfer in a multi-style scenario. However, its intercross training scheme does not guarantee each combination of style classes is seen in training, leading to a missed opportunity to learn disentangled representations of styles and sub-optimal results on disjoint datasets. To address the above problems, Whitehill et al. [23] propose an adversarial cycle consistency training scheme to ensure the use of information from all style classes. Different from intercross training, the scheme sweeps across all combinations of style classes via paired and unpaired triplets. This provides disentanglement of multiple style classes, enabling the model to transfer style in a more faithful manner than the existing methods.

However, similar to  [22], the method of  [23] still suffers a limitation that can only transfer the style seen in training, and is inadequate to transfer the speech to a target style from a new speaker with an unknown, arbitrary style, thus narrowing down applicable scenarios for neural TTS. In addition, recording training samples in a new style (e.g., poetry style) is challenging and labor-intensive, transferring style from one dataset to another (i. e., disjoint, multi-style datasets) is appealing for TTS systems.

More recently, the first author of this paper proposes an approach to perform seen and unseen style transfer on disjoint datasets [24]. Despite the realization of seen and unseen style transfer, the method cannot distinguish different style and speaker types well, especially for unseen style transfer, resulting in sub-optional style transfer accuracy and speaker preservation. Hence, it is necessary to further enhance the performance of seen and unseen speech style transfer on disjoint, multi-style datasets.

Our approach to seen and unseen speech style transfer is built upon an encoder-decoder neural network [6, 7]. Fig. 1 illustrates our proposed framework for both seen and unseen styles on disjoint, multi-style datasets. Let $X_{s}=\{x_{s}^{(1)},\ldots,x_{s}^{(n)}\}$ be the source utterances, and $X_{t}=\{x_{t}^{(1)},\ldots,x_{t}^{(m)}\}$, the target utterances, respectively. We assume that each speech utterance $x$ can be decomposed into style representation, $z\in Z$, and speaker representation, $r\in R$. Each source utterance, $x_{s}^{(i)}\in X_{s}$, has its individual style, $z_{s}^{(i)}$, and the target utterance, $x_{t}^{(j)}\in X_{t}$, has the style, $z_{t}^{(j)}$. We use style encoder and speaker encoder, $E_{z}(x)$ and $E_{r}(x)$, to learn discriminative style representation $z$ and speaker representation $r$ of an utterance $x$, respectively: $z_{s}^{(i)}=E_{z}(x_{s}^{(i)})$, $r_{s}^{(i)}=E_{r}(x_{s}^{(i)})$, $z_{t}^{(j)}=E_{z}(x_{t}^{(j)})$, $r_{t}^{(j)}=E_{r}(x_{t}^{(j)})$. In this paper, we use Tacotron 2 [1] as the synthesizer $T$, which converts the combined encoder states (including style representations, speaker representations, and text encoder states) to generate a target Mel spectrogram with the target style and the target speaker’s timbre. The Tacotron 2 is composed of a text encoder and a decoder with attention, whose details will be described in Section III-D. We then employ the LPCNet neural vocoder [44, 45] to reconstruct the final speech waveforms from the generated Mel spectrograms, and the details of each individual component in LPCNet vocoder will be described in Section IV-B. Our network is trained with six objective loss functions, including: reconstruction ($\mathcal{L}_{rec}$), adversarial ($\mathcal{L}_{adv}$), style distortion ($\mathcal{L}_{dis}$), cycle consistency ($\mathcal{L}_{cyc}$), style classification ($\mathcal{L}_{stycls}$) and speaker classification ($\mathcal{L}_{spkcls}$) losses. Next, we introduce the learning of latent style space, speaker encoder network, synthesizer network and six objective losses.

The style encoder $E_{z}(x)$ constructs a latent style space, and then outputs a sampled style representation $z$ to the synthesizer $T$ to guide style generation. As discussed above, the generated style space via VAE network [25], usually a Gaussian distribution, may not be expressive enough to transfer the style effectively. In this paper, we resort to the IAF [27, 46], a potent technique to construct sophisticated distributions, to learn discriminative and expressive style representations. In other words, IAF can map a simple initial variable to a more complex one by applying a chain of cascaded invertible transformations. As shown in Fig. 2, we let a reference encoder network output $\mu_{0}$ and $\sigma_{0}$, in addition to an extra output $h$, which serves as an additional input to each subsequent step in the flow. We draw a random sample $\epsilon\sim\mathcal{N}(0,I)$, and initialize the chain with:

Afterward, the initial variable $z_{0}$ along with hidden output $h$ is provided to $k$ steps of inverse autoregressive transformations to obtain flexible posterior probability distribution with latent variable $z_{k}$:

In each step of the flow transformations, we adopt an autoregressive neural network with inputs $z_{k-1}$ and $h$, and outputs $\mu_{k}$ and $\sigma_{k}$. And amortization is performed by using $h$ as input to autoregressive networks of flow transformations [47]. These autoregressive transformations are invertible if $\sigma_{i}\textgreater 0$ condition is satisfied for $i^{th}$ value of $D$ dimension. Autoregressive structure of flow allows simple computation of the Jacobian determinant of each transformation as a change in global posterior probability density of reference encoder network denoted as $\log q(z_{K}|x)$, where $z_{K}$ is output of the last flow step. Eq.(3) provides a tractable change of the probability density, and its detailed derivation can be found in  [27]. The flexibility of the distribution of the final iteration $z_{K}$ , and its ability to closely fit to the true posterior, increases with the expressivity of the autoregressive models and the depth of the chain. See Fig. 3 for an illustration of flow transformations.

Different from [24], we then plug a style classifier to the style encoder shown in Fig. 2, which helps to learn more discriminative style embeddings for better discriminations of different styles. In detail, the classifier includes three fully connected (FC) layers, all with ReLU activation, and a softmax layer to output the probability of different style types, i. e., reading, broadcasting, talking, story-telling, customer-service, poetry, and game styles. We use the output of the third FC layer as the style representation, $z_{s}$ or $z_{t}$. In this way, we can obtain more discriminative and expressive style embeddings to guide style generation.

In previous works, speaker embeddings are usually obtained by using a speaker recognition model [31, 48] trained on the voxceleb corpus available in the Kaldi toolkit [49, 50] or a text-independent speaker verification task [32, 51], where the speaker extraction models need to be pre-trained. In our proposed network, we add a well-designed speaker encoder, $E_{r}(x)$, to learn the speaker representations. Different from  [33], we propose to jointly train the speaker encoder network $E_{r}(x)$ with the rest of the neural TTS modules. We expect that the use of the speaker encoder can learn discriminative speaker embeddings to capture the characteristics of individual speakers, even the speaker is not seen in the training data.

Fig. 4 shows the architecture of speaker encoder in our approach, which maps a sequence of acoustic features of a speech utterance, into a fixed-dimensional embedding vector. First, we feed the features extracted from an utterance $x$ into a 3-layer LSTM network. Two FC layers both with ReLU activation are connected to the last LSTM layer as an additional transformation of the last frame response of the network. Different from [24], a speaker classifier is added into speaker encoder to learn more discriminative speaker representations. As shown in Fig. 4, the classifier has the same structure as the style classifier, consisting of three FC layers, all with ReLU activation, and a softmax layer to output the probability of different speaker. Similarly, the output of the third FC layer is used as the speaker representation $r_{s}$ or $r_{t}$, which is then used as a condition for the synthesizer $T$ to guide speaker identity generation.

In this paper, we leverage Tacotron 2 as the synthesizer $T$. Tacotron 2 is an attention-based sequence-to-sequence network [6, 7, 52], composed of a text encoder and a decoder with attention, which generates a Mel spectrogram as a function of an input text sequence and conditions the signal generated by the auxiliary encoder networks (e.g., style encoder and speaker encoder). It closely follows the network architecture of Tacotron 2 [1]. Input phonemes are represented using a learned 512-dim phoneme embedding, which is passes through a stack of three convolutional layers each containing 512 filters with shape $5\times 1$, followed by a bidirectional LSTM of 256 units for each direction. In this work, to condition the output on additional attribute representations (e.g., style representations and speaker representations), the resulting text encodings are concatenated with the generated style embeddings $z$ from the style encoder and the extracted speaker embeddings $r$ from the speaker encoder, and then are accessed by the decoder through a location sensitive attention mechanism [53], which takes attention history into account when computing a normalized weight vector for aggregation.

The base Tacotron 2 autoregressive decoder network takes the attention-aggregated text encoding, and the bottlenecked previous frame (processed by a pre-net comprised of two FC layers of 256 units) at each step as input. The decoder input is then passed through a stack of two uni-directional LSTM layers with 1024 units. The output from the stacked LSTM is concatenated with the new decoder input (as a residual connection [54]), and linearly projected to predict the Mel spectrogram of the current frame, as well as the end-of-sentence token. Finally, the predicted spectrogram frames are passed to a post-net, which predicts a residual that is added to the initial decoded sequence of spectrogram frames, to predict the spectrograms by minimizing the overall mean squared errors.

For the synthesizer $T$, we form a reconstruction loss $\mathcal{L}_{rec}$ to encourage the utterance from $T$, given style representation $z$ and speaker representation $r$ of an utterance $x$, to reconstruct $x$ itself:

where $\theta$ denotes the parameter of the corresponding module. In this way, we can maintain reconstruction fidelity of an utterance with the synthesizer $T$.

Besides, for a sample $x_{s}$, we enforce the decoded sequence, given its speaker representation $r_{s}$ and target style representation $z_{t}$, should be in the target domain $X_{t}$. Following GAN [37, 55], we introduce an adversarial loss $\mathcal{L}_{adv}$ to be minimized in decoding and adopt a discriminator $D$, as shown in Fig. 1, to distinguish from $T(r_{s},z_{t})$ and $T(r_{t},z_{t})$. The task of the synthesizer is to fool the discriminator. Specifically, the adversarial loss $\mathcal{L}_{adv}$ is defined as

But, for a sample $x_{s}\in X_{s}$, its $z_{s}$ can be an arbitrary value that minimizes the above reconstruction loss and adversarial loss, which may not necessarily capture the utterance style. This will affect the speaker representation, which is critical for representing the speaker, and it should be invariant against the transferred style. To address the issue, we introduce a style distortion loss $\mathcal{L}_{dis}$ to constrain style representation of a source utterance to be closer to the target style representation. As shown in Fig. 1, a discriminator, $D_{s}$, is first trained to predict whether a given utterance $x$ has the target style with an output probability, $p_{Ds}(x\in X_{t})$. When learning the style representation $z_{s}$, we then force the distortion between this style representation $z_{s}$ and target style representation $z_{t}$ to be consistent with the output probability of $D_{s}$. Here, we use the $L_{2}$ norm to measure the style distortion, $d(z_{s},z_{t})=\|z_{s}-z_{t}\|_{2}$, and make the style distortion positively correlated with $1-p_{Ds}(x_{s}\in X_{t})$. To incorporate this idea into our model, we model it with a standard normal distribution to evaluate the style distortion loss. Intuitively, when an utterance $x_{s}$ have a large output probability $p_{Ds}(x_{s}\in X_{t})$, our model can result in a small style distortion. That is, $z_{s}$ will be closer to $z_{t}$, and the style distortion loss $\mathcal{L}_{dis}$ is:

where $D_{s}$ is a pre-trained model trained with a portion of the training data. Here, if we integrate $D_{s}$ into our training, we may start with a $D_{s}$ with a low accuracy, and then our model is inclined to optimize a wrong style distortion loss for many epochs and gets stuck into a poor local optimum.

Unfortunately, the style distortion loss $\mathcal{L}_{dis}$ can only constrain the generated utterance to be aligned with the target style, but cannot guarantee to keep the speaker identity of a source utterance intact. To address this issue, we introduce a cycle consistency loss [56, 57], $\mathcal{L}_{cyc}$, to our model shown in Fig. 1, which requires a transferred utterance to preserve the speaker identity of its source utterance, and enables the recovery of the source utterance in a cyclic manner. The cycle consistency loss $\mathcal{L}_{cyc}$ is defined as

where $\widetilde{x}_{s}$ is the transferred utterance from a source sample $x_{s}$ and has the target style $z_{t}$. We encode the $\widetilde{x}_{s}$ with the speaker encoder $E_{r}(\widetilde{x}_{s})$ to obtain its speaker representation $\widetilde{r}_{s}$, which is then combined with its source style $z_{s}$ for decoding. Here, we expect that the source utterance can be generated with a high probability. For a target sample $x_{t}$, although we do not aim at changing its style in our model, similar to $x_{s}$, we still calculate its cycle consistency loss for additional regularization.

In addition, to learn more discriminative style and speaker representations that can distinguish different styles and capture the characteristics of different speakers, we introduce a style classification loss $\mathcal{L}_{stycls}$ and a speaker classification loss $\mathcal{L}_{spkcls}$, as shown in Fig. 1. In this paper, we let the style classifier and the speaker classifier have the same architecture, and train the two classifiers with the cross-entropy loss, or the softmax loss. The details of the two classifiers will be described in Section IV-B. Specially, the softmax loss $\mathcal{L}_{softmax}$ (i. e., the style classification loss $\mathcal{L}_{stycls}$ and the speaker classification loss $\mathcal{L}_{spkcls}$) is formulated as

where $i$ refers to the style/speaker index of the input style/speaker embedding and $j$ refers to the style/speaker index upon which the classification occurs. The $y_{i,j}$ is the ground-truth style/speaker class for the $i$-th embedding in the $j$-th style/speaker dimension, and $\hat{y}_{i,j}$ is the predicted style/speaker class. For $i=j$, the two classifiers encourage a style/speaker embedding to contain the correct information of the $i$-th style/speaker. For $i\neq j$, the two classifiers discourage the use of information about the other style/speaker.

By joint-training to minimize the weighted six specific objectives, or the overall objective function $\mathcal{L}$

where $\alpha/\beta/\gamma/\lambda/\kappa/\omega$ are coefficients of the reconstruction term $\mathcal{L}_{rec}$ in Eq.(4), the adversarial loss $\mathcal{L}_{adv}$ in Eq.(5), the style distortion loss $\mathcal{L}_{dis}$ in Eq.(6), the cycle consistency loss $\mathcal{L}_{cyc}$ in Eq.(7), the style classification loss $\mathcal{L}_{stycls}$ and the speaker classification loss $\mathcal{L}_{spkcls}$ in Eq.(8), respectively.

We have carried out experiments to evaluate the performance of the proposed approach. We focus on disjoint, multi-style datasets, where datasets of different styles are recorded, and each style is recorded by one speaker with multiple utterances. Here, an internal Chinese corpus is used in experiments, which is divided into source data and target data. Source data contains examples of four styles: reading (standard reading speech), broadcasting (news broadcasting speech), talking (conversational speech) and story-telling (audiobook speech) styles, from four different speakers whereas target data contains samples of three styles: customer-service (spontaneous speech with fast speed), poetry (classical Chinese poetry speech with rich prosody variations), and game (exaggerated speech for role dubbing in the game) styles, from three other different speakers. This represents a minimalistic scenario of the disjoint, multi-style datasets: a single model must be able to properly transfer an arbitrary or unknown style to target style while keeping a minimal perceived change in the speaker’s timbre. The corpus contains 22,212 samples ($\sim$ 25 hours) and each style contains 4,000 samples except for poetry and game styles. There are 698 samples in poetry style while 1514 samples in game style. For samples of each style, we respectively use 90% as the training set, 5% as the validation set and the rest 5% for the test. We remove long silence ($\textgreater$ 0.1 sec) at the beginning and ending of each utterance. 80-dim Mel spectrogram is extracted as target speech representations with a Hanning window of 50 ms and 12.5 ms frame shift. Phoneme sequences are used as the input, which are obtained by text normalization and G2P pre-processing modules.

For all different systems in our experiments, we train $\sim$ 220k steps with a single Nvidia Tesla P40 GPU, and a batch size is 32. The models trained and tested in our experiments include:

• •

  GST: we introduce “global style tokens” (GST) [20] into Tacotron 2 [1] to uncover expressive factors of variation in speaking style to perform the style transfer task, and make a fair comparison;

• •

  VAE: we incorporate VAE [21] into Tacotron 2 [1] to learn the latent representation of speaking style to guide the style in synthesizing speech;

• •

  MRF-IT: we augment a multi-reference encoder into GST-Tacotron 2 [22] to model multiple styles simultaneously, and adopt intercross training to extract and separate different classes of speech styles, thus achieving the transfer for desired speech styles;

• •

  MRF-ACC: we adopt an adversarial cycle consistency training scheme for multi-reference neural TTS stylization [23] to ensure the use of information from all style classes, thus performing multi-reference style transfer on disjoint datasets;

• •

  Proposed model: we introduce an IAF technique [27] to improve variational inference and learn expressive style representations, and develop a joint-training speaker encoder network to obtain discriminative speaker representations. Six loss functions with different purposes in network training are used together for enhancing the performance of seen and unseen style transfer on disjoint, multi-style datasets.

The style encoder contains a reference encoder, an IAF flow, and a style classifier, which forms a more discriminative and expressive latent style representation. Similar to  [18, 21], the reference encoder consists of a stack of six 2-D convolutional layers cascaded with one unidirectional 128-unit GRU layer. In each IAF step, we use the structure proposed in  [58] as the autoregressive neural network. As shown in Fig. 2, the style classifier contains three FC layers, all with ReLU activation, and a softmax output layer. We use the output of the third FC layer in the style classifier as the style embeddings. For the speaker encoder, we use a 3-layer LSTM with the projection operations followed by a speaker classifier shown in Fig. 4. In this paper, we let the style classifier and the speaker classifiers have the same architecture, to learn discriminative style representations and speaker embeddings that can make both style and speaker to be more distinguishable.

In our model, we adopt Tacotron 2 [1] as the synthesizer, which takes the concatenation of the speaker and style representations as the initial hidden state. As for the discriminator $D$, we follow the architecture of the discriminator in  [59]. The pre-trained discriminator $D_{s}$ used in the style distortion loss has the same structure as the style encoder followed by a sigmoid output layer. As for the objective function in Eq.(9), we empirically set the coefficients $\alpha$, $\beta$, $\lambda$, $\kappa$ and $\omega$ to 1.0, and $\gamma$ to 5.0, respectively. The six coefficients ($\alpha/\beta/\gamma/\lambda/\kappa/\omega$) are preset weights for balancing the different loss terms ($\mathcal{L}_{rec}$/$\mathcal{L}_{adv}$/$\mathcal{L}_{dis}$/$\mathcal{L}_{cyc}$/$\mathcal{L}_{stycls}$/$\mathcal{L}_{spkcls}$).

In this paper, we use LPCNet, a variant of WaveRNN [60], as a neural vocoder to convert the rendered Mel spectrograms by the synthesizer network into time speech waveforms. The architecture is the same as that described in  [45], composed of a sample rate network and a frame rate network. Different from  [45], we adopt the Mel spectrogram as the input, rather than the Bark-scale cepstral coefficients [61] and pitch parameters, thus performing a direct connection with the synthesizer network. The training pipeline of our LPCNet is the same as that of  [45]. Here, we use the same LPCNet for fair comparisons across all different systems.

The performance of seen and unseen speech style transfer is evaluated in speech naturalness, style similarity, and speaker similarity. We conduct Mean Opinion Score (MOS) and preference listening tests (ABX) of speech naturalness to evaluate the reconstruction performance of different experimental systems. An ABX test of style similarity is also conducted to assess the style conversion performance, where subjects are asked to choose which speech sample (A or B) sounds closer to the target style (X) in terms of style. We further conduct a Comparative Mean Opinion Score (CMOS) test of speaker similarity to evaluate how well the transferred speech matches that of the source speaker. For each system, we randomly select the reading style as the seen style and make three experiments: from Reading style to Customer-service style (R2C), Reading style to Poetry style (R2P), and Reading style to Game style (R2G). We randomly choose an unique Taiwanese-reading style from a new female speaker as the unseen style, and conduct three tests from Taiwanese-reading style to Customer-service style (TR2C), Taiwanese-reading style to Poetry style (TR2P), and Taiwanese-reading style to Game style (TR2G) to assess the performance of unseen style transfer.

In addition, style classification accuracy and visualization of style embedding space are adopted to further evaluate the style similarity. Speaker classification accuracy and cosine similarity are calculated to measure the speaker similarity objectively.

We use the seen and unseen style evaluation sets, which respectively contain 20 sentences in each style, randomly selected from the test set, to compare the performance of all models in speech naturalness with the MOS and ABX listening tests¹¹1Samples can be found at https://xiaochunan.github.io/disentangling/index.html. Table I summarizes the results of MOS from different models, where 15 subjects are requested to carefully listen and evaluate with rating scores from 1 to 5 in 0.5 point increments. It can be seen that our proposed approach outperforms the reference models on both seen and unseen style transfer tasks. The performance of the proposed model on unseen style transfer is much better than other models. Specially, most of subjects find that there are more unintelligible parts in an utterance synthesized by using the GST, VAE, MRF-IT and MRF-ACC models. This phenomenon is more obvious on unseen style transfer task. The results show a better generalization of the proposed model on the unseen style transfer. Seen style transfer performs with better speech naturalness than the unseen style transfer. Partially due to the small size of the speech data set in poetry and game styles, hence their MOS scores are slightly lower than that of the customer-service style.

Table II shows the results of ABX tests between the proposed model and all reference systems for speech naturalness on seen and unseen style transfer, where the same 15 subjects are asked to choose the speech samples with higher speech naturalness. We can observe that our proposed framework consistently outperforms four other systems of the prior art, which is also consistent with the MOS results. These observations verify the effectiveness of the proposed approach in terms of speech naturalness.

We also calculate the character error rate (CER) of all models on the same evaluation sets via a Conformer based ASR system [62] to evaluate speech intelligibility objectively. As shown in Table III, we show that our proposed model is superior to the four reference models, i.e., GST, VAE, MRF-IT and MRF-ACC, on both seen and unseen style transfer. This verifies objectively the effectiveness of our proposed approach in terms of speech intelligibility.

We first investigate the style classification accuracy via a speech style classifier, which is independently trained using the seven style samples (i. e., reading, broadcasting, talking, story-telling, customer-service, poetry and game styles) from the training data, to objectively evaluate the style conversion performance. The classifier has the same architecture as the style encoder in our model. Its final validation accuracy is 95.1%. We then synthesize the transferred speech adopting the same evaluation sets, and predict their style labels using the trained classifier. Table IV shows the results of style classification for seen and unseen style transfer using different models. Our proposed model achieves greater accuracy on both seen and unseen style transfer tasks, showing its ability to transfer style in synthesized samples. However, the reference models perform poorly on style classification. Specially, for the unseen style transfer (i. e., TR2C, TR2P, and TR2G), many transferred samples from the GST, VAE, and MRF-IT models are difficult to distinguish as their samples after the transfer always have poor style expression, demonstrating a poor unseen style transfer performance. The best reference system, MRF-ACC, still obtains a much lower rate of seen and unseen style transfer than the proposed model. These results validate the effectiveness of the proposed approach for seen and unseen style transfer.

We further visualize the style embedding space using t-SNE [63] to evaluate the style similarity, where different colors correspond to different styles. The style embeddings are respectively extracted from 20 real target speech and 20 transferred utterances for each style. As shown in Fig. 5, our proposed model produces much closer and more separable clusters than the other four reference systems. Particularly, in our model, different styles are well separated from each other in the style embedding space, and transferred utterances tend to lie close to real target speech from the same style in the embedding space. However, the transferred utterances are still easily distinguishable from the real human speech as demonstrated by the t-SNE visualization, where utterances from each transferred style form a nice cluster which is adjacent to a cluster of real target utterances of the corresponding style. For the GST, VAE, MRF-IT, and MRF-ACC models, real sentences form distinct clusters, but transferred utterances cannot be well separated by style, and appear distant from the corresponding real target style utterances. These observations suggest that our approach can learn effective and discriminative representations in the style space, thus enhancing the performance of both seen and unseen style transfer.

We also conduct an ABX test of style similarity between the proposed model and each reference system to subjectively assess the style conversion performance, where the same 15 listeners are asked to choose the speech samples which sound closer to the target style in terms of style expression. Here, the listeners follow the instructions: “You should not judge the content, grammar, audio quality, or speaker identity of the sentences; instead, just focus on the similarity of the style to one another.”. Higher preference means more style similarity is perceived. The results of style similarity on seen and unseen style transfer are shown in Table V, where the listeners give higher preference to the proposed system, showing the proposed approach improves the performance of seen and unseen style transfer. For the unseen style transfer, we find that the GST, VAE and MRF-IT models, in most cases, fail to transfer unseen style of the Taiwanese-reading style to the target style of customer-service style or poetry style or game style. The MRF-ACC system, the best of all references, is still significantly inferior to the proposed model in the style similarity test. These results demonstrate the effectiveness of our proposed approach for both seen and unseen style transfer.

To evaluate how well the transferred speech matches that of the source speaker’s timbre, we respectively conduct CMOS tests between the proposed model and each reference system (i. e., proposed vs GST, proposed vs VAE, proposed vs MRF-IT, and proposed vs MRF-ACC) by using the same evaluation sets. Here, CMOS is used to make a comparison in speaker similarity between two voices, ranging from -3 to 3. Generally, a positive score indicates the reference voice is better than the proposed voice in speaker similarity, worse for a negative score. The same 15 listeners are asked to just focus on the speaker identity of utterances while not judging the content, grammar, audio quality, or style information of the utterances. Table VI reports CMOS results for speaker similarity, where score of the proposed model is fixed to 0 on both seen and unseen style transfer tasks. We can observe that each reference system obtains negative CMOS scores, demonstrating that they perform worse in speaker similarity than the proposed model. This is also evident that our proposed approach delivers better speaker similarity performance than all reference systems, on both seen and unseen style transfer tasks.

We also adopt the cosine similarity to calculate the similarity between the speaker embedding of a transferred sample and the speaker embedding of a randomly selected ground truth utterance from the same speaker to objectively measure the speaker similarity. The results are shown in Table VII, where we can find that the proposed model delivers a higher speaker similarity than the other reference systems, reflecting that our approach has learned a more discriminative speaker representation. Specially, our proposed model obtains a greater cosine similarity, which demonstrates that the transferred utterances via our method are nearly always even closer to the speaker identity of source speaker. On the unseen style transfer of TR2C, TR2P and TR2G, the GST, VAE and MRF-IT models are not capable of keeping the speaker’s timbre, resulting in lower speaker similarity. The best reference system, MRF-ACC, still performs significantly worse than the proposed approach.

We further train a speaker verification model to calculate speaker classification accuracy for the same evaluation sets to objectively assess the speaker similarity. In this experiments, we follow the architecture of speaker verification model in  [64], which consists of a small size Thin ResNet-34 [65] with SE-block [66]. Table VIII summarizes the results of speaker classification accuracy on both seen and unseen style transfer tasks. Our proposed approach obtains a higher speaker classification accuracy than the other reference systems. Specially, on the unseen style transfer task (i. e., TR2C, TR2P, and TR2G), the proposed model achieves a much higher accuracy than the GST, VAE, and MRF-IT models, showing its ability to retain speaker identity in the transferred samples. The MRF-ACC system, the best reference model, is still significantly inferior to the proposed method in the speaker classification test. From these results we can conclude that our proposed model can preserve speaker identity and generate speech that resembles the source speaker, but still not as good as the real source speaker.

It is meaningful to investigate the effect of not using IAF in learning style representations for speech style transfer. In this experiments, we remove the IAF structure in the style encoder and test the performance of its seen and unseen style transfer, which is denoted as P-woIAF. Adopting the same evaluation sets, we conduct an ABX test of style similarity between the P-woIAF and the proposed approach to subjectively compare their style conversion performance. Here the same 15 listeners are also asked to choose the speech samples that sound closer to the target style in terms of style expression. Fig. 6 presents the ABX results for style similarity on both seen and unseen style transfer tasks. We can find that the P-woIAF model obtains lower preference, showing lower style similarity is perceived. However, when we plug the IAF to the style encoder, i. e., our proposed model, the listeners give higher preference to the proposed approach. The performance gain is essentially contributed by the IAF scheme in learning expressive style representations. In summary, we should construct sophisticated distributions with IAF to learn discriminative style representations, for possible improved performance in seen and unseen style transfer.

As mentioned in Section I, our proposed model can be considered as a variant of our previous model [24], which also uses style encoder and speaker encoder but in a different way. Here, we denote the previous model as the Pre-model^[24]. Specially, in the Pre-model^[24], no style classifier and speaker classifier are used in the corresponding style and speaker encoders, hence no constraints of style and speaker classification losses are imposed in the model training process. The Pre-model^[24] is therefore unable to distinguish the different styles and speakers in a direct manner. In the experiments, we compare the new proposed approach and the Pre-model^[24] with ABX subjective tests of style similarity and speaker similarity. The ABX results are shown in Fig. 7. We show that listeners give higher preference to the new proposed model which has improved the corresponding style and speaker similarities. Similar to Section V-B, we also show the scatter plots of the style embedding space for the Pre-model^[24] and the proposed model. As shown in Fig. 8, our proposed model produces more compact and highly separable clusters than the Pre-model^[24] on both seen and unseen style transfer. All evidence shows that the style and speaker classifiers proposed in this paper contribute to better style and speaker similarities for both seen and unseen style transfer, where the different styles and speakers are more distinguishable. The results confirm that the style and speaker classifiers are advantageous to make the transfer process more discriminative in styles and speakers.

In this paper, we conduct ablation studies on seen and unseen speech style transfer to validate the effectiveness of different objectives. Specifically, we respectively conduct two subjective tests: a style classification test and a speaker classification test, on the same evaluation sets where the same 15 listeners are asked to focus on the style or speaker identity. The style classification accuracy and the speaker classification accuracy are shown in Table IX and Table X, respectively. For each testing sample, we let the listeners select one style from eight style categories (i.e., reading, broadcasting, talking, story-telling, customer-service, poetry, game, and Taiwanese-reading styles), or select one speaker from eight speaker categories (i.e., seven speakers seen in training and a new speaker unseen in training). We can notice that the addition of the adversarial loss ($\mathcal{L}_{adv}$), the style distortion loss ($\mathcal{L}_{dis}$), the cycle consistency loss ($\mathcal{L}_{cyc}$), and the style and speaker classification losses ($\mathcal{L}_{stycls}$ and $\mathcal{L}_{spkcls}$) can bring substantial gain for the style classification accuracy and the speaker classification accuracy. The combination of all losses achieves the best accuracy, which outperforms the Pre-model^[24] model (i. e., without the $\mathcal{L}_{stycls}$ and $\mathcal{L}_{spkcls}$) by a large margin. In addition, the listeners find that it is much easier to distinguish the category of each style and each speaker after the addition of $\mathcal{L}_{stycls}$ and $\mathcal{L}_{spkcls}$.