EMOTIONAL VOICE CONVERSION WITH CYCLE-CONSISTENT
ADVERSARIAL NETWORK

Human speech is a complex signal that contains rich information, which includes linguistic information, para- and non-linguistic information. While linguistic information and para-linguistic information controlled by the speaker help make the expression convey information precisely, non-linguistic information such as emotion accompanied with speech plays an important role in human social interaction. Compared with Voice Conversion (VC) [1, 2, 3, 4], which is a technique to convert one speaker’s voice to sound like that of another, emotional Voice Conversion, or emotional VC, is a technique of converting speech from one emotion state into another one, keeping the basic linguistic information and speaker identity.

Most of the VC approaches focus more on the conversion of short-time spectral features, but less on the conversion of the prosody features such as F0 [5, 6, 7, 8, 3]. In these works, F0 features are usually converted by a simple logarithmic Gaussian (LG) normalized transform. For emotional VC, however, parametrically modeling the prosody features is important, since the prosody plays an important role in conveying various types of non-linguistic information, such as intention, attitude and mood, which represent the emotions of the speaker [9]. Recently, there has been tremendous active research in modeling prosodic features of speech for emotional VC, most of which involve modeling two prosodic elements, namely the F0 contour and energy contour. A Gaussian mixture model (GMM) and a classification regression tree model were adopted to model the F0 contour conversion from neutral speech to emotional speech in [10]. A system for transforming the emotion in speech was built in [11], where the F0 contour was modeled and generated by context-sensitive syllable-based HMMs, the duration was transformed using phone-based relative decision trees, and the spectrum was converted using a GMM-based or a codebook selection approach.

Prosody is inherently supra-segmental and hierarchical in nature, of which the conversion is affected by both short- and long-term dependencies, such as the sequence of segments, syllables, words within an utterance as well as lexical and syntactic systems of a language [12, 13, 14, 15, 16]. There have been many attempts to model prosodic characteristics in multiple temporal levels, such as the phone, syllable and phrase levels [17, 18, 19, 20]. Continuous wavelet transform (CWT) can effectively model F0 in different temporal scales and significantly improve speech synthesis performance [21]. The CWT methods were also adopted for emotional VC. CWT was adopted for F0 modeling within the non-negative matrix factorization (NMF) model [22], and for F0 and energy contour modeling within a deep bidirectional LSTM (DBLSTM) model [23]. Using CWT method to decompose the F0 in different scales has also been explored in [9, 24], where neural networks (NNs) or deep belief networks (DBNs) were adopted.

While previous work has shown the efficacy of using GMMs, DBLSTMs, NNs and DBNs to model the feature mapping for the spectral and the prosodic features, they all need parallel data and parallel training, which means the source and target data should have parallel scripts and a dynamic time warping (DTW) method is used to temporally align the source and target features before training the models. Parallel training data is more difficult to collect than non-parallel data in many cases. Besides, the use of DTW may introduce alignment errors, which degrades VC performance. Moreover, previous emotional VC approaches often define a minimum generation loss as the objective function, such as L1 or L2 loss. One of the issues using a minimum generation loss is an over-smoothing effect often observed in the generated speech parameters. Since this loss may also be inconsistent with human’s perception of speech, directly optimizing the model parameters using a minimum generation loss may not generate speech that sounds natural to human. Generative adversarial networks (GANs) [25] have been incorporated into TTS and VC systems [26], where it is found that GAN models are capable of generating more natural spectral parameters and F0 than conventional minimum generation error training algorithm regardless of its hyper-parameter settings. Since any utterance spoken by a speaker with the source or target emotional state can be used as training sample, if a non-parallel emotional VC model can achieve comparable performance with the parallel counterparts, it will be more flexible, more practical and more valuable than parallel emotional VC systems. The recently emerged cycle-consistent generative adversarial network (CycleGAN) [27], which belongs to the large family of GAN models, provides a potential way to achieve non-parallel emotional VC. The CycleGAN was originally designed to transform styles in images, where the styles of the images are translated while the textures remain unchanged. CycleGAN models have been used successfully for developing non-parallel VC systems [28, 29].

In this paper, we investigate the efficacy of using CycleGAN models for emotional VC tasks. Emotional VC is similar to image style transformation, where we can regard the underlying linguistic information as analogous to image content and the accompanying emotion as analogous to image style. Rather than attempting to learn a mapping between parallel training data using a frame-to-frame minimum generation loss, in this paper, the CycleGAN uses two discriminators and one classifier to guide the learning process–the discriminators aim to differentiate between natural and converted speech and the classifier aims to classify the underlying emotion from the natural and converted speech. The spectral features, F0 contour and energy contour are simultaneously converted by the CycleGAN model. We utilize CWT or logarithmic representation of the F0 and energy features. Although the training data we use is parallel, a non-parallel training process is adopted to learn the CycleGAN model, which means that source and target features are randomly paired during training, without any temporal alignment process. The objective and subjective evaluation results confirm the effectiveness of using CycleGAN models for emotional VC. The advantages offered by the CycleGAN model include (i)utilizing GAN loss instead of minimum generation loss, (ii)getting rid of source-target alignment errors and (iii) flexible non-parallel training, etc. The non-parallel training for a CycleGAN indicates its potential for non-parallel emotional VC.

The rest of this paper is organized as follows: Section 2 introduces the CycleGAN model for emotional VC and Section 3 describes the details of implementation. Section 4 gives the experimental setups and evaluations. Conclusions are drawn in Section 5.

The CycleGAN model consists of two generators ($G_{AB}$ and $G_{BA}$), two discriminators ($D_{A}$ and $D_{B}$) and one emotion classifier (C), as shown in Fig. 1, where we denote spectral and prosodic features in the domain of emotion $A$ as $S_{A}$, spectral and prosodic features in the domain of emotion $B$ as $S_{B}$, respectively. $S_{AB}$ is the converted spectral and prosodic features from emotion $A$ to emotion $B$ by the generator $G_{AB}$, while $S_{ABA}$ is the features converted back to emotion $A$ by the generator $G_{BA}$ from $S_{AB}$. To effectively learn parameters of the generators, discriminators and classifier, several losses are defined as follows.

Adversarial Loss: Generator $G_{AB}$ serves as a mapping function from emotion domain $A$ to emotion domain $B$, while generator $G_{BA}$ do the opposite, serving as a mapping function from emotion domain $B$ to emotion domain $A$. The discriminators, $D_{A}$ and $D_{B}$, aim to distinguish between genuine and converted spectral and prosodic features, i.e., discriminator $D_{A}$ distinguishes between $S_{A}$ and $S_{BA}$, and discriminator $D_{B}$ distinguishes between $S_{B}$ and $S_{AB}$. To this end, an adversarial loss, which measures how distinguishable the converted features $S_{AB}$ from the genuine target domain features $S_{B}$, is defined as

	$\displaystyle\mathcal{L}_{adv}($	$\displaystyle G_{AB},D_{B})=\mathbb{E}_{S_{B}\sim p_{data}(S_{B})}[\log D_{B}(S_{B})]$		(1)
		$\displaystyle+\mathbb{E}_{S_{A}\sim p_{data}(S_{A})}[\log(1-D_{B}(G_{AB}(S_{A}))].$

The adversarial loss distinguishing the converted features $S_{BA}$ from the genuine source domain features $S_{A}$ has a similar formulation.

Cycle Consistency Loss: The adversarial loss makes $S_{A}$ and $S_{BA}$ or $S_{B}$ and $S_{AB}$ as similar as possible while the cycle consistency loss guarantees that an input $S_{A}$ can retain its original form after passing through the two generators $G_{AB}$ and $G_{BA}$. Using the notation in Figure 1, $S_{ABA}$, which equals to $G_{BA}(G_{AB}(S_{A}))$, should not diverge too much from $S_{A}$. This is very important for emotional voice conversion, since we do not want to change the linguistic and speaker information during the conversion process. The cycle consistency loss is defined as

	$\displaystyle\mathcal{L}_{cyc}($	$\displaystyle G_{AB},G_{BA})$		(2)
		$\displaystyle=\mathbb{E}_{S_{A}\sim p_{data}(S_{A})}[||G_{BA}(G_{AB}(S_{A}))-S_{A}||_{1}]$
		$\displaystyle+\mathbb{E}_{S_{B}\sim p_{data}(S_{B})}[||G_{AB}(G_{BA}(S_{B}))-S_{B}||_{1}],$

where $||\cdot||_{1}$ means L1 norm.

Emotion Classification Loss: To explicitly guide the CycleGAN model to learn the emotion conversion function, we add additional emotion classification loss to the original model. Specifically, an accompanying emotion classifier $C$ as shown in Fig. 1 is trained, which determines whether $S_{A}$, $S_{BA}$ and $S_{ABA}$ match the desired emotion label $A$, as well as whether $S_{B}$, $S_{AB}$ and $S_{BAB}$ match the desired emotion label $B$. To achieve this, the following emotion classification loss is introduced:

	$\displaystyle\mathcal{L}_{emo}(G_{AB},G_{BA},C)$	$\displaystyle=\mathcal{L}_{emo-A}(G_{AB},G_{BA},C)$		(3)
		$\displaystyle+\mathcal{L}_{emo-B}(G_{AB},G_{BA},C),$

where

	$\displaystyle\mathcal{L}_{emo-A}$	$\displaystyle(G_{AB},G_{BA},C)=d(C(S_{A}),label_{A})$		(4)
		$\displaystyle+d(C(S_{AB}),label_{B})+d(C(S_{ABA}),label_{A}),$

and

	$\displaystyle\mathcal{L}_{emo-B}$	$\displaystyle(G_{AB},G_{BA},C)=d(C(S_{B}),label_{B})$		(5)
		$\displaystyle+d(C(S_{BA}),label_{A})+d(C(S_{BAB}),label_{B}).$

In equation (4) and (5), $d(\cdot)$ can be any divergence function used for classification problem, e.g., the binary cross-entropy loss function.

Full Objective: Combining the above adversarial loss, cycle consistency loss and emotion classification loss, the full training objective is:

	$\displaystyle\mathcal{L}(G_{AB},G_{BA},D_{A},D_{B},C)$	$\displaystyle=\mathcal{L}_{adv}(G_{AB},D_{B})$		(6)
		$\displaystyle+\mathcal{L}_{adv}(G_{BA},D_{A})$
		$\displaystyle+\lambda_{1}\mathcal{L}_{cyc}(G_{AB},G_{BA})$
		$\displaystyle+\lambda_{2}\mathcal{L}_{emo}(G_{AB},G_{BA},C),$

where $\lambda_{1}$ and $\lambda_{2}$ are trade-off parameters, adjusting the relative weights of these loss terms.

In this paper, speech features including Mel-cepstral coefficients (MCCs) and F0 are computed using WORLD [30]. The spectral features for conversion are Mel-cepstral coefficients (MCCs), which have dimension of 36. The energy contour and the F0 contour, as well as their corresponding CWT decomposition, are computed as in [23]. We use the CWT or logarithmic representation for the F0 and energy features. For convenience, we denote the CWT representation of F0 and energy contour as $\mathbf{LF0}_{cwt}$ and $\mathbf{LE}_{cwt}$, respectively, while denote the logarithmic F0 as $\mathbf{LF0}$. Network architecture of the generators, discriminators and classifier are shown in Table 1. The DBLSTM models, the baseline, are set to have the same network architecture as in [23]. The hyper-parameters and training details are made available here¹¹1Source code: https://github.com/liusongxiang/CycleGAN-EmoVC but left out for space limitation.

In the training and conversion stages, MCC features and prosodic features are concatenated, so the model maps these features together. The features are normalized to zero mean and unit variance before being fed into the CycleGAN and DBLSTM models. During conversion, the aperiodicity component remains intact and is directly copied over. We first compute the logarithm-scale F0 and energy contour from the converted CWT-represented F0 and energy features, respectively. Then a mean-variance denormalization and an exponential operation are adopted to compute the linear-scale F0 and energy contours of the target emotion from the normalized logarithm-scale ones. If we denote the converted spectral feature as $\mathbf{SP}^{c}$, which is computed from the converted $\mathbf{MCC}$, and the linear-scale energy contour as $\mathbf{e}^{c}$, the final converted spectral $\tilde{\mathbf{SP}}^{c}$ is computed as follow: (i) Compute the energy contour $\mathbf{e}^{t}$ of $\mathbf{SP}^{c}$. (ii) Compute the element-wise ratio $\mathbf{r}=\mathbf{e}^{c}/\mathbf{e}^{t}$. (iii) Scale the $i$-th frame vector $\mathbf{SP}_{i}^{c}$ by $\mathbf{r}_{i}$ to obtain $\tilde{\mathbf{SP}}_{i}^{c}$.

This paper investigates the efficacy of using CycleGAN for emotional VC tasks. Rather than attempting to learn a mapping between parallel training data using a frame-to-frame minimum generation loss, the CycleGAN uses two discriminators and one classifier to guide the learning process, where the discriminators aim to differentiate between the natural and converted speech and the classifier aims to classify the underlying emotion from the natural and converted speech. The training process of the CycleGAN models randomly pairs source-target speech parameters, thus DTW process is not needed. The objective and subjective evaluation results confirm the effectiveness of using CycleGAN models for emotional VC. To sum up, the advantages offered by the CycleGAN model include (i)utilizing GAN loss instead of minimum generation loss, (ii)getting rid of source-target alignment errors and (iii) flexible non-parallel training, etc. The non-parallel training process also indicates the potential to use non-parallel emotional speech data for developing emotional VC systems, which will be our future work.