Controlling your Attributes in Voice

Fig. 1 (a) illustrates the overall workflow of our method. First, we apply the approach described in [28] to extract sparse phonetic posteriorgrams (SPPGs) and pitch sequences from the source speech, which serve as the semantic features. Additionally, a WavLM-based SR model [29] is used to extract the original speaker vector from the source speech. Next, SRAVE generates a non-existent speaker vector based on the original speaker vector and predefined attribute labels. To capture more commonality and variation across different speakers, we use the encoder of SRAVE to decompose the synthetic speaker vector into age, gender, and identity embeddings as the conditions for TSVC, rather than directly using the speaker vector. Finally, the TSCV model conditions on these embeddings and the predefined attribute labels to transform the semantic features into speech with modified attributes.

Fig. 1 (b) illustrates the detailed architecture of SRAVE, a variational autoencoder trained with GAN. Specifically, we design an encoder consisting of three independent branches, which transforms the speaker vector into three latent embeddings: $\mathbf{z_{age}}$, $\mathbf{z_{gender}}$, and $\mathbf{z_{identity}}$. Each branch is composed of multiple residual blocks, each containing two linear layers with layer normalization and ReLU activation. After the age and gender branches, we employ a linear layer to output the probabilities of attribute labels for classification training. As for the identity branch, we introduce a contrastive loss to maximize the cosine similarity of identity embeddings between the same speaker and minimize the cosine similarity of identity embeddings between different speakers. On the decoder side, inspired by [30], we use attribute labels, $\mathbf{l_{age}}$ and $\mathbf{l_{gender}}$, as input instead of attribute embeddings to improve training stability. After the decoder module, we introduce a discriminator for generative adversarial training. Both the decoder and discriminator share the same backbone structure as the encoder branches.

Unlike previous works [15, 16, 17], we do not introduce operations, such as gradient reversal or mutual information minimization, to enforce independence between attribute and identity embeddings. On the contrary, we design a cyclic consistency training strategy to enable attribute sharing between speakers, as shown in Alg. 1. In the consistency training step, the encoder is frozen to guide the non-existent speaker vector generated by the decoder as close as possible to the real speaker vector domain.

We treat speech generation as a two-stage process: 1) generating average speech based on predefined ages and genders, and 2) refining the average speech into speech from a specific speaker based on the given attribute and identity embeddings. As illustrated in Figure 1 (c), in the first stage, we design an average generator that uses attribute labels, SPPGs, and pitch sequences to produce average acoustic features. The network of this generator consists of a Bi-LSTM layer and multiple convolutional residual blocks with AdaIN [31]. Notably, we do not use attribute embeddings in the average generator. Since no independence-guided operations are applied in SRAVE, the attribute embeddings are speaker-dependent. We believe these embeddings can help the second-stage module share the expression of attributes in the speech of similar speakers, but they are not suitable for generating speaker-independent average acoustic features.

In the second stage, we design a detail generator, an ordinary differential equation (ODE) model trained with flow matching [32], to model the mapping between the average acoustic features $X_{0}$ and the speaker-specific acoustic features $X_{T}$. The backbone of this generator follows the transformer-based network from SF-Speech [18]. During training, a random Gaussian noise $\xi$ is added to $X_{0}$ to obtain a continuous middle state distribution, expressed as ${X_{t}=(1-t)(X_{0}+\xi)+tX_{T}}$. The attribute embeddings, identity embeddings, and differential step $t$ are then repeated along the time axis and added to $X_{t}$, expressed as $X^{{}^{\prime}}_{t}$. Finally, to improve pronunciation accuracy, we concatenate SPPGs, pitch, and $X^{{}^{\prime}}_{t}$ along the hidden dimension, serving as the input to the transformer-based network to predict the direction of ODE, $dX_{t}$.

We conduct our experiments on a subset of Voxceleb2[33] with age labels from [34]. This subset contains about 5,000 speakers from 168K videos. We divided age into seven bands: less than 12, greater than 12 and less than 25, greater than 25 and less than 40, greater than 40 and less than 55, greater than 55 and less than 65, greater than 65 and less than 75, and greater than 75.

We set up three models for comparison, one baseline method and two ablation methods. Method 1 is an ODE model with attribute labels and the original speaker vector as conditions. Method 2 removes the average generator in the proposed TSVC model and employs an ODE model to map Gaussian noise to the target acoustic features directly. Method 3 eliminates the SRVAE, instead using attribute labels and the original speaker vector as conditions to train the TSVC.

We employ 3 residual blocks with 512 hidden dimensions to form the encoder and decoder of SRVAE, and 2 for its discriminator. For TSVC, we employ 3 convolutional residual blocks with 512 hidden dimensions for the average generator and 8 transformer layers with 1024 hidden dimensions for the detail generator. A HiFi-GAN [35] vocoder was trained to convert this acoustic feature into speech waveform. During training, the pitch sequences are normalized sentence by sentence to remove speaker-specific information.

All models were trained on 2 Nvidia V100 GPUs. We employed an AdamW optimizer with a learning rate starting at 0.0001 and linearly decaying to train SRVAE for 800K steps, and another AdamW optimizer with a peak learning rate of 0.0001, linearly warmed up for 5000 steps and decayed in cosine annealing over the rest steps, to train TSVC for 300K steps.

We assessed the extracted attribute embeddings by measuring their classification accuracy on the classifier in the SRVAE encoder. As for the extracted identity embedding, we compared the cosine similarity of those from the same speaker (CSIS) with the cosine similarity of those from different speakers (CSID). The classification accuracy was computed for 2,000 speaker vectors, while the cosine similarity was calculated for 2,000 pairs of speaker vectors.

As shown in Tab. I, both the attribute embeddings extracted from real speaker vectors and those extracted from generated non-existent speaker vectors demonstrate high classification accuracy. Notably, since the SRVAE encoder is trained exclusively on real speaker vectors, these similar accuracies suggest that the non-existent speaker vectors generated by SRVAE reside in the same representation space as real speaker vectors. Furthermore, the CSIS and CSID values for the real speaker vectors show a gap of 0.653, demonstrating that the identity embeddings extracted by SRVAE can accurately represent speaker identity. Meanwhile, the gap for the generated speaker vectors is 0.633, a slight decrease, yet still sufficient to indicate that the proposed SRVAE is capable of extracting accurate identity embeddings from the modified non-existent speaker vectors.

At the speech level, we conducted several subjective tests to assess the generated speech in terms of speech quality, attribute consistency, and identity consistency. For speech quality, we used Quality Comparative Mean Opinion Scores (QCMOS) to assess sound quality and naturalness, and Intelligibility Comparative Mean Opinion Scores (ICMOS) to evaluate pronunciation, as the Voxceleb2 dataset does not include text transcriptions. For gender consistency, we randomly assigned the generated and real samples for participants to predict the gender of the speaker. For age evaluation, we first asked participants to predict the speaker age of real speech without interval division. The most accurate age interval for human perception was then determined as 0-12, 12-25, 25-55, and $>$55, based on the prediction results. Finally, given the original speech and its age interval as a reference, participants were asked to predict the age interval of the modified speech. For identity consistency, participants were presented randomly with a modified or real speech sample from speaker A as a reference and were asked to select the speech with the same speaker identity from a test pair, which included speech from speaker B and another speech from speaker A. All subjective experiments involved at least 15 participants, with each being assigned 100 test questions. Additionally, we used the WavLM-based SR model to calculate speaker similarity between the speech from the same speaker (SIMS) and different speakers (SIMD), as in the speaker vector evaluation.

Results on speech quality: As shown in Tab. II, the proposed model performs best in terms of intelligibility and quality of generated speech, but there is a 0.29 gap compared to real audio in ICMOS. We believe this is due to the SPPG model’s lack of robustness when dealing with in-the-wild speech. In addition, the baseline model ranks second in this evaluation. We observe that it overfitted the training data, resulting in more than half of the modified speech being very similar to the source speech. Finally, the comparison with the ablative models demonstrates the effectiveness of our proposed approaches in modeling the natural expression of speaker attributes in speech.

Results on attribute consistency: Our method achieves a gender classification accuracy that is second only to ground truth, 49.9% higher than the baseline model, as shown in Tab. II. In age evaluation, our method obtains a confusion matrix most similar to the ground truth, while the confusion matrix of baseline exhibits a more diffuse distribution, as illustrated in Fig. 2. Additionally, after ablating SRVAE and TSVC, the predictive accuracy of gender decreases by 26.7% and 39.0%, respectively. Similarly, the consistency between the subjectively predicted age intervals and the predefined labels also decreases. These results further demonstrate the effectiveness of SRVAE and TSVC for attribute control.

Results on identity consistency: Tab. II shows the objective results using SR model. All methods achieve similar SIMD scores, while their SIMS exhibit a decreasing trend as attribute consistency increases. This trend is more pronounced when testing on speech with modified gender. This phenomenon can be explained by the fact that it is challenging for humans to recognize a speech pair with a large gap in fundamental frequency as belonging to the same identity. Although there is a decreasing trend in SIMS scores, they still exhibit a significant gap compared to SIMD, indicating that modified speech can still be recognized as originating from the source speaker by the SR model. Fig. 3 shows the subjective results. The probability of participants selecting the correct audio is positively correlated with the gap between SIMS and SIMD in Tab. II, except for ”- TSVC”. We believe this is because the poor quality of the speech generated by it affects the perception of participants.