Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignment

VC is a task that automatically converts the speech from a source speaker to a speech sound like being spoken by a target speaker while preserving the linguistic content (Mohammadi and Kain, 2017; Sisman et al., 2020). This task can be categorized into two categories: parallel and non-parallel conditions. Since parallel data are not always available, many non-parallel VC techniques have been proposed, including the methods based on variational auto-encoder (VAE) (Kameoka et al., 2019a; Saito et al., 2018), generative adversarial network (GAN) (Kaneko and Kameoka, 2018; Kameoka et al., 2018; Wang et al., 2020; Lee et al., 2021), recognition-synthesis (Chen et al., 2022; Mohammadi and Kim, 2019; Saito et al., 2018) and disentanglement (Zhang et al., 2019).

As a special case of non-parallel VC, zero-shot VC has attracted widespread attention in recent years. Zero-shot VC methods usually follow auto-encoder frameworks, where the encoder extracts content and speaker representations from speech respectively, and the decoder reconstructs speech by combining the above representations. Hence, speech representation disentanglement is crucial for this task (Yang et al., 2022a; Wang et al., 2021). Recently, several zero-shot VC methods (Yuan et al., 2021; Wang et al., 2021; Yang et al., 2022a) based on information theory have emerged, with the aim of disentangling the content-related and speaker identity-related information. IDE-VC (Yuan et al., 2021) employed mutual information (MI) with speaker labels as supervision for disentanglement. VQMIVC (Wang et al., 2021) combined vector quantization with contrastive predictive coding (VQCPC) (van Niekerk et al., 2020; Baevski et al., 2019) and MI for fully unsupervised training.

In recent years, learning the voice-face association has aroused the interest of researchers. As face and voice are inherently correlated, various cross-modal generation tasks involving both face and voice have been proposed, in addition to voice conversion supported by face images. Examples of such tasks include generating the talking face video from the audio (Park et al., 2022; Zhou et al., 2019; Chen et al., 2019; Song et al., 2018; Zhou et al., 2021), synthesizing speech from the talking face images (Prajwal et al., 2020; Kim et al., 2021; Wang et al., 2022b), reconstructing the face image from the corresponding voice (Oh et al., 2019; Wu et al., 2022a; Choi et al., 2020; Wen et al., 2019; Bai et al., 2022) and synthesizing the speaker’s voice with a face image during text-to-speech (FaceTTS) (Goto et al., 2020; Yang et al., 2023; Plüster et al., 2021; Wang et al., 2022a; Wu et al., 2022b; Yang et al., 2022b).

The most relevant task to voice conversion based on face images is FaceTTS, as they both utilize face images to extract speaker identities for controlling voice characteristics. As far as we know, Face2Speech (Goto et al., 2020) was the first work to address FaceTTS, which pretrained a face encoder with the supervisedly generalized end-to-end (GE2E) (Wan et al., 2018) loss and then replaced the speaker encoder with the face encoder in a multi-speaker TTS model. Following the Face2Speech framework, more elaborate model structures and training strategies (Wang et al., 2022a; Wu et al., 2022b; Plüster et al., 2021) have been proposed to promote the quality of synthetic speech. Recently, 3D face shapes and refined face attributes have also been utilized to generate speech (Yang et al., 2023, 2022b), which provided a referable approach to voice editing. However, the voice-face alignment in FaceTTS has yet to be explored. This paper elaborately designs a memory-based module for the alignment between these two modalities for zero-shot FaceVC, which can also be inserted into the FaceTTS framework for voice control.

The alignment between face and voice is intended to retrieve the corresponding speaker embedding when only a face image is available. The retrieved speaker embedding from a face image is referred to as the recalled face embedding. We introduce the MFVA module to improve the modeling of voice-face alignment, thereby promoting the performance of zero-shot FaceVC.

As shown in Figure 2, during the training phase, the MFVA module takes a pair of face embedding ${\bm{h}\in\mathbb{R}^{D}}$ and speaker embedding ${\bm{s}\in\mathbb{R}^{D}}$ as input, and generates a recalled face embedding ${\bm{\hat{h}}\in\mathbb{R}^{D}}$for voice control, where the face embedding ${\bm{h}}$ and the speaker embedding ${\bm{s}}$ are extracted by the face encoder and the speaker encoder respectively, and $D$ represents the dimension of the projected face or speech embedding. MFVA is composed of a voice-value memory ${\bm{M}_{voice}=[\bm{m}_{v}^{1},\bm{m}_{v}^{2},\cdots,\bm{m}_{v}^{N}]^{\intercal}}$ ${\in\mathbb{R}^{N\times D}}$ and a face-key memory ${\bm{M}_{face}=[\bm{m}_{f}^{1},\bm{m}_{f}^{2},\cdots,\bm{m}_{f}^{N}]^{\intercal}}$ ${\in\mathbb{R}^{N\times D}}$, where $N$ denotes the number of the slots and $D$ is the dimension for each slot, which equals to the dimension of the projected speaker embedding or face embedding. The training of the MFVA module contains two objectives, i.e., (1) storing sufficient voice characteristics information in voice-value memory, and (2) minimizing the distance between the distributions of two modalities.

The sufficiency of the voice characteristics information. Voice-value memory ${\bm{M}_{voice}}$ is made up of a bank of trainable slots ${\{\bm{m}_{v}^{i}}\}_{i=1}^{N}$, where $\bm{m}_{v}^{i}\in\mathbb{R}^{D}$ is the $i$-th slot. The voice-value memory is designed to exclusively capture the voice-related information and expected to generate any voice. Specifically, when we take a speaker embedding ${\bm{s}}$ as a query, the attention weight between the query and each slot is computed with cosine similarity and softmax normalization function as follows,

where $w_{v}^{i}$ represents the degree of relevance between the $i$-th slot $\bm{m}_{v}^{i}$ and the speaker embedding ${\bm{s}}$. Then we can get the attention weight vector ${\bm{w}_{voice}}=[w_{v}^{1},w_{v}^{2},\cdots,w_{v}^{N}]\in\mathbb{R}^{N}$ by computing attention weight with all slots. In the end, we reconstruct the speaker embedding from all slots,

and minimize the MSE between the input speaker embedding ${\bm{s}}$ and the recalled speaker embedding $\hat{\bm{s}}$,

In this way, the slots in the voice-value memory can be used as basis vectors for building the voice characteristics space, and various combinations of the slots can represent arbitrary voices.

The alignment between voice and face space. We utilize the slots as a streamlined bridge to map the face embedding onto the voice space. In detail, given the face embedding ${\bm{h}}$, we generate the recalled face embedding ${\hat{\bm{h}}}$ in a similar way as the recalled speaker embedding, where the attention weight is calculated with the face-key memory ${\bm{M}_{face}}$ and aligned with slots in voice-value memory ${\bm{M}_{voice}}$ as follows,

where $\bm{m}_{f}^{i}\in\mathbb{R}^{D}$ the $i$-th trainable slot in the face-key memory ${\bm{M}_{face}}$. Consequently, we combine the slots that are solely related to the voice to generate the recalled face embedding $\hat{\bm{h}}$. In addition, face images often contain various background noises and irrelevant information to the voice, such as shooting angle and image background. Voice-value memory ${M_{voice}}$ can impose an information bottleneck to remove these non-essential details. In the end, we align the slot-weight distributions between two modalities using Kullback-Leibler divergence,

In this way, during the inference phase, only given the face embedding as input, we can generate reasonable recalled face embedding to support the voice characteristics information for zero-shot FaceVC.

Content Encoder uses vector quantization (VQ) (van Niekerk et al., 2020) with contrastive predictive coding (CPC) (Baevski et al., 2019) to extract content embedding from mel-spectrograms, where VQ can be seen as an information bottleneck to remove inconsequential content information and CPC is used to explore the local structure of speech.

The mixed supervision strategy contains intra-speaker supervision and inter-speaker supervision. The left part in Figure 1 shows the intra-speaker supervision following the traditional auto-encoder paradigm. During the training phase, we firstly encode the mel-spectrograms $\bm{X}_{A}$ of input utterance from Speaker $A$ into a speaker-independent content embedding ${\bm{c}_{A}}$, normalized fundamental frequencies ${\bm{f}_{A}}$ and a length-fixed speaker embedding ${\bm{s}_{A}}$. Meanwhile, we utilize the corresponding face image $\bm{Z}_{A}$ to get the face embedding ${\bm{h}_{A}}$. Then, we feed the face embedding $\bm{h}_{A}$ and the speaker embedding ${\bm{s}_{A}}$ into the MFVA module to get recalled face embedding $\hat{\bm{h}}_{A}$. In the end, the decoder $D$ maps the above representations to reconstruct the mel-spectrograms ${\hat{\bm{X}}_{A}=D(\bm{c}_{A},\bm{\hat{h}}_{A},\bm{f}_{A})}$. The decoder is jointly trained with the MFVA and the self-attention module in the face encoder by minimizing the following reconstruction loss,

However, the training-inference inconsistency phenomenon exists when only adopting intra-speaker supervision, because the content embedding and the recalled face embedding are from the same speaker during the training phase while different during the inference phase. We introduce a simple but efficient inter-speaker supervision strategy when the parallel corpus is unavailable. In this strategy, the voice converted by the target speaker embedding is used as the pseudo-parallel corpus of the face embedding for supervised training. Specifically, as shown in the middle part of Figure 1, in addition to the speaker $A$, a pair of face image and mel-spectrograms of an extra speaker ${B}$ is as input, and the speaker embedding ${\bm{s}_{B}}$ and the recalled face embedding $\hat{\bm{h}}_{B}$ are obtained in the same way as inference phase. Then speaker A is treated as the source speaker and speaker $B$ is viewed as the target speaker to convert voice as follows,

where $\bm{X}_{speech}$ refers to the converted mel-spectrograms that are supported by the speaker embedding $\bm{s}_{B}$, and $\bm{X}_{face}$ denotes the converted mel-spectrograms obtained using the recalled face embedding $\hat{\bm{h}}_{B}$. Then, we optimize the MFVA by minimizing the reconstruction loss,

In summary, the final loss function during the training phase of zero-shot FaceVC is as follows,

where $\lambda_{1},\lambda_{2}$, and $\lambda_{3}$ are constant weights to control how the importance of each term, and $\mathcal{L}_{store}$ and $\mathcal{L}_{align}$ are described in Section 3.1.

It is widely recognized that the content encoder may encode the speaker identity-related information. Only when the speaker and content representations are disentangled, the voice characteristic of the utterance can be converted by changing the speaker-identity representation (Qian et al., 2019; Yuan et al., 2021). Hence speech representation disentanglement is a critical factor that significantly impacts the performance of the zero-shot FaceVC. Taking this into considerations, we first pretrain a zero-shot VC model and then transfer the content encoder, speaker encoder, and decoder to the zero-shot FaceVC for better performance. Specifically, during the training phase of the zero-shot FaceVC, the pretrained content encoder and the pretrained speaker encoder are fixed, while the pretrained decoder is further optimized with other modules.

In order to achieve speech representation disentanglement, mutual information (MI) is introduced to evaluate the dependency between different representations. We minimize the MI between content embeddings, speaker embeddings and fundamental frequencies utilizing the variational contrastive log-ratio upper bound (vCLUB) (Cheng et al., 2020) during the training phase of zero-shot VC. Except for the MI loss, reconstruction loss, InfoNCE loss (Oord et al., 2018) and VQ loss (van Niekerk et al., 2020) are used to optimize the zero-shot VC model. For more information on these functions, please refer to the VQMIVC (Wang et al., 2021) paper.

Zero-shot FaceVC tasks place high demands on datasets, requiring not only a large volume of background-noise-free speech from various speakers but also clear face images of the corresponding individuals. Based on the above considerations, we conducted the experiments on the LRS3-TED (Afouras et al., 2018) dataset to evaluate the zero-shot FaceVC task. This dataset includes 5,594 TED and TEDx talks in English, totaling over 400 hours of video content. The cropped face tracks in the video are provided at a resolution of 224×224 with a frame rate of 25 frames per second. The audio tracks are also available in a single-channel 16-bit 16 kHz format. In this dataset, the duration of speech from different speakers follows a long-tail distribution, which means that the majority of speakers have only a small number of utterances. To address the issue of uneven video distribution among speakers, we opted to use the top 200 speakers with the highest number of videos as our training set and validation set. During inference, we randomly selected a total of 12 newly coming speakers not in the training and validation sets: 8 target speakers (4 female and 4 male) and 4 source speakers (2 female and 2 male) for evaluation.

To extract acoustic features, we first extracted the audio from the video clips with the FFmpeg tool (Tomar, 2006). Then 80-dim mel-spectrograms and normalized fundamental frequencies were calculated with a 25ms Hanning window, a 10ms frame shift and a 400-point short-time Fourier transform (STFT). We extracted a 512-dimensional face embedding from each frame of the video using MTCNN (Zhang et al., 2016) and FaceNet (Schroff et al., 2015) subsequently. The dimension of face embeddings was projected into ${D=256}$, which was the same as the dimension of speaker embeddings and the slot dimension. The number of slots in MFVA was $N=96$. We applied the pretrained Parallel WaveGAN (PWG) vocoder (Yamamoto et al., 2020) to convert the mel-spectrograms to waveforms.

For the pretraining strategy, the zero-shot VC model was trained for 1000 epochs using a batch size of 256. The mini-batch Adam optimizer was initialized with a learning rate of 1e-6 and was warmed up to 1e-3 after 2000 iterations. The learning rate was then decayed by a factor of 0.5 at epochs 300, 400, and 500. At the zero-shot FaceVC training stage, the model was updated for 2000 epochs using a batch size of 256. Similar to pretraining the zero-shot VC model, the learning rate of the Adam optimizer was initialized as 1e-6 and warmed up to 2.5e-4 after 3000 iterations. The learning rate was then decayed by a factor of 0.5 at epochs 800, 1200, and 1600. The constant weights ${\lambda_{1}}$, ${\lambda_{2}}$, and ${\lambda_{3}}$ in Equation 12 were respectively set to be 1, 10, and 0.2.

As we are the first to attempt the zero-shot FaceVC task and there are no existing comparable methods, we compared our proposed method with the following systems to evaluate its performance:

(1) Ground Truth: This method transferred the natural mel-spectrograms of target speakers to waveforms using the pretrained PWG vocoder. Since there are no parallel utterances between source and target speakers, the ground truth results can’t be compared directly with the converted results, and are just used for indicating the upperbound of various metrics.

(2) SpeechVC: This is our pretrained zero-shot VC model using natural reference utterances of target speakers for inference.

(3) Auto-FaceVC (Lu et al., 2021): This method originally adopted AutoVC (Qian et al., 2019) as the backbone. To better adapt the model to the zero-shot FaceVC task, we altered its backbone AutoVC to VQMIVC while preserving its original training strategy.

(4) attentionCVAE (Yang et al., 2022b): This method used the face attributes to control the voice characteristics in the multi-speaker text-to-speech task. To adapt it to our zero-shot FaceVC task, we inserted its face attributes-based voice control module to replace the face encoder and the MFVA module in our proposed model.

We developed several objective metrics to evaluate the homogeneity, diversity and consistency of the converted voice. The underlying motivation for measuring homogeneity is that the zero-shot FaceVC system should generate homogeneous voice characteristics with different face images from the same target speaker, regardless of the image’s shooting angle and background. We applied the well-known open-source speaker verification toolkit, Resemblyzer³³3https://github.com/resemble-ai/Resemblyzer, to extract the speaker embedding from converted utterances of the same speaker and calculate the cosine similarity between them. The greater the value of cosine similarity, the higher the homogeneity between different utterances. Based on the descriptions above, we employed two methods to match the utterances and calculate the cosine similarity between them. (1) We employed a randomized approach to match utterances converted by different face images of the same target speaker. To avoid chance, we shuffled the utterances 500 times and calculated the average cosine similarity between all pairs, which we referred to as the speaker homogeneity score by random matching (SHR). (2) In addition to the random matching, we also conducted one-to-one matching of all utterances converted from the same utterance to the same target speaker by different face images. We then averaged cosine similarity between these pairs to obtain the speaker homogeneity score by one-to-one matching (SHO).

Apart from homogeneity, it is crucial for the voice characteristics converted from different target speakers to be diverse rather than uniform and indistinguishable. Similar to homogeneity, we obtained the speaker embedding of converted utterances from the same source speaker using different target speakers’ face images by Resemblyzer and calculated the cosine similarity between them. We hypothesize that a lower similarity indicates higher diversity in the voice characteristics between target speakers. To measure speaker diversity, we also matched the utterance in two ways. (1) We randomly matched the converted utterances from the same source speaker to different target speakers and averaged the cosine similarity between them in 100 shuffles, which we referred to as the speaker diversity score by random matching (SDR). (2) We conducted a one-to-one matching of utterances converted from the same utterance to different target speakers and averaged cosine similarity between them to obtain the speaker diversity score by one-to-one matching (SDO).

When assessing the consistency between the voice characteristics and the corresponding face images, the gender attribute is the primary factor to consider. Hence, we used the open source speech segments toolkit, inaSpeechSegmenter⁴⁴4https://github.com/ina-foss/inaSpeechSegmenter, to calculate gender accuracy (GA) for each converted utterance. Specifically, a speech segmenter (Doukhan et al., 2018) was firstly used to discard segments that did not contain any speech. Next, the remaining speech segments were classify into either male or female using the convolutional neural networks. The gender of a converted utterance was consistent with the target speaker if all the speech segments in this utterance were classified as the gender of the target speaker.

For subjective evaluation, we adopted two mean opinion scores in terms of face-voice consistent degree (MOS-FVC) and speech naturalness (MOS-SN). MOS-FVC was used to evaluate whether the face image and the voice characteristics were consistent with each other, e.g., a middle-aged man’s face with the little girl’s voice would be considered inconsistent. MOS-SN was used to quantitatively measure the naturalness of the converted voice. The listeners were asked to score each converted utterance on a scale from 1 (completely unnatural or completely inconsistent) to 5 (completely natural or completely consistent) for two metrics.

We chose 6 utterances from each of the 4 source speakers and randomly selected one face frame in 3 videos from each of the 8 target speakers for inference. Then we matched them pairwise and converted a total of 576 utterances for objective evaluation. Two subjective metrics were evaluated on the Amazon Mechanical Turk platform⁵⁵5https://www.mturk.com/. 20 converted utterances were randomly selected from each system and a total of 20 listeners participated in the test. All objective and subjective evaluation results are reported in Table 1.

We can observe that the proposed FVMVC outperformed the Auto-FaceVC and attentionCVAE systems on all objective metrics significantly (${p<0.05}$ in paired t-tests). Compared with Auto-FaceVC, the slots in MFVA quantify the voice characteristics space, which makes the voice control via face images more homogeneous. Additionally, our proposed FVMVC incorporates the MFVA module to alleviate the problem of over-smoothing, resulting in a more diverse range of voice characteristics. In attentionCVAE, facial attributes can only provide limited information such as gender, age, and ethnicity. As a result, while this method ensures relatively consistent voice characteristics and accurate gender, it also tends to generate very similar voices for different target speakers, resulting in a loss of diversity. In addition, following the method described in attentionCVAE (Yang et al., 2022b), we found that the facial attributes may vary across different face images of the same speaker, which can lead to heterogeneity among the voice characteristics of the same target speakers. With regard to GA, our proposed FVMVC has demonstrated a significant improvement, increasing from 90.85% and 91.66% in the two aforementioned methods to 97.91%. The MOS-FVC has a strong correlation with the GA. In cases where the voice and the face image display a clear gender mismatch, the consistency score between them tends to be significantly low. Since the three methods employed the same backbone, their performance in terms of MOS-SN is quite comparable.

We can observe that our proposed FVMVC performs better than SpeechVC with respect to SHR and SHO. This could be attributed to the presence of stable identity information in face images and several speaker-independent factors included in natural reference speech, such as prosody and emotions. These factors can influence the homogeneity between the converted utterances of the same target speakers. In terms of SDR, SDO and MOS-FVC, our proposed method is less effective than SpeechVC, which is caused by the limited amount of voice characteristics information contained in the face image compared to the one contained in the natural reference speech. Additionally, for GA and MOS-SN, the performance of our proposed FVMVC and SpeechVC is essentially similar.

We utilized Resemblyzer to extract speaker embeddings from the utterances generated by the Ground Truth and those converted by three other systems, i.e., SpeechVC, Auto-FaceVC, and FVMVC. We present their t-SNE (Chan et al., 2019) visualization in Figure 4. For the Auto-FaceVC system, some embedding clusters contained both male and female target speakers, as shown in the red box of Figure 4(c). This led to that a single converted utterance may contain both male and female voices in different segments. On the other hand, our proposed FVMVC model produced embeddings with a clear boundary between two genders, which further demonstrates the effectiveness of our method on GA and MOS-FVC metrics. In addition, the embeddings of different target speakers overlapped a lot for the Auto-FaceVC system. While similar to SpeechVC, our proposed FVMVC model had much less embedding overlap across target speakers, which also further confirms the better speaker diversity achieved by our method.

We selected 2 target speakers’ 6 face images taken from different perspectives for voice conversion, as shown in Figure 3. The first three columns belong to the first target speaker, and the last three columns belong to the second target speaker. The slot weights and corresponding mel-spectrograms of converted utterances based on the face images are visualized. We discover that the distributions of slot weights remain consistent across different face images of the same speaker, regardless of the angle or expression displayed in the images. This finding suggests that the speaker’s facial features are of decisive importance in the process of aligning face and voice, and are minimally affected by external factors such as camera position, background, and other sources of noise. As we can see from the third row in Figure 3, with the aid of stable recalled face embeddings, the mel-spectrograms converted by different face images exhibit a high level of uniformity.

Additionally, we attempted to achieve voice characteristics interpolation by manipulating the slot weights in the MVFA module. We chose a male and a female target speakers for creating new voices by interpolation, as depicted in Figure 5. Specifically, we blended the slot weight of two face images with distinct weights to obtain the new recalled face embedding. From left to right, as the slot weights of the male speaker $B$ increases, the voice characteristics gradually shift from female to male, and the fundamental frequencies gradually decreases. This further validates the effectiveness of the MFVA module for face-based voice control.

In this section, we conducted ablation experiments on our proposed FVMVC to explore the effectiveness of each module. As shown in Table 2, we conducted experiments by removing the inter-speaker supervision, the MFVA module and the pretraining strategy from the proposed FVMVC, respectively. The results show that when inter-speaker supervision was removed, the model’s performance on SDR, SDO and GA decreased. This suggests that alleviating the inconsistency between training and inference phases can help to fit the recalled face embedding to the decoder, resulting in more diverse voice generation. After removing the MFVA module, the output of the face encoder was directly fed into the decoder without any constraints imposed by speaker embeddings. We find that the model’s performance decreased in all aspects, highlighting the crucial role of alignment between face and voice in the model’s performance. We also attempted to train the model from scratch without pretrained strategy. Our results show that the pretraining on the zero-shot VC task has a significant positive impact on the proposed FVMVC model.