Towards Automatic Face-to-Face Translation

We make use of recent works on Automatic Speech Recognition (ASR)(Amodei et al., 2016) to convert the speech of the source language L_A into the corresponding text. Speech recognition for English has been extensively investigated, owing to the existence of large open-source speech recognition datasets(Panayotov et al., 2015; Rousseau et al., 2012) and trained models(Amodei et al., 2016). We employ the DeepSpeech 2 model to perform English speech recognition in this work.

NMT is often modelled as a sequence to sequence problem which was first introduced with neural networks in  Sutskever et al. (2014). Further improvements were brought about with attention mechanisms by  Bahdanau et al. (2014) and  Luong et al. (2015). More recently,  Vaswani et al. (2017) introduced transformer network which relies only on an attention mechanism to draw global dependencies between input and output. The transformer network outperforms its predecessors by a healthy margin and hence we decided to adopt this into our pipeline. It has also been observed in works like Johnson et al. (2017) that training multilingual translation systems also improve the performance especially for low resource languages. Thus, in this work, we also follow a similar path where we use state of the art architectures extended to multilingual learning setups.

There has been a lot of work in the area of text-to-speech (TTS) synthesis, starting with the most commonly used HMM-based models(Zen et al., 2009). These models can be trained with lesser data to produce fairly intelligible speech, but fail to capture aspects like prosody that is evident in natural speech. Recently, researchers have achieved natural TTS by training neural network-based architectures (Shen et al., 2018; Ping et al., 2017) to map character sequences to mel-spectrograms. We adopt this approach, and train Deep Voice 3(Ping et al., 2017) based models to achieve high-quality text-to-speech synthesis in our target language L_B. Our implementation of DeepVoice 3 also makes use of a recent work on guided-attention (Tachibana et al., 2018) allowing it to achieve high-quality alignment and faster convergence.

Multiple recent works (Ping et al., 2017; Arik et al., 2018) make use of multi-speaker TTS models to generate voice conditioned on speaker embeddings. While these systems offer the advantage of being able to generate novel TTS voice samples given a few seconds of reference audio, the quality of TTS is inferior (Ping et al., 2017) compared to single-speaker TTS models. In our system, we employ another recent work(Kaneko and Kameoka, 2017) that uses a CycleGAN architecture to achieve good voice transfer between two human speakers with no loss in linguistic features. We train this model to perform a cross-language transfer of a synthetic TTS voice to a natural target speaker voice. We evaluate our models and show that by using just about ten minutes of a target speaker’s audio samples, we can emulate the speaker’s voice and significantly improve the experience of a listener.

Lip synthesis from a given audio track is a fairly long-standing problem, first introduced in the seminal work of  Bregler et al. (1997). However, realistic lip synthesis in unconstrained real-life environments was only made possible by a few recent works (Suwajanakorn et al., 2017; Kumar et al., 2017). Typically, these networks predicted the lip landmarks conditioned on the audio spectrogram in a time window. However, it is important to highlight that these networks fail to generalize to unseen target speakers and unseen audio. A recent work by  Chung et al. (2017) treated this problem as learning a phoneme-to-viseme mapping and achieved generic lip synthesis. This leads them to use a simple fully convolutional encoder-decoder model. Even more recently, a different solution to the problem was proposed by  Zhou et al. (2018), in which they use audio-visual speech recognition as a probe task for associating audio-visual representations, and then employ adversarial learning to disentangle the subject-related and speech-related information inside them. However, we observed two major limitations in their work. Firstly, to train using audio-visual speech recognition, they use $500$ English word-level labels for the corresponding spoken audio. We observed that this makes their approach language-dependent. It also becomes hard to reproduce this model for other languages as collecting large video datasets with careful word-level annotated transcripts in various languages is infeasible. Our approach is a fully self-supervised approach that learns a phoneme-viseme mapping, making it language independent. Secondly, we observe that their adversarial networks are not conditioned on the corresponding input audio. As a result, their adversarial training setup does not directly optimize for improved lip-sync conditioned on audio. In contrast, our LipGAN directly optimizes for improved lip-sync by employing an adversarial network that measures the extent of lip-sync between the frames generated by the generator and the corresponding audio sample. Additionally, both  Zhou et al. (2018) and  Chung et al. (2017) normalize the pose of the input faces to a canonical pose, thus making it difficult to blend the generated faces in the original input video. Proposed LipGAN tackles this problem by providing additional information about the pose of the target face as an input to the model thus making the final blending of the generated face in the target video fairly straightforward.

We use publicly available state-of-the-art ASR systems for generating text in language L_A. A publicly available pre-trained model using Deep Speech 2 is used for speech recognition in English. This model was trained on LibriSpeech dataset and achieves WER% of 5.22% on the LibriSpeech test set. Once we have text, recognized in a source language, we translate it into a target language using an NMT model, which we discuss next.

We use the re-implementation of Transformer-Base (Vaswani et al., 2017) available in fairseq-py³³3https://github.com/pytorch/fairseq. The language pairs we attempt our problem on contains a low resource language, Hindi. To create a nmt system which works well for Hindi as well as English, we resort to training a multiway model to maximize learning(Neubig and Hu, 2018; Aharoni et al., 2019). We closely follow Johnson et al. (2017) in training a multi-way model whose parameters are shared across all seven languages - Hindi, English, Telugu, Malayalam, Tamil, Telugu, Urdu. Details of the translation system has been reported in (Philip et al., 2019).

In Table 1, we report evaluation metrics for language directions which are within the scope of this paper. We indicate the size of training data used and the evaluated scores using the widely used Bilingual Evaluation Under Study (BLEU) obtained on the test split of IIT-Bombay Hindi-English Parallel Corpus (Kunchukuttan et al., 2018). We compare against Google Translate⁴⁴4compared in the first week of April, 2019 in this test set, which is indicated in Table 1 as Online-G. We achieve an increase of  3 BLEU points on the test set compared to Google Translate.

Next, we describe our methods of generating speech from the target text in L_B, obtained after translating source text in language L_A.

For our Hindi text-to-speech model, we adapt a re-implementation of the DeepVoice 3 model proposed by  Ping et al. (2017). Due to the lack of publicly available large scale dataset for Hindi, we curate a dataset similar to LJSpeech by recording Hindi sentences from crawled news articles.

We adopt the nyanko-build ⁵⁵5https://github.com/r9y9/deepvoice3_pytorch implementation of DeepVoice 3 to train our Hindi TTS model. We trained on about $10,000$ audio-text pairs and evaluated on $100$ unseen test sentences. Griffin-Lim algorithm (Griffin and Lim, 1984) was used to generate waveforms from the spectrograms produced by our model. We evaluate this model by conducting a user study with $25$ participants using our unseen test set. The average Mean Opinion Scores (MOS) scores with 95% confidence intervals are reported in Table 2. In the next section, we describe how we can modify the voice of the TTS model to a given target speaker.

Voice of a speaker is one of the key elements of her acoustic identity. As our TTS model only generates audio samples in a single voice, we personalize this voice to match the voice of different target speakers. As collecting parallel training data for the same speaker across languages is infeasible, we adopt the CycleGAN architecture (Kaneko and Kameoka, 2017) to work around this problem.

For a given speaker we collect about $10$ minutes of audio clips, which can be easily obtained as we need only a non-parallel dataset. Using our trained TTS model, we generate $5000$ samples amounting to about 3 hours worth of synthetic TTS speech. For each speaker, we train a CycleGAN for about $50K$ iterations with a batch size of $16$. The other hyperparameters are the same as used in  Kaneko and Kameoka (2017). During inference, given a TTS generated audio sample, the model preserves the linguistic features and generates speech in the voice of the speaker it was trained on.

We evaluate our Voice Transfer models in a similar fashion to  Kaneko and Kameoka (2017), with the help of $30$ participants. We use 20 generated TTS samples each of which are voice transferred across $5$ famous personalities. Table 3 reports the result of this study. In the next section, we describe how we generate realistic talking face videos.

We formulate our talking face synthesis problem as “learning to synthesize by testing for synchronization”. Concretely, our setup contains two networks, a generator $G$ that generates faces by conditioning on audio inputs and a discriminator $D$ that tests whether the generated face and the input audio are in sync. By training these networks together in an adversarial fashion, the generator $G$ learns to create photo-realistic faces that are accurately in sync with the given input audio. The setup is illustrated in Figure 3.

The generator network is a modification of  Chung et al. (2017) and contains three branches: $(i)$ Face encoder, $(ii)$ Audio encoder and a $(iii)$ Face Decoder.

While using only an L2 reconstruction loss for the generator can generate satisfactory talking faces (Chung et al., 2017), employing strong additional supervision can help the generator learn robust, accurate phoneme-viseme mappings and also make the facial movements more natural.  Zhou et al. (2018) employed audio-visual speech recognition as a probe task to associate the acoustic and visual information. However, this makes the setup language-specific and offers only indirect supervision. We argue that directly testing whether the generated face synchronizes with the audio provides a stronger supervisory signal to the generator network. Accordingly, we create a network that encodes an input face and audio into fixed representations and computes the L2 distance $d$ between them. The face encoder and audio encoder are the same as used in the generator network.

Our training process is as follows. We randomly select a $T$ millisecond window from an input video sample and extract its corresponding audio segment $A$, resampled at a frequency $F$ Hz. We choose the middle video frame $S$ in this window as the desired ground-truth. We mask the mouth region (assumed to be the lower-half of the image) of a person in the ground truth frame to get $S_{m}$. We also sample a negative frame $S^{\prime}$, i.e., a frame outside this window which is expected to not be in sync with the chosen audio segment $A$. At each training batch to the generator, the unsynced face $S^{\prime}$ concatenated channel wise with the masked ground truth face $S_{m}$ and the target audio segment $A$ is provided as the input. The generator is expected to generate the synced face $G([S^{\prime};S_{m}],A)\approx S$. Each training batch to the discriminator contains three types of samples: $(i)$ Synthetic samples from the generator $(G(S^{\prime},A),A);y_{i}=1,(ii)$ Actual frames synced with audio $(S,A);y_{i}=0$ and $(iii)$ Actual frames out of sync with audio $(S^{\prime},A);y_{i}=1$. The third sample type is particularly important to force the discriminator to take into account the lip synchronization factor while classifying a given input pair as real / synthetic. Without the third type of sample, the discriminator would simply be able to ignore the audio input and make its decision solely on the quality of the image. The discriminator learns to detect synchronization by minimizing the following contrastive loss:

where $m$ is the margin, which we set to $2$. The generator learns to reconstruct the face image by minimizing the L1 reconstruction loss:

We train the generator $G$ and discriminator $D$ using the following GAN objective function:

where $z\in\{S,S^{\prime}\}$. Here, $G$ tries to minimize $L_{a}$ and $L_{Re}$ and $D$ tries to maximize $L_{a}$. Thus, the final objective function is:

We use the LRS 2 dataset (Afouras et al., 2018) which contains over $29$ hours of talking faces in the provided train split in the dataset. We train on four NVIDIA TITAN X GPUs with a batch size of $512$. We extract $13$ MFCC features from each audio segment $(T=350,F=100)$ and discard the first feature similar to  Chung et al. (2017). We detect faces in our input frame using dlib (King, 2009) and resize the face crops to $96$x$96$x$3$. We use the Adam (Kingma and Ba, 2014) optimizer with an initial learning rate of $1e{-3}$ and train for about $20$ epochs.

We evaluate our novel LipGAN architecture quantitatively and also with subjective human evaluation. During inference, the model generates the talking face video of the target speaker frame-by-frame. The visual input is the current frame concatenated with the same current frame with the lower-half masked. That is, during inference, we expect the model to morph the input shape and preserve other aspects like pose and expression. Along with each of the visual inputs, we feed a $T=350ms$ audio segment. In Figure 4, we compare the talking faces generated by $3$ models on audio segments actually spoken by Narendra Modi and Elon Musk.

Finally, our Face-to-Face translation pipeline with all the components put together is evaluated based on its impact on the end-user experience. We choose $5$ famous identities and generate talking face videos in Hindi of Andrew Ng, Obama, Modi, Elon Musk and Chris Anderson using our complete pipeline. We do this by choosing short videos of each of the above speakers speaking in English. We use our ASR and NMT modules to recognize the speech in English and translate it to Hindi. We use our Hindi TTS model to obtain speech in Hindi. We convert this speech to the voices of each of the above speakers using our CycleGAN models. Using these final voices, we generate talking face videos using our LipGAN network. We compare these videos against videos with $(i)$ English speech and automatically translated subtitles $(ii)$ automatic dubbing to Hindi $(iii)$ automatic dubbing with voice transfer and $(iv)$ automatic dubbing with voice transfer + lip synchronization. Additionally, we also benchmark human performance for speech-to-speech translation: $(v)$ Manual dubbing and $(vi)$ Manual Dubbing + automatic lip synchronization. We ask the users to rate the videos on a scale of $1-5$ for two attributes. First one is ”Semantic consistency” to check whether the automatic pipelines preserve the meaning of the original speech and the second attribute is the ”Overall user experience”, where the user considers factors such as the realistic nature of the talking face and his/her comfort level. The results of this study are reported in Table 7.

The results present three major takeaways. Firstly, we observe that there are significant scopes for improvement in each of the modules of automatic speech-to-speech translation systems. Future improvements in each of the speech and text translation systems will improve the user study scores. Secondly, the increase in user scores by using lip synchronization after manual dubbing again validates the effectiveness of the LipGAN model. Finally, note that adding each of our automatic modules increases the user experience score, emphasizing the need for each of them. Our complete proposed system improves the overall user experience over traditional text-based and speech-based translation systems by a significant margin.

Movies are generally dubbed by dubbing artists manually. The dubbed audio is then overlaid to the original video. This causes the actors’ lips to be out of sync with the audio, thus affecting the viewer experience. Our pipeline can be used to automate this process at different levels with different trade-offs. We demonstrate that we can synthesize and synchronize lips in manually dubbed videos, thus automatically correcting any dubbing errors.

We also show a proof-of-concept for performing automatic dubbing using our translation pipeline. That is, given a movie scene in a particular language our system can potentially be used to dub it to a different language. However, as shown by the user study scores in Table 7, significant improvements to the speech-to-speech pipeline is necessary to achieve realistic dubbing of complex speech present in movies.

As reported before in (Jha et al., 2019), a large amount of online educational content is present in English in the form of video lectures. They are often aided with subtitles of foreign languages. But this increases the cognitive load of the viewer. Dubbing these videos with just speech-to-speech systems creates a visual discrepancy between the lip motion and the dubbed audio. However, with the help of our face to face translation system, it is possible to automatically translate such educational video content and also ensure lip synchronization.

Automatic Face-to-Face Translation systems can potentially allow viewers to access and consume important information from across the globe irrespective of the underlying language. For example, a Hindi or German viewer can watch an English interview of Obama in the language of his/her choice with lip synchronization.