Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

While initial approaches [23, 20] to this problem extracted the visual features from sensors or active appearance models, the recent works employ end-to-end approaches. Vid2Speech [10] and Lipper [22] generate low-dimensional LPC (Linear Predictive Coding) features given a short sequence of $K$ frames $(K<15)$. The facial frames are concatenated channel-wise and a 2D-CNN is used to generate the LPC features. We show that this architecture is very inadequate to model real-world talking face videos that contain significant head motion, silences and large vocabularies. Further, the low dimensional LPC features used in these works do not contain a significant amount of speech information leading to robotic, artificial sounding speech.

A follow-up work [9] of Vid2Speech does away with LPC features and uses high-dimensional melspectrograms along with optical flows to force the network to explicitly condition on lip motion. While this can be effective for the GRID corpus that has no head movements, optical flow could be a detrimental feature in unconstrained settings due to large head pose changes. Another work [36] strives for improved speech quality by generating raw waveforms using GANs. However, both these works do not make use of the well-studied sequence-to-sequence paradigm [31] that is used for text-to-speech generation [30]; thus leaving a large room for improvement in speech quality and correctness.

Finally, all the above works show results primarily on the GRID corpus [7] which has a very narrow vocabulary of $56$ tokens and very minimal head motion. We are the first to study this problem in a large vocabulary setting with thousands of words and sentences. Our datasets are collected from YouTube video clips and hence contain a significant amount of natural speech variations and head movements. This makes our results directly relevant to several real-world applications.

In this space as well, several works [6, 38, 37, 28] are limited to narrow vocabularies and small datasets, however, unlike lip to speech, there have been multiple works [5, 2] that specifically tackle open vocabulary lip to text in-the-wild. They employ transformer sequence-to-sequence [35] models to generate sentences given a silent lip movement sequence. These works also highlight multiple issues in the space of lip-reading, particularly the inherent ambiguity and hence the importance of using a language model. Our task at hand is arguably harder, as we not only have to infer the linguistic content, but also generate with rich prosody in the voice of the target speaker. Thus, we focus on extensively analyzing the problem in a single-speaker unconstrained setting, and learning precise individual speaking styles.

Over the recent years, neural text-to-speech models [30, 27] have paved the way to generate high-quality natural speech conditioned on any given text. Using sequence-to-sequence learning [31] with attention mechanisms, they generate melspectrograms in an auto-regressive manner. In our work, we propose Lip2Wav, a modified version of Tacotron 2 [30] that conditions on face sequences instead of text.

Prior works in lip to speech regard their speech representation as a 2D-image [10, 36] in the case of melspectrograms or as a single feature vector [10] in the case of LPC features. In both these cases, they use a 2D-CNN to decode these speech representations. By doing so, they violate the ordering in which they model the sequential speech data, i.e. the future time steps influence the prediction of the current time step. In contrast, we formulate this problem in the standard sequence-to-sequence learning paradigm [31]. Concretely, each output speech time-step $S_{k}$ is modelled as a conditional distribution of the previous speech time-steps $S_{<k}$ and the input face image sequence $I=(I_{1},I_{2},\dots,I_{T})$. The probability distribution of each output speech time-step is given by:

Lip2Wav, as shown in Figure 3 consists of two modules: (i) Spatio-temporal face encoder (ii) Attention-based speech decoder. The modules are trained jointly in an end-to-end fashion. The sequence-to-sequence approach enables the model to learn an implicit speech-level language model that helps it to disambiguate homophenes.

There are multiple output representations from which we can recover intelligible speech, but each of them have their trade-offs. The LPC features are low-dimensional and easier to generate, however, they result in robotic, artificial sounding speech. At the other extreme [36], one can generate raw waveforms but the high dimensionality of the output (16000 samples per second) makes the network training process computationally inefficient. We take inspiration from previous text-to-speech works [27, 30] and generate melspectrograms conditioned on lip movements. We sample the raw audio at $16$kHz. The window-size, hop-size and mel dimension are $800,200,$ and $80$ respectively.

Our visual input is a short video sequence of face images. The model must learn to extract and process the fine-grained sequence of lip movements. 3D convolutional neural networks have been shown to be effective [18, 33, 36] in multiple tasks involving spatio-temporal video data. In this work, we try to encode the spatio-temporal information of the lip movements using a stack of 3D convolutions (Figure 3). The input to our network is a sequence of facial images of the dimension $T\times H\times W\times 3$, where $T$ is the number of time-steps (frames) in the input video sequence, $H,W$ correspond to the spatial dimensions of the face image. We gradually down-sample the spatial extent of the feature maps and preserve the temporal dimension $T$. We also employ residual skip connections [14] and batch normalization [16] throughout the network. The encoder outputs a single $D$-dimensional vector for each of the $T$ input facial images to get a set of spatio-temporal features $T\times D$ to be passed to the speech decoder. Each time-step of the embedding generated from the encoder also contains information about the future lip movements and hence helps in the subsequent generation.

To achieve high-quality speech generation, we exploit the recent breakthroughs [30, 27] in text-to-speech generation. We adapt the Tacotron 2 [30] decoder which has been used to generate melspectrograms conditioned on text inputs. For our work, we condition the decoder on the encoded face embeddings from the previous module. We refer the reader to the Tacotron 2 [30] paper for more details about the decoder. The encoder and decoder are trained end-to-end by minimizing the L1 reconstruction loss between the generated and the ground-truth melspectrogram.

In the initial stages of training, up to $\approx 30K$ iterations, we employ teacher forcing similar to the text-to-speech counterpart. We hypothesize that this enables the decoder to learn an implicit speech-level language model to help in disambiguating homophenes. Similar observations are made in lip to text works [2] which employ a transformer-based sequence-to-sequence model. Over the course of the training, we gradually decay the teacher forcing to enforce the model to attend to the lip region and to prevent the implicit language model from over-fitting to the train set vocabulary. We examine the effect of this decay in sub-section 7.3.

The size of the visual context window for inferring the current speech time-step helps the model to disambiguate homophenes [10]. We employ about $6\times$ larger context size than prior works and show in sub-section 7.1 that this design choice results in significantly more accurate speech.

The primary focus of our work is on single-speaker lip to speech synthesis in unconstrained, large vocabulary settings. For the sake of comparison with previous works, we also train the Lip2Wav model on the GRID corpus [7] and the TCD-TIMIT lip speaker corpus [13]. Next, we train on all five speakers of our newly collected speaker-specific Lip2Wav dataset. Unless specified, all the datasets are divided into 90-5-5% train, validation and unseen test splits. In the Lip2Wav dataset, we create these splits using different videos ensuring that no part of the same video is used for both training and testing. The train and test splits are also released for fair comparison in future works.

We prepare a training input example by randomly sampling a contiguous sequence of $3$ seconds which corresponds to $T=75$ or $T=90$ depending on the frame rate (FPS) of the video. The effect of various context window sizes is studied in Section 7.1. We detect and crop the face from the video frames using the $S^{3}FD$ face detector [40]. The face crops are resized to $48\times 48$. The melspectrogram representation of the audio corresponding to the chosen short video segment is used as the desired ground-truth for training. For training on small datasets like GRID and TIMIT, we halve the hidden dimension to prevent over-fitting. We set the training batch size to $32$ and train until the mel reconstruction loss plateaus for at least $30K$ iterations. In our experiments for unconstrained single-speaker, convergence was achieved in about $200K$ iterations. The optimizer used is Adam [21] with an initial learning rate of $10^{-3}$. The model with the best performance on the validation set is chosen for testing and evaluation. More details, specifically a few minor speaker-specific hyper-parameter changes can be found in the publicly released code²²footnotemark: 2.

During inference, we provide only the sequence of lip movements and generate the speech in an auto-regressive fashion. Note that we can generate speech for any length of lip sequences. We simply take consecutive $T$ second windows and generate the speech for each of them independently and concatenate them together. We also maintain a small overlap across the sliding windows to adjust for boundary effects. We obtain the waveform from the generated melspectrogram using the standard Griffin-Lim algorithm [12]. We observed that neural vocoders [34] perform poorly in our case as our generated melspectrograms are significantly less accurate than state-of-the-art TTS systems. Finally, the ability to generate speech for lip sequences of any length is worth highlighting as the performance of the current lip-to-text works trained at sentence-level deteriorates sharply for long sentences that barely last over just $4$ - $5$ seconds [2].

We start by evaluating our approach against previous lip to speech works in constrained datasets, namely the GRID [7] corpus and TCD-TIMIT lip speaker corpus [13]. For the GRID dataset, we report the mean test scores for $4$ speakers which are also reported in the previous works. Tables 2 and 3 summarize the results for GRID and TIMIT datasets respectively.

As we can see, our approach outperforms competing methods across all objective metrics by a significant margin. The difference is particularly visible in the TIMIT [13] dataset, where the test set contains a lot of novel words unseen during training. This shows that our model learns to capture correlations across short phoneme sequences and pronounces new words better than previous methods.

We now move on to evaluating our approach in unconstrained datasets that contain a lot of head movements and much broader vocabularies. They also contain a significant amount of silences or pauses between words and sentences. It is here that we see a more vivid difference in our approach compared to previous approaches. We train our model independently on all $5$ speakers of our newly collected Lip2Wav dataset. The training details are mentioned in the sub-section 5.2. For comparison with previous works, we choose the best performing models [36, 9] on the TIMIT dataset based on STOI scores and report their performance after training on our Lip2Wav dataset. We compute the same metrics for speech intelligibility and quality that are used in Table 3. The scores for all five speakers for our method and the two competing methods across all three metrics are reported in Table 4.

Our approach produces much more intelligible and natural speech across different speakers and vocabulary sizes. Notably, our model has more accurate pronunciation, as can be seen in the increased STOI and ESTOI scores compared to the previous works.

In addition to speech quality and intelligibility metrics, it is important to manually evaluate the speech as these metrics are not perfect [9] measures.

Given the superior performance of our Lip2Wav approach on single-speaker lip to speech, we also obtain baseline results on the highly challenging problem of multi-speaker lip to speech synthesis for random identities. Note that the focus of the work is still primarily on single-speaker lip to speech. We adapt the approach presented in [19] and feed a speaker embedding as input to our model. We report our baseline results on the LRW [6] dataset intended for word-level lip-reading, i.e. it is used to measure the performance of recognizing a single word in a given short phrase of speech. We do not demonstrate on the LRS2 dataset [5] as its clean train set contains just $29$ hours of data, which is quite small for multi-speaker speech generation. For instance, multi-speaker text-to-speech generation datasets [39] containing a similar number of speakers contain several hundreds of hours of speech data.

In Table 7, we report the speech quality and intelligibility metrics achieved by our multi-speaker Lip2Wav model on the LRW test split. As none of the previous works in lip to speech tackle the multi-speaker case, we do not make any comparisons with them. We also report the WER by getting the text using the Google ASR API. For comparison, we also report the WER of the baseline lip to text work on LRW [6]. Note that the speech metric scores shown in Table 7 for word-level lip to speech cannot be directly compared with the single-speaker case which contains word sequences of various lengths along with pauses and silences.

We end our experimental section here. Apart from showing significant increases in performance from previous lip to speech works, we also achieve word-level multi-speaker lip to speech synthesis. In the next section, we conduct ablation studies on our model.

As stated before, the lip to speech task is highly ambiguous to be inferred solely from lip movements. One of the ways to combat this, is to provide reasonably large context information to the model to disambiguate a given viseme. Previous works, however, use only about $0.3-0.5$ seconds of context. In this work, we use close to $6\times$ this number and provide a context of $3$ seconds. This helps the model to disambiguate by learning co-occurrences of phonemes and words and the resulting improvement is evident in Table 8.

We plot the activations of the penultimate layer of the spatio-temporal face encoder in Figure 4 to show that our encoder is highly attentive towards the mouth region of the speaker. The attention alignment curve in Figure 5 shows that the decoder conditions on the appropriate video frame’s lips while generating the corresponding speech.

To accelerate the training of a sequence-to-sequence architecture, typically, the previous time step’s ground-truth (instead of the generated output) is given as input to the current time-step. While this is highly beneficial in the initial stages of training, we observed that gradually decaying the teacher forcing from $\approx 30K$ iterations significantly improves results and prevents over-fitting to the train vocabulary. A similar improvement is also observed in lip to text works [2]. In Table 9, we show the significant improvement in test scores by gradually decaying teacher forcing.

While using a 3D-CNN worked best in our experiments to capture both the spatial and temporal information in unconstrained settings, we also report in Table 10 the effect of using different kinds of encoders. We replace the encoder module while keeping the speech decoder module intact. We see that the best performance is obtained with a 3D-CNN encoder.