LiFi: Towards Linguistically Informed Frame Interpolation

Video frame interpolation and extrapolation is the task of synthesizing new video frames conditioned on the context of a given video [1]. Contemporary applications of such interpolation are video playback software for increasing frame rates, video editing software for creating slow motion effects, and virtual reality software to decrease resource usage. From the early per-pixel phase-based shifting approaches such as by [2], current approaches have shifted to generating video frames using techniques like optical flow or stereo methods [1, 3]. These types of approaches typically involve two steps: motion estimation and pixel synthesis. While motion estimation is needed to understand the movement of different objects of the images across frames, pixel synthesis focuses on the generation of the new data.

A related task to video frame interpolation is talking face generation. Here, given an audio waveform, the task is to automatically synthesize a talking face [4, 5]. In recent times, these approaches have become popular with both academic and non-academic purposes [6]. While, on one hand, they are being used to extend speechreading models to low resource languages [7], on the other, many of them are also used to generate fake news and paid content as well.

The literature on talking face generation, as well as video interpolation and extrapolation use the conventional video quality metrics such as root mean squared (RMS) distance, structural similarity index, etc. to measure the quality of generated videos [8]. Although this evaluates the quality of generated pixels well, but none of the research works show how it translates to linguistically plausible video frames. This is important since speech being a linguistic problem,f cannot be solely addressed by non-linguistic metrics like RMS distance. The goal of this paper is to investigate how the different video interpolation and extrapolation algorithms are able to capture linguistic differences between the generated videos.

Speech as a natural signal is composed of three parts [9]: visual modality, audio modality and the context in which it was spoken (crudely the role played by language). Correspondingly, there are three tasks for modeling speech: speech-reading (or popularly known as lipreading) [7, 10], speech recognition (or ASR) [11] and language modeling [12]. The part of speech which is closest to the speech video generation task is the visual modality of speech; and visemes are the fundamental units of this part of speech. Calculating metrics such as mean squared error (MSE) over the whole video does not directly yield any information on these aspects of speech which makes us question the faithfulness of the thus attained reconstruction. Therefore, the focus of this work is to investigate video generation model’s understanding of the visual speech modality. To this end, we propose six tasks for looking at different aspects of the visual speech modality. Hence, with this work, we try to make the following contributions: 1. We explore different video interpolation and extrapolation networks for usage on speechreading videos and propose a new auxiliary method of ROI loss using an visemic ROI unit for attaining faithful reconstructions.
2. For the first time in literature, we test the video frame generation algorithms on different aspects of speech such as visemic completion, generating prefix and suffix from context, word-level understanding, etc. These facets of language are critical to a model’s language understanding. We show that most of these networks are not able to capture the language aspect of a speech video.
3. We release six new challenge datasets corresponding to different language aspects. These have been verified automatically and manually and are meant to facilitate reproduction and follow-up testing and interpretation.

Most of the previous research on speech video reconstruction and interpolation focuses on conventional metrics like MSE, SSIM and PSNR, which although ensure good quality reconstructions [13, 3], can be misleading in terms of the nuances of the underlying dataset. Videos of the same person saying two different things can have fairly high values of the mentioned metrics thus indicating faithful reconstruction, which makes the evaluation difficult (Figures 1, 2)¹¹1We provide two examples in the Figures 1 and 2 and a few more in the Figure 5 in the Appendix.. We propose the following evaluation methods and corresponding datasets ²²2We make the datasets available at https://sites.google.com/view/yaman-kumar/speech-and-language/linguistically-informed-lrs-3-dataset. Training scripts for the models in Section 3.2 are available at https://github.com/midas-research/linguistically-informed-frame-interpolation/ (Table 3) for the speech videos to take into account the underlying language and speech information.

1. RandomFrame: In this type of frame corruption, the frames at random timestamps are replaced with white noise. In a real scenario, the missing frames are dropped however the indices of the dropped frames are generally available [14]. To indicate the missing frames, we insert white noise images in their place. We do this type of corruption in two settings: 40% and 75% corruption rate. We call the 75% case as one with the extreme sparsity. A model that can work in such conditions can deal with extremely sparse speech videos and high dropping rates. Random corruption tests if the model effectively interpolates between different poses, facial expressions and probable word completions.

2. Visemic Corruption: A viseme is a specific facial image corresponding to a particular sound [9]. In this type of corruption, we choose visemically equivalent sub-words in two or more words and corrupt the first viseme within the subword. For example, consider the words ‘million’ and ‘billion’, both are visemically equivalent. We corrupt the first common viseme - ‘p’ and show the results for such corruption in Table 5. The dataset also includes the cases where the visemic equivalence does not occur at the start of the word. For example, the first common viseme which is corrupted is ‘@’, in ‘Probably’ and ‘Possibly’, when the phonetic phrase “ebli” is spoken in both. The timestamps of specific phonemes are obtained through the Montreal Forced Aligner [15]. After that, the phonemes were converted to visemes. Through this type of corruption, we test the model’s knowledge of specific visemes given the context. We list the SSIM metrics after reconstructing each viseme is detailed in Table 8 of the Appendix.

3. InterWord and IntraWord Corruptions: In this type of corruption, we corrupt 80% of approximately 20,000 unigram and bigram combinations. Corruption was done such that for a particular gram, 10% each of the prefix and suffix of the overall gram remains in the original state after the corruption. For example, consider the bigram ‘United States’, the corruption of this bigram results in ‘Un*********es’. A model that works well on such corruptions understands the context of the words well and is robust to corruptions during the transition from one word to another.

4. Prefix Completion: For this corruption, we remove the start frames of a video. We do this for two levels of corruptions- a low level, 40% and a high level corruption (75%).

5. Suffix Completion: Similar to the Prefix Completion test, we corrupt the ending frames from the sequence of input, which is also performed for 40% and 75% corruption. These two tests make sure that the model understands both the word start and word end and is not biased towards either. While predicting word start requires context from the previous word and the remaining part of the word, predicting the end of the word requires context from the prefix and the next word.

6. POS Tagged Corruptions: In this type of corruption, we identify nouns, pronouns, verbs, adjectives and determinants using Spacy’s English language POS tagger³³3https://spacy.io/api/tagger/. The corresponding timestamps for all such tagged words were obtained through MFA. Following this, we decided to corrupt these words for further study. For example, in a video from LRS3, the speaker says ‘The short answer to the question is that no, it’s not the same thing’. The POS tagger identifies the three words ‘answer’, ‘question’ and ‘thing’ as nouns. Thus all of the frames belonging to these three words were corrupted while inducing corruptions to nouns.

By introducing such corruptions, we test how well do the models understand semantic features of the language spoken. For example, for a particular model proper nouns may be more difficult to predict than pronouns. This test helps to bring such nuances into forefront.

In this section, we present the experimental results for the models given in the Section 3.2 on the tests explained in the Section 2. Results of the test suite are given in the Table 2. We notice that for random corruption, though fully convolutional models beat bi-directional LSTM for low levels of noise, the bi-directional LSTMs gives a much better performance for longer sequential frame reconstructions. We see that the ROI extraction unit helps boost the performance of the FCN model. The ROI stream network helps attain a better PSNR and SSIM comparable with LSTM. This can be attributed to the model’s ability to focus more on visemes indirectly by learning a better reconstruction of the ROI for speech videos. The secondary stream is a very shallow network consisting of two convolutions and a spatial transformer which helps extract the mouth ROI from the already generated output. Thus, ROI loss further helps the network learn more about the visemic structure of the reconstruction and help attain better scores on the metrics.

We also show a quantitative comparison with Super SloMo [3]. Since Super SloMo only allows for the prediction of the contiguous frame we had to significantly modify the experimental setup to make any comparisons with it. The architecture did not allow the first and last frame of the batch from being corrupted. Therefore, though the quantitative results are the best for it, Super-Slomo suffers from severe experimental restrictions unlike the other models presented. In the case of continuous corruption, it will require the start and the end frames to reconstruct the intermediate ones, hence it is difficult to implement it in real time while taking into account the random nature of the noise/corruption.

If we look at the results on viseme and word-level datasets (Table 5), it can be seen that FCN3D with ROI outperforms SuperSlomo in terms of SSIM in word level corruption and PSNR for Viseme corruption attaining a PSNR of 26.54 against 25.2605 by Super SloMo. This is due to the smaller duration of corruption in these datasets as compared to those in Table 2. This is also likely to be the practical corruption distribution for realtime speech video streaming where networks like Super SloMo cannot function since they generate frames for only third time $t_{3}$ step between $t_{1}$ and $t_{2}$ (one of which may not exist). The proposed models are not restricted by this limitation and for the speech videos, capture a longer temporal context (not just the frames at timestamps $t_{1}$ and $t_{2}$) besides focusing on the mouth region using the ROI unit. As evident from Table 2 for random corruption FCN3D+ROI gives a high PSNR value of 24.75 beating BDLSTM and coming in close with Super SloMo with a PSNR value of 28.26, while LSTM outperforms FCN3D+ROI for prefix and suffix corruptions however with a smaller difference of 2.21, while Super SloMo outperforms both with a higher margin due to its ability to incorporate optical flow. This highlights a key limitation of such methods for speech video interpolation tasks due to their inability to capture context also reinforcing the need for a better test suite for a more faithful evaluation of speech video interpolation.

In this papaer, we demonstrate the necessity of a comprehensive linguistically-informed test suite that encompasses all the major aspects of speech in the task of speech video interpolation and reconstruction. We release six challenge datasets for this purpose. We also compare several different contemporary deep learning models for the different tasks proposed. We show the importance of incorporating visemic loss and provide a natural proxy to judge the visemic nature of reconstruction. In the future, we would like to cover more such linguistic aspects of speech. A parallel task would also be to take the audio-modality into account while reconstructing the corrupted and missing frames in video interpolation.