Data Standardization for Robust Lip Sync

As depicted in Fig. 2, the visual front-end, audio front-end, and task-related back-end are the three parts that make up modern lip sync model architectures [1]. The DSP should be placed between the raw images and the visual front-end. It takes raw images as inputs and produces standardized images, which are then fed into the visual front-end. Further, the DSP is built using the parametric image formation model described in Sec. III and thus has two components: a network for estimating the required coefficients from the input, and a renderer that uses the coefficients to synthesize images. Their implementations should fulfill the design requirements.

1. 1.

   The DSP is robust on its own in the wild.

2. 2.

   Preserving lip motion information from the raw images.

3. 3.

   Reducing compound distracting factors.

4. 4.

   No significant new distracting factors are introduced.

The image formation model is based on physical rules, which are stable in the wild, hence the DSP meets requirement 1. To satisfy requirement 2, a network that focuses on consistently resolving expression-related ambiguities is customized by modifying [11], see Sec. IV-B to Sec. IV-E. To satisfy requirement 3, standardized images are sythesized with modified coefficients, see Sec. IV-F. Regarding requirement 4, missing teeth in synthesized images might indeed introduce a new distracting factor. However, experimental results presented in Sec. V suggest that this has negligible negative effects on lip sync, as the cues for lip motion remain intact.

For training, several image collections are necessary. First, a large number of single-speaker videos are gathered, and then each video is processed for face detection/tracking and 3D landmark detection [11] in order to create face tracks [2]. Each face track is a sequential sequence of a subject’s facial images, and a subject may show up in more than one face track. Then, around $M$ facial images are sampled from each of the high-quality face tracks [8]. Finally, the dataset contains $K$ collections, and each has $M_{k}$ pairs of facial images and landmarks $\{l_{n}\}$ of a subject, i.e. $\boldsymbol{C}_{k}=\left\{\left(\boldsymbol{I}_{i},\{l_{n}\}_{i}\right)\right\}_{i=1}^{M_{k}}$.

We use speaking videos for the following reasons: 1) Facial images from short-duration videos are more likely to fulfill the assumption that they are of the same underlying subject-specific attributes. 2) They enhance training by introducing diversified lip motions and implicit temporal stability.

The model consists of a regression network and two trainable matrices, as depicted in Fig. 3(a). The estimation of preimage attributes is carried out using the regression network, termed as E-Net. Similar to [11] [12], the E-Net is implemented as a ResNet-50 [22] with the last fully-connected layer of 97 dimensions, for regressing a 97-dimensional vector of preimage attributes ($\boldsymbol{\beta}\in\mathbb{R}^{64}$, $\boldsymbol{\gamma}\in\mathbb{R}^{27}$, $\boldsymbol{p}\in\mathbb{R}^{6}$). One of those trainable matrices is the shared identity matrix of dimension 80-by-$K$, where 80 is the dimension of identity ($\boldsymbol{\alpha}\in\mathbb{R}^{80}$) and $K$ is the number of collections in the dataset. It is denoted as shared because the identity shared by all of the facial images in the $k$-th collection is represented in its $k$-th column. Similarly, the other is the shared texture matrix of dimension 80-by-$K$ since $\boldsymbol{\delta}\in\mathbb{R}^{80}$.

To measure the discrepancy for an estimated coefficient $\boldsymbol{\hat{x}}=\{\boldsymbol{\hat{\alpha}}_{k},\boldsymbol{\hat{\delta}}_{k},\boldsymbol{\hat{\beta}},\boldsymbol{\hat{p}},\boldsymbol{\hat{\gamma}}\}$, different losses are used, following [11].

The photometric loss. It measures the photometric consistency between the input $\boldsymbol{I}$ and the synthesized one $\boldsymbol{I}^{\prime}$.

where $\boldsymbol{\mathcal{M}}$ and $\boldsymbol{A}$ are two masks indicating the facial area and the per-pixel skin probability [11], respectively.

The landmarks distance (LMD) loss. It measures the geometric consistency between detected landmarks $\{l_{n}\}$ and ones $\{l_{n}^{\prime}\}$ reprojected from predefined 3D shape vertices [11].

where $\omega_{n}$ is the weight of the $n$-th landmark. Lip landmarks are weighted to 10, otherwise 1.

The regularization loss. It adds zero-mean Gaussian distribution prior constraints on 3DMM coefficients.

where $\omega_{\boldsymbol{\alpha}}=0.5$, $\omega_{\boldsymbol{\beta}}=2.0$ and $\omega_{\boldsymbol{\delta}}=0.5$.

In total, all terms are weighted combined as

where $\lambda_{pho}=1.92$, $\lambda_{lan}=1.6e{-3}$, and $\lambda_{reg}=3.0e{-4}$.

Most hyper-parameters are directly adopted from [11], with stronger regularization on the expression term to prevent the identification of all variations in facial shapes as expressions, thereby making the identity attribute trivial [13]. Note that we do not explicitly regularize the disentanglement between texture and illumination, as they will be replaced during face synthesis. However, a similar idea can be applied.

The whole model, including the E-Net and two trainable matrices, is trained jointly with built collections in a way similar to [11], with the following modifications made. 1) As depicted in Fig. 3(a), instead of being regressed, subject-specific attributes are trainable parameters indexed from shared matrices and shall be updated during training. 2) For a batch, we sample multiple collections first, then several data from each collection. Data from the same collection constrain one another and assist in resolving ambiguities as they share the same subject-specific attributes but differing preimage ones.

We propose to synthesize a standardized face with depth as inputs for subsequent AV tasks. As described in Sec. IV-A, it is expected that the effects of compound distracting factors will be reduced in the synthesized images while the expression information from the raw input will be preserved. This can be done by setting appropriate coefficients to the image formation model, as stated in Sec. III.

In specific, we utilize modified coefficients for rendering instead of all coefficients estimated from the raw input. As depicted in Fig. 3(b), the trained E-Net estimates the expression singly from the input. Meanwhile, attributes corresponding to distracting factors are explicitly set to predefined default values, regardless of inputs. By rendering with modified coefficients, the synthesized images are devoid of the effects of compound distracting factors while preserving the expression from the raw images. For the ease of subsequent AV tasks, we set uniform white illumination, mean identity, mean texture, and frontal pose as defaults, so that $\boldsymbol{\hat{x}}=\{\boldsymbol{\alpha}=\boldsymbol{0},\boldsymbol{\delta}=\boldsymbol{0},\boldsymbol{\hat{\beta}},\boldsymbol{\gamma}=\boldsymbol{\gamma}_{\boldsymbol{0}},\boldsymbol{p}=\boldsymbol{p}_{\boldsymbol{0}}\}$, and render an expressive image $\boldsymbol{I}^{\prime}(\boldsymbol{\hat{x}})$. Examples are given in Fig. 4, where raw images (column a) vary greatly whereas synthesized images (column e) are standardized and free of distracting factors (pose, illumination, etc.).

We can also render a corresponding pseudo depth map $\boldsymbol{D}^{\prime}$ in order to more accurately describe 3D lip motions. This is done by rendering with $\boldsymbol{Z}^{\prime}_{\boldsymbol{S}}$ instead of illuminated $\boldsymbol{T}^{\prime}$ as the texture (see (4)), where $\boldsymbol{Z}^{\prime}_{\boldsymbol{S}}$ is the normalized Z-coordinator of $\boldsymbol{S}$,

where $\boldsymbol{Z}_{min}=\min(\boldsymbol{Z}_{\boldsymbol{S}_{i}})$ and $\boldsymbol{Z}_{max}=\max(\boldsymbol{Z}_{\boldsymbol{S}_{i}})$ are the min depth and max depth of the video clip, respectively.

Overall, we synthesize standardized expressive RGB/RGBD images from raw facial images. Notably, we also reduce the effect of the often poor image quality found in raw data via synthesis.

We develop the proposed model and the lip-sync model using VoxCeleb2 [23]. Its dev-split has over 1 million utterances from 145569 videos of 5994 subjects and its test-split has 36237 utterances from 4911 videos of 118 subjects.

To develop the proposed model, we collect 5000 subjects from the dev-split. Ten utterances are sampled from each of the two videos for each subject, resulting in $K=50k$ collections with each having about $M=50$ samples. Separately, a minidev-split consisting of 5000 utterances from 500 subjects is sampled from the dev-split for training the lip-sync model, while the entire test split is used for evaluation. Note that the minidev-split is less than ${1}/{7}$ the size of the test-split.

Regarding the ASD task, we directly evaluate lip-sync models models trained with VoxCeleb2 on the test split of ASW dataset [2]. The test split consists of 53 videos (51 accessible), with 4.5 hours of active (i.e., sync) face tracks and 3.4 hours of inactive ones.

According to Sec. IV-E, we first train the proposed model. More specifically, two matrices are initialized with Gaussian noise $\mathcal{N}(\mu=0,\,\sigma=0.01)$ and the E-Net is ImageNet-pretrained [24]. Two collections and eight samples from each are sampled for each batch. With a constant learning rate of $1.0e{-4}$, the model is trained using the Adam optimizer for 20 epochs.

Due to the lack of datasets with ground truth (GT) expressions of $\boldsymbol{\beta}\in\mathbb{R}^{64}$, we use 5-fold cross-validation for the evaluation. Particularly, $50k$ collections are split into 5 groups. Each time, we train with four groups and evaluate the holdout group, whose GT coefficients are obtained by overfitting it with the proposed method. We use the LMD (see (6)) on lip landmarks as a metric. The group LMD is defined as the mean of the collection LMDs, and each is defined as the mean of the data LMDs. We form mixed coefficients in order to isolate the effects of certain estimated attributes. For instance, the mixed coefficients are formed as $\boldsymbol{\hat{x}}=\{\boldsymbol{\alpha}_{gt},\boldsymbol{\delta}_{gt},\boldsymbol{\hat{\beta}},\boldsymbol{p}_{gt},\boldsymbol{\gamma}_{gt}\}$ in order to evaluate the estimated expression.

We compare with Deep3D [11] and MGCNet [12]. The min, max, and average group LMDs are reported in Tab. I, and results with GTs are also included as references. For Deep3D and MGCNet, results with estimated expressions perform worse than counterparts with GT expressions, and additionally performing image-wise identity estimation can improve results. These results suggest that they incorrectly ascribe some parts of face morphings to identity variations (see Fig. 4(b)(c)), and as a result, their estimated expressions when used alone cannot adequately describe lip motions. Our results using the E-Net estimated expressions, otherwise, are more in line with those using GTs, showing that the E-Net can more consistently attribute facial morphings to expressions.

We conduct experiments on lip sync and ASD tasks. We adopt the self-supervised PerfectMatch [1] for lip sync, using synthesized images as the visual input and Mel-frequence cepstral coefficients (MFCCs) as the audio input. We first train a model on the dev-split of VoxCeleb2 using the raw face tracks as a baseline for comparisons. Then, a variety of versions are trained using the minidev split, including raw face tracks, synthesized images with E-Net estimated expressions, and synthesized images with estimated identities and expressions from Deep3D.

For evaluation, we follow [1] to extract audio and visual features from every 0.2s segment with a stride of 0.04s. Then, we calculate the cosine similarity between each visual feature and each audio feature within a $\pm 15$ frame window, and we determine the offset providing the min distance. If the determined offset is within $\pm 1$ frame of the GT offset, it is correct.

The evaluation results are listed in Tab. II. With less data, models trained with synthesized images outperform those trained with raw face tracks. Further, models trained with RGBD images yield better results. This may be the case because depth information makes 3D lip motions more distinct, see Fig. 4. Particularly, models trained with E-Net estimated expressions consistently outperform others, suggesting that improved lip motion description may directly benefit lip sync.

In the ASD task, effective lip-sync models are directly applied to ASW’s test split [2] without training or finetuning on ASW. Following [2], an AV pair exceeding a threshold distance is deemed active. Our pipeline preprocesses face tracks, synthesizes images, and averages cosine similarities across 15 frames for robustness as suggested in [1]. Results in Tab. III reveal that lip-sync models exhibit great generalizability by matching the strong baseline [2] in all metrics, including average precision (AP), the area under the receiver operating characteristic (AUROC), and equal error rate (EER). This evaluation is challenging as our models are evaluated across datasets and trained with less data.