Learning Facial Representations from the Cycle-consistency of Face

We start with the introduction of facial motion cycle-consistency in the following. We denote that the expression representations $R_{\textrm{exp}}$ are learned by an encoder $E_{\textrm{exp}}$ from the input face image $F$:

$$R_{\textrm{exp}}=E_{\textrm{exp}}(F)\,.$$

(2)

As the idea described in the previous section, we model a facial expression as the optical flow field between the neutral face and the face with expression. Therefore, the forward ($F\to\hat{F}$) optical flow field $flow^{\textrm{fw}}\in\mathbb{R}^{2\times H\times W}$, where $H$ and $W$ is the height and width of the input image, is learned by the decoder $D_{\textrm{flow}}$ from expression representations. Moreover, according to the well-known forward-backward flow consistency [1, 19], we can compute the backward ($\hat{F}\to F$) optical flow field $flow^{\textrm{bw}}$ according to $flow^{\textrm{fw}}$, which basically is inverse $flow^{\textrm{fw}}$ by a warp function $\mathcal{W}$:

$$\begin{split}flow^{\textrm{fw}}=&D_{\textrm{flow}}(R_{\textrm{exp}})\,,\\ flow^{\textrm{bw}}=&-\mathcal{W}(flow^{\textrm{fw}},flow^{\textrm{fw}})\,.\end{split}$$

(3)

We use bilinear interpolation to implement the warping operation $\mathcal{W}$ as in [39]. By using the forward optical flow field $flow^{\textrm{fw}}$ we can warp $F$ pixel-wisely to obtain an intermediate facial image, denoted as $\tilde{F}$. Followed by using the corresponding backward optical flow field $flow^{\textrm{bw}}$, we are able to warp back from $\tilde{F}$ to reconstruct $F$. This procedure straightforwardly lead to a reconstruction loss $\mathcal{L}_{\textrm{flow}}$ which is defined as:

$$\mathcal{L}_{\textrm{flow}}=|F-\mathcal{W}(\mathcal{W}(F,flow^{\textrm{fw}}),flow^{\textrm{bw}})|\,.$$

(4)

Furthermore, we exploit a general image feature extraction to represent a face image, that is, the coarse-to-fine feature maps $feat_{F}$ obtained from layers conv2$\_$1, and conv3$\_$1 of VGG19 network pre-trained on ImageNet [37]. Given a forward flow field $flow^{\textrm{fw}}$, we simply use the bilinear interpolation function $d_{s}(\cdot)$ to obtain $d_{s}(flow^{\textrm{fw}})$ of the size equal to $feat_{F}$. The de-expression is then achieved by first warping $feat_{F}$ with $d_{s}(flow^{\textrm{fw}})$ and then adopting a decoder $D_{\textrm{exp}}$ to generate neutral face image $\hat{F}$:

$$\hat{F}=D_{\textrm{exp}}(\mathcal{W}(feat_{F},d_{s}(flow^{\textrm{fw}})))\,.$$

(5)

Moreover, we argue that the image features $feat_{\hat{F}}$ of a neutral face obtained by VGG19 could be warped back via the downsampled backward flow $d_{s}(flow^{\textrm{bw}})$ and then be fed into the decoder $D_{\textrm{exp}}$ for reconstructing a face with expression, denoted as $F^{{}^{\prime}}$, which ideally should be identical to the original face $F$. This process is exactly the re-expression:

$$F^{{}^{\prime}}=D_{\textrm{exp}}(\mathcal{W}(feat_{\hat{F}},d_{s}(flow^{\textrm{bw}})))\,.$$

(6)

The change on a face image $F$ caused by a facial motion can be expressed in terms of a spatial image transformation $T$, where we denote the corresponding face image with different motion but the same identity as $F_{T}$. As both $F$ and $F_{T}$ are with the same identity, their corresponding neutral faces should be also identical. That is, their decoded neutral faces after performing de-expression are invariant to each other, which leads to the constraint:

$$\hat{F}=\hat{F_{T}}\,.$$

(7)

Following the concept of this invariance, we should be able to apply the re-expression operation on $feat_{\hat{F_{T}}}$ (the features for the decoded neutral face of $F_{T}$) via the downsampled backward flow $d_{s}(flow^{\textrm{bw}}_{F})$ of $F$ (related to the expression of $F$) to reconstruct a face image denoted as $F^{{}^{\prime\prime}}=D_{\textrm{exp}}(\mathcal{W}(feat_{\hat{F_{T}}},d_{s}(flow^{\textrm{bw}}_{F})))$, which ideally is quite similar to the original $F$ due to the hypothesis that $feat_{\hat{F}}=feat_{\hat{F_{T}}}$ as $\hat{F}=\hat{F_{T}}$. The similar story holds for performing re-expression on $feat_{\hat{F}}$ by $d_{s}(flow^{\textrm{bw}}_{F_{T}})$ to reconstruct $F^{{}^{\prime\prime}}_{T}=D_{\textrm{exp}}(\mathcal{W}(feat_{\hat{F}},d_{s}(flow^{\textrm{bw}}_{F_{T}})))$, which is almost identical to $F_{T}$. The illustration of this invariance, also named as facial motion cycle-consistency, is shown in Figure 3(a).

The reconstruction derived from the invariance (that is $F^{{}^{\prime\prime}}$ versus $F$ and $F_{T}^{{}^{\prime\prime}}$ versus $F_{T}$) builds up the objectives $\mathcal{L}_{\textrm{exp}}$ for learning the expression representations $R_{\textrm{exp}}$, where we utilize both the L1 loss and the perceptual loss [11, 21] to evaluate the error of reconstruction:

	$\displaystyle\mathcal{L}_{\textrm{exp}}(F,F_{T})=$	$\displaystyle|F-F^{{}^{\prime\prime}}|+|F_{T}-F_{T}^{{}^{\prime\prime}}|$		(8)
	$\displaystyle+$	$\displaystyle\lambda(\Phi(F,F^{{}^{\prime\prime}})+\Phi(F_{T},F^{{}^{\prime\prime}}_{T}))\,,$

where $\lambda$ is set to $0.05$ to balance the L1 and perceptual losses. The perceptual loss is defined as $\Phi(F,F^{{}^{\prime\prime}})=\sum_{l}\lVert\mathit{\phi}^{l}(F)-\mathit{\phi}^{l}(F^{{}^{\prime\prime}})\rVert_{2}+\sum_{l}\lVert\mathcal{G}(\mathit{\phi}^{l}(F))-\mathcal{G}(\mathit{\phi}^{l}(F^{{}^{\prime\prime}}))\rVert_{2}$. The function $\phi^{l}(\cdot)$ extracts VGG19 features from layer $l$, in which the conv2_1, conv3_1, and conv4_1 layers are used here. The function $\mathcal{G}(\cdot)$ calculates the Gram matrix of the feature map.

In terms of identity representation $R_{\textrm{id}}$, we utilize the encoder $E_{\textrm{id}}$ to extract $R_{\textrm{id}}$ from the input face image $F$:

$$R_{\textrm{id}}=E_{\textrm{id}}(F)\,.$$

(9)

Based on the idea described previously, we argue that the identity representation could be deprived from the neutral face to obtain the mean face. To implement the de-identity operation, we design a decoder $D_{\textrm{id}}$ to generate the mean face $\bar{F}$ from the modulated VGG features $feat_{\hat{F}}$ of a neutral face $\hat{F}$, which is similar to the idea of feature modulation idea proposed in the AdaIN paper [18]:

$$\bar{F}=D_{\textrm{id}}(\frac{feat_{\hat{F}}-\mu({feat_{\hat{F}}})}{\sigma(feat_{\hat{F}})}\sigma^{\textrm{m}}+\mu^{\textrm{m}})\,,$$

(10)

where $\mu(\cdot)$ and $\sigma(\cdot)$ are used to compute the mean and standard deviation respectively, and $\mu^{\textrm{m}}$ and $\sigma^{\textrm{m}}$ are learned from $R_{\textrm{id}}$ by the multi-layer perceptron $\textrm{MLP}_{\textrm{de}}$:

$$\mu^{\textrm{m}},\sigma^{\textrm{m}}=\textrm{MLP}_{\textrm{de}}(R_{\textrm{id}})\,.$$

(11)

Furthermore, the re-identity can be achieved in a similar manner but reversely with the decoder $D_{\textrm{id}}$:

$$\hat{F^{{}^{\prime}}}=D_{\textrm{id}}(\frac{feat_{\bar{F}}-\mu({feat_{\bar{F}}})}{\sigma(feat_{\bar{F}})}\sigma^{\textrm{id}}+\mu^{\textrm{id}})\,,$$

(12)

where the $\mu^{\textrm{id}}$ and $\sigma^{\textrm{id}}$ are also learned from $R_{\textrm{id}}$ but by another multi-layer perceptron $\textrm{MLP}_{\textrm{re}}$.

We hypothesize that the mean face is global for all the faces. In other words, no matter starting from which neural face of any identity, we should always obtain the same mean face after performing de-identity operation. Given the neutral faces $\hat{F_{1}}$ and $\hat{F_{2}}$ of different identities, we can derive the invariance related to identity as:

$$\bar{F_{1}}=\bar{F_{2}}\,.$$

(13)

Therefore, we should be able to reconstruct $\hat{F_{1}}$ by using its corresponding $\{\mu_{1}^{\textrm{id}},\sigma_{1}^{\textrm{id}}\}$ to apply the re-identity operation on the mean face obtained from $\hat{F_{2}}$. The result of this reconstruction is denoted as $\hat{F}_{1}^{{}^{\prime\prime}}$:

$$\hat{F}_{1}^{{}^{\prime\prime}}=D_{\textrm{id}}(\frac{feat_{\bar{F}_{2}}-\mu(feat_{\bar{F}_{2}})}{\sigma(feat_{\bar{F}_{2}})}\sigma^{\textrm{id}}_{1}+\mu^{\textrm{id}}_{1})\,.$$

(14)

Again, similar story holds to perform re-identity operation (with $\{\mu_{2}^{\textrm{id}},\sigma_{2}^{\textrm{id}}\}$) on the mean face obtained from $\hat{F_{1}}$ to reconstruct $\hat{F_{2}}$. We denote the reconstruction result as $\hat{F}_{2}^{{}^{\prime\prime}}$:

$$\hat{F}_{2}^{{}^{\prime\prime}}=D_{\textrm{id}}(\frac{feat_{\bar{F}_{1}}-\mu(feat_{\bar{F}_{1}})}{\sigma(feat_{\bar{F}_{1}})}\sigma^{\textrm{id}}_{2}+\mu^{\textrm{id}}_{2})\,.$$

(15)

The illustration of this invariance related to identity representations, also named as identity cycle-consistency, is shown in Figure 3(b).

As the way that $\mathcal{L}_{\textrm{exp}}$ is defined, the reconstruction derived from the invariance (that is, $\hat{F}_{1}^{{}^{\prime\prime}}$ versus $\hat{F}_{1}$ and $\hat{F}_{2}^{{}^{\prime\prime}}$ versus $\hat{F}_{2}$) leads to the objectives $\mathcal{L}_{\textrm{id}}$ for learning the identity representations $R_{\textrm{id}}$:

	$\displaystyle\mathcal{L}_{\textrm{id}}(\hat{F}_{1},\hat{F}_{2})=$	$\displaystyle|\hat{F}_{1}-\hat{F}_{1}^{{}^{\prime\prime}}|+|\hat{F}_{2}-\hat{F}_{2}^{{}^{\prime\prime}}|$		(16)
	$\displaystyle+$	$\displaystyle\lambda(\Phi(\hat{F}_{1},\hat{F}_{1}^{{}^{\prime\prime}})+\Phi(\hat{F}_{2},\hat{F}_{2}^{{}^{\prime\prime}}))\,.$

Moreover, we additionally introduce a margin loss $\mathcal{L}_{\textrm{m}}$ to constrain the mean face:

$$\mathcal{L}_{\textrm{m}}(\bar{F},\hat{F})=\max({\left\|\bar{F}-\hat{F}\right\|}-\alpha,0)\,,$$

(17)

where we set $\alpha=0.1$ in all experiments. The main motivation behind this margin loss is that we would like to constrain the difference between the mean face and the neutral face to be within a margin. Otherwise the obtained mean face could potentially become an arbitrary image far from a face image.

The facial motion cycle-consistency described in Section 2.2 involves an image pair of faces with different expressions/poses but of the same identity. Fortunately, this type of data can be easily available from the video recording of human faces, for instance, the video of interview or talk-show, which exists widely on the Internet nowadays. Given any two frames in this type of video clip of a person, we can easily obtain a pair of facial images showing different expressions. Therefore, we can take the advantage of this type of video sequences (as the dataset described in the following) and collect training data for learning both the expression and identity representations in an unsupervised manner.

Figure 4 illustrates several examples of the intermediate results obtained from our model, including the input faces, forward flow fields, neutral faces, backward flow fields, mean faces, and the faces reconstructed from their neutral ones. We demonstrate that the proposed method can handle face images with large variation in poses and can preserve facial attributes such as wearing glasses or beard.

Visualization of the facial motion flows presents both the head motion and the movement of facial muscles. The neutral faces are deprived of facial motions in comparison to their original facial images. Moreover, the mean faces obtained from different input images are almost identical to each other, which is in line with our assumption of identity invariance.

Given the trained model, we investigate the learnt expression representation by evaluating the performance of its applications on expression recognition and head pose regression. The goal is to verify whether the expression representation successfully encodes the information related to the facial motions and poses as our definition (i.e. the expression factor contains all the variations between a face image and its corresponding neutral face of the same identity, including facial motions and head poses). We conduct linear-protocol evaluation scheme for demonstrating the effectiveness of our method.

We also investigate the applications of identity representations learned by using the proposed method on the VoxCeleb dataset. Good performance of person recognition demonstrates that our identity representations do contain rich information related to identities.

Frontalization is the process of synthesizing the frontal facing view of a single facial image. In this work, there are two ways to obtain the neutral face in the frontal view: de-expression and re-identity. The de-expression operation removes the head motion and facial expressions from facial images and thus generates the neutral faces with frontal view. On the other hand, the re-identity operation recovers the neutral face by adding the identity to the mean face which is already in frontal view. As shown in Figure 5(a), the proposed method is able to synthesize the neutral faces from the facial images with various poses by the de-expression operation. The input images are from the LFW dataset [17] which are never seen during the training of our model. We also show a state-of-the-art approach in Figure 5(b) which additionally uses facial landmarks [14] for qualitative comparison.

We notice that the synthesized images from the proposed method are a little bit blurry, we hypothesize that it might be caused by the plenty blurry training images in the Voxceleb dataset. We believe that further improvements can be obtained by using other high-quality datasets.

The proposed model can naturally be used to perform image-to-image translation by transferring the facial motion of the source image into the target one. To this end, we simply calculate and then apply the backward flow field of the source image to warp the neutral face of the target image via the re-expression operation. As shown in Figure 6, our method can transfer the head pose and expression from the source to the target without noticeable artifacts. On the other hand, the results of X2Face method [42] reveal visible artifacts when the pose difference between source and target is large.