FlowFace++: Explicit Semantic Flow-supervised End-to-End Face Swapping

In many previous face-swapping methods, the source face is typically mapped into an ID embedding using a pre-trained face recognition model. The ID embedding is trained on recognition tasks, which leads to indistinguishable intra-class variance for a specific person. This means that the ID embedding may not fully capture the personalized appearances of a face during face swapping. This could potentially result in a loss of important details in the final face-swapping output. Additionally, previous methods use another face encoder to extract the target face attributes. This means that they propose different face representations for the source and the target faces. The distinguishable face representations bring challenges to their fusion, which may damage the fine granularity of the resulting image.

In our work, the above two issues are addressed by a pretrained face-MAE model (MAE encoder) that provides a prior fine granularity of facial representation. The pretrained face-MAE model is built with image reconstruction, instead of ID recognition. It is shared by the source and target faces, so our work provides a unified face representation for the source and target faces.

Specifically, we opt to employ a shared encoder that projects both the source and target face into a unified latent representation. The encoder architecture is based on the MAE [29] framework and was pre-trained on a large-scale face dataset [9, 10, 11, 12] consisting of 2.17 million images, utilizing a masked training strategy. When contrasted with the compact latent code of StyleGAN2 [13] and the identity embedding, the latent representation of MAE has the ability to more effectively capture both facial appearances and identity information. This is due to the use of masked training, which necessitates the reconstruction of masked image patches from visible neighboring patches. As a result, each patch embedding contains rich topology and semantic information. Using the pre-trained MAE encoder $E_{f}$, we can project a given facial image $I_{*}$ into a latent representation, commonly referred to as patch embeddings:

where $e_{*}\in R^{N*L}$. $N$ and $L$ denote the number of patches and the dimension of each embedding, respectively.

Through the use of the shared MAE encoder, both the source and target faces are projected into a representational latent space. The next step is to fuse the source identity information with the target attribute within this latent space. It’s assumed that related patches, such as the nose to nose, would convey identity information during the transfer process. To account for this, we developed a cross-attention fusion module (CAFM) that can dynamically aggregate identity information from the source and blend it into the target in an adaptive and patch-wise manner.

As shown in Figure 2, CAFM comprises a cross-attention block and two standard transformer blocks [30]. To begin, we calculate the $Q,K,V$ values for each patch embedding in the source ($e_{s}$) and target ($e_{t}$) sets. Then the cross attention is computed by:

In this equation, $\operatorname{CA}$ denotes Cross Attention, $Q_{*},K_{*},V_{*}$ are predicted using attention heads, and $d_{k}$ denotes the dimension of $K_{*}$. The cross-attention mechanism characterizes the relationship between each target patch and the source patches. Subsequently, we aggregate the source identity information using the computed $\operatorname{CA}$ and fuse it with the target values through addition:

After that, $V_{fu}$ are normalized through a layer normalization (LN) and processed by multi-layer perceptrons (MLP). Both the Cross Attention and MLP are accompanied by skip connections. The fused embeddings $e_{fu}$ are then fed into two transformer blocks, resulting in the final output $e_{o}$.

Finally, a convolutional decoder is utilized to generate the final swapped face image $I_{o}$ from the output $e_{o}$. In contrast to the ViT decoder in MAE, we find that the convolutional decoder produces more realistic results.

To train our face-swapping network $F^{swa}$, two human face images (i.e. $I_{s}$ and $I_{t}$) will be used as the source face and target face, respectively, serving as the two inputs of $F^{swa}$. Generally, there is no ground truth available for the results of face swapping. To further constrain the output distribution of the swapping result $I_{o}$, the training data include a portion $25\%$ of $\{$($I_{s}$, $I_{t}$)$\}$ where $I_{s}=I_{t}$. This portion of data allows using pixel-wise reconstruction loss, as done in [7, 23, 3, 26, 2]. The other $75\%$ of the training data consists of $\{$($I_{s}$, $I_{t}$)$\}$ where $I_{s}$$\neq$$I_{t}$.

We design seven loss functions from the aspects of facial shape, posture, texture, expression, and identity, to constrain the result faces generated by our $F^{swa}$:

where $\lambda_{rec}$, $\lambda_{id}$, $\lambda_{exp}$, $\lambda_{ldmk}$, $\lambda_{attr}$ are hyperparameters for each term. We set $\lambda_{rec}$=10, $\lambda_{id}$=5, $\lambda_{exp}$=10, $\lambda_{ldmk}$=5000, $\lambda_{attr}$=2 and $\lambda_{flow}$=3 in our experiment.

Adversarial Loss. To enhance the realism of the face swapping results, we employ the hinge version adversarial loss[31] for training, denoted by $L_{adv}$:

where $D^{swa}$ is the discriminator which is trained with:

Reconstruction Loss. As mentioned above, in 25% of the training data, $I_{s}$ and $I_{t}$ are identical to each other. It means that $I_{t}$ is the desired result of $I_{o}$. For these data, $I_{o}$ is supervised by an additional pixel-wise reconstruction loss:

It is noteworthy that the reconstruction loss is not existed when $I_{s}$$\neq$$I_{t}$.

Posture Loss. To maintain proper face posture during face swapping, we utilize the landmark loss as a constraint:

where $P_{o}$ represents the landmarks of $I_{o}$. It is worth noting that only the 51 landmarks of the inner-face are included in $P_{t}$ and $P_{o}$, and the facial shape is determined by the facial shape loss (see below) rather than the landmarks of the facial contour.

Perceptual Loss. As high-level feature maps contain semantic information, we utilize the feature maps from the final two convolutional layers of a pre-trained VGG as the representation of facial attributes. The loss is formulated as:

Expression Loss. We adopt a novel fine-grained expression loss [32] that penalizes the $\mathcal{L}_{2}$ distance of two expression embeddings:

Identity Loss. The identity loss is utilized to enhance the identity similarity between $I_{s}$ and $I_{o}$:

where $E_{id}$ denotes a face recognition model [33] and $cos$ denotes the cosine similarity.

Facial Shape Loss. The facial shape loss is used to constrain $I_{o}$ to have a similar facial shape as $I_{s}$. The shape-aware semantic flow generated by $D_{shape}$ explicitly quantifies the discrepancies in facial contour between the two input faces by modeling the pixel-level motion trend on facial shape. Consequently, as the face shape of $I_{o}$ approaches that of $I_{s}$, the semantic flow between $I_{s}$ and $I_{o}$ tends towards zero:

where $V_{so}$ denotes the semantic flow between $I_{s}$ and $I_{o}$, $zeros\_like(V_{so})$ denotes a full-zero matrix with the same dimensions as $V_{so}$ and $\left\|*\right\|_{2}$ denotes the euclidean distance.

Since our facial shape discriminator needs to warp the face shape pixel-wisely while being trained, we choose the explicit facial landmarks as the shape representation. We use a 3D face reconstruction model (3DMM) [36] to obtain facial landmarks. As shown in Figure 3, the 3D face reconstruction model $E_{3D}$ extracts 3D coefficients of the source and target:

where $\beta_{*},\theta_{*},\psi_{*},c_{*}$ are the FLAME coefficients [37] representing the facial shape, pose, expression, and camera, respectively. $*$ is $1$ or $2$, representing the first or the second input face. With these coefficients, the second input face can be modeled as:

where $M_{2}$ represents the 3D face mesh of the $I_{2}$. $W$ is a linear blend skinning (LBS) function that is applied to rotate the vertices of $T_{P}$ around joint $J$. $\mathcal{W}$ is the blend weights. $T_{P}$ denotes the template mesh $\overline{T}$ with shape, pose, and expression offsets [37].

Then, we reconstruct $I_{1}$ similarly, except that the pose and expression coefficients are replaced with the $I_{2}$’s ones. The obtained 3D face mesh is denoted as $M_{12}$. Finally, we sample 3D facial landmarks from $M_{2}$ and $M_{12}$ and project these 3D points to 2D facial landmarks with the target camera parameter $c_{2}$:

where $M_{*}^{i}$ is a vertex in $M_{*}$, $\Pi$ is an orthographic 3D-2D projection matrix, and $s$ and $t$ are parameters in $c_{2}$, indicating isotropic scale and 2D translation. $P_{*}$ denotes the 2D facial landmarks. It should be noted that we only use the landmarks at the facial contours as the shape representation since inner facial landmarks contain identity information that may influence the reshaping result.

The relative displacement between $P_{2}$ and $P_{12}$ only describes sparse movement. To accurately perceive the discrepancies of the two faces, we need to obtain dense motion between them. Therefore, we propose the semantic flow, which models the semantic correspondences between two faces. To achieve a more shape-aware semantic flow, $D_{shape}$ warps the target face according to the semantic flow during training, and constrains the warped target face to be consistent with the source face in terms of facial shape. We design a semantic guided generator $G^{res}$ to estimate the semantic flow. Specifically, $G^{res}$ requires three inputs: $P_{12}$, $P_{2}$ and $S_{2}$, where $P_{12}$ and $P_{2}$ are the 2D facial landmarks obtained above. $S_{2}$ is the second face segmentation map that complements the semantic information lost in facial landmarks. The output of $G^{res}$ is the estimated semantic flow $V_{t}$, the formulation is:

Then, a warping module is introduced to generate the warped faces using $V_{t}$. We find that an inaccurate flow is likely to produce unnatural images, On the contrary, imposing constraints on the warped images can lead to a more precise semantic flow, and therefore, we design a semantic guided discriminator $D^{res}$ that ensures $G^{res}$ to produce a more accurate flow. The warping operation is conducted on $I_{2}$:

Where $F$ is the warping function in the warping module. We feed the warped face $I_{2}^{res}$ to $D^{res}$. Thus, $D^{res}$ is able to discriminate whether the input is real or fake. It should be noted that $D^{res}$ and warping module are only used during training of $D^{shape}$.

We adopt three loss functions for $D^{shape}$:

where $\lambda_{ldmk}$ and $\lambda_{rec}$ represent hyperparameters associated with each term. In our experimental setup, we have designated the value of $\lambda_{ldmk}$ as 800 and that of $\lambda_{rec}$ as 10.

As in the training of $F^{swa}$, the reconstruction loss between $I_{2}^{res}$ and $I_{2}$ is used for self-supervision since there is also no ground-truth for face swapping results.

Adversarial Loss. The more realistic the resultant images are, the more accurate the generated shape-aware semantic flows are, therefore we employ the hinge version adversarial loss[31] for training, denoted by $L_{adv}$:

where $D^{res}$ is the discriminator which is trained with:

Landmark Loss. Since there is not pixel-wised ground truth for $I_{2}^{res}$, we rely on the 2D facial landmarks $P_{12}$ to regulate the shape of $I_{2}^{res}$. To be specific, we employ a pre-trained facial landmark detector [38] to forecast the facial landmarks of $I_{2}^{res}$, which are denoted as $P_{2}^{res}$. Then the loss is computed as:

At this point, our designed facial shape discriminator is able to generate a shape-aware semantic flow which finely perceives discrepancies of facial shape between two input faces. Subsequently, the semantic flow can be utilized to enforce similarity between the facial shape of the face-swapped faces and that of the source faces during training of the $F_{swa}$.

Our method is compared with seven methods including Deepfakes [22], FaceSwap [20], FlowFace [6], FSGAN [23], FaceShifter [2], SimSwap [3], and HifiFace [7]. For Deepfakes, FaceSwap, FaceShifter, and HifiFace, we use their released face swapping results of the sampled 10,000 images. For FlowFace, FSGAN and SimSwap, the face swapping results are generated with their released codes.

Table I demonstrates that the proposed FlowFace++ outperforms the other methods in most evaluation metrics, including ID Acc, shape error, and expression error (Expr Error). These results validate the superiority of FlowFace++. FlowFace++ produces slightly worse results in terms of pose error compared to other methods, which may be attributed to our manipulation of face shape that poses greater challenges in pose. The employed head pose estimator is sensitive to face shapes, which might have impacted the results.

The qualitative comparisons are conducted on the same FF++ test set collected in the quantitative comparisons. As shown in Figure 4, our FlowFace++ (or FlowFace) maintains the best face shape consistency. Most methods result in face shapes similar to the target ones since they do nothing to transfer the face shape.

Although HifiFace is intentionally designed to manipulate the face shape, our method still outperforms it in terms of evaluation metrics. Compared to HifiFace, Figure 4 illustrates that our generated face shapes are more similar to the source faces. Since HifiFace injects the shape representation into the latent feature space, and directly uses facial landmarks generated by the 3D face reconstruction module as supervision for facial shape, it may be harder to achieve a fine perception of facial shape than our explicit supervision of facial shape differences by the semantic flow. Additionally, our proposed method excels at preserving fine-grained target expressions, as marked indicated by the red boxes in Rows 1 and 4 of Figure 4.

We further compare our methods with more six SOTA face swapping methods:MegaFS [25], FaceInpainter [5], HighRes [27], SmoothSwap [8], HifiFace [7] and RAFSwap [26]. The source, target faces and results used in this comparative experiment are cropped from the original papers of these methods. Figure 5 demonstrates that our method is capable of better transferring the shape of the source face to the target. Although the results exhibited in the paper of HifiFace demonstrate its effectiveness in transferring facial shape, it suffers from the problem of facial expression leakage from the source face to the result. While SmoothSwap can change the facial shape, it often destroys the target attributes (e.g., hairstyle, and hair color).

The qualitative comparisons above also demonstrate that our results exhibit higher similarity to the source face in terms of inner facial features (e.g., beard), confirming that our MAE encoder is more efficacious in effectively representing facial appearances than the identity embedding used in [7, 5, 8] or the latent code of StyleGAN2 used in [25, 27, 26]. Moreover, our approach demonstrates a higher degree of fidelity in preserving target attributes (e.g., skin color, lighting, and expression), in comparison to other other methods.

In order to further validate our FlowFace++, we conduct a subjective comparison study with two of the state-of-the-art face swapping methods, SimSwap and HifiFace, both of which have publicly shared their codes or results. We randomly select 30 instances of swapped faces generated from each of the three aforementioned methods. Participants are instructed to choose the best results in terms of shape consistency, identity consistency, expression consistency, or image realism.

For each image, a maximum of 39 participants are recruited for evaluation. Each participant selects their preferred option. And for a given option, assigns a value of 1 if they select it and 0 if they do not. Under the condition of a confidence threshold of 80%, if the left endpoint of a confidence interval for one given option is greater than 0.5, this image is considered to have completed the evaluation process. Otherwise, the number of participants is increased until the maximum of 39 participants was reached. Table II shows that our method outperforms the two baselines in terms of shape consistency, identity consistency and image realism, validating the superiority of our method. In terms of expression consistency, our FlowFace++ slightly lags behind SimSwap, which could be attributed to the coupling between facial shape and expression, where changes in facial shape may affect the discriminability of expressions.

As shown in Figure 6, FSGAN and FaceSwap struggle to produce convincing results from the large angular deviations of the source faces. On the other hand, FaceShifter and SimSwap generate resulting faces that are not as clear and accurate. Although other results generated by RAFSwap based on StyleGAN2 are still clear, it cannot effectively extract the identity information of source faces, resulting in poor identity similarity between the result and source faces. In contrast, our FlowFace++ is still able to transfer the attributes and facial shape of the source faces to the target faces accurately.

As shown in Figure 7, FSGAN and FaceSwap mistakenly transfer the lighting of the source faces to the result faces, while SimSwap encounters difficulties in extracting facial features from unevenly lit source faces, resulting in blurred results. And RAFSwap generates some incorrect transferings (note the red box markings). Our FlowFace++ is still capable of generating high-quality face-swapping results even in the presence of uneven illumination.

Our FlowFace++ achieves remarkable performance in robustness testing, primarily because we utilize the MAE encoder which is designed following MAE and pre-trained on a large-scale face dataset using the masked training strategy. Even under extreme input conditions with various interferences, the MAE encoder is able to extract rich features from the input faces. Furthermore, our CAFM module facilitates the adaptive aggregation of identity and facial shape information from the source, allowing us to effectively eliminate the interfering information.

To maintain the expression in the generated image consistent with the target image, we introduce an effective expression embedding [32] which employs a continuous and compact embedding space to represent the fine-grained expressions. The distance between two expression embedding reflects the similarity between them. Therefore, we formulate the expression loss as a L2 loss that computes the expression embedding distance between the generated and target images, so as to ensure that they are consistent in terms of expression.

We conduct ablation experiments to validate the design of end-to-end face-swapping framework $F^{swa}$. We remove the $D^{shape}$ (w/o $D^{shape}$). Figure 8 shows that in the absence of $D^{shape}$’s constraints, the network lacks the ability to transfer facial shape (note the cheeks and jaw angles). Hence, $D^{shape}$’s constraints are crucial in ensuring our FlowFace++’s proficiency in facial shape warping.

To further validate the effectiveness of $D^{shape}$, we attempt to warp the input faces using the semantic flows generated by $D^{shape}$. As shown in Figure 9, after warping with the semantic flow, the facial contours of $I_{2}$ are changed, becoming closer to $I_{1}$, while maintaining the inner-facial appearances. This fully demonstrates that the semantic flow generated by our designed $D^{shape}$ effectively represents the pixel-wise motions in facial contours explicitly.

Our previous work, FlowFace[6], is a two-stage framework. It uses the $D^{shape}$’s warped results on the target faces as the first stage, then utilizes the $F_{swa}$ which is trained without $D^{shape}$’s supervision to transfer the non-shape identity as the second stage.

As shown in Figure 10, the performance of FlowFace[6] on the target faces via $D^{shape}$ is not entirely perfect and may result in distortions in details (e.g., eyebrow, mouth and low jawbone), as highlighted by the red circle in the image. Such distortions may emerge in the second stage, potentially leading to flaws in the final results. This convincingly underscores the rationale behind our end-to-end network architecture of FlowFace++, which incorporates $D^{shape}$ as a form of supervision.

We design four ablation experiments to validate the effectiveness of our proposed $D^{shape}$:

A. Sparse Landmarks vs. Dense Flow. During the training process of $F_{swa}$, we adopt the dense flow $V_{s}o$ generated by $D_{shape}$ to calculate the shape discrepancies between $I_{s}$ and $I_{o}$. To demonstrate the rationality of dense flow, similar to the approach of HifiFace[7], we replace it with sparse landmarks as the supervision for facial shape, where the sparse landmarks $P_{12}$ can be obtained by calculating Equation 16. Then the new facial shape loss can be computed by:

The $P_{o}$ represents the landmarks of the $I_{o}$ and only 17 landmarks on the contour are involved in the calculation of ${L}_{sparse\_ldmks}$. As seen in Figure 14, when sparse landmarks are used as supervision, residual ghosts may appear on the facial contour during the process of transferring facial shape. This phenomenon can be attributed to that sparse landmarks cannot represent pixel-wise dense motion.

B. Removing $S_{2}$ of $G^{res}$. We Remove the semantic input $S_{2}$ of $G^{res}$ ($G^{res}$ w/o Seg) to validate our proposed semantic guided generator $G^{res}$. It can be seen from Figure 14 that some inaccurate flow occurs in the generated face, which implies that only facial landmarks cannot guide $G^{res}$ to produce accurate dense flow due to the lack of semantic information. The results also demonstrate that the semantic information is beneficial for accurate flow estimation and validates $G^{res}$.

C. Adding the semantic inputs ($S_{2}$ and $S_{2}^{res}$) of $D^{res}$ ($D^{res}$ w Seg). FlowFace [6] proposes that adding semantic inputs to the adversarial loss can improve the reconstruction of fine details in the facial region. We attempt to add semantic inputs to $D^{res}$ as well, however, as shown in the experimental results in Table III suggest that this does not result in any significant accuracy improvements. The implication is that in FlowFace++, $D^{res}$ does not require semantic inputs and is still capable of effectively discriminating between real and generated faces.

D. Removing $D^{res}$ (w/o $D^{res}$). As observed in Figure 14, the generated images produced by w/o $D^{res}$ exhibit a few artifacts, and the estimated semantic flow is also marred by substantial noise.

The aforementioned observations serve to corroborate the efficacy of our proposed $D^{shape}$.

Three ablation experiments are conducted to evaluate the design of $F^{swa}$:

A. Choices on CAFM, Addition and AdaIN. To verify the effectiveness of our proposed CAFM, we conduct a comparative analysis between the CAFM and two other methods, namely, Addition (which simply involves adding the source values to the target values) and AdaIN (which first averages source patch embeddings and then injects them into the target feature map using AdaIN residual blocks). As shown in Figure 12 and Table IV, Addition simply blends all information of the source face to the target face, thus resulting in severe pose and expression mismatch. AdaIN, due to its global modulation, impacts not only the facial features but also the non-face parts such as hair. By contrast, $F^{swa}$ with CAFM achieves a high ID Acc and effectively preserves the target attribute. This result demonstrates the highly accurate identity information extraction capabilities of CAFM from the source face and its adaptive infusion into the target counterpart.

To further validate the efficacy of our CAFM, we visualize the cross-attention computed by CAFM. As depicted in Figure 13, when a specific part of the target face is given (marked with red boxes), our proposed CAFM precisely focuses on the corresponding regions of the source face, thus validating its ability to adaptively transfer the identity information from the source patches to the respective target patches.

B. Latent Representation vs. ID Embedding (ID Embed.). To verify the superiority of using the latent representation of MAE, We design an experiment that use a face recognition network [44] to extract the features of the source faces and continue to use the original MAE encoder to extract the features of the target faces, while keeping the other network structures unchanged. As can be seen from Figure 12, ID Embed. misses some fine-grained face appearances, such as eyebrow edge. In contrast, $F^{swa}$ contains richer identity information and achieves higher ID Acc, as shown in Tab IV.

C. Convolutional Decoder vs. ViT Decoder. We try two different decoders to determine the better one. As shown in Table IV, compared to Vit Decoder, Convolutional Decoder exhibits superior performance in terms of ID accuracy, expression error, and pose error, while performing roughly on par in the aspect of shape error. As can be seen in Figure 12, the result of Vit Decoder exhibits partial blurry regions and erroneous leakage of the source face’s hair.