High-resolution Face Swapping via Latent Semantics Disentanglement

Given two high-resolution face images, we aim to construct a face image with the identity of the source face $x_{s}$ and the attributes of the target face $x_{t}$ such as the pose, expression, illumination and background. To do so, we first construct a side-output swapped face using a StyleGAN generator, by blending the structure attributes of the source and the target in the latent space while reusing the appearance attributes of the target (Sec. 3.2). To further transfer the background of the target face, we use an encoder to generate multi-resolution features from the target image, and blend them with the corresponding features from the upsampling blocks of the StyleGAN generator. The blended features are fed into a decoder to synthesis the final swapped face image (Sec. 3.3). An overview of our pipeline is provided in Fig. 1. The networks are trained using a set of loss functions that enforce desirable properties such as the preservation of source identity and target appearance (Sec. 3.4). We also extend our approach to video face swapping, using additional spatio-temporal constraints to enforce coherence (Sec. 3.5).

A face image typically contains different classes of attributes. For example, the pose and expression are related to the face structure, while the lighting and colors are related to the appearance. Previous face swapping techniques, such as FaceShifter [26] and MegaFS [55], do not explicitly distinguish different classes of attributes when transferring them to the output image. We instead argue that it is beneficial to transfer structure and appearance attributes separately: the structure attributes of the output can be jointly determined from their counterparts in the source and target images, to obtain the same pose and expression as the target face while retaining the identity of the source face. Meanwhile, the appearance attributes of the target face can be directly reused to achieve similar facial appearance in the output.

To this end, we note that StyleGAN [20, 21], a state-of-the-art generative model, provides a suitable representation for such separate treatments. In particular, to use the StyleGAN generator, it is common practice to encode an image in an extended latent space $W+$ where a latent code consists of multiple high-dimensional vectors, one for each input layer of StyleGAN [3, 40]. As noted in [40, 51], different input layers of StyleGAN correspond to different levels of details. Thus we treat the first $K$ vectors of the latent code, which correspond to the shallow layers, as the encoding for the structure attributes. The remaining vectors, which correspond to the deeper layers, are used for appearance. For $1024\times 1024$ images, the latent code consists of 18 different 512-dimensional vectors, and we follow [40] and choose the first 7 vectors as the structure part. With such disentanglement, the structure part and the appearance part of the latent code can be transferred separately. Specifically, we first use the pre-trained pSp encoder [40] to invert the source face $x_{s}$ and the target face $x_{t}$ to obtain their $W+$ latent codes $w_{s}=(g_{s},h_{s})$ and $w_{t}=(g_{t},h_{t})$ respectively, where $g_{s},g_{t}$ are the structure parts and $h_{s},h_{t}$ are the appearance parts. To construct the structure attributes of a swapped face that has the same identity of the source with the pose and expression of the target, we note that they should be derived from the source structure attributes with modifications that take into account the pose and expression difference between the source and the target. Therefore, we compute the structure part $\widehat{g}_{s}$ of the latent code for a swapped face by applying a structure transfer latent direction $\overrightarrow{{n}}$ that is derived from the source and target structures:

To derive $\overrightarrow{{n}}$, we note that the face structure can be indicated by the facial landmarks. Thus we train a landmark encoder $E_{le}(\cdot,\cdot)$ that generates $\overrightarrow{{n}}$ from the heat-map encodings of the source landmarks $l_{s}$ and the target landmarks $l_{t}$:

To transfer the appearance attributes from the target face to the swapped face, we directly reuse the appearance part $h_{t}$ of the target latent code. It is then re-integrated with $\widehat{g}_{s}$ to form a latent code for the swapped face:

where $\operatorname*{Cat}(\cdot,\cdot)$ denotes the concatenation operator. The code is fed into a pre-trained StyleGAN generator to obtain a side-output swapped face $y_{s}$.

For face swapping applications, the background of the target face also needs to be retained in the output image. This often cannot be guaranteed for the swapped face $y_{s}$ computed in Sec. 3.2. A common solution is to apply Poisson Blending as the post-process that blends the swapped inner-face with the target image. However, this can lead to unnatural appearance around the boundary of the inner-face.

To address this issue, we discard the side-output face $y_{s}$, but retain the features $F_{s}=\{f^{0}_{s},f^{1}_{s},...f^{N}_{s}\}$ produced by each upsampling block of the StyleGAN generator from the latent code $\widehat{w}_{s}$ in Eq. (2). We note that $F_{s}$ can be regarded as a representation for different levels of details from the side-output face image. Accordingly, we apply an encoder $E_{t}$ to the target face $x_{t}$ such that its layers generate corresponding features $F_{t}=\{f^{0}_{t},f^{1}_{s},...f^{N}_{s}\}$, where $f^{i}_{t}$ is of the same dimension as $f^{i}_{s}$ and represent the details of the target face image at the same resolution. We then aggregate each pair of corresponding features $(f^{i}_{s},f^{i}_{t})$ by replacing the components of $f^{i}_{t}$ for the inner-face region with their counterparts in $f^{i}_{s}$. All aggregated features are fed into a decoder to produce the final face image $y_{f}$, which can be written as:

where $\operatorname*{Dec}(\cdot,\cdot,\cdot)$ denotes the decoder, and $m_{t}$ is the inner-face mask for the target face image. In this way, the decoder transfers the background in multi-level features, which not only eliminates the need for explicit background blending but also enables the code $\widehat{w}_{s}$ to focus on the facial region and facilitates attributes transfer.

We tailor several losses that are applied on the side-output swapped face $y_{s}$ or the final face $y_{f}$ for efficient attributes transfer, as explained in the following. We also introduce a style-preservation loss on the final output to narrow the style gaps between the swapped one and the target.

Our method can be extended to video face swapping. Given a source face $x_{s}$ and a sequence of target faces with $M$ consecutive frames $S_{t}=\{x_{t}^{0},x_{t}^{1},...,x_{t}^{M-1}\}$, we would like to obtain a swapped face sequence $Y=\{y^{0},y^{1},...,y^{M-1}\}$. Most of the existing works apply the image-based face swapping methods to each video frame separately, which can lead to incoherent results between neighboring frames and cause artifacts such as flickering. Such artifacts are particularly noticeable in high resolutions. To address this issue, we need to enforce consistency between neighboring frames in terms of both the structure and the appearance, such that these attributes vary smoothly across the frames. Existing temporal consistency works [25, 9] only take the appearance consistency into account and cannot be used directly for face swapping. We propose two spatio-temporal constraints for the latent space and the image space respectively, to achieve consistency in both the structure and the appearance.

We utilize the StyleGAN2 generator [21] pre-trained on the FFHQ dataset [20] with the resolution of $1024\times 1024$. The pSp encoder is also pre-trained on this dataset. We use a modification of pSp as our landmark (more details of the structure can be seen in the supplementary materials). We use the Adam optimizer [23] to train the model with a learning rate of $1\times 10^{-4}$, and the exponential decay rates for the $1_{st}$ and $2_{nd}$ moment estimate being ${\beta}_{1}=0.9$ and ${\beta}_{2}=0.999$ respectively, and ${\epsilon}=1\times 10^{-8}$. The batch size is set to be 8 and the model is trained with 500,000 iterations. We empirically set the weights in Eq. (3) as $\lambda_{1}=1$, $\lambda_{2}=2$, $\lambda_{3}=0.1$, $\lambda_{4}=2$ and $\lambda_{5}=0.2$. We implement our method in Pytorch with four Tesla V100 GPUs. It takes about two days to train the whole model.

To allow comparing with other face swapping methods that can be only applied to low-resolution images, we further evaluate our method on the Face-Forensics++ dataset. Fig. 3 presents qualitative comparison results against MegaFS as well as three non-GAN prior based and low-resolution methods: FaceSwap [2], Deepfakes [1], and FaceShifter [26]. We can see that FaceSwap and MegaFS can lead to noticeable blending boundaries or artifacts, while our method can eliminate them due to our background transfer. However, the quality of our swapped results is not that high as high-resolution face swapping in the Fig. 2. This is potentially due to domain gap: the styleGAN generator and the pSp encoder used in our method are both pre-trained on high-resolution data, while the majority of the data from FaceForensics++ are of lower resolutions. The large domain gap between low- and high-resolution datasets can potentially decrease the performance of our method.

We also perform quantitative evaluation on this dataset. In particular, we follow MegaFS [55] that samples 10 frames from each video evenly then processed by MTCNN [53]. After filtering the repeated identities, we obtain 885 videos with 88500 frames in total. Tab. 2 shows the average values of evaluation metrics for different methods. We can see that our method preserves the target poses and expressions more effectively, thanks to our structure attributes transfer that takes the landmarks as input and provides a strong guidance signal to the final swapped faces. Meanwhile, our model is outperformed by FaceShifter and MegaFS on ID retrieval rate. We conjecture that this is due to the domain gap between the low- and high-resolution images as mentioned above, which makes our inversion model less effective in preserving the identity information from low-resolution images.

In this section, we perform ablation study and use the CelebA-HQ dataset to evaluate the effectiveness of our disentangled approach in transferring the attributes. We use MegaFS as a baseline, since they swap faces solely on two latent codes without disentanglement between attributes. We further include three variants of our approach with modifications on the modules and the loss functions. For the first variant (Var.1), we keep the appearance part of source code $w_{s}$ unchanged for producing the side-output face and discard the style-transfer loss and the latent code swapping operation for appearances transfer, i.e., there is no explicit transfer guidance that contributes to the appearance attributes. For the second variant (Var.2), we discard our background transfer module and directly blend the side-output face with the target face image as the final output. The qualitative comparison between different variants and the baseline is shown in Fig. 4. We can see that the MegaFS baseline leads to blurry faces and sharp boundaries. As mentioned previously, this is due to the highly entangled identity and attributes in the latent code, which can lead to information loss during the transfer. The first variant presents the unnatural styles, which verifies the necessity of our appearance attributes transfer that provides explicit appearance guidance in the swapping process. The second variant presents a notable blending boundary between the swapped face region and the background (see the $2_{nd}$ and $3_{rd}$ samples), due to the lack of the background transfer module which fuses multi-resolution features from the source and the target for a natural blending. Meanwhile, our target encoder-decoder structure also can tolerance the structure difference between source and targets, and produce the final plausible results. Thanks to our disentangled attributes transfer, our final swapped faces present the successful semantic transfer from the targets, with the source identity well preserved.

Fig. 5 shows qualitative results of our method on high-resolution videos. Applying our method individually to each frame leads to incoherence between neighboring frames, while our code and flow trajectory constraints improve the coherence and visual quality considerably.

We use a modification of pSp [40] as our landmark encoder. pSp inverts the input image to the $W+$ latent space with a feature pyramid through three levels: the coarse, medium, and fine, which controls different level of attributes. We discard pSp’s higher layers and only use the coarse and medium layers to produce the semantic transfer direction $\overrightarrow{{n}}$.

Our target encoder consists of 8 downsample blocks, each block contains a convolutional layer and a ‘Blur’ layer [21]. The target encoder produces the multi-resolution target features, which includes: $8\times 8\times 512$; $16\times 16\times 512$; $32\times 32\times 512$; $64\times 64\times 512$; $128\times 128\times 256$; $256\times 256\times 128$; $512\times 512\times 64$; $1024\times 1024\times 32$. Then they blend with the source features produced by StyleGAN generator with the corresponding resolution.

The decoder has a mirror structure with target encoder, and we use the transpose convolution for upsampling. In the first upsampling block, we take the blending results of target and source features as input. And in each rest block, we concatenate the blended features with the output of last block as input.

To test the effectiveness of the landmark encoder, we implement a variant, named Var.3, that infers the structure transfer latent direction $\overrightarrow{n}$ from the face images directly without using the landmarks. We replace the landmark encoder with an image encoder, and feed the concatenation of the source and target face images to the encoder.

Its results are shown in Tab. 3 and Fig. 6. In Tab. 3, it has worse performance for the metrics Pose Err. and Exp. Err., which indicates that the landmark encoder plays a vital role in pose and expression preservation. Besides, without the landmark guidance, Var.3 cannot effectively blend the target background with the source inner face using target mask; this leads to lower quality results than our method in Fig. 6.

Tab. 3 shows quantitative evaluation results on CelebA-HQ. We can see that both Var.1 and Var.2 have a higher FID value than our approach, indicating worse quality of face images from both variants. This is because our appearance transfer can narrow the color gap while our background transfer eliminates the blending artifacts, both helping to improve the quality of our results. On the other hand, Var.1 and Var.2 have similar performance as our approach on the other three metrics related to identity, pose and expression. This is because they also utilize the landmark encoder and the identity-preservation loss, which help to transfer the pose and expression while preserving the identity. The analysis of Var.3 can be seen in last subsection.