ShapeEditer: a StyleGAN Encoder for Face Swapping

As one of the main contents of deepfake, face swapping declares to the world today that seeing is not always believing. Face swapping refers to transferring the identity of the source image to the face of another target image while keeping the illumination, head posture, expression, dress, background and other attribute information of the target image unchanged. Face swapping has received widespread attention since its birth, catering to the rich needs of social life, such as hairstyle simulation, film and television shooting, privacy protection[1] and so on.

Face swapping is accompanied not only by its interesting and operational application prospects, but also by various challenges between reality and vision. The early face swapping methods[2, 3] require a large number of images of source and target characters to provide sufficient facial information, otherwise the models will not have a suitable reference basis to convert reasonable results. Some 3D-based methods[4, 5, 6] make use of the advantage of fitting 3D face model to deal with the problems of large angle and small samples. At the same time, due to the limitation of the accuracy of 3D face model, it is impossible to generate works with more perfect details and higher fidelity. Recently, with the continuous tapping of the potential of Generative Adversarial Network (GAN), some face-swapping methods based on GAN[7, 8, 9, 10, 11] can achieve a good fusion of identity and attribute information with only a small number of samples, reflecting the effect of great creativity. Unfortunately, the surprising creativity of these methods does not offset the negative effects of their frequent artifacts and low resolution limitation.

On another track, the most advanced face image generation methods have been able to generate facial images with high resolution and realistic texture. Most notably, StyleGAN[12] can randomly generate a variety of clear faces with a resolution of up to $1024\times 1024$. On this basis, some works[13, 14, 15, 16] deeply explore the StyleGAN potential vector space; some[17, 18, 19] find a linear direction to control the change of a single facial attribute; some[20] realize the control of facial expression and posture in the original StyleGAN image domain; some other[13, 21] perform well in dealing with the difficult task of facial super-resolution.

We propose a many-to-many face swapping method based on pretrained StyleGAN[12] model, which aims to ensure the clarity and fidelity of the results while fusing identity and attribute information. In view of the inherent ability of pretrained StyleGAN[12] model to generate random high-quality face images, the difficulty of this task is how to accurately generate the corresponding latent vectors. In order to achieve this goal, we first design a novel encoder, ShapeEditor, to find the corresponding codes in the $\mathcal{W+}$ vector space. The work flow of the encoder is divided into two stages, the first is the respective extraction of identity and attribute codes; the second is to map the combination of two-channel codes into the potential input vector domain of the pretrained model. Moreover, we design a set of loss functions with strong monitoring ability to urge ShapeEditor to update parameters so as to learn to map into the latent space of StyleGAN[12] step by step. As verification, we have done a lot of qualitative and quantitative comparisons with the existing face swapping methods in the experiments, which shows the unique advantages of our method.

In summary, the main contributions of this paper are:

• 1.

  A new StyleGAN latent space encoder. Our encoder can effectively mix two channels and map them into $\mathcal{W+}$ space of StyleGAN.

• 2.

  A novel face swapping method. To the best of our knowledge, our method is the first to swap face based on StyleGAN, while having higher clarity and authenticity, comparable fidelity than other state-of-the-art works.

Face swapping has always been a focus of attention, which is applied for privacy protection, entertainment and art, mainly using face alignment, facial key point detection, 3D face modeling, GAN and other technologies. Recently, StyleGAN[12], which has excellent performance in generating real and clear faces, has been proposed and further used in a variety of face editing operations.

Different from other works, the first criterion we pursue is that the images after face swapping have both resolution of up to $1024\times 1024$ and high authenticity. In order to ensure the high quality of human face, our framework is based on facial pretrained StyleGAN[12], which has rich semantic expression ability. StyleGAN has three potential spaces: initial potential space $\mathcal{Z}$, intermediate potential space $\mathcal{W+}$ and extended potential space $\mathcal{W+}$. It is proved[22] that the concatenation of 18 different 512-dimensional vectors is the easiest to embed the image and get a more reasonable result, so the operation space we choose is $\mathcal{W+}$.

Our method requires two images as input: $I_{attr}$ and $I_{id}$. We expect the output of the model to reflect the identity of $I_{id}$, as well as the facial expression, head posture, hairstyle, lighting and other attribute information of $I_{attr}$. Therefore, the main challenge of this work is to obtain StyleGAN potential vectors which are consistent with the $\mathcal{W+}$ spatial distribution and better integrate attributes and identity. In order to solve this problem, we design a two-step coding process. As shown in Figure 2, the entire mapping process is divided into two phases: ID-ATTR Encoding and Latent Space Encoding. In the first stage, $E_{id}$ extracts the identity vector of $I_{id}$, and $E_{attr}$ extracts the attribute vector of $I_{attr}$. Inspired by pSp[13], $E_{attr}$ consists of a pyramid-shaped three-layer feature map extraction structure and a set of Convolutional Mappers (CM). In the second stage, we input the concatenation of $E_{id}(I_{id})$ and $E_{attr}(I_{attr})$ into the Multilayer Perceptron (MLP) of each layer, and map the vectors containing identity and attribute information directly to the $\mathcal{W+}$ potential vector space. In summary, the whole image conversion process can be represented as :

where $G(\cdot)$ represents the pretrained StyleGAN[12] model, $MLP(\cdot)$ represents the Multilayer Perceptron, $[\cdot,\cdot]$ represents the concatenation of two vectors.

We compare our method with FSGAN[10], FaceShifter[11], Nitzan et al.[20] on the CelebAMask-HQ[26] test dataset. As shown in Figure 3, as expected, because our method is based on a pretrained StyleGAN[12] with high-quality face generation capabilities, all the generation results (Figure 3, column 6) are stable and clear enough that there are no errors such as artifacts and abnormal illumination.

Almost every output image (Figure 3, column 3) of FSGAN[10] shows unnatural lighting transition and lack of facial details, and the abnormal region of the face is caused by directly extracting and filling the internal region of the face (Figure 3, row 3, column 4), which is completely avoided in our method.

Because there is no pretrained model as backbone, it is difficult for FaceShifter[11] to avoid facial blur, and some results even show facial illumination confusion (Figure 3, row 3, column 4) and eye ghosting (Figure 3, row 7, column 4), showing that its authenticity is far behind our method.

Similar to our method, Nitzan et al.[20] uses StyleGAN[12] as backbone. But it can not accurately integrate identity and attribute information because of its simple encoder structure and the constraint of $\mathcal{W}$ potential space. Therefore, although it can generate high-quality images (Figure 3, column 6), it is not as good as our method in semantic information fusion, which is reflected in that the attributes of the target image, such as hairstyle and background, are not contained.

In addition to the excellent performance in terms of authenticity and fidelity, our method can also deal with extreme lighting condition (Figure 3, row 2, column 6) and even keep the sense of age (Figure 3, row 3, column 6). Thanks to that we use the facial recognition module to extract the identity vector instead of directly utilizing the pixels in the facial area, we can extract the identity information very well even if the source image has facial occlusion (Figure 3, row 4, column 6); our model understands the knowledge that whether its output should have glasses (Figure 3, column 6, rows 5-6), which is embedded in the potential space of the pretrained StyleGAN[12] model.

In this paper, we propose a new face-swapping framework, including ShapeEditor and pretrained StyleGAN model. The pretrained model gives our framework the potential to generate clear and real faces, while the ShapeEditor encoder effectively extracts and integrates the attribute and identity information of the input images, and accurately maps them into the $\mathcal{W+}$ space, thus controlling the StyleGAN to output appropriate results. Extensive experiments show that our method performs better than other existing frameworks.