MFIM: Megapixel Facial Identity Manipulation

We introduce our facial attribute encoder. As shown in Figure 2(a), it first extracts hierarchical latent maps from a given image like pSp encoder [34]. Then, map-to-code (M2C) and map-to-map (M2M) blocks produce the style codes and style maps respectively, which are forwarded to the pretrained StyleGAN generator.

Our model can create a new identity by mixing multiple identities. We call this operation ID mixing. In order to allow the user to semantically control the identity creation process, we design a method to extract the ID style codes from multiple source images and then mix them like style mixing [21]. Here, we describe ID mixing using two source images, but it can be generalized to use multiple source images more than two. Specifically, when using two source images, the user can take global ID attributes (e.g., face shape) from one source image and local ID attributes (e.g., eyes) from the other source image and mix them to synthesize an ID-mixed image, $x_{mix}$.

Figure 2(b) describes this process. The ID-irrelevant style codes and style maps are extracted from $x_{tgt}$ (red arrow in Figure 2(b)). However, the ID style codes are extracted from two source images, global and local source images. We denote them as $x_{src}^{gb}$ and $x_{src}^{lc}$, respectively, and the style codes extracted from them are called global (light blue arrow in Figure 2(b)) and local ID style codes (dark blue arrow in Figure 2(b)), respectively. These ID style codes transform the specific generator feature maps. In particular, the global ID style codes transform the ones with coarse spatial resolution (e.g., $32\times 32$), while the local ID style codes are for the ones with fine spatial resolutions (e.g., from $64\times 64$ to $1024\times 1024$). In this manner, the global ID style codes transfer the global ID attributes (e.g., face shape) of $x_{src}^{gb}$, while the local ID style codes transfer the local ID attributes (e.g., eyes) of $x_{src}^{lc}$ due to the property of style localization [21].

MegaFS [48] which exploits pretrained StyleGAN also has the potential to perform ID mixing. However, MegaFS struggles with transforming the face shape (Section 4.2), so it is difficult to effectively perform ID mixing.

The generated images of our model can be seen in Figure 3. The qualitative and quantitative comparisons between our model and the baselines are presented in Figure 4 and Tables 2 and 3, respectively. We first compare our model to the baselines on FaceForensics++ [35], following the evaluation protocol of SmoothSwap [25]. As shown in Table 2, our model is superior to the baselines in all metrics except for pose-HN. It is noteworthy that our model outperforms the baselines for the shape, expression, and pose at the same time, whereas the existing baselines do not perform well for all those three metrics at the same time. For example, among the baslines, SmoothSwap [25] and HifiFace [41] achieve good scores in the shape, but the expression scores of these baselines are not as good. On the other hand, FaceShifter [27] and SimSwap [10] achieve good scores in the expression and pose, but the shape scores of these baselines are not as good. However, our model accomplishes the state-of-the-art performance for the shape, expression, and pose metric at the same time.

In addition, we compare our model to the previous megapixel model, MegaFS [48], on CelebA-HQ. We generate 300,000 images following MegaFS [48]. Then, each model is evaluated with the same metrics used in the evaluation on FaceForenscis++. For FID, we use CelebA-HQ for the real distribution following MegaFS [48]. As shown in Table 3, our model outperforms MegaFS [48] in the all metrics.

We conduct an ablation study on CelebA-HQ to demonstrate the effectiveness of each component of our model following the evaluation protocol of the comparative experiment on CelebA-HQ (Section 4.2). The qualitative and quantitative results are presented in Figure 5 and Table 4, respectively.

The configuration (A) is trained by using only the ID-irrelevant and ID style codes. The style maps and 3DMM supervision are not used in this configuration. In Figure 5, the configuration (A) generates an image that has the overall structure and pose of $x_{tgt}$, but has the identity of $x_{src}$ (e.g., eyes and face shape). This is because the ID-irrelevant style codes transform the generator feature maps with the coarser spatial resolutions (from $4\times 4$ to $16\times 16$) than the ID style codes (from $32\times 32$ to $1024\times 1024$), so the ID-irrelevant style codes synthesize more global aspects than the ID style codes do. However, the configuration (A) fails to reconstruct the details of $x_{tgt}$ (e.g., expression, hair style, and background). This is because the ID-irrelevant style codes, which do not have the spatial dimensions, lose the details of $x_{tgt}$.

To solve this problem, we construct the configuration (B) by adding the style maps to the configuration (A). In Figure 5, the configuration (B) reconstructs the details of $x_{tgt}$ better than configuration (A). It is also supported by the improvement of the expression score in Table 4. These results show that the style maps, which have the spatial dimensions, can preserve the details of $x_{tgt}$. However, the generated image by configuration (B) does not have the same face shape with that of $x_{src}$, but with that of $x_{tgt}$.

Therefore, for the more effective identity transformation, we improve our model by adding the 3DMM supervision to the configuration (B). First, we construct the configuration (C) by adding $\mathcal{L}_{shape}$ to the configuration (B). As a result, the generated image by the configuration (C) has the same face shape with that of $x_{src}$ rather than that of $x_{tgt}$. It leads to the improvement of the shape score in Table 4. However, the expression and pose scores are degraded. This result is consistent with Figure 5 in that the generated image of configuration (C) has the same expression with $x_{src}$, not $x_{tgt}$, which is undesirable. We assume that this is because the expression and pose of $x_{src}$ are leaked somewhat while the face shape of $x_{src}$ is actively transferred by $L_{shape}$. It means that the ID and ID-irrelevant representations of MFIM are not perfectly disentangled. Improving our model to solve this problem can be future work.

In order to restore the pose and expression scores, we first construct the configuration (D) by adding $\mathcal{L}_{pose}$ to the configuration (C), and then construct the configuration (E) by adding $\mathcal{L}_{exp}$ to the configuration (D). As a result, as shown in Table 4, the pose and expression scores are restored to the similar scores to the configuration (B). Finally, the generated image by the configuration (E) in Figure 5 has the same face shape with that of $x_{src}$, while the same pose and expression with that of $x_{tgt}$.

Although the configuration (E) can faithfully reconstruct the details of $x_{tgt}$ such as background and hair style, we can further improve our model to reconstruct the high-frequency details by adding ROI only synthesis to the configuration (E) at the inference phase. This configuration is denoted as (E+). It allows our model to generate only the face region, but it does not require any segmentation label. More details on this are in the supplementary material. In Figure 5, the configuration (E+) reconstructs the high-frequency details on hair. We use the configuration (E) for all the quantitative results, and the configuration (E+) for all the qualitative results.

Figure 6 shows the qualitative results of ID mixing using our model. In Figure 6, $x_{mix}$ has the new identity with the global ID attributes (e.g., face shape) of $x_{src}^{gb}$, but the local ID attributes (e.g., eyes) of $x_{src}^{lc}$. This property of ID mixing allows the user to semantically control the ID creation process. We also compare our model with MegaFS [48] in terms of ID mixing in the supplementary material.

We quantitatively analyze the properties of ID mixing on CelebA-HQ. We prepare 30,000 triplets by randomly assigning one global source image and one local source image to each target image. Then, we define Relative Identity ($R\text{-}ID$) distance and Relative Shape ($R\text{-}Shape$) distance following SmoothSwap [25]. For example, $R\text{-}ID(gb)$ is defined as $R\text{-}ID(gb)=\frac{D_{ID}(x_{mix},~{}x_{src}^{gb})}{D_{ID}(x_{mix},~{}x_{src}^{gb})+D_{ID}(x_{mix},~{}x_{src}^{lc})}$ where $D_{ID}$ means $L_{2}$ distance on the feature space of the face recognition model [9]. This measures how similar the overall identity of $x_{mix}$ is to that of $x_{src}^{gb}$ compared to $x_{src}^{lc}$. $R\text{-}ID(lc)$ is defined similarly, so $R\text{-}ID(gb)+R\text{-}ID(lc)=1$. In addition, $R\text{-}Shape(gb)$ and $R\text{-}Shape(lc)$ are defined in the same manner with $R\text{-}ID(gb)$ and $R\text{-}ID(lc)$, respectively, but they are based on the 3DMM [37] shape parameter distance to measure the similarity of face shape.

In Table 5, the two rows denoted by local and global show the results of conventional face swapping, not ID mixing, which uses a single source image. In particular, the row denoted by local is the result of conventional face swapping using only $x_{src}^{lc}$ as the source image without using $x_{src}^{gb}$. For this reason, $R\text{-}ID(lc)$ and $R\text{-}Shape(lc)$ are smaller than $R\text{-}ID(gb)$ and $R\text{-}Shape(gb)$, respectively, which means that the generated image has the same overall identity and face shape as $x_{src}^{lc}$, not $x_{src}^{gb}$. Similarly, the row denoted by global shows that $x_{mix}$ has the same overall identity and face shape as $x_{src}^{gb}$, not $x_{src}^{lc}$.

On the other hand, the row denoted by ID mixing shows the results of ID mixing, which uses both the $x_{src}^{gb}$ and $x_{src}^{lc}$ as described in Section 3.3. In contrast to when only one of $x_{src}^{lc}$ or $x_{src}^{gb}$ is used, $R\text{-}ID(gb)$ is similar to that of $R\text{-}ID(lc)$. It means that the overall identity of $x_{mix}$ by ID mixing is like a new identity, a mixed identity of $x_{src}^{lc}$ and $x_{src}^{gb}$. Furthermore, $R\text{-}Shape(gb)$ has a smaller value than $R\text{-}Shape(lc)$. It means that the face shape of the generated image is more similar to that of $x_{src}^{gb}$ than that of $x_{src}^{lc}$, which is consistent with Figure 6.

Our facial attribute encoder, which is based on the psp [34] encoder, uses the same encoder backbone (blue structures denoted as ‘Encoder Blocks’ in Figure 2(a)) as the psp encoder. As shown in Figure 2(a), the encoder backbone extracts the hierarchical latent maps from the given image. The M2C and M2M blocks of our facial attribute encoder extract the style codes and style maps from the hierarchical latent maps extracted from the backbone, respectively. The details of encoding process are as follows.

We use the pretrained generator of StyleGAN [22], so we use the same architecture with StyleGAN without modification except for the mapping network that maps a random vector $z\in\mathcal{Z}$ to an intermediate latent space $\mathcal{W}$. We replace the mapping network with the facial attribute encoder which produces the ID-irrelevant style codes, ID style codes and style maps. These are forwarded appropriately to each layer of the pretrained StyleGAN generator, as shown in Tables 7 and 8. Table 7 describes the process of face swapping, which uses a single source image, $x_{src}$, but Table 8 describes the process of id mixing, which uses the global and local source images, $x_{src}^{gb}$ and $x_{src}^{lc}$.

We use the pretrained discriminator of StyleGAN [22], so we use the same architecture with StyleGAN without modification.

We use FFHQ [21], which consists of 70,000 human faces at $1024\times 1024$ resolution, for the training dataset. It is noteworthy that the most of the previous face-swapping models [27, 41, 48, 17] extend the training dataset by combining multiple datasets, but we only use FFHQ. Therefore, our model can be trained more efficiently because our model does not require any additional preprocess steps such as image alignment to combine the multiple datasets.

For training, we basically follow the image preprocess protocol of pSp [34]. However, for $\mathcal{L}_{adv}$ and $R_{1}$, we use the images with the size of $1024\times 1024$. Furthermore, for 3DMM supervision, we preprocess the images by following the image preprocess protocol of DECA [16] before forwarding the images to DECA encoder.

Our model can faithfully reconstruct the background or hair style of $x_{tgt}$, but we can further improve our model to reconstruct the high-frequency details of the background or hair style via ROI only synthesis.

Note that it does not require a segmentation label at all. This process is depicted in Figure 7. Assuming that the image is aligned, we use a mask, which has a size of $1024\times 1024$, with a fixed box at the expected location of the face. Specifically, we set the size of the box to a width of 512 and a height of 608 and top-left coordinates, $(top,~{}left)$, to $(384,~{}256)$. The inside of the box has a value of one, and the outside has a value of zero. Then, we blur the boundary by downsampling the mask to the size of $16\times 16$ and upsampling it to the size of $1024\times 1024$ again. With this mask, the final output image is generated as

where $m$ is a mask and $\odot$ is the element-wise product. Note that it is not used at the training phase, only at the inference phase. Also, we use it only in the qualitative results, not in the quantitative results at all.

For the 3DMM supervision, our model utilizes the 3DMM parameter reconstruction loss which is formulated as

where $\mathcal{L}_{shape}$, $\mathcal{L}_{pose}$, and $\mathcal{L}_{exp}$ are described in the main manuscript, and $\lambda_{shape}$, $\lambda_{pose}$, and $\lambda_{exp}$ are weights for each loss.

On the other hand, HifiFace utilizes the landmark reconstruction loss. Note that 3DMM can reconstruct a 3D face using 3DMM parameters and extract landmark keypoints corresponding to the 3D face. Using this capability, HifiFace encourages the landmark keypoints of the generated image, $\left\{q^{gen}_{k}\right\}_{k=1}^{K}$ to be equal to its ground-truth landmark keypoints, $\left\{q^{gt}_{k}\right\}_{k=1}^{K}$. Here, when using DECA [16], $K=68$, and the ground-truth landmark keypoints are extracted from the reconstructed 3D face using the shape parameter of the source image and the pose, expression, and cam parameters of the target image. We apply this method to our model to formulate the landmark reconstruction loss as

In Table 11, we compare $\mathcal{L}_{lm}$ and $\mathcal{L}_{param}$ on CelebA-HQ [20]. Here, unlike the quantitative experiment on CelebA-HQ in the main manuscript, we use 30,000 face-swapped images instead of 300,000. Specifically, we randomly assign an image to each image in CelebA-HQ and make 30,000 pairs of the source image and target image.

The configuration (B) in Table 11 is the same with that in the main manuscript. Then, we construct the configurations (B+$\mathcal{L}_{lm}$) and (B+$\mathcal{L}_{param}$) by adding $\mathcal{L}_{lm}$ and $\mathcal{L}_{parm}$ to the configuration (B), respectively. The configuration (B+$\mathcal{L}_{param}$) is the same with the configuration (E), our proposed model, in the main manuscript.

As shown in Table 11, adding $\mathcal{L}_{lm}$ to the configuration (B) does not improve the shape score while $\mathcal{L}_{param}$ improves the shape score. However, we can see that $\mathcal{L}_{lm}$ improves the pose score by comparing the configurations (B) and (B+$\mathcal{L}_{lm}$). We think that this may be because the pose, which is the more global attribute than the shape and expression, has a greater effect on the landmark regression than the shape or expression. For this reason, the most effective way to decrease $\mathcal{L}_{lm}$ can be to match the pose of the generated image to that of the target image. As a result, the model focuses on matching poses, and may not be sufficiently motivated to improve the shape score.

In contrast, we use a separate loss for each attribute. In particular, to decrease $\mathcal{L}_{shape}$, the face shape of the generated image should be the same as that of the source image. Due to this difference, $\mathcal{L}_{param}$ can improve the shape score, while $\mathcal{L}_{lm}$ cannot. Although the pose score is somewhat degraded after applying $\mathcal{L}_{param}$, transforming the face shape rather than preserving the pose is one of our important goals. Furthermore, the configuration (B+$\mathcal{L}_{param}$) still shows the visually plausible results in terms of the pose. Therefore, we propose $\mathcal{L}_{param}$ as our 3DMM supervision method.

Based on the results in Table 11, we further improve our model by combining $\mathcal{L}_{param}$ and $\mathcal{L}_{lm}$ as shown in Table 12. For the results in Table 12, we use 300,000 face-swapped images, which is the same setting with that of the quantitative experiment on CelebA-HQ in the main manuscript.

In Table 12, we construct the configuration (F) by adding $\mathcal{L}_{lm}$ with the weight for this loss of 1,000 (i.e., $\lambda_{lm}=1000$) to the configuration (E). Here, we use only some of the landmark keypoints instead of the full landkark keypoints to encourage our model to further focus on matching the pose. Specifically, we use $\left\{q^{gen}_{k}\right\}_{k\in\left\{9,31,37,46,49,55\right\}}$ and $\left\{q^{gt}_{k}\right\}_{k\in\left\{9,31,37,46,49,55\right\}}$ . As shown in Table 12, the configuration (F) achieves the better pose and pose-HN scores than the configuration (E) without deterioration on the shape and expression scores. As a result, the configuration (F) achieves the better shape, expression, and pose scores than the configuration (B) at the same time. However, $\mathcal{L}_{lm}$ is not our contribution and the configuration (E) also shows the visually plausible results in terms of pose, so we propose the configuration (E) as our final model.

Figure 8 shows the qualitative results for several configurations. We construct the configuration (E+) by adding ROI only synthesis (Section Appendix. C.2) to the configuration (E). As shown in Figure 8, the configuration (E+) transfers the ID attributes (e.g., eyes and face shape) of the source image actively while preserving the ID-irrelevant attributes (e.g., pose and expression) of the target image. In Figure 8, the differences between the configurations (A) and (B) show the effectiveness of the style maps, and the differences between the configurations (B) and (E+) show the effectiveness of the 3DMM supervision.