IntereStyle: Encoding an Interest Region for Robust StyleGAN Inversion

Our architecture is shown in Figure 4, which is based on $pSp$ [24] model, combined with the iterative refinement [3, 31]. At $i$-th iteration of the iterative refinement, our encoder $E$ receives a latent calculated at the previous step, $\textbf{w}_{i-1}$²²2When $i=1$, we utilize $\textbf{w}_{0}$ as an average latent of StyleGAN., together with a pair of images. The pair consists of $(\hat{y}_{i-1},I_{i})$, where $\hat{y}_{i-1}$ is a decoded result of $\textbf{w}_{i-1}$ via generator $G$, i.e., $\hat{y}_{i-1}=G(\textbf{w}_{i-1})$, and $I_{i}$ is a preprocessed ground truth image, $I$, by our proposed method in Section 3.4. $E$ targets to encode the difference between $\hat{y}_{i-1}$ and $I_{i}$ into the latent, $\Delta_{i}$. Consequently, $G$ can yield an image $\hat{y}_{i}$, which is more similar to $I_{i}$ than $\hat{y}_{i-1}$, by decoding the latent $\textbf{w}_{i}=\textbf{w}_{i-1}+\Delta_{i}$. Our model iteratively refines the latent with a total of $N$ iterations. Finally, we utilize a loss function $L_{image}$ for training, consisting of the weighted sum of $L_{2}$, $LPIPS$ [34], and $L_{ID}$ [24]. We explain the details of each loss in Appendix A.

To guide the model to focus on the interest region, we need to label the interest region first. The interest region can be designated arbitrarily according to the usage of the inversion model. For instance, facial and hair regions for the facial domain, and the whole body for the animal domain can be set as the interest region. For labeling this interest region, the off-the-shelf segmentation networks are used, which is described in Section 4. However, directly using the segmentation masks from networks causes the distortion of facial boundaries in the generated image. To accommodate the boundary information, we use the dilated segmentation mask containing the interest region, as shown in Figure 5. Without dilation, the loss term on the interest region cannot penalize the inverted face on the uninterest region. Consequently, we dilate the mask to penalize the overflowed reconstruction of the interest region boundary. Though the small part of the uninterest region would be included in the interest region through the dilation, we empirically find that our model still precisely generates the interest region without any distortion of boundaries. We visually show the effect of mask dilation at the ablation study in Section 4.1.1.

To enforce the model to invert the interest region into the latent space precisely, we should train the model to concentrate on the interest region regardless of the uninterest region. However, due to the spatial-agnostic feature of Adaptive Instance Normalization (AdaIN) [21], inverted style latents considering the uninterest region may deteriorate the inversion quality of the interest region. To prevent the encoded style of the uninterest region from deforming the inverted result of the interest region, the inversion of each region should be disentangled.

As $E$ encodes the difference of the input pair of images in ideal, the latent $\Delta$ obtained by encoding the pair of images that only differ on the uninterest region does not contain the information of the interest region. In other words, the decoding results from the latents w and $\textbf{w}+\Delta$ should be the same on the interest region. Motivated by the above, we propose a simple method named Interest Disentanglement (InD) to distinguish the inversions of the interest region and uninterest region. We construct the pair of input images for InD as follows: the original image $I$, and the same $I$ but multiplied with the interest region mask. Then, as shown in Figure 4, we can yield the pair of images which only differs in the uninterest region, and the corresponding latent from $E$, $\Delta_{b}$. Ideally, the information in $\Delta_{b}$ is solely related to the uninterest region, which implies $\textbf{w}_{N}$ generates the interest region robustly even $\Delta_{b}$ is added. Consequently, we define InD loss, $L_{InD}$ as follows;

where $\hat{y}$ is the inversion result from the latent $\textbf{w}=\textbf{w}_{N}+\Delta_{b}$. We apply Interest Disentanglement only at the training stage, which enables the inference without the interest region mask. We empirically find that IntereStyle focuses on the interest region without any prior mask given, after the training.

At the early steps of the iterative refinement, $E$ focuses on reducing the distortion of the uninterest region [3]. Due to the spatial-agnostic feature of AdaIN, we claim that excessively focusing on the uninterest region hinders the inversion of the interest region. We propose a method named Uninterest Filter (UnF), to make $E$ concentrate on the interest region at every iteration consistently. UnF eliminates details of the uninterest region, which is inherently infeasible to reconstruct. Thus, $E$ can reduce the redundant attention on the uninterest region for the low distortion. In detail, UnF eases calculating $\Delta_{i}$ by blurring the uninterest region of $I$ at each iteration, with a low-pass Gaussian Filter with radius $r$, $LPF_{r}$. As shown in Figure 4, UnF gradually reduces the radius of Gaussian filter of $LPF$ as iterations progress, with the following two reasons; First, the redundant attention on the uninterest region is considerably severe at the early stage of the iterations [3]. Consequently, we should blur the image at the early iterations heavily. Second, excessive blur results in the severe texture difference between the interest and uninterest region. We claim that $E$ is implicitly trained to encode the difference of the whole region, which can be biased to produce the blurred region when the blurred images are consistently given. For the realistic generation, the input at the $N$-th iteration, $I_{N}$ is deblurred, i.e., identical to $I$. We calculate the input image at the $i$-th iteration as below:

At the training stage, we jointly train the model with the off-the-shelf encoder training loss [24] $L_{image}$ and $L_{InD}$. However, in contrast to Restyle [3], which back-propagates $N$ times per batch, we back-propagate only once after the $N$-th iteration is over. Thus, ours show relatively faster training speed compared to Restyle. Our final training loss is defined as below:

Our proposed methods, InD and UnF are synergistic at the training; While InD disentangles the inversion of the uninterest region, UnF forces to look at the interest region. Though applying $L_{image}$ to the images multiplied with $I_{mask}$ inherently drives $E$ to focus on the interest region, InD is essential for robust training. Without $L_{InD}$, we find $E$ implicitly contains information of the uninterest region into $\Delta_{i}$, which affects the inversion of the interest region by AdaIN.

Inversion of GAN is deeply related to the image manipulation on the latent space. In this section, we compare the quality of edited images produced by various StyleGAN inversion models [24, 30, 3], manipulated via InterFaceGAN [28] and StyleCLIP [22] methods, and style mixing [18, 19]. Figure 8 shows the results of editing real-world images via InterFaceGAN and StyleCLIP, together with the inversion results. We changed three attributes for each method; smile, age, and pose for InterFaceGAN, and “smile”, “lipstick”, and “Mohawk hairstyle” for StyleCLIP. Our model showed high inversion and perceptual qualities consistently among various editing scenarios, even with strong makeups or obstacles. However, $pSp$ and $e4e$ missed important features of images, such as makeups or eye shapes. Moreover, $pSp$ produced artifacts in several editing scenarios. In the cases of $\text{Restyle}_{pSp}$ and $\text{Restyle}_{e4e}$, they failed to robustly handle obstacles. In the right side of Figure 8(a), $\text{Restyle}_{pSp}$ produced severe artifacts around the mouth, while $\text{Restyle}_{e4e}$ totally changed the shape. Moreover, the Restyle-based models showed low editability in specific cases, such as “Mohawk”.

To attribute to the superior disentanglement feature of StyleGAN latent space [28], we can separately manipulate the coarse and fine features of images. Following the settings from the StyleGAN [18] experiment, we took styles corresponding to either coarse, middle, or fine spatial resolution, respectively, from the latent of source B, and the others were taken from the latent of source A. Moreover, we mixed more than one hard case, e.g., obstacles on faces and extreme poses, to evaluate the robustness of our model. As shown in Figure 9, our model showed higher perceptual quality on the interpolated images compared to $\text{Restyle}_{pSp}$. $\text{Restyle}_{pSp}$ produced images with texture shift, i.e., images with cartoon-like texture, distorted facial shape, and undesirable artifacts during the style mixing. In contrast, our model generated stable facial outputs. Additional qualitative results are shown in Appendix D.2.

We compared the progress of iterative refinement between Restyle [3] and IntereStyle in Figure 10. Restyle reconstructed most of the coarse features within a few steps, while the variance of Restyle increased consistently as iteration progressed. In other words, the reduction of distortion is marginal, though Restyle excessively focuses on this. Consequently, the latent from Restyle was located far from $W$, which yields an image with low perceptual quality. In contrast, IntereStyle concentrated on the interest region that could be generated without a broad extension from $W$. Consequently, IntereStyle effectively reduced distortion on the interest region by iteration while maintaining high perceptual quality.