ReGANIE: Rectifying GAN Inversion Errors for Accurate Real Image Editing

As shown in figure 3, the rectifying network employs a dual-pathway encoder that processes the difference image $\Delta X=X-\hat{X}$ and the primary image $\hat{Y}$ in separate branches. Since the $Y-\hat{Y}$ pair and the $X-\hat{X}$ pair are thought to suffer similar loss of information caused by the same inversion model, we assume that the difference between $X$ and $\hat{X}$ provides sufficient information for perfect reconstruction of $Y$ from $\hat{Y}$. We build separate encoders ($E_{p}$ and $E_{d}$) to process the primary image and the difference image respectively: see Figure 3 (bottom).

Considering an image of resolution $512\times 512$, Encoder $E_{p}$ and Encoder $E_{d}$ consist of 4 downsampling layers respectively. Specifically, $E_{p}$ extracts a latent feature map $f_{x}$ of size $32\times 32$ from the primary image. $E_{d}$ extracts a spatial style code $sp$ of size $32\times 32$ and a global style vector $gl$ of dimension $2048$, which is written as

where $sp$ is the style codes with spatial dimensions, and $gl$ is the global style vector.

A style-based Generator $G_{p}$ is then used to fuse the extracted style codes ($sp$ and $gl$) and latent features ($f_{x}$), and generates an image with the inversion errors rectified. The generator starts from the latent feature $f_{x}$ and generates a $512\times 512$ image by applying 4 upsampling layers. The style codes ($sp$ and $gl$) are joined to the generator through style modulation as used in (Park et al. 2020). Note that $X$ and $Y$ are both generated images, they are actually exchangeable to each other, just as

The style codes $sp$ and $gl$ are designated to convey spatial and global information respectively. The original StyleGAN architecture projects an image from a 512-d vector that causes loss of spatial information and leads to an optimization upper bound for inversion accuracy. Therefore, we modify the generator structure to allow for joint spatial and global modulation.

The style modulation module slightly different from that used in StyleGAN2 (Karras et al. 2020b) to allow for alternative modulations of the spatial codes and the global vector: see Figure 3 (bottomright). The equations are written as

where $f^{l}$ represents the feature map of the $l$th layer. $w^{l}$ and $b^{l}$ represent the weights and biases of the $l$th layer, respectively. “$\cdot$” denotes element-wise multiplication and “$*$” refers to convolution.

Spatial intermediate features are used in (Wang et al. 2022; Kim et al. 2021; Yao et al. 2022) for improved reconstruction. However, they are prone to artifacts when deformation occurs in the editing, as the spatial features are not well aligned with the edited image. To avoid the issue, we alternatively place the spatial modulation and global modulation in each layer of the generator. Please refer to supplement for detailed architecture. We observe that the encoder and generator will adaptively learn to re-align the spatial information based on global information.

The rectifying network is trained with an objective consisting of three loss terms, which is defined as:

where $\mathcal{L}_{1}$ loss is used to suppress the pixel-wise disparities between the reconstructed image $I_{r}$ and the original image $I$. $\mathcal{L}_{lpips}$ (Zhang et al. 2018) loss measures the features similarities between the two based on a pre-trained VGG16 network (Simonyan and Zisserman 2014), which enforces reconstructions at the feature level. The image realism is encouraged using an adversarial loss $\mathcal{L}_{GAN}$ with $R_{1}$ regularization (Karras et al. 2020b), where the discriminator $D$ is trained adversarially against the rectifying network based on Loss $\mathcal{L_{D}}$. We set $\lambda_{GAN}=1.0$, $\lambda_{l1}=1.0$ and $\lambda_{lpips}=1.0$ in our experiments for the best performance.

As in Algorithm 1, in the first phase, we use off-the-shelf pre-trained models provided in (Karras et al. 2020b; Li et al. 2021; Tov et al. 2021). We train the rectifying network with generated samples from Phase I. Specifically, the training triplets ($\Delta I$, $\hat{I}$ and $I$) can be obtained in three ways to encourage reconstructions of both edit-free or edit-involved scenarios. First, $\Delta I=Y-\hat{Y}$, $\hat{I}=\hat{X}$ and $I=X$. Second, $\Delta I=X-\hat{X}$, $\hat{I}=\hat{Y}$ and $I=Y$. Third, $\Delta I=X-\hat{X}$, $\hat{I}=\hat{X}$ and $I=X$. The former two encourage reconstructing from its inversion and the inversion errors of its edited partner. The third way encourages reconstructing from its own inversion and inversion errors. During training, the attribute set intended for manipulation are those provided by DyStyle (Li et al. 2021) network, including pose, age, glasses, expression, etc.

During inference, the rectifying network is expected to reconstruct an ideal edited image $Y_{r}$ from its inversion $\hat{I}=\hat{Y}$ and the inversion errors of the input image $\Delta I=X_{o}-\hat{X}$. It is also encouraged to reconstruct the input image itself $X_{r}$ from its own inversion $\hat{I}=\hat{X}$ and inversion errors $\Delta I=X_{o}-\hat{X}$.

To verify the reconstruction accuracy of the proposed method, we compare our approach with state-of-the-art GAN inversion methods including e4e (Tov et al. 2021), ReStyle (Alaluf, Patashnik, and Cohen-Or 2021b), HFGI (Wang et al. 2022), HyperStyle (Alaluf et al. 2022) and PTI (Roich et al. 2022): see Figure 4.

We can observe that the encoder-based methods e4e and ReStyle cannot restore difficult cases well (long-tail information loss). HFGI, HyperStyle and PTI are able to recover more details in the original image due to use of spatial information encoding or generator optimization, but the inversion accuracy is far from being perfect. Our approach achieves the best reconstruction accuracy and the reconstruction errors are mostly unnoticeable. We provide user study and more visualizations, please see the supplementary material.

We perform a quantitative comparison of related GAN inversion methods and ours in terms of the reconstruction accuracy and inference time: see Table 1. The similarity between the reconstructed image and the original was measured with L2 distance, LPIPS (Zhang et al. 2018) and MS-SSIM (Wang, Simoncelli, and Bovik 2003) scores. Additionally, the identity similarity is measured using a pre-trained face recognition model provided by (Huang et al. 2020). Note that, unlike other methods, we do not explicitly apply the identity preserving loss during training. Finally, the inference time is also tested. e4e (Tov et al. 2021), HFGI (Wang et al. 2022) and our method are single-pass methods, while ReStyle (Alaluf, Patashnik, and Cohen-Or 2021b) and HyperStyle (Alaluf et al. 2022) employ multi-pass forwarding (5 based on the official recommendation). Since PTI (Roich et al. 2022) is optimization-based, it is iterated for 450 times to search the initial code, followed by 350 iterations to update the generator weights. PTI costs 2-3 orders of magnitude more time than the encoder-based methods. As shown in Table 1, our method outperforms other methods in terms of the reconstruction accuracy by large margin, while the speed is among the best two.

To visually compare different image editing methods, we demonstrate their results as for manipulating three types of facial attributes: age, pose and expression. The selected attributes are representatives of three typical editing scenarios. Specifically, age involves whole-image texture editing, pose involves global geometric deformation, and expression involves local deformation. As our rectifying network is trained with the editing results of DyStyle (Li et al. 2021), for fair comparisons, we use InterfaceGAN (Shen et al. 2020) as the style editor in the test phase for all methods.

It can be observed from Figure 5 that ReStyle (Alaluf, Patashnik, and Cohen-Or 2021b), HyperStyle (Alaluf et al. 2022), and PTI (Roich et al. 2022) see limited attribute control accuracy: see the failure to produce wrinkles while designated for aging. HFGI (Wang et al. 2022) produces silhouette artifacts, which can be caused by its naive injection of spatial information into the generator. The e4e (Tov et al. 2021) model obtains stable editing results, but suffer noticeable reconstruction errors and unwanted information loss. In general, our method achieves the best reconstruction quality while not compromising the attribute control accuracy. User study and more visualizations see the supplement.

We quantitatively evaluate the editing quality of different methods, by measuring the attribute control accuracy and identity preservation. As InterfaceGAN edits image attributes by linearly manipulating the StyleGAN latent space $\mathcal{W}$. Specifically, $w_{edit}=w_{init}+\alpha*d$, where ${d}$ is the normal vector of the separation plane (or edit direction) for a specific attribute, and $\alpha$ is the edit magnitude. The attribute of an image can be continuously manipulated by controlling the value of $\alpha$.

We use two numeric attributes, age and yaw angle, for quantitative evaluation. We intend to see how the perceived attribute value responds to the value of $\alpha$. It is believed that the greater the perceived attribute value varies given the same amount of $\alpha$ shift, the better the editability is. We employ the official pre-trained HopeNet (Ruiz, Chong, and Rehg 2018) model for pose estimation, and the age regressor is trained by ourselves based on the official implementation of Dex VGG (Alaluf, Patashnik, and Cohen-Or 2021a). In addition, we also calculated the average identity similarity (Huang et al. 2020) score as the indicator of identity preservation. As shown in Table 2, our method and e4e (Tov et al. 2021) achieve the best attribute control competency, while our method achieve the best identity preservation, implying that our method faithfully manipulates the target attribute without producing unwanted changes along other attributes.

In this work, we use DyStyle (Li et al. 2021) to generate the editing results required for the training of the rectifying network. However, what is inspiring is that the performance of the rectifying network for manipulations of unseen attributes such as hair color, bangs, or face shape is also plausible: see Figure 7. This implies that the generator has learned to align the two inputs regardless of the edit types.

All networks in Phase I and Phase II are trained with images from the same domain. However, we discover that our rectifying network works well on certain unseen images out of the target domain. As shown in Figure 8, the editing results on the artistic portrait dataset can be rectified with the rectifying network trained on FFHQ dataset without any fine-tuning. This implies that our rectifying network has learned to fuse and repair the inputs without overfitting into a specific domain.