Content-Aware Preserving Image Generation

This training phase has two primary objectives: to optimize $G_{\mathbf{\phi}}$ and $E_{\mathbf{\psi}}$ to generate high-fidelity images that closely match real-world image distributions, and to optimize $D_{\mathbf{\zeta}}$ to accurately distinguish the generated images as fake. The generator $G_{\mathbf{\phi}}$ is optimized using the loss determined by the evaluation of generated images:

where $N_{b}$ is the number of images in a mini-batch, $j$ denotes the image index, and $\sigma\left(\cdot\right)$ denotes the sigmoid function. Subsequently, the discriminator of this module is trained using the adversarial loss with a regularizer of the squared norm of the gradients of the discriminator on real images (Mescheder et al., 2018):

where $\lambda$ is a weighting factor to control the influence of each loss on the total loss function, $\nabla$ denotes the gradient of a function, and $\lvert\lvert\cdot\rvert\rvert_{\scriptscriptstyle 2}$ denotes $\ell 2$-norm.

The goal of the second training phase is to optimize the parameters of all components in the frequency encoding module. This is achieved by utilizing frequency-refined images that emphasize high-level content features while suppressing undesired fine details. The procedure for refining these frequency components is outlined in the section “Frequency Selection.” Using these refined inputs, the encoder modules analyze the patterns to produce intermediate content guide vectors (as detailed in the section of “Guiding Vector Generation”). These intermediate vectors are then used to produce the final content guide vectors (See the “Content Fusion” section). This process ensures that the extracted features align closely with the embeddings of reference images. The loss function used to measure this alignment is discussed in the “Loss Design” section.

In the inference process, a real image $\mathbf{y}$ is used as input instead of generated images $\mathbf{x}_{\scriptscriptstyle f}$. The guiding vector $\mathbf{q}$ is injected into the coarse blocks of the generator, and the style vector $\mathbf{w}$ is inputted into the remaining blocks. This approach enables the resulting image to capture attribute features from the input image (e.g., shape, age, and hairstyle in face image generation) while maintaining a similar-looking background or color distribution as the image generated by inputting $\mathbf{w}$ into all layers.

To scrutinize our hypothesis, we employ the two-dimensional discrete Fourier transform (DFT) to extract frequency features, which is a widely used tool for frequency analysis. The DFT is a transform that converts a finite sequence of samples into a representation that describes frequency components. Given an intensity (grayscale) image $\widetilde{\mathbf{x}}\left(m,n\right)\in\mathbb{R}^{M\times N}$ converted from $\mathbf{x}$, followed by the definition of luminance in  (Series, 2011), the two-dimensional DFT $\>\mathbf{X}:\mathbb{R}^{M\times N}\rightarrow\mathbb{C}^{M\times N}$ is defined as follows (Gonzalez, 2009):

where $u=0,1,\cdots,M-1$ and $v=0,1,\cdots,N-1$ denote equally spaced frequency variables, and $\left(m,n\right)$ denotes a spatial location of $\widetilde{\mathbf{x}}$. To reduce storage and bandwidth requirements, we resize the input intensity image to half its original size, resulting in $\widetilde{\mathbf{x}}\in\mathbb{R}^{128\times 128}$, where $M$ and $N$ equal to $128$. This resizing operation has been found to have no significant impact on the model’s performance.

Each computed DFT component in (7) corresponds to a specific frequency, which is determined by its position $u$ and $v$. The magnitude of each component represents the strength or intensity of that frequency in $\widetilde{\mathbf{x}}$. Multiplying the magnitude of each DFT component by $(-1)^{m+n}$ shifts the direct current (DC) component $X(0,0)$ to the center $(\lfloor M/2\rfloor,\>\lfloor N/2\rfloor)$ of the frequency spectrum. The shifted DFT is then denoted as $\mathbf{Y}(k,\ell)=\mathbf{X}(u-\lfloor M/2\rfloor,v-\lfloor N/2\rfloor)$, where $k$ and $\ell$ represent both positive and negative frequencies, i.e., $k=-\lfloor M/2\rfloor,\cdots,\lceil M/2\rceil-1$ and $\ell=-\lfloor N/2\rfloor,\cdots,\lceil N/2\rceil-1$. In the experiments of this paper, we used this shifted version of the frequency spectrum.

To conduct a more rigorous analysis of the influence of frequency components across different bands, the frequency spectrum denoted as $\mathbf{Y}_{f}$ can be adjusted by selectively reducing specific frequency components, where the index $f\in\left\{L,H\right\}$ denotes the low-pass ($L$) and high-pass ($H$) filtering operations, defined as follows:

where $u_{c}$ and $v_{c}$ are the positions of zero frequency; and the hyper-parameters $b_{\scriptscriptstyle L}$ and $b_{\scriptscriptstyle H}$ denote the cutoff values for highlighting some range of frequencies.

Once frequency characteristics change by (10) or (13), the frequency-selected image $\mathbf{x}_{\scriptscriptstyle f}\left(m,n\right)\vspace{0.1cm}$ is acquired by the inverse DFT, $\>\mathbf{X}^{-1}:\mathbb{C}^{M\times N}\rightarrow\mathbb{R}^{M\times N}$ to preserve both spatial information and refined frequency information:

for $m=0,1,\cdots,M-1$ and $n=0,1,\cdots,N-1$.

The impacts of frequency selection in image composition by (10) and (13) are visually demonstrated in Figure 4. The figure comprehensively depicts how different frequency bands emphasize specific characteristics when composing images. This visualization provides compelling evidence for the significant impact of frequency components in image composition. In particular, the inverse DFT image in (b) preserves low-frequency components by utilizing a filter described by (10), smoothing some details compared to the original images. On the other hand, the inverse DFT image in (c) highlights edges by applying a filter based on (13), suppressing homogeneous components in the original image. Based on these observations and theoretical reasoning (Strang and Nguyen, 1996; Gonzalez, 2009; Lee et al., 2018), we can prioritize low-frequency components that are directly related to the content of images, while suppressing high-frequency components that are mainly related to apperance. This approach allows us to focus on and transfer only the content of the image, providing a strong rationale for using frequency analysis in image generation. By emphasizing content and minimizing stylistic influence, frequency analysis and filtering improve content transfer and reduce stylistic interference. These advantages are further supported by the experimental results presented in Section 4.

To evaluate the similarity of the generated image distribution with real images, the performance of the proposed framework was compared with that of the that of the current state-of-the-art GAN model, StyleGAN2 (Karras et al., 2020b), as well as with diffusion models including ADM (Dhariwal and Nichol, 2021) and Diffusion-GAN (Wang et al., 2023b). Note that since StyleGAN2 does not provide evaluation results under the same configuration as the proposed framework, we re-trained StyleGAN2 using the configuration detailed in Section 4.2 to ensure a fair comparison. For this purpose, we utilized the official PyTorch implementation of StyleGAN2-ADA (Karras et al., 2020a), developed by the same authors as StyleGAN2 (Karras et al., 2020b), which has been confirmed to produce results consistent with the TensorFlow version.

The performance of models was primarily evaluated using Precision (P) (Kynkäänniemi et al., 2019) on a set of $50,000$ generated images for each dataset. Precision measures the percentage of generated images that accurately match specific real images in the dataset, indicating how realistic the generated images are. Therefore, Precision is the most suitable metric for assessing the content similarity between generated and reference images, aligning with the objectives of this work. Nevertheless, Fréchet Inception Distance (FID) (Heusel et al., 2017) scores were also provided, as it is a commonly used metric in image generation. FID focuses more on quantifying the diversity of the generated images compared to the desired distribution (Kynkäänniemi et al., 2019).

This evaluation results are provided in Table 1. It is important to note that in the experiments conducted in this section, a lowpass filter with a cutoff frequency of $b_{\scriptscriptstyle L}=10$ was applied to all test datasets. The filter type and cutoff value were selected based on the experiments described in Section 4.4.3, where they provided the best precision suitable for the task. According to the results in Table 1, the precision of the proposed framework showed significant improvements compared to StyleGAN: $17\%$ on FFHQ, $16\%$ on AFHQ Cat, $25\%$ on AFHQ Dog, $20\%$ on LSUN Bedroom, and $35\%$ on LSUN Church. Additionally, performance improved by approximately $15\%$ on FFHQ, $4\%$ on AFHQ Cat, $5\%$ on LSUN Bedroom, and $56\%$ on LSUN Church compared to ADM. Compared to Diffusion-GAN, performance improved by approximately $18\%$ on FFHQ, $34\%$ on AFHQ Cat, $36\%$ on AFHQ Dog, $14\%$ on LSUN Bedroom, and $36\%$ on LSUN Church. In contrast, the FID scores of the proposed framework were higher than those of the comparison method. This discrepancy arose because the proposed method emphasizes preserving the attributes of reference images, resulting in generated images that closely resemble specific images in the real dataset. As a consequence, the proposed framework might not fully capture the overall statistical properties of the real dataset. This highlights the trade-off between similarity and diversity in the training of GANs. Further analysis of the tradeoff relationship between Precision and FID in Section 4.4.2 suggests that as Precision increases, FID inevitably increases. Therefore, higher Precision values confirm that the proposed framework can produce images preserving content attributes of reference images, even as FID increases, aligning with our intended goal.

We also visually demonstrate the effectiveness of the proposed framework in Figs. 5, 6, and 7. In the figures, the real input images are displayed in the first column, and the images generated from a projected vector $\mathbf{w}$ in Figure 3 are shown in the first row. The examples of the generated images obviously demonstrate that they preserved the content of the corresponding input real images within the same row. In particular, the generated face images Figure 5 exhibited similar attributes such as age, face shape, pose, direction, hairstyle, and accessories, while other attributes such as skin color, hair color, and ethnicity matched those in the first row. This observation indicates that the color distribution and backgrounds of the generated images were influenced by the injected vector $\mathbf{w}$ in the fine layers of $G_{\scriptscriptstyle\phi}$, as expected.

As validated by the experiments, the proposed frequency encoding module effectively produces output images that resemble the content of the reference images, thereby providing users with control over the content of the generated images.

As specific evaluation metrics to quantify the transfer of content from real images to generated images are not available, we assessed the effectiveness of the proposed framework by examining its ability to preserve attributes of guiding images. However, since attribute labels for other benchmark datasets were unavailable, we employed the CelebA dataset to train attribute classifiers and applied these classifiers to the images of FFHQ. The CelebA dataset includes 40 facial attribute annotations, covering aspects such as gender, hairstyle, and accessories, as detailed in Section 4.1.

To conduct the experiment, we trained multiple binary classifiers for all attributes. These classifiers were then applied to both the guiding images and the generated images to determine the percentage of attributes that matched between them. The binary classifiers were implemented using EfficientNet-B4 (Tan and Le, 2019) and trained with the Adam optimizer. We set the optimizer’s parameters as $\beta_{1}=0.9$, $\beta_{2}=0.999$, and a weight decay of 0.01. The training process lasted for $200$ epochs, using a batch size of $512$. To ensure a smooth learning rate transition, we employed a warm-up strategy, gradually increasing the learning rate from $0$ to $0.00001$ (Goyal et al., 2017) until the first epoch. The training images were obtained by cropping to $90\%$ of their original size. Data augmentation techniques, such as horizontal flipping, random brightness and contrast adjustments, and cutout (DeVries and Taylor, 2017) with four holes (each hole having a maximum edge length of $32$) were also applied.

The attribute classification results are exemplified for the output images of Figure 5(a) in Figure 5(b). Each cell of Figure 5(b) contains abbreviations representing attributes corresponding to the images’ positions in Figure 5(a). The attributes include Attractive (A), Bald (B), Bushy Eyebrows (BE), Black Hair (BH), Big Lips (BL), Big Nose (BN), Brown Hair (BrH), Bags Under Eyes (BUE), Chubby (C), Double Chin (DC), Eyeglasses (E), High Cheekbones (HC), Heavy Makeup (HM), Male (M), Mouth Slightly Open (MSO), No Beard (NB), Narrow Eyes (NE), Pale Skin (PS), Smiling (S), Sideburns (Sb), Straight Hair (SH), Wearing Lipstick (WL), Wavy Hair (WaH), Wearing Hat (WH), Young (Y). As expected, the attributes of the output images aligned with the attributes of the images in the same row. For instance, the hairstyle, gender, and glasses of the real images were transferred to the output images, while the background and color characteristics of the generated images were preserved in the outputs. In fact, the average matching accuracy across all attributes is higher than $85\%$. This demonstrates the effectiveness of the proposed framework in controlling the content of the generated images while retaining other attributes influenced by the injected random vector.

To determine the structure of CFM, we conducted experiments to assess the impact of varying quantities of EM blocks on its performance, as shown in Figure 8. Based on the experiment, we configured the CFM with 4 EM blocks. Configurations employing more than 4 EM blocks could decrease the FID score; however, they also showed a considerable escalation in model complexity, offering only marginal performance improvements compared to the 4-block structure. Consequently, the experimental result suggested that the CFM with 4 EM blocks is the optimal choice, as it ensures both satisfactory performance and manageable model complexity.

The truncation trick (Brock et al., 2019; Karras et al., 2019; Marchesi, 2017) refers to the practice of restricting the range of random noise input to enhance the similarity between generated images and real images at the expense of reducing diversity. This tradeoff between similarity and diversity is commonly observed in GANs training. When the truncation is maximized, i.e., $\psi=0$, the variation in the generated images decreases due to the reduced influence of random noise. Conversely, as the truncation threshold increases, the diversity increases, but the perceptual quality of the generated images improves, making them more similar to real images (Kynkäänniemi et al., 2019). This is clearly demonstrated by the analysis results in Figure 9. In the figure, we plotted FID and (1-Precision) instead of Precision to clearly illustrate the tradeoff relationship between them. This choice is made because better performance is represented by a lower FID value, while higher values indicate better performance for Precision.

The observed tradeoff relationship can be directly applied to interpret the performance of the proposed framework, as reported in Section 4.3.1, in the context of its objective. The proposed framework aims to generate output images that closely resemble user-selected, specific real-world images. As a result, the precision of the generated images is significantly improved, indicating a higher percentage of accurate matches with the reference images. However, this emphasis on content preservation might come at the expense of overall statistical diversity, leading to higher FID scores compared to other methods. The intentional tradeoff aligns with the primary objective of the framework, which is to afford users control over the content of the generated images while maintaining an acceptable level of quality, as observed in the examples in Figures 5, 6, and 7. This interpretation is supported by the experimental results presented in Table 1 and Figure 10.

In order to analyze the influence of frequency components on image generation, we experimented by generating images using different cutoff frequencies ($b_{\scriptscriptstyle L}$ and $b_{\scriptscriptstyle H}$) with (10) or (13), along with (7) and (14). We applied this to input images with different frequency limits, enabling us to examine the correlation between frequency manipulation and the resulting generated images.

Figure 10(a) shows the ratio of FID corresponding to different bandwidths of low-pass filtering using (10) and high-pass filtering employing (13) with respect to the FID of images without filtering. The values next to the datasets’ names represent the FID of the unfiltered images for reference purposes. Overall, encoding frequency-selected images into feature embeddings tended to result in improved FID scores. Interestingly, the FID scores improved as more high-frequency components were removed through lowpass filtering. On the other hand, maintaining only high-frequency components did not have a significant impact on the improvement of FID scores, although it yielded better results compared to full-band images. Likewise, increasing the bandwidth of high-pass filtered images led to better FID scores. Based on these observations, we conclude that restricting certain frequency bands improves the quality of generated images in image generation. In particular, maintaining a narrow band of low-frequency components tends to produce higher-quality images. This could be attributed to the characteristics of neural networks, which may face challenges in learning high-frequency components during training.

As already pointed out in Section 4.3, the tradeoff relationship between FID scores and Precision can also be observed in the graphs in Figure 10(a) and (b). This relationship was particularly evident in the models utilizing low-passed images, as depicted in Figure 10(b). Specifically, as Precision increased, the FID score also increased, indicating a decline in the similarity of the overall distribution of real data. Although the tradeoff in high-pass filtered images was less clear, there were instances where higher Precision corresponded to higher FID scores.

In addition, Figure 11 displays the attribute classification accuracy on the FFHQ dataset for various cutoff values and classification thresholds (0.3, 0.5, 0.7). To examine the influence of frequency bands on content preservation, we conducted experiments similar to those described in Section 4.3.2, but using different frequency-selected images. The attribute classification results indicate that wider bandwidths of low-pass filters tended to preserve the content characteristics to a slightly greater extent. This observation is consistent with the analysis of Precision, where increasing precision with wider low-pass bandwidth allowed for the preservation of more attributes. Conversely, narrow bandwidth high-pass filters showed a slight advantage in transferring content attributes. While the performance differences may not be significant, the experimental results suggest that manipulating low-frequency components, which contain more abundant spatial information, can assist in controlling the content of generated images. This provides valuable insight into content manipulation techniques.

We also visualize how the output images look like with different cutoff frequencies in Figure 12. These examples provide qualitative evidence that retaining low-frequency components was more effective in aligning the content with the guiding images compared to keeping high-frequency components. This reaffirms the fact that preserving only low-frequency components can enhance image quality, as indicated by the FID scores, and result in higher Precision, although the impact on Precision is minimal, as observed in the graphs of Figure 10.