Rethinking Image Skip Connections in StyleGAN2

GANs [1] consist of two neural networks, the generator (G) and the discriminator (D), which are trained simultaneously through a competitive process. The role of G is to generate data that is indistinguishable from real data, while the job of D is to distinguish between the fake data synthesized by G and actual data. This adversarial process is akin to a forger trying to create a counterfeit painting, with D acting as the art critic trying to spot the fake. Through this competition, G could produce fake data that looks real theoretically. The traditional objective functions of G and D are defined as follows:

However, a notable challenge inherent in the original objective functions is the instability during the training phase. This instability often manifests as difficulty in convergence, leading to sub-optimal generation of data and sometimes causing the training process to fully fail. To address these issues, many papers [40, 13, 15, 41, 26] have been actively proposing and evaluating alternative objective functions that can offer more stable training dynamics.

In the current landscape of GAN research and application, a particularly effective approach has gained prominence, known as the non-saturating logistic loss with $R_{1}$ regularization [41, 26, 24, 27]. This method enhances the training stability and the quality of the generated images by carefully balancing the learning process of G and D. The non-saturating logistic loss ensures that the gradients of G do not vanish or explode, maintaining a steady pressure on the generator to improve. Concurrently, the $R_{1}$ regularization adds a penalty based on the first-order gradient of the D for real data, encouraging smoother decision boundaries and further stabilizing the training dynamics. The non-saturating logistic loss with $R_{1}$ regularization is defined as below:

where $\gamma$ is a hyper-parameter that constrains the gradient magnitude of the discriminator. By using this function, many GAN-based image synthesis methods have achieved more reliable and higher-quality results.

StyleGAN2 [26] is an advanced version of the original StyleGAN architecture, which represents a significant advancement in the field of GANs. StyleGAN2 introduces several key improvements over its predecessor to enhance image quality and address specific issues observed in StyleGAN-generated images, such as unnatural artifacts and inconsistencies. One of the notable changes in StyleGAN2 is the redesign of the normalization method that maintains stylistic control while reducing artifacts. In detail, the original StyleGAN used a feature called AdaIN (Adaptive Instance Normalization) at each layer of G to inject style information. However, this led to certain artifacts and issues related to style control. StyleGAN2 introduces a revised approach to normalization, called weight demodulation. This technique replaces the previous modulation mechanism and ensures that the scaling of the convolutional weights does not introduce undesired artifacts in the generated images. Weight demodulation helps in producing cleaner and more coherent textures, significantly improving image quality.

Another improvement is in the handling of the progressive growing technique in StyleGAN. Instead of progressively increasing the resolution during training, StyleGAN2 modifies the network architecture and training process to maintain a fixed resolution, which helps reduce training instability and eliminates phase artifacts. Specifically, they conducted extensive experiments to evaluate the effectiveness of different strategies that improve the information flow within the network, such as residual connections [42, 12, 13], image skip connections [43, 44], and hierarchical methods [45, 46]. Based on the experimental results, they concluded that integrating G with image skip connections and D with residual connections leads to effective performance. It is reliable to draw conclusions based on the results of various experiments, but we believe that it is necessary to consider the theoretical reasons why image skip connections yield good results. Therefore, in this brief, we first analyze the mathematical meaning of image skip connections and propose a new image skip connection method that shows impressive performance.

As shown in Fig. 1, StyleGAN2 generates the output image by aggregating the images derived from the intermediate layers. While this appears to be a coarse-to-fine technique where images are created from intermediate layers and then progressively added together, it would not be the case. The reason is as follows: Unlike the convolution layer that conducts both modulation and demodulation (yellow box in Fig. 1), the toRGB layer performs only modulation. In other words, since demodulation is not performed, the operation of the toRGB layer is equivalent to first modulating the feature and then performing a $1\times 1$ traditional convolution layer without activation function. Therefore, we can express the toRGB layer with the following formula:

where $f_{j}^{\prime}\in\mathbb{R}^{c_{j}\times h\times w}$ indicates j-th intermediate feature modulated by scaling each channel along with the given Style, and $I_{j}\in\mathbb{R}^{3\times h\times w}$ and $W_{j}\in\mathbb{R}^{c_{j}\times 3}$ are the j-th intermediate image and weight matrix of $1\times 1$ convolution, respectively. In addition, x and y represent the pixel coordinates of the feature map, while $c_{j}$ denotes the channel dimension of $f_{j}^{\prime}$. Note that there is no non-linearity in Eq. 5 since the modulation and the $1\times 1$ convolution in each pixel are linear operations.

Using Eq. 5, we can express in mathematical terms how the output image is formed. For simplicity, let us assume that the output image has $8\times 8$ resolution. Then, the output image $I\in\mathbb{R}^{3\times 8\times 8}$ can be formulated as

where $Up(\cdot)$ indicates the $2\times$ up-sampling function such as nearest neighbor and bilinear interpolation methods, and $I_{1}$ and $I_{2}$ are intermediate images having $4\times 4$ and $8\times 8$ resolution, respectively. Using Eq. 5, we can rewrite the Eq. 6 as

where $W_{a}\in\mathbb{R}^{(c_{1}+c_{2})\times 3}$ is an integrated matrix formed by the combination of $W_{1}$ and $W_{2}$, and $f_{a}^{\prime}\in\mathbb{R}^{(c_{1}+c_{2})\times h\times w}$ also indicates an combined feature formed by concatenating $Up(f_{1}^{\prime})$ and $f_{2}^{\prime}$ along the channel dimension. That means the existing image skip connection technique achieves mathematically identical results by concatenating the modulated intermediate features, i.e. $f_{j}^{\prime}$, in the channel direction, followed by performing a 1x1 convolution operation.

Based on this analysis, we point out that the conventional image skip connection method may encounter problems when generating high-resolution images. For instance, when producing images with a resolution of $256\times 256$, image skip connections are executed from a resolution of $4\times 4$ up to $256\times 256$. As mentioned earlier, the image skip connection technique, which has the same mathematical meaning with concatenating intermediate features at each resolution followed by a 1x1 convolution, leads to an issue where the dimension of the concatenated features increases. In essence, when generating a $256\times 256$ image, it is equivalent to projecting modulated features with 2,496 dimensions onto a three-dimensional image using a single linear operation, i.e. $1\times 1$ convolution layer without actviation function. However, this method faces challenges in achieving optimal performance due to the difficulty of directly projecting high-dimensional features into a lower-dimensional space.

To alleviate this problem, we propose a method that is simple but surprisingly effective. Fig. 2 illustrates the overall framework of the proposed method, which replaces the two $3\times 3$ convolution layers present in each resolution block of StyleGAN2 (Two convolution layers of StyleGAN2 are described in Fig. 1). Specifically, we first produce the feature $f_{i}\in\mathbb{R}^{c\times h\times w}$ by using $3\times 3$ convolution layer with an up-sampling technique. Then, $f_{i}\in\mathbb{R}^{c\times h\times w}$ is transformed into a feature $f_{s}\in\mathbb{R}^{c/r\times h\times w}$, which has a lower channel dimension compared to a $f_{i}$, through a 3x3 convolution operation (i.e. , squeeze process), where r is a user parameter that represents the channel-squeeze ratio. Here, $f_{s}$ is used to generate an intermediate image via the toRGB layer; this process means reducing the dimension of the features that are concatenated to generate the output image (recall Eq. 7). Subsequently, through another 3x3 convolution operation, we produce a feature $f_{e}\in\mathbb{R}^{c\times h\times w}$, which has the same channel dimension as $f_{i}$ (i.e. excitation process). We agree that there might be an information loss owing to the squeeze and excitation process. To mitigate this issue, we produce the output feature $f_{o}\in\mathbb{R}^{c\times h\times w}$ by first concatenating $f_{i}$ and $f_{e}$ and then applying a 1x1 convolution operation. The effectiveness of this process will be proved in Section IV-D. Note that all convolution operations, except the toRGB layer, incorporate both modulation and demodulation processes.

These simple structural changes also help in reducing the number of network parameters. In the case of the original StyleGAN2, each resolution requires $18c^{2}$ convolution weight parameters, whereas the proposed method needs $(10+18/r)c^{2}$ parameters (assuming the number of input and output channels is c, and excluding the number of Style parameters). Thus, if the user parameter r is greater than 2.25, it is possible to reduce the number of network variables, leading to improvements in memory efficiency. For instance, setting r to 8 can reduce the convolution weight parameters by approximately 32%. Furthermore, our framework is straightforward and can be easily integrated into existing StyleGAN2-based models. In the following section, we demonstrate the effectiveness of our method in unconditional image generation across various datasets.

We assess the effectiveness of the proposed method across multiple datasets, including CIFAR-10 [33], FFHQ [24], multiple LSUN [34] categories such as cat, horse, and church, and AFHQ [35]. CIFAR-10 is comprised of 50,000 small color images distributed across 10 classes, while FFHQ encompasses 70,000 high-quality facial images. AFHQ contains approximately 5,000 images per cat, dog, and wild animal face. The LSUN church, horse, and cat datasets depict respective scenes, and we have used 200,000 images per each dataset [2]. Consistent with prior research, we have implemented horizontal flips for the CIFAR-10, FFHQ, and AFHQ datasets. All images were resized to $256\times 256$, except CIFAR-10 images, which were maintained at $32\times 32$. The proposed method has a single user parameter r that determines the channel squeeze ratio. In our experiments, we empirically set the r value as 8, except for the ablation studies.

To prove the superior performance and generalizability of the proposed method, we executed comparative experiments against various baseline models, including StyleGAN2 [26], ADA [47], GGDR [2], and GLeaD [3]. StyleGAN2 is extensively recognized in this domain for its robust performance and stable generator training. Both GGDR and GLeaD are evolved iterations of StyleGAN2, highlighting significant performance improvements in recent years. ADA applies data augmentation strategies and is particularly effective in scenarios involving datasets with limited numbers of images such as CIFAR-10 and AFHQ.

In our experiments, we systematically substituted the generator block with our proposed method across all baseline models. We maintained consistency in hyperparameters, such as learning rate and regularization weights, following the previous papers. We conducted the experiments using the official source code of each baseline models¹¹1https://github.com/NVlabs/stylegan2-ada-pytorch^{, 22}2https://github.com/naver-ai/GGDR^{, 33}3https://github.com/EzioBy/glead. Detailed parameter settings can be found in Table I.

In this study, we utilized widely recognized metrics, namely the Fréchet Inception Distance (FID) [37] and Precision/Recall [38, 39], to evaluate the realism of the generated images. The FID assesses the Wasserstein-2 distance between the feature distributions of real and generated images, derived from the pre-trained Inception model [54]. FID is defined as follows:

where $\{\mu_{p},C_{p}\}$ and $\{\mu_{q},C_{q}\}$ are the mean and covariance of the samples with distributions of real and generated images, respectively. A lower FID score indicates that the generated images have better quality. Precision measures the proportion of generated images resembling those in the training set, while Recall indicates the proportion of training images that the model can replicate. For CIFAR-10, we also use Inception Score (IS) [36] following the previous papers [2, 47]. IS can be expressed as

where l is the class label predicted by the Inception model [54] trained by using the ImageNet dataset [55], and $p(l|X)$ and $p(l)$ represent the conditional class distributions and marginal class distributions, respectively. Unlike the FID, the higher the IS score, the better the quality of the generated image. For our analysis, we generated 50,000 samples and calculated these metrics accordingly.

To evaluate the performance enhancement brought by our method, we first conducted experiments on the CIFAR-10 dataset. We compare the performance of our method not only with the leading GANs-based models [51, 52, 26, 53, 26, 2] but also with the latest diffusion-based models [48, 49, 50]. Table II shows the FID and IS scores of ours and comparison methods on the CIFAR-10 dataset. Specifically, we incorporated our method into ADA [47], recognized as a benchmark model for unconditional image generation, particularly effective on small datasets. As shown in Table II, the proposed method successfully improves the ADA performance in terms of both FID and IS scores. Moreover, we applied our proposed method to GGDR [2] which already outperforms ADA. As demonstrated by our experimental results, the proposed method not only significantly boosts the GGDR performance but also achieves state-of-the-art performance. Fig. 3 shows the samples of our method on the CIFAR-10 dataset.

To prove the enhanced capability of the proposed method in generating high-resolution images, we conducted experiments using multiple datasets such as FFHQ, LSUN-Church, LSUN-Horse, and LSUN-Cat. We set the StyleGAN2, GGDR, and GLeaD [3] methods as our baseline models following the previous papers [2, 3]. Table III presents the FID and Precision/Recall scores for both the proposed method and the comparative methods. Specifically, we first assess the performance enhancement brought by the proposed method, using StyleGAN2 as the baseline model. As shown in Table III, the proposed method not only achieves superior performance across all datasets used in the experiments compared to prior works such as UT [56] and Polarity [57] but also marginally enhances the performance of StyleGAN. In particular, the proposed method improves the Recall with the large gap compared to StyleGAN2, which indicates that the proposed method generates more diverse images and is less prone to mode collapse. Moreover, we proceeded with experiments using GGDR and GLeaD, which enhance the performance of StyleGAN2, as our baseline models. As depicted in Table III, the proposed method improves performance across all datasets, and notably, in the case of the GLeaD baseline, both FID and Recall metrics exhibit significant enhancements. Therefore, these experimental results confirm that the proposed method consistently boosts performance, regardless of the baseline. The samples of the generated image are presented in Fig. 4. As shown in Fig. 4, the proposed method produces high-quality synthetic images that appear remarkably real.

To show the robustness of the proposed method against the small dataset, we conducted experiments using the AFHQ dataset [35] by setting the ADA [47] as the baseline model. Table IV presents the experimental results. Since the existing paper [53] only compared FID scores, this brief also focuses solely on FID comparisons. As shown in Table IV, the proposed method improves the baseline in terms of FID scores with a large gap; the proposed method enhances the performance of the baseline model successfully, even on datasets with a smaller number of images. These results indicate that the proposed method is not only effective but also highly adaptable to various types of data, which in turn, enhances its utility and relevance in the various applications. Fig. 5 shows samples of our method on the AFHQ datasets.

The proposed method includes a user-defined variable r that determines the degree of channel compression. While compressing the channels can reduce the number of network parameters, it may also decrease performance. Thus, making the selection of an appropriate r value is crucial. To determine the optimal value, we conducted ablation studies on the FFHQ and LSUN church datasets. We use the StyleGAN2 as the baseline model. As summarized in Table V, we can observe that as the r value increases, the number of parameters in G decreases. However, there is a corresponding decline in performance in terms of FID metric as the number of G parameters decreases. For instance, at $r=16$, the performance on the LSUN church dataset shows weak performance compared to the baseline model. Conversely, $r=4$ and $r=8$ not only show superior performance compared to the baseline model but also the added benefit of reduced parameter count. In terms of the Recall metric, the proposed method consistently outperforms the baseline model, irrespective of the r value. We chose $r=8$ as it significantly reduces the parameters of the network while still maintaining respectable performance.

As mentioned in Section III-B, there is a need to validate the necessity of the feature blending process, which involves the concatenation of features followed by a $1\times 1$ convolution. To this end, we build a modified structure without the feature blending process (as shown in Fig. 6(a)) and evaluate its performance. Table VI summarizes the experimental results. The method without the feature blending process shows lower performance compared to the proposed method, yet it still exhibits comparable or superior performance relative to the baseline model. These results indicate that the significant contribution to performance improvement stems from the image squeeze connection rather than the feature blending process. We believe that these results indirectly highlight the limitations of the existing image skip connection method and the superiority of our approach.

Furthermore, we also measured the performance variations according to the placement of the toRGB layer (Figs. 6(b) and  6(c)). Specifically, Fig 6(b) illustrates the use of the toRGB layer before the squeeze process, while Fig. 6(c) depicts its application after the excitation process. It is important to note that both methods employ features that have high-dimension to the toRGB layer. In other words, similar to the traditional image skip connection, high-dimensional intermediate features are concatenated through the toRGB layer. As shown in Table VII, it can be observed that both cases exhibit lower performance compared to the proposed method. Additionally, they also show decreased performance compared to the baseline model. These results indicate that the performance improvement in the proposed method is not merely due to the presence of squeeze and excitation processes. Instead, it suggests that utilizing low-dimensional features to the toRGB layer enhances performance. Based on these experimental results, we consider the proposed method as a solution that addresses the issues with the traditional image skip connection and significantly aids in improving the performance of StyleGAN2-based models.