Spectral Normalization and Dual Contrastive Regularization for Image-to-Image Translation

GANs [10] [20] [19] have obtained great success, especially in I2I translation and the key idea is adversarial loss [10]. I2I translation can be categorized into two groups: a paired setting [18] [31] [22] (supervised) and an unpaired setting (unsupervised)[42][36][17][1] . Paired setting means that each image from the source domain has a corresponding label, which can be regarded as classic GAN. In order to further improve the quality of the generated images, SPADE introduces a the spatially-adaptive normalization layer. However, it is difficult to obtain paired training data, as a result, current methods [42] are usually based on unpaired settings, which are developed based on one assumption: cycle-consistency. For example, CycleGAN [42], Dual-GAN [36] and MUNIT [17] train cross-domain GANs with cycle-consistency loss. CycleGAN learns two mappings simultaneously via translating an image to the target domain and back preserving the fidelity of the input and the reconstructed image. However, the assumption of cycle consistency that two domains can be mapped in both directions is too strict to obtain sufficient context. To alleviate the issue, many methods have tried to break the cycle consistency. DistanceGAN [1] proposes a distance constraint that allows unsupervised domain mapping to be one-sided. GC-GAN [8] enforces geometry consistency as a constraint for unsupervised domain mapping. CUT [23] introduces patch-wise contrastive learning into I2I translation, which significantly improves the quality of translation. However, only patch-wise contrastive loss is unable to effectively maintain the global structure and texture of the generated images.

Recent studies of the self-supervised learning [13] [5] [29] [38] show its strong ability to represent an image without labels, particularly with the help of the contrastive loss [3] [6]. Its idea is to perform the instance-level discrimination and learn the feature embedding, by pulling the features from the same image together and pushing those from different ones away. It has been observed that contrastive loss has been utilized in several low-level visual tasks, resulting in outstanding performance in various applications such as style transfer [39], image generation [24], image smoothing [43], image super-resolution and Deraining [34, 4]. PatchNCE [23] proposes patch-based contrastive learning, which uses a noise-contrastive estimation framework by learning the correspondence between the patches of the input image and the corresponding generated image patches. Excellent results are achieved and the recent methods [12] [41] [16] [32] [40] also obtained better performance by utilizing the idea of patch-wise contrastive learning. DCLGAN [12] proposes a method based on patch-wise contrastive learning and dual learning settings (using two generators and two discriminators). Although DCLGAN improves the quality of the generated images, an additional generator and discriminator need to be trained, which increases the training costs. QS-Attn [16] designed a query-selected attention module by intentionally choosing significant anchors for patch-wise contrastive learning. MCL [11] designed multi-feature contrastive learning (MCL) to construct a patch-wise contrastive loss using the feature information of the discriminator output layer. In parallel to these various designed methods, we mainly explore global regularization by introducing the idea of global contrastive loss. Therefore, we propose a new unpaired I2I translation framework based on dual contrastive regularization and spectral normalization to improve the global information of the generated images, which consistently shows better results.

Given an input image $\mathrm{~{}x~{}}\in\mathbb{~{}R}^{H\times W\times 3}$ from the source domain X, our goal is to translate it into G(x) in the target domain Y via the adversarial loss, having no apparent difference with the real image$\mathrm{~{}y~{}}\in\mathbb{~{}R}^{H\times W\times 3}$ from the domain Y. The framework of SN-DCR is shown in Fig.2, which consists of a generator G and a discriminator D. We divide the generator into two parts, the first part is defined as the encoder E, and the other part is defined as the decoder. SN-DCR includes three loss functions, the adversarial loss , patch-wise contrastive loss and dual contrastive regularization. In order to perform better patch-wise contrastive loss, we introduce the qs-attn module (global) in our proposed framework.

We expect the generated image G(x) to be as similar as possible to the real image $\mathrm{~{}y~{}}\in\mathbb{~{}R}^{H\times W\times 3}$ from the domain Y , and the generator G can be mapped from the domain X to the domain Y through the adversarial loss. The adversarial loss is as follows:

We adopt the real images of the target domain and source domain as the positives and negatives to improve better quality of the generated images. DCR constrains the generator by two different feature spaces. Note that, to ensure the flexibility of our proposed method, these positives and negatives are randomly chosen. To achieve a constraint of the global structure, we propose a semantic contrastive loss, which aims to encourage the generated image G(x) to be close to the positives P while keeping away from the negatives N. For the feature space, inspired by AECR-Net [35] ,we employ a pre-trained VGG-16 network to extract the feature maps $f\in\mathbb{R}^{C\times H\times W}$.

The semantic contrastive loss can be expressed as:

where $F_{i}$ refers to extracting the i-th hidden features from the VGG-16 network pre-trained on ImageNet. $n$ refers to n layers that we choose. P refers to the real images of the target domain, and N refers to the real images of the source domain. Here we choose the 1st, 3rd, 5th, 9th and 13th layers. $\omega_{i}$ are weight coefficients, and we set $\omega_{1}$ = 1/16, $\omega_{2}$ =1/8, $\omega_{3}$ = 1/4, $\omega_{4}$=1/2, $\omega_{5}$=1.

Besides, to achieve a constraint of the global texture, we use a special feature space. We can build this feature space in any layer of the network, which consists of the correlations between the different filter responses. These feature correlations are given by the Gram matrix:

where $M^{l}_{ij}$ is the inner product between the feature maps $i$ and $j$ in the layer $l$. $k$ refers to the vector length. We employ a feature extraction network to extract the feature maps [c, h, w]. We can obtain the feature maps $i$[ c, h*w] and the feature maps $j$[ c, h*w] via flatten and matrix transpose. Gram Matrices is the inner product between the feature maps $i$ and $j$. We then get a set of Gram matrices $\{M^{1},M^{2},\ldots,M^{L}\}$ from layers {1,2,3…L} in the feature extraction network. The Gram matrix $M$ is a quantitative description of latent image features. We expect that the distance $d$ (we use L2 to measure this distance) between the generated image G(x) and negatives $N$ is much greater than the distance between the generated image G(x) and positives $P$:

Therefore, our proposed style contrastive loss can be expressed as:

where $\alpha$ are hyperparameters (similar to the triplet loss [26] ), we set it to 0.04 in our experiments. Finally, our dual contrastive Regularization is formulated as:

where $\lambda_{1}$ and $\lambda_{2}$ are weight coefficients, and we set $\lambda_{1}$ = 1, $\lambda_{2}$ =0.5.

Following the setup of CUT, we employ a patch-wise contrastive loss to maximize the mutual information between the inputs and outputs. The key idea is to pull the query (patch from the generated image, yellow box in Fig.2) and positives (corresponding from the input image, blue box) together, and push the query and negatives (non-local patches from the input image, red box) away. We employ E to extract features from the input image and the generated image. Then we introduce QS-Attn module to deliberately choose important anchors for patch-wise contrastive loss. The formula can be expressed as:

where $q$ refers to the anchor feature (yellow box in Fig.2) from G(x), $k^{+}$ refers to a positive (blue box) and $k^{-}$ refers to (N-1) negatives (red box). Here $\tau$ indicates a temperature parameter used to measure the distance between query and other samples. The default value is $0.07$ . Note that the positive $k^{+}$ is the corresponding of the anchor feature $q$ in the input image x , and $(N-1)$ negatives are randomly selected in x.

We utilize E to extract features from the generated image, select the L layers of interest from E, and sent it to Q (QS-Attn moudle) to get the features we need. The resulted features can be denoted by $\{\hat{\mathbf{z}}_{l}\}_{\text{L}}=\{Q_{l}(E^{l}(G(\mathbf{x})))\}_{L}$, where $E^{l}$ represents l-th layer we choose. We use qs-attn module to select important anchors in each selected layer , $s\in\left\{1,...,S_{l}\right\}$ ($S_{l}$ represent the number of patches selected in each layer). In the same way, the corresponding patches of L layer are obtained from the input image, $\{{\mathbf{z}}_{l}\}_{\text{L}}=\{Q_{l}(E^{l}(G(\mathbf{x})))\}_{L}$. We take the corresponding patches obtained from the input image as positives, and the other features as negatives.

The patch-wise contrastive loss can be expressed as:

In summary, the overall objective function of SN-DCR can be formulated as:

where $\mathcal{L}^{X}_{patch}$ is the patch-wise contrastive loss defined in Eq.(8), and $\mathcal{L}^{Y}_{patch}$ is the identity loss, in which the positive $k^{+}$ and negatives $k^{-}$ are extracted from a real image y from the domain Y, and the anchor q is from G(y). We refer to the CUT settings to add the identity loss. The goal of this identity loss is to prevent generator G from making changes on the target domain images.

Table 1 shows the quantitative results of our proposed SN-DCR compared with all baselines on three datasets, including QS-Attn [16], DCLgan [12] , FseSim [41], CUT [23], FastCUT [23] and Cyclegan [42]. We mainly use FID and SWD scores as our quantitative metrics. For the metrics, our algorithm outperforms all the baselines obviously, and our SN-DCR achieves state-of-the-art performance on unpaired I2I translation tasks. As illustrated in the Table 2, SN-DCR demonstrates superiority over DCLgan and Cyclegan concerning both performance and computational parameters. Despite having computational parameters comparable to QS-Attn and CUT, SN-DCR outperforms them in terms of performance. Although SN-DCR falls slightly short of FastCUT and FSeSim in computational parameters, its performance significantly surpasses that of FastCUT and FSeSim. Despite these limitations in specific computational metrics, SN-DCR distinguishes itself by exhibiting notable efficacy in the image translation task. Fig.1 shows the visual results comparison with all baselines on the Van Gogh$\rightarrow$Photo dataset. Compared with other methods, our SN-DCR is able to preserve the global structure information and generate the photos with more natural details, proving that our DCR is effective to improve the global information. To better illustrate the superiority of our SN-DCR, we display some visual results in Fig.5. We randomly pick four samples on the Horse $\rightarrow$ Zebra and Cat $\rightarrow$ Dog datasets. QS-Attn, DCLGAN and FseSim fail to preserve details and textures of the generated image. CUT, FastCUT, Cyclegan generate unsatisfactory content. Compared with all baselines, our SN-DCR not only performs better in terms of global structure and texture, but also generates more real images with more natural details. Fig 6 shows the visual results comparison with all baselines on the CityScapes. Compared with other methods, our SN-DCR is able to clearly generate a wide variety of cars with natural details, and performs better in terms of global structure and texture, proving that our DCR is able to enhance consistency of the global texture. Moreover, our SN-DCR can produce more real images with more color details. Table 2 shows the quantitative results of our proposed SN-DCR compared with all baselines on the CityScapes datasets. Obviously, our algorithm outperforms all the baselines. For structural similarity (SSIM) metrics, our proposed SN-DCR performs better than other methods, proving that our DCR is effective to enhance the information of global structure.

In comparison experiments, SN-DCR shows better performance than all baseline methods. In SN-DCR, we apply the FCA and SN module in design of the generator, QS-Attn module in patch-wise contrastive learning, and dual contrastive regularization.

For effect of the weights( hyperparameter $\lambda_{1}$ and $\lambda_{2}$ ) in DCR, ablation experiments are performed on the Horse $\rightarrow$ Zebra dataset to ensure their optimum values, as shown in Table 3. To evaluate the measures separately, we mainly conduct the ablation study on the Horse $\rightarrow$ Zebra dataset. Our baseline is CUT. Visual results and metrics for ablation study are listed in Fig 7 and Table 4.

The metric of models A and B performs better than CUT, reflecting that our proposed global contrastive loss functions are useful for I2I translation tasks. Model C outperforms A and B, indicating the effectiveness of our dual setting. Model D performs better than C, indicating that the addition of QS-Attn module is beneficial to the model’s performance. Model E without DCR to exhibit the different influence between global constraints and local constraints. Model E outperforms QS-Attn, proving that our designed generator is effective. Model F outperforms D, indicating the effectiveness of the SN ResBlock in our generator. Finally, Model G, which combines all the measures, achieves the best performance on I2I translation tasks, validating the efficacy of our overall approach.