Real-World Image Super Resolution via Unsupervised Bi-directional Cycle Domain Transfer Learning based Generative Adversarial Network

In recent years, deep learning-based methods have exhibited exceptional ability to enhance SISR performance. However, most of these methods rely on supervised settings, where specific pre-defined and paired LR and HR images are required in the training process. The pioneer method was introduced by Dong $et\ al.$ [14], namely SRCNN. It is able to recover HR images from bicubic downsampled LR images in an end-to-end manner. Moreover, the enhanced deep SR method EDSR [5] was proposed by Lim $et\ al.$, which greatly enlarges the network to be deeper and wider, resulting in learning richer feature representations. Later, generative adversarial learning was introduced in SISR. SRGAN [15] proposed by Ledig $et\ al.$ incorporates adversarial loss and perceptual loss to update the generator and discriminator respectively, which can obtain photo-realistic SR images. Furthermore, the ESRGAN [7] introduces the new generator architecture and adopts enhanced adversarial loss, achieving promising performance in the visual aspect.

The blind SISR is defined as the task that supposes the LR image is produced by an unknown degradation kernel from its HR version. The Iterative Kernel Correction (IKC) [16] was proposed by Gu $et\ al$. to estimate blur kernel and eliminate artifacts caused by kernel discrepancy. Zhang $et\ al$. [17] proposed IRCNN which includes a set of CNN denoisers to estimate the blur model. However, the results of IRCNN indicate that it is difficult to estimate a comprehensive degradation kernel in real-world conditions. The blind SR methods have the limited capability to approximate unknown degradations. Thus, there is still room for improvement in dealing with blind image super-resolution.

Recently, many unsupervised methods have been proposed to satisfy real-world conditions where the paired LR-HR image data is unavailable, and the input image is corrupted by unknown degradation kernels. By employing the internal recurrence of information inside an image, ZSSR [18] can use pseudo image pairs to recover LR images with diverse blur kernels. Inspired by CycleGAN, Yuan $et\ al$. [10] proposed CinCGAN, which first generates bicubic downsampled LR images from the input and then super-resolves LR images to HR images. However, CinCGAN only considers single bicubic degradation, resulting in poor generalization in complicated real-world SR tasks. By contrast, our method utilizes UBCDTN to simulate diverse real-world degradation types, making it perform proficiently in the real world.

The designed UBCDTN is able to translate the source domain $I^{LR}_{degraded}$ to the target domain $I^{LR}_{real}$. It provides an effective bi-directional cycle solution to reduce the domain gap between $I^{LR}_{degraded}$ and $I^{LR}_{real}$. The forward-cycle module contains $G_{A}$ where it aims to learn a mapping $I^{LR}_{degraded}\rightarrow\widehat{I}^{LR}_{real}$. The goal of $G_{B}$ in backward-cycle module is to learn another mapping $I^{LR}_{real}\rightarrow I^{LR}_{syn}$.

As shown in Fig. 1, the forward-cycle module comprises generator $G_{A}$, discriminator $D_{B}$ and Feature Extractor $FE_{A}$. The backward-cycle module consists of $G_{B}$, $D_{A}$, and $FE_{B}$, where these networks have the same architecture as the forward-cycle module’s corresponding components, but for different purposes. We utilize U-Net as the basic architecture for $G_{A}$ and $G_{B}$. $D_{A}$ and $D_{B}$ are standard convolutional neural networks, where each is composed of nine convolutional layers followed by BatchNormalization (BN) layers and Leaky ReLU layers. The pre-trained VGG19 is exploited as feature extractor $FE_{A}$ and $FE_{B}$.

The UBCDTN trains two generators simultaneously, where these two generators should be translated in bi-direction and inverted to each other. We involve adversarial learning in both modules, where $G_{A}$ and $G_{B}$ are trained on discriminator $D_{B}$, $D_{A}$ respectively. However, as mentioned before, the SISR task is an ill-posed problem in which there exist many possible SR images reconstructed from one given LR input. Thus, applying adversarial loss alone to train UBCDTN leads to the problem of model collapse, and it cannot produce the desired generated images which have the same characteristics and patterns as target domain images. It is necessary to employ regularization on UBCDTN to improve the quality of translated images. Thus, we propose a cycle-consistency constraint to guarantee the domain correlation between ($\widehat{I}^{LR}_{real}$, $I^{LR}_{real}$) and ($I^{LR}_{syn}$, $I^{LR}_{degraded}$). In addition, in order to avoid color variation, identity loss is applied to preserve color composition between the input image and output image [12]. Moreover, we introduce the cycle-perceptual loss as an additional constraint to preserve the sharper edges and finer structures in the reconstructed images. The following sections provide more details of the forward-cycle module and backward-cycle module.

In this section, we present the proposed SESRN, which aims to generate the desired $I^{SR}$ from the $\widehat{I}^{LR}_{real}$ produced by UBCDTN. First, we describe four main components of SESRN: $G_{SR}$, $SE$, $D_{SR}$, $CE$. Second, we describe the optimization loss function for SESRN. The architecture of SESRN is illustrated in Fig. 2. In addition, the structure of the joint discriminator and generator proposed in GAFH-RIDN [13] are shown in Fig. 3.

The full objective loss for the UBCDTL-GAN, which is the combination of $L^{UBCDTN}_{total}$ and $L^{SESRN}_{total}$, can be defined as:

The complete objective loss $L^{UBCDTL-GAN}_{total}$ encourages the proposed method UBCDTL-GAN can solve the unpaired real-world image super-resolution problem.

At the training stage, we conduct experiments on the DF2K dataset [25, 5], which is a merge of the DIV2K and Flickr2K datasets, containing a total of 3450 images. Specifically, as for the LR images, we use the real-world LR images from DIV2K NTIRE 2017 unknown degradation 4× dataset, and Flickr2K LR images collected from NTIRE 2020 Real-World Track 1 training source dataset, where all the LR images are corrupted with unknown degradation, and downsampled 4× by an unpredicted operator to satisfy real-world conditions, resulting in sensor noise, compression artifacts, etc. Since the goal of our method is to solve an unsupervised super-resolution problem without LR-HR paired images, we select the first 1725 images (number: 1-1725) from the DF2K HR dataset as our HR training dataset, and the LR training dataset is formed by the other 1725 images (number: 1726-3450) obtained from DF2K real-world LR dataset. Overall, our method is trained on such an unpaired real-world LR-HR dataset.

To evaluate the proposed method on real-world data, at the testing stage, we use the validation dataset from the NTIRE 2020 Real-World SR challenge Track 1. This dataset contains 100 testing LR images (scaling factor: 4×), where all LR images are processed with unknown degradation operations to simulate the realistic artifacts and natural characteristics. In order to compare the qualitative and quantitative results fairly, we use the same validation dataset for all experiments.

The training procedure is divided into three steps. Instead of randomly initializing model weights, we pre-train the UBCDTN and SESRN in the first and second steps, and then we jointly train the whole method in an end-to-end manner. First, we train the UBCDTN with unpaired artificially degraded images $I^{LR}_{degraded}$ and real-world images $I^{LR}_{real}$, which aims to transfer the LR image from the artificially degraded LR domain to the real-world LR domain. Second, we pre-train the SESRN using the approximated real-like images $\widehat{I}^{LR}_{real}$ and their HR version $I^{HR}$ to generate realistic super-resolved images $I^{SR}$. As for pre-trained UBCDTN and SESRN, we use the same optimization strategy where the Adam optimizer [26] is applied to train both networks by setting $\beta_{1}=0.9$, $\beta_{2}=0.999$. The learning rate is initialized as $10^{-4}$ and the minibatch size is set as 8. We train the UBCDTN and SESRN 50K epochs separately. After the pre-training process, both UBCDTN and SESRN are able to acquire exceptional initialization weights, which are beneficial to training stability and fast training speed. In the third step, we jointly train two networks together. The proposed UBCDTL-GAN takes an unpaired HR image $I^{HR}$ and $I^{LR}_{real}$ as the input to generate the final result $I^{SR}$. The UBCDTL-GAN applies the Adam optimizer with an initial learning rate = $10^{-4}$, $\beta_{1}=0.9$ and $\beta_{2}=0.999$ to the whole training process. The model is trained for 100K epochs with the minibatch size 8.

The quantitative results presented in Table I demonstrate that our method has promising superiority, achieving the highest 26.83dB/0.789 in terms of PSNR/SSIM values. The results obtained from SRFBN [27] place the second-best with 25.37dB/0.642. Our method improves the PSNR/SSIM values by 1.46dB/0.147 over their method. EDSR [5] and SRFBN [27] cannot achieve desired numerical results since they are PSNR-oriented methods and are merely trained on simple degradation images. Moreover, we found that the performances of ESRGAN [7] and SRGAN [15] are lower than most of the methods in Table I. Besides, from Fig. 5, the visual results of ESRGAN and SRGAN show over-smoothed textures and unrealistic artifacts. Interestingly, we found such a phenomenon has also been presented in [10, 18, 30]. The underlying reason is that these methods only train on the simple and clean pre-defined LR images, ignoring the difference of domain distribution between the real-world LR domain and the pre-defined LR domain. Benefiting from the proposed UBCDTN and SESRN, our method has the ability to solve the problem of domain distribution shift and boost quantitative performances when dealing with real-world SR tasks.

The visual comparisons are provided in Fig. 4. We compare with various SR methods such as Bicubic, Nearest Neighbor, SRGAN [15], ESRGAN [7], CycleGAN [12], and ZSSR [18]. For the Bicubic and Nearest Neighbor methods, it is obvious that the results lack high-frequency contents, producing overly smooth edges and coarse textures. Regarding the SRGAN, it fails to alleviate the blurring details on the lines and edges of the SR results. The results of ESRGAN suffer from apparently broken artifacts and dramatic degradation problems, which are unfaithful to human perception. As for the unsupervised method CycleGAN and ZSSR, the SR results are improved, but there are still far away from the ground truth. Although the SR images of CycleGAN present better shapes than the previously compared methods, the results still remain unnatural edges and distortions, leading to poor visual effects. Besides, the blind method ZSSR was also evaluated, but it fails to reduce visible corruptions in some degree since there are still over-smoothed textures and noise-like characteristics existing in the images.

Compared with the aforementioned methods, the SR results of our method superiorly outperform all other methods, super-resolving visually pleasant SR images with sharper edges and finer textures. Our SR results are more realistic than SRGAN and ESRGAN since these two methods are only trained on simple degradation data (e.g., bicubiced LR images) without introducing any complicated noise and artifacts from the real-world images while our method trains on approximated real-like LR images consisting of similar characteristics as real-world LR images. The unsupervised method CycleGAN is less effective in super-resolve unclear LR images. Although it involves the cycle translation model, it lacks a powerful super-resolution network as the one (SESRN) in our method. Besides, the other unsupervised method ZSSR also fails to achieve the expected results, since it does not take into account the domain gap between noise-free LR images and real-world images. In contrast, benefiting from the domain transfer network (UBCDTN), our method is able to successfully eliminate the domain gap and produce real-like LR images comprising real-world patterns. Overall, The SR results verify the powerful unsupervised learning strategy used in the proposed method for super-resolving photo-realistic SR images.

In ablation studies, we design several variants which consist of different proposed components. We adopt the SESRN as the baseline variant in the following experiments and pay more attention to investigating the elements used in the UBCDTN. To comply with the single variable principle, we gradually add one of the components to the baseline variant. We describe the details of designed variants, all of which are specified as follows:

1) VariantA: The VariantA is designed as the baseline variant, which only contains SESRN. As shown in Table II, VariantA can be considered as removing all UBCDTN components from the ultimate proposed method. In the following variants, we successively add each of the components to VariantA.

2) VariantB: In VariantB, we introduce $G_{A}$ and $D_{B}$ while $G_{B}$, $D_{A}$ and both $FE_{A}$, $FE_{B}$ are removed. Because $G_{A}$ and $D_{B}$ are essential components of the forward cycle module in UBCDTN, VariantB can be considered as composed of the forward cycle module of UBCDTN and baseline model, removing the backward cycle module.

3) VariantC: Besides the baseline model, it consists of two generators $G_{A}$, $G_{B}$ and two feature extractors $FE_{A}$, $FE_{B}$ of UBCDTN, eliminating discriminators $D_{A}$ and $D_{B}$ involved in UBCDTN.

4) VariantD: It is constructed by the four components $G_{A}$, $G_{B}$, $D_{B}$ and $D_{A}$ of UBCDTN, while it removes feature extractors $FE_{A}$ and $FE_{B}$ of UBCDTN.

5) VariantE (Proposed): The VariantE represents the ultimate proposed method which comprises a baseline model and all components of UBCDTN.

This experiment is conducted by VariantA and VariantE. Specifically, because of removing UBCDTN, VariantA is trained on bicubic downsampled LR images directly while VariantE takes real-like LR images obtained by UBCDTN as the training LR inputs. According to the analysis of the performance between VariantA and VariantE, we can demonstrate advantages originating from UBCDTN. As shown in Fig. 5, VariantA produces over-smoothed SR images missing high-frequency details while the SR results of VariantE contain naturally desired edges and textures. In addition, from Table III, the quantitative results of VariantA decrease dramatically from 26.83dB/0.789 to 25.97dB/0.757 after removing UBCDTN. The reason is that VariantA simply trains on bicubic data, ignoring the domain distribution difference between bicubic data and real-world data when solving the real-world SR task. By incorporating the UBCDTN in the variant, there is a noteworthy improvement in terms of both qualitative and quantitative performance, which is able to verify that UBCDTN plays an important role in the super-resolution procedure. Thus, we can validate the effectiveness of the proposed UBCDTN and its necessity.

In this experiment, we compare VariantB and VariantE to verify the effect of $G_{B}$ and $D_{A}$, which is also identical to show the effectiveness of the backward cycle module. Note that VariantB is equivalent to UBCDTN for domain transformation without the backward cycle module. In this setting, VariantB takes the artificially degraded images as the input to $G_{A}$ and generates real-like LR images through the supervision of $D_{B}$ in the forward cycle module without the backward cycle module. It can be observed from Table III that with the absence of the backward cycle module, VariantB performs worse than VariantE which contains the whole UBCDTN in terms of PSNR/SSIM values, since there is no restriction to prevent the forward cycle module and backward cycle module from contradicting each other. A huge enhancement can be observed after integrating the backward cycle module into VariantE, where it is able to greatly improve quantitative performance and further produce high quality SR images with desirable details. This is due to the presence of a backward cycle module. The variant is capable of utilizing cycle consistency constraint, which guarantees the correction between two inverse modules in an unsupervised manner. In a word, by introducing $G_{B}$ and $D_{A}$, the whole bi-directional cycle consistency learning strategy can be established to produce real-like LR images which maintain the same characteristics as real-world LR images. These results validate the significance of involving $G_{B}$ and $D_{A}$, which also reveals that the backward cycle module can improve quantitative results and visual quality.

In this experiment, we aim to demonstrate the contribution of $D_{A}$ and $D_{B}$, which can also reflect the importance of an adversarial loss in the UBCDTN. The VariantC and VariantE are compared to investigate the effect of $D_{A}$ and $D_{B}$. In this case, VariantC is trained on cycle-consistency loss, identity loss, and cycle-perceptual loss, while the adversarial loss is not involved resulting from removing $D_{A}$ and $D_{B}$. As shown in Table III, it is obvious that performance severely decreases by removing discriminators from VariantC. By incorporating $D_{A}$ and $D_{B}$ into VariantE, we observe that VariantE significantly outperforms VariantC by a large margin of 1.39dB/0.06 in terms of PSNR/SSIM. From Fig. 5, the visual results of VariantC are degraded, where the SR images contain over-smoothed textures and unclear artifacts, while VarantE is able to generate visually realistic SR images with more nature-looking details. The underlying reason for poor performance is that we cannot employ adversarial loss on VariantC, which leads to the lack of adequate iterative adversarial training. According to the quantitative and qualitative comparisons, it can be verified that by applying adversarial loss performed by $D_{A}$ and $D_{B}$ to the variant, the SR performance is greatly enhanced, indicating the effectiveness of $D_{A}$, $D_{B}$ and adversarial loss.