From Face to Natural Image: Learning Real Degradation for Blind Image Super-Resolution

Different from the complex textures in natural images, the specific structure in face images make it feasible to well handle the real-world LQ face images [60, 18, 8, 24, 7, 5]. To alleviate the sensibility for the unknown degradation, reference images or component features are suggested for guiding the blind restoration process [29, 28, 27]. Most recently, generative face prior [22, 23] based methods [47, 4, 53] are proposed to improve and stabilize the restoration quality, which can robustly restore the real-world LQ face images in most scenarios. Their great generalization on face images inspires us to explore the possibility of extending the restoration performance from the local region (i.e., face) to the whole image.

The real-world LQ images are mainly corrupted with unknown degradation parameters, so some works focus on estimating these degradation parameters and then apply non-blind restoration methods to recover it. Bell-Kligler et al. [2] firstly propose the image-specific KernelGAN to predict the blur kernels and feed them to ZSSR [41] for non-blind restoration. Gu et al. [16] introduce iterative kernel correction method to estimate the blur kernel which further benefits the restoration results. Luo et al. [34] alternate the optimization of restoring HQ images with the predicted kernel and estimating the blur kernel with the restored results, both of which can compensate each other. Wang et al. [46] suggest a degradation-aware super-resolution network that learn the degradation related parameters to guide the restoration process. However, real-world LQ images usually have high frequency noises or compression artifacts, and these methods are sensitive with them, which brings adverse effect for parameter prediction.

The main challenge of blind image super-resolution task can be ascribed to the lack of suitable training pairs. So a straightforward way is to collect the real-world LQ and HQ pairs. Cai et al. [3] adjust the focal length of the digital cameras to capture the paired LQ/HQ images on the same scene. Wei et al. [50] build a larger dataset with a large-scale diverse benchmark by zooming the digital cameras. Except for the cumbersome capturing process, the spatial and brightness misalignment easily leads to uncontrollable errors. Moreover, although these images are realistic, they are more suitable for the specific super-resolution task that has the similar capturing scenarios. These types of collecting data occupies very few of these complex real-world degraded images, resulting in the failure cases when handling other real degradation, e.g., noise or compression.

To alleviate the difficulties in synthesizing real-world LQ images, recent works tend to learn the restoration mapping with unpaired LQ and HQ images. Yuan et al. [54] suggest a Cycle-in-Cycle network by firstly mapping the LQ input to noise-free space and then super-resolving it through a pre-trained super-resolution model. Similarly, Lugmayr et al. [32] adopt the cycle consistent loss to learn a domain distribution network to generate new LQ/HQ pairs for supervised restoration. Fritsche et al. [12] also propose the unsupervised DSGAN model to generate the degraded LQ images with the same characteristics as the original ones. To constitute more realistic LQ images, Ji et al. [19] extract the blur kernels via KernelGAN [2] and noise injection through [6, 59], which perform on HQ images to simulate the real degradation process. Although these methods achieve great performance in most cases, they still show limited generalization ability in super-resolving real-world LQ images, because 1) the estimated degradation parameters from only a single image is highly ill-posed and they are not enough to infer how the HQ images degraded (Figure 1 (a)), and 2) the real-world LQ images usually suffer from complex degradation, which is challenging to model due to the lack of paired data. In contrast, our ReDegNet adopts the pairs of real-world LQ and pseudo HQ face images to explore the real degradation process.

Another way is to extend the degradation space. Instead of the traditional degradation process that degrades the HQ image with Gaussian blurring, followed by the bicubic downsampling operation, and the injection of Gaussian noise and JPEG compression, Zhang et al. [56] propose a practical degradation model with randomly shuffled orders of these operations which tremendously cover the diverse degradation space. Similarly, Wang et al. [48] suggest a high order degradation model with several repeated degradation process. Although these two methods show great generalization in handling real-world images, they are still incapable for those images corrupted with complex degradation like the halftone image in Figure 1 (a). Traditional methods remove these continuous noisy dots mainly through filters [26, 33, 36], look-up-tables [9], dictionary learning [11], or maximum a posteriori estimation [44]. Recent CNN-based inverse halftoning methods [52, 51, 13, 43] and even these estimation or data-driven based methods still fail to generate photo-realistic results on these types of real-world LQ images, which can be ascribed to the difficulties in synthesizing proper LQ images.

Instead of predicting the degradation related representations from only a single LQ image [19, 46], we take the LQ and HQ pairs as input to explore the degradation process about how the HQ image is degraded to the LQ one. The degradation representation network (DegNet) shown in Figure 2 (a) is stacked with several convolutional layers, each of which followed by spectral normalization [39] and LeakyReLU activation. A fully convolutional (FC) layer is incorporated in the last to predict the degradation representation vector $\Omega$, which has the size of $1\times 512$. This sub-network is optimized through two terms, i.e., the disentanglement loss in Eqn. 7 and the gradient back propagated from the following SynNet.

After obtaining the degradation representation vector $\Omega$, the remaining problem is about how to utilize it to control the degradation process. Inspired by the StyleGANs [22, 23] that control the style of the generated image with one vector within $\mathcal{W}$ space, we adopt the similar structure to map the degradation representation $\Omega$ to $\mathcal{W}$ space through several fully convolutional (FC) layers. Then, instead of feeding the broadcast noise in StyleGAN, the image content of our SynNet is provided by the features of the input HQ images. Finally, with the degradation styles D and image content C, the degraded image is reconstructed with the modulated convolution operation (MCBlock) in which the degradation styles serve as the convolutional weights to control the degradation process of the given image content [23]. With several cascaded MCBlocks, the final LQ result which is expected to have the similar degradation types with the given degradation representation can be synthesized. Since the degradation vector $\Omega$ should be a global representation without any spatial information, here we randomly crop the HQ image as the input of SynNet to alleviate the spatial dependency.

To introduce different scales of textures in the training phase, we adopt the random rotation, resampling, and cropping on face images in DegNet and SynNet, simultaneously. The proposed SynNet combined with DegNet constitutes our ReDegNet that can be jointly optimized in a supervised end-to-end manner.

After training on face images, our ReDegNet can not only extract the real degradation representation from pairs of face images, but also generate the corresponding LQ image with the expected degradation styles. So as for general restoration, we store large amounts of degradation representations that are extracted from real-world LQ and their pseudo HQ face images, which will be sampled to imitate the real degradation process on natural HQ images (Figure 2 (c)). Here we also resample and rotate the LQ/HQ face pairs to augment the degradation space. Notably, our ReDegNet can be utilized in some specific restoration, in which the degradation types are not easy to synthesize with current degradation model. For the intractable old photos (e.g., Figure 1) or old films, we can obtain their degradation representations with DegNet through the pairs of LQ face region within them and its pseudo HQ result. Then the HQ natural images can be utilized to generate the corresponding LQ image by SynNet to synthesize these degradation types of natural training pairs, which can be used to fine-tune the specific restoration on the whole image.

Two types of loss functions are collaborated together to constrain the learning of our ReDegNet, i.e., (i) disentanglement loss that is introduced to extract the degradation-related representations, and (ii) reconstruction loss that is suggested to constrain the synthetic results close to the ground-truth.

i) The MSE loss $\mathcal{L}_{\textit{mse}}$ contains two terms and is formulated as:

where $\hat{I}^{L}_{f}$ is the generated LQ face image in Eqn. 2 and $I^{\textit{ReaL}}_{f}$ is the collected real-world LQ image. $\mathcal{C}_{*}$, $\mathcal{H}_{*}$, $\mathcal{W}_{*}$ are the dimensions and $\Phi_{i}$ is the $i$-th convolution layer of the pre-trained VGG-19 model [42]. This objective constrains the synthetic LQ images close to the real-world LQ images in both pixel and feature space [21].

ii) The realistic loss $\mathcal{L}_{real}$ mainly considers two types of constraints, i.e., style loss [14] and adversarial loss [15]. The first one is computed with the Gram matrix on the feature spaces of VGG-19 model and can be formulated as:

in which the variants have the same definitions as these in Eqn. 8. The second one is the widely used adversarial loss which is effective in constraining the results within the natural manifold. In this paper, we adopt the discriminator from SNGAN  [39] by incorporating the spectral normalization behind each convolutional layer. It is worth noting that the result $\hat{I}^{L}_{f}$ is expected to be a LQ image and visually blur in most cases, which is difficult for discriminator to distinguish whether it is a real LQ or fake LQ image due to the wider space of LQ types. So instead of only taking the synthetic result into the discriminator, we take the HQ image and their degradation representation as additional conditions [37]. The hinge version of adversarial loss [39, 55] is given by:

Combining the two terms together, the final realistic loss is formulated as:

iii) The third one is the degradation-consistent loss. As analyzed before, the degradation representation $\Omega^{Syn}_{f}$ and $\Omega^{Syn}_{n}$ in Eqns. 5 and 6 are obtained from the face and natural pairs that are corrupted by the same degradation process. Therefore, switching $\Omega^{Syn}_{f}$ and $\Omega^{Syn}_{n}$ should have the same LQ results. Thus the degradation-consistent loss is suggested as:

where $\ell_{\textit{mse}}$ is the MSE loss defined in Eqn. 8. With the constraints on the degradation representation $\Omega^{\textit{Syn}}_{f}$ ($\Omega^{\textit{Syn}}_{n}$) that is extracted from face (natural) images and performed on natural (face) ones, we can further optimize the disentanglement learning, and benefit the training process of the SynNet.

To sum up, the final learning objective is formulated as:

where $\lambda_{\textit{disen}}$, $\lambda_{\textit{mse}}$, $\lambda_{\textit{real}}$ and $\lambda_{\textit{cons}}$ are set to 5, 1, 0.1, and 2, respectively.

We collect real-world LQ face images from Internet, and then adopt GPEN [53] to obtain their pseudo HQ counterparts. These images cover diverse degradation types, from slightness to severeness, oldness to present, etc. Among them, 10,000 images are used for training, 1,000 images for validating, and the remaining 5,000 images for testing. Except these collected images, we also introduce the synthetic LQ face images from FFHQ [22] with common degradation, e.g., blur, noise, JPEG compression, and downsampling operation, etc, to improve the generalization ability. During the inference, we conduct the degradation representation pool $\{\Omega^{ReaL}_{f}\}^{N}$ from these face pairs, which will be sampled to constitute the natural pairs for training our general restoration network (i.e., F2N-ESRGAN).

As for the natural image, we follow BSRGAN [56], and adopt DIV2K [1], Flick2K [30, 45] and FFHQ [22] for training our ReDegNet and F2N-ESRGAN. Adam optimizer [25] with $\beta_{1}=0.5$ and $\beta_{2}=0.999$ is adopted to train ReDegNet and F2N-ESRGAN. The initial learning rate is set to $2\times 10^{-4}$ and will decrease by 0.5 when the MSE loss $\mathcal{L}_{mse}$ on the validation set tends to be stable. All the experiments are implemented on a PC server with 4 Tesla V100 GPUs.

Table 1 lists the quantitative results. One can see that (i) as for these real-world pairs (RealSR Canon and Nikon [3], and DRealSR [50]), although the PSNR and SSIM of Ours is comparable against others, the LPIPS of Ours obtains the best performance, which indicates that our results are more consistent with human perception [58]. The best LPIPS of Ours in turn validates the effectiveness of our ReDegNet in synthesizing the realistic training pairs. (ii) As for the non-reference image quality metric, we collect two groups of real-world images, i.e., RealSRSet proposed in BSRGAN [56], and RealLQSet that contains LQ images collected from Internet and LQ frames extracted from 480P videos. We can see that results of Ours are better than others in most cases, but inferior to RealSR [19] in RealSRSet [56]. We analyze that the RealLQSet (1,000 images) covers more types of common real-world LQ images than RealSRSet (only 20 images), which indicates RealLQSet is more suitable in evaluating the performance of super-resolving the real-world LQ images. The better NIQE of Ours may be attributed to the usage of degradation that are learned from real-world LQ face images.

Except the quantitative metrics, visual comparison appears to be critically important in evaluating the restoration performance, especially for these real-world LQ images. In this paper, we select these real-world LQ images from three types of datasets, i.e., RealSRSet from BSRGAN [56], RealSR dataset [3], and the collected real-world LQ images from our RealLQSet. Visual results of the competing methods are shown in Figure 3. One can see that results of Ours are much clearer than others, not only in the smooth regions (1st row), but also in these with rich and complex textures (2$\sim$4th rows). Due to the limited ability in predicting the kernel and noise from the real-world LQ images, RealSR [19] fails to generate plausible and photo-realistic textures when handling the input with complex degradation. Although BSRGAN [56] and Real-ESRGAN [48] show great generalization due to the wider handcrafted degradation spaces, our F2N-ESRGAN performs comparable against them with these degradation representations that are extracted from the real-world LQ face images, which indicates the effectiveness of our method in synthesizing the photo-realistic LQ images, and in turn contributes to the better restoration performance.

Except the general super-resolution task mentioned above, our method can also fine-tune the restoration model on specific scenarios which have face images in them. Figures 1 and 4 show the specific cases. We can observe that (1) although they are similar to the halftone degradation, the restoration result by the inverse halftone method [43] can not well handle it (see (b)) due to the complex degradation that these real-world LQ images usually suffer from. (2) The general restoration methods, i.e., RealSR [19], BSRGAN [56], and Real-ESRGAN [48] also fail to generate plausible results on these unsynthesizable degradation (see (c$\sim$e)), while Ours perform favorable but still contain obvious artifacts (see (g)). (3) By fine-tuning BSRGAN with the synthetic halftone degradation [13], BSRGAN* has slight improvement in reducing the artifacts, but still can not generate photo-realistic structures (see (f)). (4) By restoring the face region with GPEN [53] and synthesizing the similar degradation types on natural images (see our suppl.), results of Ours^s are much better than others, which indicates the effectiveness of our method in learning the degradation from face images and transferring to natural ones. Compared with BSRGAN and Real-ESRGAN, our method can not only handle the general restoration with limited real degradation, but also fine-tune the model for some specific scenarios that have face images, which are common in the consumer photography and old photos or films.

Firstly, to illustrate the degradation extraction ability of our DegNet, we introduce t-SNE [35] to visualize the degradation representation $\Omega$ for different degradation types. To this end, we generate four groups of LQ face pairs by separately degrading 5,000 HQ test images with Gaussian blurring, downsampling, Gaussian noise and JPEG compression. Then DegNet is utilized to extract their degradation representations. The visualization of each group mapping to 2D space by t-SNE is shown in Figure 5 (a). We can observe that these four groups of degradation representations are embedded into four clusters completely, which indicates that our DegNet can well capture and distinguish the different degradation types.

Secondly, we explore the degradation space of these competing methods. With the 5,000 real-world test LQ and HQ face pairs, we synthesize the LQ images by utilizing the degradation models of RealSR [19], BSRGAN [56], Real-ESRGAN [48] and our ReDegNet on the pseudo HQ images. Among them, the kernel and noise of RealSR are extracted from the real-world images. BSRGAN and Real-ESRGAN are used with their default settings from their released models. As for ours, we randomly sample from the degradation representation pool $\{\Omega^{ReaL}_{f}\}^{N}$ via SynNet to generate the LQ images. Note that $\{\Omega^{ReaL}_{f}\}^{N}$ have no overlap with the 5,000 test pairs. The visualization of the degradation representations of these five groups, i.e., RealSR, BSRGAN, Real-ESRGAN, Ours and the real-world LQ/HQ pairs, is shown in Figure 5 (b). One can see that our synthetic LQ images are more consistent with the real-world LQ ones than the competing methods, indicating the effectiveness of our method in extracting the degradation from real pairs of face images. Albeit BSRGAN and Real-ESRGAN have more diversities due to the random/high orders and handcrafted degradation, only few LQ images are similar to the real-world LQ ones within 5,000 pairs.

Finally, to evaluate the necessities of the disentanglement loss and our pairs of LQ and HQ face images, we design two variants, i.e., Ours $(U)$ by using unpaired data which feeds only the LQ images into DegNet and random HQ images into SynNet, respectively, and adopts the discriminator to distinguish whether the result has the similar degradation with the LQ input or not, and Ours (-$D$) by removing the disentanglement loss. The comparisons on real-world LQ images are shown in Table 1 and Figure 5 (c). We can see that compared with Ours $(U)$, results of Ours are clearer and more photo-realistic, indicating the effectiveness of our supervised manner in predicting the real degradation from the pairs of face images. Besides, by removing the disentanglement loss, results of Ours (-$D$) easily have distorted structures and obvious artifacts, which may be caused by the inaccurate degradation representation that may contain the face related content.

This work is intuitively motivated by the observation that the face region usually shares the similar degradation with the non-face region. However, the background is sometimes out of the depth of field, which easily has the inconsistent degradation with face region, thereby bring limited benefits for the specific restoration. Besides, our general restoration model performs not obviously superior to the competing methods, especially on these camera captured test sets in Table 1. It is better to collect face images under the similar scenarios to augment the degradation space.