Degradation-Guided Meta-Restoration Network for Blind Super-Resolution

As described in E.q. (1), an HR image is sequentially degraded by blur, down-sampling and noise. Inspired by this process, we attach the MNM to the head of the network and the MBM to the end of the network, which inversely handles the degradation.

Meta-denoise module (MNM). There is a meta-layer at the beginning of the meta-denoise module. The guidance for the meta-layer in MNM is the estimated noise map $N_{est}$ from our degradation extractor instead of a noise level $\sigma$ which is commonly used in  [31, 35]. Considering the widely-used noise type, Additive White Gaussian Noise (AWGN), the probability density for each pixel $x$ is denoted as:

where $\sigma_{n}$ is the noise level parameter and $p(x)$ represents the probability density distribution. From this formulation, we can see that the parameter $\sigma_{n}$ only influences the probability density distribution of the noise in pixel $x$. However, it lacks the ability to express the exact noise value in that pixel. Therefore, we choose the noise map which provides more dense information in our model setting. Ideally, the estimated noise map is the additive noise of the input LR image and the network parameters need to learn an inverse operation. In such a case, the noise map can be regarded as the biases for each spatial position. A concatenation operation followed by a convolution layer are adopted to implement MNM, which can be formulated as:

where “Conv” and “Concat” represent the convolution layer and the concatenation operation, respectively. $I^{LR}$ is the input degraded LR image and $N_{est}$ indicates the estimated noise map from our degradation extractor. $F$ is the output feature maps of MNM. When the convolution kernel size is set to 1, each value in the noise map just influences the corresponding position’s bias in this convolution process. In other words, the meta-layer in MNM generates meta-biases according to the estimated noise map for the network to better handle noise removal.

Deep feature learning and up-sampling. Between MNM and MBM, there is a network for deep feature learning and up-sampling. We adopt the residual group structure proposed in SOTA SISR method RCAN [37]. Each residual group is composed of several sequential residual channel attention blocks (RCABs) with a long skip connection. The residual path in each RCAB consists of “convolution + ReLU + convolution + channel attention”. Finally a convolution layer and a pixelshuffle layer enlarge the feature resolution according to the magnification scale factor.

Meta-deblur module (MBM). In general, a Gaussian blur process can be formulated as a convolution process:

where $X$ and $V$ are input features and blurred features, respectively. $K$ is the blur kernel with the size of $k\times k$ and $\otimes$ represents the convolution operation. Since convolution is a operation on local area, so each pixel value in $V$ is calculated from a local patch with the size of $k\times k$ in $X$. Considering deblur as an inverse process of the above convolution, each piexel value in $X$ is also related to a local patch with the size of $k\times k$ in $V$, which can be also expressed by a convolution process.

In our blind SR network, MBM is to process such an inverse process. To further enhance the expression capability of the module, we make the convolution kernels spatially variant and each kernel can be regarded as a meta-weight which is generated from the meta-layer in MBM. The estimated blur kernel $K_{est}$ serves as a guidance of this meta-layer because the blur kernel influences the deblur process.

Therefore, the target of the meta-layer in MBM is to generate proper meta-weights for this dynamic convolution process. Specifically, the estimated blur kernel will be firstly dimension-reduced to $C_{k}$ by a fully connected layer. This fully connected layer is initialized by the PCA matrix [35] which records the principle information estimated from a series of random sampled blur kernels. Then we repeatedly stretch such $1\times 1\times C_{k}$ tensor to the features with $H\times W\times C_{k}$ where $H$ and $W$ indicate the feature resolution. The stretched features are concatenated with the network features to generate the meta-weights of this dynamic convolution layer. The meta-weights have the shape $k^{2}\times H\times W$. Each spatial position with size $k^{2}\times 1\times 1$ represents a convolution kernel in that position. In such a design, the meta-layer in MBM predicts dynamic convolution parameters according to the estimated blur kernel $K_{est}$ for the network to address blur recovery.

It is essential to extract accurate degradation for blind SR task, since degradation mismatch will produce unsatisfactory results [9]. The goal of the degradation extractor is to estimate accurate degradation which can provide solid guidance to the meta-restoration modules (MNM and MBM).

The structure of the degradation extractor is shown in Figure 3. One convolution layer followed by two residual blocks are firstly used to extract features of the input LR image. Then there are two branches to extract noise maps and blur kernels respectively. The upper branch estimates the noise map of the input LR image by an additional convolution layer, while the lower branch extracts blur kernel. Two convolution layers and a global average pooling layer are used in the end of the module. The resulting features are reshaped to the size of the blur kernel. We use a softmax layer at the end of the blur branch to ensure there is no value shift before and after blur degradation. To accurately estimate blur kernels and noise maps, we introduce a degradation reconstruction loss and a degradation consistency loss. These two losses are discussed in Section 3.3.

There are three types of loss functions in our model. The overall loss is denoted as:

where $\mathcal{L}_{RE}$ represents reconstruction loss, which is:

where ($C,H,W$) is the size of the HR image. We utilize $L_{1}$ loss which has been demonstrated to produce sharper results compared to $L_{2}$ loss.

The last two losses are shown in Figure 3. $N_{est}$ and $K_{est}$ are the estimated noise map and blur kernel, while $N_{GT}$ and $K_{GT}$ represent the ground truth noise map and the ground truth blur kernel, respectively. The degradation reconstruction loss can be interpreted as:

The degradation reconstruction loss directly constraints the estimated noise map and blur kernel closer to the accurate ones, which provides a direct supervision. To further enhance the accuracy of degradation estimation, we propose a novel degradation consistency loss. We degrade the original HR image by the estimated degradation to get a simulated degraded LR image $I^{LR}_{sim}$, then we also use the degradation extractor to estimate the blur kernel $K_{sim}$ and noise map $N_{sim}$ from such simulated LR image. The degradation consistency loss can be described as:

where the first term of degradation consistency loss aims to constraint the consistency between the input LR image and the simulated degraded LR image. Similar $I^{LR}_{sim}$ and $I^{LR}_{est}$ indicates the accuracy of the degradation estimation to some extent. Similarly, the second and third terms in degradation consistency loss constraint the consistency of the estimated noise map and blur kernel, which enhance the accuracy of the degradation estimation.

Training setups. To synthesize degraded images for training, we use isotropic blur kernels, bicubic down-sampling and additive white Gaussian noise following the common settings used in previous works [31, 35]. Specifically, the blur kernel size is set to $15\times 15$, and the kernel width is randomly and uniformly sampled from the range of [0.2, 3.0]. The noise level $\sigma$ varies in the range of [0, 75]. During training, we augment images by random horizontal flipping and rotating $90^{\circ}$, $180^{\circ}$ and $270^{\circ}$. Each mini-batch contains 32 LR patches with size $48\times 48$.

We set the channel number of residual blocks in the degradation extractor as 64 and the kernel size of all the convolution layers as $3\times 3$. There are 5 residual groups with each containing 20 RCABs in our full model. The global average pooling is used at the end of the blur branch. The weight coefficients for $\mathcal{L}_{RE}$, $\mathcal{L}_{DR}$ and $\mathcal{L}_{DC}$ are 1, 10 and 1, respectively. Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and $\epsilon=1\times 10^{-8}$ is used with initial learning rate of $1\times 10^{-4}$. We train the model for $5\times 10^{5}$ iterations and the learning rate is halved every $2\times 10^{5}$ iterations.

Datasets and Metrics. For fair comparisons, we use DF2K [1, 28] as training set following the common settings used in previous works [9, 31]. All models are evaluated on both standard benchmarks (i.e., Set5 [3], Set14 [33] and B100 [20]) and real-world cases. Specifically, the real-world cases we used for comparisons include the commonly-used real image Flower [15] and the test set of “NTIRE 2020 Real World Super-Resolution” challenge [19]. The test set of the challenge contains 100 unknown-degraded test images without ground truth.

We report quantitative results in terms of PSNR and SSIM metrics, which are calculated on Y channel of YCbCr space. Since the ground truths of the degraded images of real cases are unavailable, we conduct qualitative evaluations and a user study for fair comparisons.

In this section, we compare our model with SOTA blind SR methods, i.e., ZSSR [25], IKC [9] and SFM [7] on standard benchmarks. ZSSR is the SOTA method to learn internal distribution of the LR image. IKC is the SOTA blind SR method and SFM achieves performance improvement based on IKC backbone. For IKC and SFM which assume a fixed noise level, we additionally train IKC-vn and SFM-vn models with variant noise training for more fair comparison. We attach all the websites of the implementation in the footnote¹¹1Implementation of the methods in blind SR comparison:
ZSSR [25]: https://github.com/assafshocher/ZSSR
IKC [9]: https://github.com/yuanjunchai/IKC
SFM [7]: https://github.com/majedelhelou/SFM
.. For fair comparisons, we train IKC and SFM with $15\times 15$ blur kernels on DF2K dataset. We use IKC as the backbone of SFM and the percent rate of training for SFM is 50%. We sample isotropic Gaussian blur kernels from {0.2, 1.3, 2.6} and AWGN levels from {15, 50}.

Quantitative Comparisons. In Table 1, ZSSR which learns the internal distribution of the LR image performs badly since the noise and blur degradation in LR images will make its performance drop significantly. For IKC and SFM which have fixed noise level assumption zero, the performance drops significantly when noise level becomes larger. Even with variant noise training, the performance of SOTA methods is increased but still limited since they do not have specially-designed modules to handle variant noises. Among these blind SR methods, our DMSR can achieve the best performance on all benchmarks in different degradation situations.

To further validate the superior performance of our model, we also conduct experiments in the noise-free situation to meet the assumption of IKC and SFM, where IKC and SFM have significantly better performance. To get the noise-free version of our model “DMSR-nf”, we fine-tune the DMSR where we fix the noise level to 0. We train another $1\times 10^{5}$ iterations with learning rate $1\times 10^{-4}$. As shown in the bottom row in Table 1, even in this narrow range of degradations, our model can still achieve the best performance. This further verifies the real-world super-resolution capability of our model.

Qualitative Comparisons. We also show visual comparison results among different blind SR methods in Figure 4, where degradation parameters are set as isotropic blur kernel 2.6, noise level $\sigma$ 15 with scale factor $\times$4. ZSSR learns the noise distribution of the LR image so its SR results obviously contain noise. IKC and SFM fails to handle such degraded images beyond their degradation assumption. With variant noise training, IKC-vn and SFM-vn can generate better SR results, but not clear as ours. Our DMSR model can handle both noise and blur degradation well and achieves the best visual performance.

In this section, we further evaluate different methods on real cases. We compare our DMSR model with different kinds of SR methods to show the overall practical application ability. These methods include two SOTA blind SR methods with variant noise training IKC [9]-vn and SFM [7]-vn. We also adopt a SOTA SISR method RCAN [37] and a SOTA non-blind SR method UDVD [31] to conduct comparison. For the non-blind SR method UDVD, manual grid search on degradation parameters are usually performed and the best result is chosen. However, such a design is time-consuming on real-world application and not fair for other methods since they manually choose the best result from all the generated results. So for fair comparison, we draw a degradation parameter window which contains different degradation types applied on one image. It has 24 cases of degradations with 6 different noise levels $\sigma_{n}$ and 4 different blur kernels $\sigma_{k}$ for each scale factor 2, 3, 4, where $\times$4 is shown in Supplementary. During inference, we first manually choose the most similar degraded image in the window and adopt its parameters as the input for UDVD. Such a design can save more real-world inference time and is more fair for comparison.

Performance on real images. We first conduct evaluation on the real image Flowers [15]. Since there is no ground truth HR image for this image, we only show the visual comparison results in Figure 5. The degradation parameters [$\sigma_{k}$, $\sigma_{n}$] of UDVD is selected as [2.1, 60] for this image. As we can see, our DMSR model can achieve the best visual performance among these methods. The visual quality of RCAN is severely influenced by the noise degradation. IKC-vn and SFM-vn fail to remove the influences of the noise. Even with manually chosen degraded parameters, the SOTA non-blind SR method UDVD cannot generate textures as natural as ours and there are some artifacts in their result.

Performance on real-world image challenge. In addition to the real image, we also run our model on the test set of “NTIRE 2020 Real World Super-Resolution” challenge [19]. There are 100 unknown-degraded test images in Track 1 and we use these images for evaluation. The degradation parameters of UDVD for this test set are set to [1.2, 15]. Visual comparison is shown in Figure 6. As shown in this figure, our DMSR model can generate the results with clearer and more realistic textures. To further validate the superiority of our model, we conduct a user study on this challenge test set where our DMSR is compared with RCAN, IKC-vn, SFM-vn and UDVD. We collect 3,200 votes from 16 subjects, where each subject is invited to compare our model with two other methods. Therefore, each of the four comparison combination is evaluated by 8 subjects. In each comparison process, the users are provided with two images, including a DMSR result and another method’s result. Users are asked to select the one with better visual quality. The user study results are shown in Figure 7, where the values on Y-axis indicate the percentage of users that prefer our DMSR model over other methods. As we can see, DMSR significantly outperforms SOTA SISR method with over 97% of users voting for ours. For SOTA blind and non-blind SR methods IKC-vn, SFM-vn and UDVD, our model still has over 88% probability of winning. Such results validate the favorable visual quality of our DMSR model. This demonstrates that the meta-restoration modules in our model can generate proper network parameters on-the-fly for different real-world degradation situations.

Degradation Extractor. In this part, we will verify the effectiveness of the proposed DR loss and DC loss. As shown in Table 2, without all losses in the degradation extractor, PSNR / SSIM performance are 24.87 / 0.7194. When adding DR loss to directly supervise degradation estimation, the performance increases to 24.92 / 0.7220. After applying DC loss on the LR image and degradation (blur kernel and noise map), the final performance is further increased to 24.98 / 0.7256. Such an ablation demonstrates the effectiveness of the losses in our degradation extractor. In addition, Figure 8 shows the blur kernel estimation results with and without the degradation consistency loss. With DC loss, the estimated blur kernel is more accurate.

Meta-restoration Modules. We also conduct ablation experiments on the two types of meta-restoration modules, MNM and MBM. The ablation results can be viewed in Table 3. For meta-denoise module, we first verify that the noise map is superior to the noise level number $\sigma$. We run a comparison model in which the degradation extractor predicts the noise level $\sigma$ and this number will be spatially repeated as the “noise map” to be used in MNM. From the table, we can see that utilizing the noise map will bring about 0.1 PSNR performance improvement, which demonstrates the superiority of the noise map. For meta-deblur module, when we add MBM into the model, the performance will be further increased. Such ablation experiments verifies the effectiveness of our two types of meta-restoration modules.

The number of MBM. We also analyze the model performance with different numbers of MBMs. As shown in Table 4, we conduct experiments on one, two and three MBMs. After one MBM equipped, adding more MBMs can not bring obvious performance increase. This is because multiple MBMs is equal to one MBM by its linear nature. Considering additional memory cost, our final model contains only one MBM as the default setting.