Image Super-Resolution Quality Assessment: Structural Fidelity Versus Statistical Naturalness

Single image super-resolution (SISR) aims to recover a high-resolution (HR) image given a single low-resolution (LR) image. SISR plays a significant role in a wide range of applications, from satellite imaging, web browsing, to video surveillance [1]. During the past decades, numerous SISR algorithms have been proposed, including interpolation-based [2, 3], dictionary-based [4, 5, 6], and deep learning-based methods [7, 8, 9, 10, 11, 12]. The visual appearance and quality of SISR generated images vary dramatically when different SISR approaches are employed. Nevertheless, there is still no consensus on how the quality of SISR created images should be assessed. This is critically important because image quality assessment (IQA) methods not only help evaluate and compare SISR algorithms, but also guide the development of future SISR methodologies.

In general, the most reliable quality assessment method is human subjective evaluation [13, 14, 15]. But subjective tests are usually expensive, time-consuming and hard to be integrated into SISR optimization frameworks. Therefore, it is highly desirable to design effective objective IQA models for SISR generated images. Depending on the availability of the original pristine image, full-reference (FR) IQA [16, 17, 18, 19, 20, 21, 22] and no-reference (NR) IQA approaches [23, 24, 25, 26, 27] may be applied. Additionally, since many SISR algorithms produce blurry reconstructed images, image sharpness assessment (ISA) or blur measures [28, 29, 30] may also be employed.

Despite the success in other IQA applications, existing FR-IQA, NR-IQA and ISA methods often fall short when evaluating the quality of SISR generated images. The gap is not only on the accuracy in predicting subjective scores, but also on effectively interpreting the nature of key quality degradation trends in SISR images. An example is given in Fig. 1, where traditional SISR methods such as VDSR [7] are highly effective at achieving high signal fidelity in terms of signal-to-noise ratio or structural similarity [16] when compared to the original images, but the resulting images often look artificial. On the other hand, recently proposed generative adversarial network (GAN) based approaches such as SRGAN [12] are impressive at producing natural-looking reconstructed images, but their signal fidelity measures are significantly lower. These observations motivate us to look at the problem in a two-dimensional (2D) space of structural fidelity versus statistical naturalness, as demonstrated at the bottom part of Fig. 1.

Multi-scale image decomposition such as Laplace and wavelet transforms have been shown to be effective at characterizing not only local perceptual degradation, but also the statistical naturalness of images. Therefore, we apply a multi-scale image transform and construct local structural fidelity and global statistical naturalness measures both in the transform domain. Given the original HR image ${\bf x}$, and the SISR generated test image ${\bf y}$, inspired by the success of MS-SSIM [17], we define subband patch level structural fidelity measure as:

where $x$ and $y$ denote the patches extracted from the $k$-th subband from ${\bf x}$ and ${\bf y}$, respectively, ${{\sigma}_{x}}$ and ${{\sigma}_{y}}$ are their standard deviations, ${{\sigma}_{xy}}$ represents the covariance between $x$ and $y$, and $C$ is a positive stabilizing constant. The scale level structural fidelity measure is then computed by spatial pooling:

where $M$ denotes the number of local patches in the subband. Finally, we fuse across scales to obtain the overall structural fidelity between ${\bf x}$ and ${\bf y}$:

where $K$ is the total number of scales/subbands, and ${{\alpha}_{k}}$ is the weight assigned to the $k$-th scale as in [17]. Furthermore, natural texture-rich content tends to have higher entropy in the transform domain [31]. Thus, we use the global entropy of transform coefficients as a statistical naturalness measure:

where $P({c_{y}})$ denotes the probability of subband coefficients of the test image ${\bf y}$ and may be approximated with histograms.

Although the proposed pair of (SF, SN) measure is rather simple, it offers a meaningful 2D illustration of the behaviors of SISR algorithms. Fig. 1 shows three original HR images with their corresponding SISR images generated by VDSR [7] and SRGAN [12]. The (SF vs. SN) plot clearly indicates the relative advantage of VDSR over SRGAN on the SF measure, and conversely the advantage of SRGAN over VDSR on the SN measure. The pattern is consistent over all three content, as indicated by the pairs of points (A1, A2), (B1, B2) and (C1, C2). Fig. 2 shows images generated by different SISR algorithms applied to LR images of different sizes and enhanced by different scaling factors. The (SF vs. SN) plots offer a platform to examine the behaviors of different SISR algorithms across scaling factors. It can be observed that the the general trend of any SISR algorithm is that both SF and SN measures drop with increasing scaling factors. However, the speed of change may vary depending on the algorithm and possibly the image content. For example, the DRRN [10] method appears to be much more sensitive to scaling factor change than VDSR [7] and DCSCN [9] for the right image.

In practice, it is often desirable to obtain a single quality score indicating the overall quality of SISR generated images. This can be achieved by collapsing the proposed 2D measure into a scalar quality prediction, e.g., by a linear combination:

where the weighting factors $w_{F}$ and $w_{N}$ adjust the relative importance of the two measures, and are set empirically at 0.9 and 0.1, respectively, in the current implementation. We name $Q$ the SFSN measure, which creates straight lines as level sets in the 2D space, as shown in the bottom plots of Figs. 1 and 2. This scalar quality prediction can then be validated by comparing against subject-ratings of SISR generated images.

We validate the proposed fused SFSN quality prediction method on three public SISR IQA databases, including WIND [13], CVIU [14], and QADS [15]. The WIND database considers 8 interpolation algorithms with scaling factors of 2, 4, and 8. It contains 312 SISR images corresponding to 13 reference images. The CVIU database consists of 30 reference HR images and 1,620 SISR generated images created by 9 algorithms with 6 pairs of (scaling factor, kernel width) combinations, where a larger scaling factor corresponds to a larger blur kernel width. The QADS database contains 20 original HR images and 980 images generated by 21 SISR algorithms, including 4 interpolation-based, 11 dictionary-based, and 6 deep learning (DL) based models applied for upsampling factors of 2, 3, and 4. In all three databases, each SISR generated image is subject-rated and annotated by a mean opinion score (MOS). We compare the proposed method with 8 FR-IQA, 5 NR-IQA, and 4 ISA models. The Spearman Rank-order Correlation Coefficient (SRCC) comparison results are reported in Table I, where the best performances are highlighted in bold. Other common evaluation criteria [32] produce similar results but are not included due to space limit. Despite its simple and straightforward construction, SFSN achieves surprisingly competitive performance against state-of-the-art IQA and ISA models.

Since different categories of SISR methods often generate drastically different appearance of the reconstructed images, it is intriguing to investigate how IQA methods perform for different SISR categories. The results on the QADS [15] database are reported in Table II, where the proposed method delivers superior performance in each of the interpolation-based, dictionary-based, and DL-based SISR categories, as well as when all three categories are evaluated together. Ablation test has also been conducted to assess the performance when only the SF or SN measure is employed. The results are shown in Tables I and II. It appears that both SF and SN measures make important contributions, but the best performance is achieved by the SFSN model that combines both of them.

In this work, we opt to a 2D approach to assess the quality of SISR generated images as a tradeoff between structural fidelity and statistical naturalness. This allows us to better understand the nature of quality degradations and better observe the varying behaviors of different SISR algorithms. We also show that a rather straightforward implementation of a local structural fidelity assessment, a global statistical naturalness measure, and a linear combination of the two, results in an SFSN model that achieves surprisingly high correlations with MOS. In the future, better structural fidelity and statistical naturalness measures, and more sophisticated combination methods may be developed. The 2D assessment idea may also be integrated into novel SISR algorithms, aiming to achieve an optimal balance between the two goals.