Can No-reference features help in Full-reference image quality estimation?

The use of image processing to improve the quality of the content to an acceptable level for human viewers has been of substantial interest to the computer vision community. For achieving this, a significant step is to accurately measure the perceptual quality of the content, and it has been found to be critical in several computer vision applications [4, 22]. The primary weakness of traditional deep learning-based convolutional neural networks is that they usually generate overly smooth images instead of highly textured images because of the unavailability of the proper metrics for training these networks and quantifying these perceptual quality effects. To an extent, Generative Adversarial Networks (GAN) [8] address this problem while learning distributions with pointed peaks and adopting the adversarial training loss for image restoration tasks. Using this, they could generate sharp and visually appealing images compared to models trained without adversarial loss. Nonetheless, such adversarially trained models often obtain lower scores than those trained without adversarial loss evaluated on multiple traditional, well-known metrics such as PSNR and SSIM [27]. Efficiency of these metrics are inferior, specifically for assessing textures and fine details in the generated image [17]. Since the ultimate goal of image enhancement models is to render visually pleasing images for humans and achieve a high Mean Opinion Score (MOS), developing a robust metric for IQA is essential. Neural perceptual image quality metrics can also be utilized as a loss function to train deep networks for image or video restoration.

Perceptual IQA methods are of two different kinds: Full-reference methods where a distorted query image is compared against a reference image and No-reference methods where the quality of a query image is evaluated without any reference image. The common approach used in Full-reference IQA (FR-IQA) task is to extract features using a network from both reference and query images and predict perceptual quality score based on interaction ($L_{2}$ distance, dot product etc.) between reference and query features. However, the contribution of features extracted only from the query image in FR-IQA task is under-explored. In this paper, we propose a multi-branch model for FR-IQA problem which utilizes both Full-reference and No-reference feature extraction. In one Full-reference branch, we compute difference of ImageNet features extracted from query and reference images, whereas learnt features are compared in other Full-reference branch. No-reference branch uses only query image to extract features. Features from these three branches are concatenated and fed to fully connected layers to predict the quality score. Our model surpasses state-of-the-art FR-IQA models on two benchmark datasets.

Image Quality Assessment (IQA) methods are used to assess the quality of pictures that may have been deteriorated during processing such as generation, compression, denoising, and style transfer. IQA algorithms may be classified into no-reference and full-reference approaches based on distinct settings. No reference methods are designed to assess image quality without the need of a reference image  [19, 34]. On the other hand, image quality assessment in full reference setting takes in account of both distorted and reference image.

SSIM [27], MS-SSIM [28], PSNR, and other full-reference approaches are extensively used. They inspired the development of FSIM [32], SR-SIM [31], and GMSD [30]. These hand-crafted approaches compare the feature difference between the deformed picture and the reference image to determine image quality. Deep learning-based full-reference algorithms [21, 2] have recently been shown to outperform hand-crafted systems in terms of model performance.

Bosse et al. [2] implemented an architecture within a unified framework which allows joint learning of local quality and local weights. In other words, the relative importance of local quality to the final quality estimate is learnt. Their proposed architecture can be used in both No-reference (NR) and Full-reference (FR) IQA setting with minor modifications. Zhang et al. [33] showed deep features extracted from internal activations networks trained for high-level classification tasks, represent human perceptual similarity remarkably well, outperforming widely accepted metrics like SSIM, PSNR, FSIM and so on. They improved the performance of their network by calibrating feature outputs from a pre-trained network with their proposed BAPPS dataset. Keyan et al. [5] proposed the first full-reference IQA model with texture resampling tolerance. Their proposed method blends spatial average correlations as texture similarity with feature map correlations as structure similarity.

Prashnani et al. [21] made a large-scale dataset annotated with the probability that humans will prefer one image over another as human perceptual error. Then, using a unique pairwise-learning framework, they further proposed a new metric, PieAPP, estimated by a deep-learning model to predict which deformed image would be preferred over the other which is well correlated with human opinion. Gu et al. [9] contributed Perceptual Image Processing ALgorithms (PIPAL) dataset, which is a large-scale IQA dataset. This dataset, in particular, contains the results of GAN-based Image Restoration methods, which were not included in earlier datasets. They also proposed the Space Warping Difference Network, which incorporates $L_{2}$ pooling layers and Space Warping Difference layers, to improve an IQA network’s performance on GAN-based distortion by explicitly addressing spatial misalignment.

Cheon et al. [3] adapt Vision transformers [7] for perceptual IQA. They have used a pretrained Inception-Resnet-v2 network [25] as feature extraction backbone and transformer encoder-decoder architecture to obtain quality score predidction. Shi et al. [23] proposed Region-Adaptive Deformable Network which uses reference-oriented deformable convolution to improve performance of the network on GAN-based distortion by adaptively accounting spatial misalignment. Their patch-level attention module contributes to enhance the interaction between distinct patch regions that were previously processed separately.

Guo et al. [10] sample random patches from both query and reference images and extract features from different scales from a feature extraction module. A quality score per scale is generated with the help of Feature Fusion and Score Regression modules and these scores are averaged to obtain image-level quality score. Ayyoubzadeh et al. [1] use a Siamese-difference network equipped with spatial and channel attention. They also develop a Surrogate Ranking loss to improve Spearman Rank correlation score. Hammou et al. [11] compute difference of features extracted from different layers of pretrained VGG16 network [24]. These features are fed to an ensemble consisting of XGBoost, LIGHTGBM and CatBoost to predict the quality score.

Given a reference image $I_{r}$ and a distorted query image $I_{q}$, our goal is to predict a quality score $p$ which correlates with the perceived quality of $I_{q}$. Our model consists of three parallel branches: (a) Full-reference pretrained (FRP) branch (b) Full-reference non-pretrained (FRNP) branch and (c) No-reference (NR) branch. For Full-reference branches (FRP and FRNP), both query and reference images are fed as input, whereas in No-reference branch only the query image is fed as input. In each of the branches, we have a convolutional neural network based encoder. In FRP branch, we use encoder from a classifier trained on ImageNet since these features are known to correlate well with perceptual quality [15, 13, 33]. Weights of FRP encoder is kept fixed throughout the training. We don’t use pretrained encoders in FRNP and NR branches to enable learning of discriminative features from the training data.

Full-reference branches focus on extracting features from both query and reference images and compute pixel-wise difference between query and reference features. But when the distortion is severe, computing pixel-wise difference, even in feature space, may not be optimal due to spatial misalignment. Hence, we use a No-reference branch to focus only on features related to the distortion present in the query image.

Let’s denote encoders of FRP branch, FRNP branch and NR branch as $E^{FRP}$, $E^{FRNP}$ and $E^{NR}$ respectively. We describe details about Full-reference and No-reference branches in the following.

Full-reference branches: In a full-reference branch $b$, we extract multi-scale features from corresponding encoder $E^{b}$ for both query and reference images. We obtain features from four different scales, $\phi^{b}_{s}(I_{t})$ where $s\in\{1,2,3,4\}$ and $I_{t}\in\{I_{q},I_{r}\}$. Spatial resolution of features in scale $s$ is $(h/2^{s}\times w/2^{s})$ where $(h,w)$ is resolution of the input images. Then we compute difference features $d^{b}_{s}$ from each scale and concatenate them to obtain $d^{b}$. Hence, $d^{FRP}$ and $d^{FRNP}$ are given by,

where $abs(.)$ is absolute value, $GAP(.)$ is Global Average Pooling layer and $\oplus$ stands for feature concatenation.

No-reference branch: Since reference image is not used as an input to No-reference (NR) branch, we obtain no-reference features from this branch. Similar to Full-reference branches, multi-scale features are extracted followed by Global Average pooling and feature concatenation.

Finally, difference features from full-reference branches and no-reference features are concatenated and passed to three fully-connected layers to predict the quality score. Overview of our model is shown in Figure 1.

In this paper, we have proposed a multi-branch network for image quality assessment in full-reference setting. Full-reference branches computes feature from both query and reference images whereas No-reference branch extracts feature only from the query image. Full reference branches utilize both Imagenet pretrained features as well as learnt features. Our experiments show that addition of no-reference branch indeed improves FR-IQA performance. The proposed model outperforms state-of-the-art algorithms on two benchmark datasets. In future, incorporation of spatial and channel attention [29] can be utilized to reweigh features in different branches and improve quality score predictions.