GMC-IQA: Exploiting Global-correlation and Mean-opinion Consistency for No-reference Image Quality Assessment

Due to the remarkable progress in vision applications, considerable attention has been focused on elevating the performance of IQA. As a pioneer, Kang et al. (2014) design a convolutional neural network (CNN) for IQA to extract image features. Then they extend this work to a multi-task CNN Kang et al. (2015). However, insufficient training samples limit effective learning of CNNs-based models. For this reason, some methods Su et al. (2020); Qin et al. (2023) employ pre-trained networks, such as ResNet He et al. (2016) and vision transformer (ViT) Dosovitskiy et al. (2020), as feature extractors. However, recent research Zhu et al. (2020); Chen et al. (2022) point out that these popular networks pre-trained for high-level tasks are not suitable for IQA. Therefore, some works pre-train models on related pretext tasks, e.g., image restoration Lin and Wang (2018); Ma et al. (2021), quality ranking Liu et al. (2017); Ma et al. (2017), and contrastive learning Zhao et al. (2023); Madhusudana et al. (2022). Some other methods enhance the IQA performance by introducing auxiliary information. For instance, Wang et al. (2023a); Saha et al. (2023) integrate textual information into the IQA. Zhang et al. (2023) explore the relationship among multiple tasks, namely the IQA, scene classification and distortion classification. Additionally, many methods utilize the idea of ensemble learning to aggregate IQA-related knowledge for more effective learning. Ma et al. (2019) collect a set of existing IQA models for annotation. The annotated samples are used for training their model to learn the quality score as well as the uncertainty. Both Wang et al. (2023b) and Zhang et al. (2020) propose a novel multi-dataset training strategy.

Although existing methods have improved IQA performance by addressing various aspects of the model, they ignore to exploit the global correlation consistency in the training process, which makes a gap between training and evaluation objectives.

Distance-based loss and rank-based loss are two representative losses used in IQA models. The distance-based loss aims to minimize the difference between predicted scores and GTs. MAE and MSE are widely used for training the IQA models. When evaluating the performance of IQA models, global correlation consistency metrics, like PLCC and SROCC, are employed. This inconsistency leads to a gap between training and evaluation objectives. To address this problem, Li et al. (2020) propose a Norm loss, which is formulated using a novel normalization approach. The Norm loss is closely connected to PLCC and RMSE. It is performed within a batch during training, while PLCC and SROCC are computed on the entire test dataset during evaluation. This also contributes to inconsistent training and evaluation objectives.

For rank-based loss, the IQA task is viewed as a quality ranking problem. Gao et al. (2013) adopt the cross-entropy loss to compute the discrepancy between predicted quality ranking and GT binary labels for each image pair. Liu et al. (2017) employ the hinge loss to formulate the optimization objective of quality ranking learning. Ma et al. (2017) use learning-to-rank algorithms, like RankNet Burges et al. (2005) and ListNet Cao et al. (2007), to train the IQA model on a number of image pairs. Most of these methods employ image pairs to compute the rank-based loss during training, while the evaluation is conducted on a large sequence of data to compute the ranking consistency. This discrepancy in training and evaluation can lead to sub-optimal performance.

The performance of an IQA model is typically evaluated using two metrics: PLCC and SROCC. Given $n$ images, the two metrics measure the correlation consistency between the predicted scores $\mathbf{P}=[P_{1},P_{2},\cdots,P_{n}]$ and the GT labels $\mathbf{G}=[G_{1},G_{2},\cdots,G_{n}]$. They can be formulated into a unified form as follows:

In PLCC, $\mathbf{X}=\mathbf{P}$ and $\mathbf{Y}=\mathbf{G}$; in SROCC, $\mathbf{X}=R(\mathbf{P}$) and $\mathbf{Y}=R(\mathbf{G})$, where $R(\cdot)$ is a ranking function. In Eq. (1), $\sigma_{\mathbf{X}}$ and $\sigma_{\mathbf{Y}}$ denote the standard deviation of $\mathbf{X}$ and $\mathbf{Y}$, respectively, and $\operatorname{cov}(\mathbf{X},\mathbf{Y})$ is the covariance.

There are two main problems preventing the IQA models to be directly optimized towards the two evaluation metrics: 1) the ranking function $R(\cdot)$ in the SROCC is non-differentiable and 2) due to the limited GPU memory, the two metric scores can only be computed within a batch during training, leading to a significant disparity compared to the results computed on the entire data set. To address these problems, we present a pairwise preference-based rank approximation approach to build a differentiable SROCC variant. Based on the PLCC and the SROCC variant, we propose a kind of global correlation consistency (GCC) loss, which enables IQA models to be optimized for the two evaluation metrics. Furthermore, we devise a queue mechanism to narrow the disparity between metric scores computed on a batch and the entire dataset.

A) Pairwise Preference-based Rank Estimation. To tackle the issue of non-differentiability of the $R(\mathbf{\cdot})$, we propose a novel approach called pairwise preference-based rank estimation, which transforms the sequence ranking into pairwise comparisons. By computing the expectation of pairwise comparison results, we build a differentiable proxy function $R^{\prime}(\cdot)$ to estimate the rank indices, which are statistically close to the true ranking results.

For the input data $\mathbf{P}$, we first normalize it by $S_{P_{i}}=(P_{i}-\bar{P})/\mathit{N}_{\mathbf{P}}$, where $\mathit{N}_{\mathbf{P}}=[\sum_{i=1}^{n}(P_{i}-\bar{P})^{2}]^{\frac{1}{2}}$ and $\bar{P}=1/n\cdot\sum_{i=1}^{n}P_{i}$. Then we obtain a normalized data $\mathbf{S_{P}}=[S_{P_{1}},S_{P_{2}},\cdots,S_{P_{n}}]$. To get the relative comparison of any two elements in $\mathbf{S_{P}}$, we propose a pairwise preference label (PPL) . The PPL $H^{P}_{ij}$ ($\forall i,j\in[1,n]$) is defined as the probability that $S_{p_{i}}$ is greater than $S_{p_{j}}$, which is mathematically expressed as follows:

where $\mathbf{\omega}(\cdot)$ denotes the probability density function of the standard normal distribution, $\mathit{b}_{ij}=S_{P_{i}}-S_{P_{j}}$ and erf$(\cdot)$ is the error function. For any two elements $S_{P_{i}}$, $S_{P_{j}}\in\mathbf{S_{P}}$, there are three possible situations:

Based on Eq. (2), we obtain a probability matrix $\mathbf{H(P)}=(H^{P}_{ij})_{n\times n}$. Based on the PPLs, we propose a ranking proxy function $R^{\prime}(\cdot)$, which estimates the rank indices for the input data $\mathbf{P}$ as follows:

where $\sigma_{P_{i}}$ denotes the estimated index for the element $P_{i}$, and $\mathbb{E}[\cdot]$ denotes the expectation. According to Eq. (3), when $P_{i}(\forall i\in[1,n])$ is higher, the PPL $H^{P}_{ik}(\forall k\in[1,n])$ also tends to increase, which leads to a greater estimated index $\sigma_{P_{i}}$. In this manner, we achieve a statistically differentiable estimation to the non-smooth ranking function. Additionally, the proposed method preserves both monotonicity and correlation characteristics of the original ranking function.

B) Queue Mechanism. To address problem brought by the limited GPU memory, we propose a queue mechanism to approximate the two losses computed on a batch to those computed on the entire data set. The core idea is to maintain an on-the-fly queue for storing the latest samples. This allows us to reuse the predicted scores of the model from the immediate preceding batches. Specifically, the queue always maintains $K$ pairs of data, each of which consists of a predicted score of previous batches and the corresponding GT. The data in the queue are progressively replaced, where the current batch of data are enqueued and the oldest data are dequeued. Removing the oldest data can be beneficial, since these data are the most outdated and thus the least consistent with the newest ones, ensuring a small discrepancy between the queue data and the current batch of data.

The queue length $K$ is an important hyper-parameter. The queue data always represent a sampled subset of the entire dataset. Increasing the value of $K$ can lead to a closer estimation of the PLCC and SROCC computed on the entire dataset. However, it’s important to note that if $K$ is too large, it may cause the queue to retain a number of old data, potentially reducing the accuracy of ranking estimation. In the experiments, we demonstrate the impact of different values of $K$ on the performance of our model.

C) Formulation of GCC loss. Combining the pairwise preference-based rank estimation and the queue mechanism, the proposed GCC loss consists of two loss functions: a PLCC-formulated and a SGCC-formulated global correlation consistency (PGCC and SGCC) loss. They are formulated as follows:

where $\rho$ is defined in Eq. (1), and $R^{\prime}(\cdot)$ is the proposed rank estimation defined in Eq. (3). The PGCC and the SGCC loss measure the monotonicity and the linear correlation between predicted scores and GT labels, respectively.

D) Theoretical Analysis. For the input data $\mathbf{X}=[X_{1},X_{2},...,X_{n}]$, the normalized data $S_{X_{i}}$ has the property $\sum_{i=1}^{n}S_{X_{i}}^{2}=1$. Thus, we can reformulate Eq. (1) to:

Then we can transform $\mathbf{\mathcal{L}_{PGCC}}$ and $\mathbf{\mathcal{L}_{SGCC}}$ to:

Eq. (6) clearly indicates that both $\mathbf{\mathcal{L}_{PGCC}}$ and $\mathbf{\mathcal{L}_{SGCC}}$ can be reformulated in the form of the MSE loss. Compared to the widely used MSE loss which computes the one-by-one distance between predicted scores and GT scores, the two losses introduce global consistency by applying the $\ell_{2}$ normalization to score values and ranking indices, which impose stronger constraints on the global data distribution and ranking correlation, respectively.

In this work, we present a novel network called mean-opinion network (MoNet), which collects various opinions by capturing diverse attention contexts to make a comprehensive decision on the image quality score. Fig. 1 shows the network architecture of the MoNet, which mainly consists of three parts: i) a pre-trained ViT is employed for multi-level feature perception, ii) multi-view attention learning (MAL) modules are proposed for opinion collection, and iii) an image quality score regression module is designed for quality estimation.

A) Multi-level Semantic Perception. Given an image $I\in\mathbb{R}^{H\times W\times 3}$, we firstly crop it into $C$ patches with the size of $S\times S$, where $H$ and $W$ denote the height and width of the image and $C=\frac{H\times W}{S^{2}}$. Then the patches are flattened and fed into a linear projection with the dimension of $D$, producing the embedding feature $\mathbf{E}\in\mathbb{R}^{C\times D}$. Subsequently, the features $\mathbf{E}$ sequentially traverses 12 transformer blocks, resulting in a set of multi-level features. Finally, the outputs of $N$ transformer blocks are selected and used as basic features, denoted as $f_{j}$ ($1\leq j\leq N$).

B) Multi-view Attention Learning Module. The critical part of the MoNet is the multi-view attention learning (MAL) module. The motivation behind it is that individuals often have diverse subjective perceptions and regions of interest when viewing the same image. To this end, we employ multiple MALs to learn attentions from different viewpoints. Each MAL is initialized with different weights and updated independently to encourage diversity and avoid redundant output features. The number of MALs can be flexibly set as a hyper-parameter. We show in our results its effect on the performance of our model. As shown in Fig. 1, the MAL starts from $N$ self-attentions (SAs), each of which is responsible to process a basic feature $\mathbf{f}_{j}$ ($1\leq j\leq N$). The outputs of all the SAs are concatenated, forming a multi-level aggregated feature $\mathbf{F}\in\mathbb{R}^{C\times D\times N}$. Then $\mathbf{F}$ passes through two branches, i.e., a pixel-wise SA branch and a channel-wise SA branch, which apply a SA across spatial and channel dimensions, respectively, to capture complementary non-local contexts and generate multi-view attention maps. In particular, for the channel-wise SA, the feature $\mathbf{F}$ is first reshaped and permuted to convert the size from $C\times D\times N$ to $D\times(C\times N)$. After the SA, the output feature is permuted and reshaped back to the original size $C\times D\times N$. Subsequently, the outputs of the two branches are added and average pooled, generating an opinion feature. The design of the two branches has two key advantages. First, implementing the SA in different dimensions promotes diverse attention learning, yielding complementary information. Second, contextualized long-range relationships are aggregated, benefiting global quality perception.

C) Image Quality Score Regression. Assuming that $M$ opinion features are generated from $M$ MALs employed in the MoNet. To derive a global quality score from the collected opinion features, we utilize an additional MAL. The MAL integrates diverse contextual perspectives, resulting in a comprehensive opinion feature that captures essential information. This feature is then processed through a transformer block, three convolutional layers with kernel sizes of $5\times 5$, $3\times 3$, and $3\times 3$ to reduce the number of channels, followed by two fully connected layers that transform the feature size from 128 to 64 and from 64 to 1. Finally, we obtain a predicted quality score from the MoNet.

Our total loss function combines the two GCC losses and the MSE loss. The two GCC losses are computed based on both the queue data ($\mathbf{P}^{q}$ and $\mathbf{G}^{q}$) and the current batch of data ($\mathbf{P}^{b}$ and $\mathbf{G}^{b}$), constituting a new sequence of data $\mathbf{P}^{a}=[\mathbf{P}^{q},\mathbf{P}^{b}]$ and $\mathbf{Q}^{a}=[\mathbf{Q}^{q},\mathbf{Q}^{b}]$. The MSE loss is only computed on the current batch of data. The total loss is defined as follows:

where $\alpha$, $\beta$ and $\gamma$ are hyper-parameters. In Eq. (7), the GCC losses and the MSE loss enhance the global correlation consistency and mean opinion consistency, respectively. Therefore, the total loss is called GMC loss. When the current batch of data are easy samples, both GCC and MSE losses tend to be small, resulting in a smaller GMC loss; otherwise, the GMC loss becomes larger. Therefore, the GMC loss can pull away easy samples from hard ones, thereby strengthening the training on hard samples. We train the MoNet using the GMC loss, forming our GMC-IQA framework.

We train and evaluate our model on four authentic datasets, namely LIVEC Ghadiyaram and Bovik (2015), BID Ciancio et al. (2010), Koniq Hosu et al. (2020), and SPAQ Fang et al. (2020), which contain 1162, 586, 10073 and 11125 images with realistic distortions, respectively. PLCC and SROCC are used for evaluating the performance of IQA models. Both metrics range from -1 to 1, with higher values indicating better performance of IQA models.

Following the settings in Su et al. (2020); Yang et al. (2022); Qin et al. (2023), we randomly divide each dataset into 80% for training and 20% for testing. For the SPAQ, the shortest side of an image is resized to 512 according to Fang et al. (2020). Let $D$ denote the number of training samples, the queue length is set to 60%$\times D$. The hyper-parameters $\alpha$, $\beta$ and $\gamma$ defined in Eq. (7) are set to 0.5, 0.5 and 1. The pre-trained vit_base_patch8_224 is used as the backbone of the MoNet. We use $N=4$ transformer blocks to extract basic features, namely the 3rd, 6th, 9th and 12th blocks. If not explicitly specified, the default number of the MAL is set to $M=3$. All experiments are conducted 10 times, and the median of the 10 scores are reported as the final score. We use the Adam optimizer with a learning rate of $1\times 10^{-5}$ and a weight decay of $1\times 10^{-5}$. The learning rate is adjusted using the Cosine Annealing for every 50 epochs. We train our model for 100 epochs with a batch size of 11.

The results on the four authentic datasets are shown in Tab. 1. The proposed GMC-IQA outperforms all the compared models on the LIVEC, BID and Koniq. On the SPAQ, GMC-IQA obtains a slightly inferior performance to QPT-ResNet50 Zhao et al. (2023) with a margin of 0.13% in terms of PLCC. It is worth noting that QPT-ResNet50 synthesizes approximately $2\times 10^{14}$ distorted images for pre-training. It exhibits a remarkable superiority over existing methods. Nevertheless, our GMC-IQA achieves better results to QPT-ResNet50 on most datasets without the complicated dataset synthesis for pre-training.