Norm-in-Norm Loss with Faster Convergence and Better Performance for Image Quality Assessment

Image quality assessment (IQA) has received considerable attention (Wang et al., 2004; Mittal et al., 2012; Ye et al., 2012; Kang et al., 2014; Ma et al., 2016; Hosu et al., 2020) and plays a key role in many vision applications, such as compression (Rippel et al., 2019) and super-resolution (Zhang et al., 2019). It can be achieved by subjective study or objective models. Subjective study uses mean opinion score (MOS) to assess image quality. This is considered as the most reliable and accurate way, whereas it is expensive and time-consuming. So the objective models that can automatically predict image quality are in urgent need. In terms of the availability of the reference image, objective IQA models can be divided into three categories: full-reference IQA (Wang et al., 2004; Zhang et al., 2014; Kim and Lee, 2017; Bosse et al., 2018), reduced-reference IQA (Ma et al., 2011; Xu et al., 2015; Bampis et al., 2017; Liu et al., 2018b), and no-reference IQA (Mittal et al., 2012; Kang et al., 2014; Liu et al., 2017; Ren et al., 2018).

Most classic learning-based IQA models are based on mapping the handcrafted features to image quality by support vector regression (SVR) (Mittal et al., 2012). Recently, deep learning-based models, which jointly learn feature representation and quality prediction, show great promise in IQA (Kang et al., 2014; Ren et al., 2018; Lin and Wang, 2018; Hosu et al., 2020). However, these models mostly treat IQA as a general regression problem. And they adopt standard regression loss functions for training, i.e., mean absolute error (MAE) and mean square error (MSE) between the predicted quality scores and the corresponding subjective quality scores. We notice a fact that the IQA models trained using MAE or MSE loss exhibit slow convergence. For example, on a dataset containing only about 10,000 images with a resolution of 664$\times$498, training the model on an NVIDIA GeForce RTX 2080 Ti GPU (11GB) takes more than one day to reach convergence. Since the size of the training dataset becomes larger and larger in the deep learning era, faster convergence is preferable to reduce the training time.

In this work, we tackle the slow convergence problem in the context of IQA. In fact, slow convergence problem is common in machine learning and computer vision, which may be due to the non-smooth loss landscape (Santurkar et al., 2018). To provide faster convergence for the learning process, it is widely-used to do input data normalization or intermediate feature rescaling (Ioffe and Szegedy, 2015; Hoffer et al., 2018). Normalizing the output predictions is rarely recommended. However, it is shown that normalizing the network output can lead to faster convergence of generative networks for image super-resolution (Mullery and Whelan, 2018). Inspired by this work, to achieve fast convergence of IQA model training, we explore normalization in the design of loss functions for IQA.

We propose a class of loss functions, denoted as “Norm-in-Norm”, for training an IQA model with fast convergence. Specifically, the predicted quality scores is firstly subtracted by their mean, and then they are divided by their norm after centralization. Similar normalization is applied to the subjective quality scores. After the normalization, we define the new loss based on the norm of the differences between the normalized values. The new loss normalizes both labels and predictions while label normalization only normalizes labels, and the new loss encourages the IQA model to make linear predictions with respect to (w.r.t.) subjective quality scores. Hence, after training, the linear relationship can be determined by applying least squares regression (LSR) on the whole training set for mapping the predictions to the subjective quality scores. In the testing phase, this learned linear relationship is applied to the model prediction to get the final predicted quality score for a test image.

There are two interesting findings about the new loss. First, we derive that Pearson’s linear correlation coefficient (PLCC)-induced loss (Ma et al., 2018; Liu et al., 2018a) is a special case of the proposed loss, where PLCC is a criterion for benchmarking IQA models. Second, after introducing a variant of the proposed loss, we show its connection to root mean square error (RMSE) — another IQA performance criterion.

Further, we conduct theoretical analysis on the property of the new loss. And it is proved that due to the embedded normalization, the new loss has stronger Lipschitzness and $\beta$-smoothness (Nesterov, 2013), which means the gradients of the loss function is more stable and more predictable. Thus, the gradient-based algorithm for learning the IQA model has a smoother loss landscape. And this is conducive to the faster convergence of the IQA model.

Generally, the proposed “Norm-in-Norm” loss can be applied to any regression problem, including IQA problems. In particular, we pick a challenging real-world IQA problem: quality assessment of in-the-wild images, to verify the effectiveness of the proposed loss. Quality assessment of in-the-wild images has two challenges, i.e., content dependency¹¹1Content dependency means human ratings of image quality depend on image content and distortion complexity. We design a no-reference IQA model based on aggregating and fusing multiple level deep features extracted from the image classification network. Deep features are considered for tackling the content dependency. And multiple level features are used for handling the distortion complexity. The model is trained with the proposed loss. The experiments are conducted on two benchmark datasets, i.e., CLIVE (Ghadiyaram and Bovik, 2016) and KonIQ-10k (Hosu et al., 2020). Figure 1 shows the convergence results on KonIQ-10k. By using the proposed loss, the model only needs to look at the training images once (i.e., in one epoch) to achieve a prediction performance indicated by the grey dash line. It is about 10 times faster than MAE loss and MSE loss. This verifies that the proposed loss facilitates faster convergence for training IQA models. We claim that it mainly benefits from the embedded normalization in our loss. Besides the faster convergence, we notice that the proposed loss also leads to a better prediction performance than MAE loss and MSE loss.

To sum up, our main contribution is that we propose a class of normalization-embedded loss functions in the context of IQA. The new loss is shown to have some connections to PLCC and RMSE. And our theoretical analysis proves that the embedded normalization in the proposed loss can facilitate faster convergence. For the quality assessment of in-the-wild images, it is experimentally verified that the new loss can provide both better prediction performance and faster convergence than MAE loss and MSE loss. What’s more, the proposed model outperforms state-of-the-art models on this challenging IQA problem.

In this section, we explore the normalization in the design of loss functions, and propose a class of “Norm-in-Norm” loss functions for training IQA models with fast convergence. The idea is to apply normalization for the predicted quality scores and the subjective quality scores respectively using their own statistics when computing the loss.

Assume we have $N$ images on the training batch. For the $i$-th image $I_{i}$, we denote the predicted quality by an objective IQA model $F(\cdot;\bm{\theta})$ as $\hat{Q}_{i}$ and its subjective quality score (i.e., MOS) as $Q_{i}$.

We introduce the concepts of Lipschitzness and $\beta$-smoothness (Nesterov, 2013). For a univariate function $f$, $f$ is $L$-Lipschitz if $|f(x_{1})-f(x_{2})|\leq L|x_{1}-x_{2}|,\forall x_{1},x_{2}$. And $f$ is $\beta$-smooth if its gradient is $\beta$-Lipschitz, i.e., $\beta$-smoothness corresponds to the Lipschitzness of the gradient. The proposed loss $l$ is a differentiate multivariate function w.r.t. the model predictions $\hat{\mathbf{Q}}$. Its Lipschitzness is indicated by its gradient magnitude and its $\beta$-smoothness in the gradient direction is indicated by the quadratic form of its Hessian matrix. Smaller gradient magnitude and quadratic form of its Hessian correspond to better Lipschitzness and $\beta$-smoothness, respectively.

In this section, we theoretically prove that, when $q$ equals 2, the embedded normalization can improve the Lipschitzness and the $\beta$-smoothness of the proposed loss $l$. That is, the gradient magnitude and the quadratic form of its Hessian are reduced by the embedded normalization, which indicates the gradients of the proposed loss is more stable and more predictable. This ensures that the gradient-based algorithm has a smoother loss landscape, so the training of the IQA model gets more robust and the model converges faster.

First, we prove a theorem about Lipschitzness. When $q=2$, based on Eqn. (8), $\hat{\mathbf{R}}=\hat{\mathbf{S}},\mu_{\hat{R}}=\frac{1}{N}\langle\mathbf{1},\hat{\mathbf{S}}\rangle=0$. Denote ${\mathbf{g}}=\frac{\partial l}{\partial\hat{\mathbf{S}}},{\mathbf{g}}_{n}=\frac{\partial l}{\partial\hat{\mathbf{Q}}}$. Thus Eqn. (9) becomes

We show that the gradient magnitude of the new loss satisfies Eqn. (20) in Theorem 3.1 (See proof in Supplemental Materials C).

The left side of Eqn. (20), $\left\|{\mathbf{g}}_{n}\right\|^{2}$, indicates the Lipschitzness with the embedded normalization. Without the normalization, the Lipschitzness is indicated by $\left\|{\mathbf{g}}\right\|^{2}$. From Eqn. (20), we derive that the Lipschitzness of the proposed loss is improved whenever the sum of the gradient ${\mathbf{g}}$ deviates from 0 or the gradient ${\mathbf{g}}$ correlates the normalized predicted quality scores $\hat{\mathbf{S}}$. In addition, ${\hat{b}}$ is larger than 1 in practice (see Supplemental Materials B), which also contributes to the improvement of the Lipschitzness. So from Theorem 3.1, we can infer that the embedded normalization improves the Lipschitzness.

Next, we prove a theorem about $\beta$-smoothness. Denote $\mathbf{H}=\frac{\partial^{2}l}{\partial\hat{\mathbf{S}}^{2}},\mathbf{H}_{n}=\frac{\partial^{2}l}{\partial\hat{\mathbf{Q}}^{2}}$. We then prove that the quadratic form of the loss Hessian in the gradient direction satisfies Eqn. (3.2) in Theorem 3.2 (The proof is provided in the Supplemental Materials D).

The left side of Eqn. (3.2), ${\mathbf{g}}_{n}^{T}\mathbf{H}_{n}{\mathbf{g}}_{n}$, indicates the $\beta$-smoothness with the embedded normalization. Without the normalization, the $\beta$-smoothness is indicated by ${\mathbf{g}}^{T}\mathbf{H}{\mathbf{g}}$. From Eqn. (3.2), we can see that the $\beta$-smoothness is improved when the normalized subjective quality scores ${\mathbf{S}}$ and normalized predicted quality scores $\hat{\mathbf{S}}$ are not linearly correlated ($\langle{\mathbf{S}},\hat{\mathbf{S}}\rangle<1$). And it is further improved if the gradient ${\mathbf{g}}$ and the normalized predicted quality scores $\hat{\mathbf{S}}$ are also not linearly correlated ($\langle{\mathbf{g}},\hat{\mathbf{S}}\rangle<\|{\mathbf{g}}\|$). In addition, ${\hat{b}}>1$ also contributes to the improvement of the $\beta$-smoothness. So from Theorem 3.2, we can infer that the embedded normalization improves the $\beta$-smoothness.

Besides theoretical analysis, we also conduct an experimental verification. Quality assessment of in-the-wild images is important for many real-world applications, but few attention is paid to it. In-the-wild images contain lots of unique contents and complex distortions. The greatest challenge for this problem is how to handle the content dependency and distortion complexity. In this section, we pick this challenging problem for verifying the effectiveness of our “Norm-in-Norm” loss in comparison with MAE loss and MSE loss. In addition, we intend to provide state-of-the-art prediction performance on this challenging problem.

Realizing that most IQA methods train models using MAE or MSE loss with empirically slow convergence, we address this problem by designing a class of loss functions with normalization. The proposed loss includes Pearson correlation-induced loss as a special case. And a special case of the loss variant is connected with RMSE between the linearly mapped predictions and the subjective ratings. We theoretically prove that the embedded normalization helps to improve the smoothness of the loss landscape. Besides, experimental verification of the proposed loss is conducted on the quality assessment of in-the-wild images. Results on two benchmark datasets (KonIQ-10k and CLIVE) show that the model converges faster and better with the proposed loss than that with MAE loss and MSE loss.

The proposed loss is invariant to the scale of subjective ratings. Facilitated with the new loss, we can easily mix multiple datasets with different scales of subjective ratings for training a universal IQA model. In the future study, we intend to verify the effectiveness of this new loss in the universal image and video quality assessment problems. Besides, it is a future direction on how to optimally choose the hyper-parameters $p$ and $q$ in the class of proposed loss functions for a specific regression task.