Deep Decomposition and Bilinear Pooling Network for Blind Night-Time Image Quality Evaluation

In the literature of traditional blind image quality assessment, natural scene statistics (NSS) and human visual system (HVS) are two main cues for designing objective BIQA models. As for NSS-based frameworks, Moorthy et al. [12] proposed the Distortion Identification-based Image Verity and INtegrity Evaluation (DIIVINE) index to evaluate perceptual image quality in a no-reference manner, which is composed of distortion identification and NSS-based quality regression. Likewise, the CurveletQA [11] also performs within a two-stage framework containing distortion classification and quality assessment. Different from DIIVINE, the quality assessment of CurveletQA is based on NSS features in curvelet domain. Except for the two-stage frameworks, other NSS-based BIQA algorithms have been developed. For example, Mittal et al. [10] presented the blind/referenceless image spatial quality evaluator (BRISQUE), which is operated in the spatial domain. Moreover, the BLIINDS-II [9] was proposed by exploiting the NSS model of discrete cosine transform (DCT) coefficients. In addition, some opinion-unaware BIQA methods based on NSS, i.e. so-called “completely blind” models, such as Natural Image Quality Evaluator (NIQE) [20] and ILNIQE [19], have shown competitive performance with the help of massive natural images.

Beyond the NSS features, many other statistical factors have been considered by researchers. In the family of two-stage framework, Liu et al. [17] proposed the Spatial-Spectral Entropy-based Quality (SSEQ) index, where local spatial and spectral entropy features are used to predict perceptual image quality. The GM-LOG method [15] extracts the joint statistics of local contrast features to assess image quality, including gradient magnitude and Laplacian of Gaussian response. Apart from statistical structural features, lumninance histogram is used in the NRSL model [13]. The NSS features are combined with contrast, sharpness, brightness and colorfulness, together forming the BIQME framework [18].

For the HVS-based objective BIQA methods, some HVS-inspired features are applied to estimate perceptual image quality. Among these methods, Gu et al. [14] proposed the No-reference Free Energy-based Robust Metric (NFERM) on the basis of free energy principle. In [21], Li et al. used contrast masking to design the BIQA model based on structural degradation. Besides, according to the similar concept regarding the HVS properties, they proposed the GWH-GLBP by computing the gradient-weighted histogram of local binary pattern [16].

However, the above-mentioned conventional BIQA methods generally need to design elaborate handcrafted features with the pre-defined NSS or HVS mechanisms. Thus, resorting to data-driven methods based on deep learning is a promising alternative.

Recently, deep learning has achieved great success in the field of blind image quality assessment. These methods can be typically divided into two categories, consisting of those using pre-trained deep features and end-to-end learning ones. For the first category, Wu et al. [29] proposed the HFD-BIQA that integrates deep semantic features from ResNet [30] into local structure features. Moreover, a Network in Network (NIN) model [31] pre-trained on ImageNet [32] was utilized to make image quality prediction. For the second category, Kang et al. [33] proposed a relatively shallow convolutional neural network (CNN) structure for BIQA. Each patch is assigned a subjective quality of the corresponding image as the ground-truth targets for training. In this way, the visual quality of whole image is then calculated by averaging predicted patch quality values. Furthermore, a BIQA model was developed based on shearlet transform and stacked auto-encoders [34]. In [35], the RankIQA was designed to synthesize masses of ranked images for training a Siamese network. Ma et al. [36] proposed an end-to-end optimized deep neural Network for BIQA. Additionally, Bosse et al. [37] presented the end-to-end WaDIQaM that can blindly learn perceptual image quality. The deep bilinear convolutional neural network called DBCNN was proposed to bilinearly pool the feature representations to a single quality score [38].

Although the deep learning-based BIQA models can deliver good performance, they are not suitable for evaluating the perceptual quality of NTIs. This is mainly because these models usually neglect the specific characteristics of NTIs, e.g. reduced visibility, low contrast, additive noises, invisible details, and color distortions.

In real-world applications, people may encounter many kinds of poor imaging environments, e.g. hazy, rainy, underwater, and so on. In such poor conditions, capturing images with high-quality is quite challenging and thus addressing the blind quality assessment issue is urgently needed.

In the quality evaluation of hazy images, Min et al. [39] proposed the haze-removing features, structure-preserving features, and over-enhancement features to construct the objective quality assessment index. They also used synthetic hazy images to build an effective quality assessment model for image dehazing [40]. For image deraining applications, a novel deep quality assessment model to predict the visual quality of real-world derained images was developed, which is achieved by designing a feature embedding network [41]. As for underwater image quality evaluation, Yang et al. [42] proposed to linearly combine chroma, saturation and contrast factors for quantifying the perceptual quality of underwater images. In [43], colorfulness, sharpness and contrast measures were fused to predict the underwater image quality. Most recently, Jiang et al. [44] first constructed a large-scale benchmark dataset for quality evaluation of underwater image enhancement and then proposed a no-reference underwater image quality metric by extracting both color and luminance features.

It should be noted that the degradation characteristics of NTIs are different from those of hazy, rainy and underwater images. Thus, these quality evaluation methods developed for hazy, rainy and underwater images cannot achieve good performance on NTIs. In this paper, we focus on designing efficient end-to-end deep network architectures for blind NTIQE.

According to the Retinex theory [28], a single image $I$ can be considered as a composition of two independent layer components, i.e., reflectance $R$ and illumination $L$, in the fashion of $I=R\otimes L$, where $\otimes$ denotes element-wise product. However, recovering two components from one single input is an ill-posed problem. Although the Retinex theory has been widely applied in relevant applications [45, 46, 47, 48], there are obvious differences between our proposed method and these works. First, the problem we focuses is different from these works, i.e., we focus on the problem of NTIQE while these works [45, 46, 47, 48] focus on low-light/night-time image enhancement. Second, none of them have successfully decomposed a single image into two independent components with an embedded deep neural network module in a fully unsupervised manner. In what follows, we will present how to design an effective deep image decomposition module to achieve this goal.

The detailed architecture of our deep image decomposition module is shown in Fig. 2. It contains two streams corresponding to the reflection component ($R$) and illumination component ($L$), respectively. The reflection stream adopts a 5-layer U-Net-like architecture, followed by two convolutional (conv) layers and a Sigmoid layer in the end, while the illumination stream is composed of two conv+ReLU layers and a conv layer on concatenated feature maps from the reflection stream, finally followed by a Sigmoid layer in the end.

Typically, there is no available ground truth reflection and illumination maps that can be used for supervised training. Therefore, designing a well-defined non-reference loss function is the key to the success for training a stable deep image decomposition module. In the literature, a basic assumption is that different shots of the same scene should have the same reflection component. Furthermore, although the illumination maps could be intensively varied, they should be of simple and mutually consistent structures. These inspire us to take a pair of images (associated with the same scene) as input and impose both reflection and illumination constraints between the image pair to train the image decomposition module without ground truth.

Specifically, during the training stage, the input to our image decomposition module is a pair of an NTI and its corresponding exposure-adjusted image (EAI). We denote the input NTI and EAI by $[I_{N},I_{E}]$. Their corresponding reflection and illumination components are denoted by $[R_{N},R_{E}]$ and $[L_{N},L_{E}]$, respectively. The image decomposition module is constrained by a hybrid loss defined upon these components. In th following, we first describe how to generate the EAI from the input NTI, and then illustrate the definitions of different loss terms.

EAI generation: Given an input NTI $I_{N}$, we first generate multiple intermediate images with different exposure levels according to a camera response model that characterizes the relationship between pixel values and exposure ratios. Then, the multiple intermediate images with different exposure levels are fused to obtain an EAI $I_{E}$.

Since there is no available camera information, an existing camera response model [49] that characterizes the general relationship between pixel value and exposure ratio is applied:

where $I$ and $e$ represent the pixel value and the exposure ratio, respectively, and the parameters $\alpha=-0.3293$ and $\beta=1.1258$ are estimated by fitting a total number of 201 real-world camera response curves provided in the DoRF database [50]. Specifically, the exposure ratios are $e,\cdots,e^{K}$, where the base ratio is empirically set to $e=2.4$ and the number of ratios is set to $K=4$, as in [51]. Based on the multi-exposure images $\{{\mathcal{E}}_{1},{\mathcal{E}}_{2},{\mathcal{E}}_{3},{\mathcal{E}}_{4}\}$, the SPD-MEF algorithm [52] is performed to reconstruct a fused image which can be used as the EAI $I_{E}$.

Inter-consistency loss: The inter-consistency loss includes reflection consistency loss and illumination mutual consistency loss. First, the reflection consistency loss ${\mathcal{L}}_{con}^{R}$ encourages the reflection similarity, which is defined as follows:

where ${\|\cdot\|}_{1}$ means the ${\ell}_{1}$ norm. Second, the illumination mutual consistency loss ${\mathcal{L}}_{con}^{L}$ is defined as follows:

where $\triangledown$ means the first order derivative operator along both horizontal and vertical directions, $c$ is a parameter controlling the shape of the above penalty curve. To facilitate understanding, we draw the penalty curves with different values of $c$ in Fig. 3. As we can see, the penalty value first increases and then decreases to zero as $M$ increases. In our implementation, we set $c=0.1$. By minimizing such an illumination mutual consistency loss, the mutual strong edges are encouraged to be well preserved and all weak edges are to be suppressed.

Intra-smoothness loss: Besides the inter-consistency loss mentioned above, we also consider intra-component smoothness loss. On the one hand, the illumination maps should be piece-wise smooth, thus we introduce a structure-aware smoothness loss ${\mathcal{L}}_{sm}^{L}$ to constraint both $L_{N}$ and $L_{E}$:

where $\tau$ denotes a small positive constant which is empirically set to $\tau=0.01$ to avoid the denominator being zero. This loss measures the relative structure of the illumination with respect to the reflection. Therefore, the illumination loss can be aware of image structure which is reflected by reflection. Specifically, for a strong edge point in the reflection map, the penalty on the illumination will be small; for a point in the flat region of the reflection map, the penalty on the illumination becomes large. On the other hand, different from the illumination maps that should be piece-wise smooth, the reflectance maps are usually tend to be piece-wise continuous. Thus, we directly use a total-variation loss ${\mathcal{L}}_{sm}^{R}$ to constraint both $R_{N}$ and $R_{E}$:

Reconstruction loss: The third consideration is that the decomposed two components should well reproduce the input in the fashion of element-wise product, which is constrained by an image reconstruction loss:

Image decomposition loss: The total loss for our image decomposition module is defined as follows:

Based on the reflection and illumination components generated by the image decomposition module, the next step is to build feature representations for each of these two components separately. In this work, we design a simple self-reconstruction-based encoder-decoder architecture to achieve this goal. Specifically, both the reflection and illumination components share the same feature encoding network architecture. However, these two feature encoding networks are constrained by different loss functions, i.e., tailored loss terms are designed to separately regularize the feature encoding of reflection and illumination components.

As shown in Fig. 4, our proposed self-reconstruction-based encoder-decoder module involves two parts namely encoder and decoder. The encoder receives either the reflection ($R$) or illumination ($L$) component as input and progressively forms a set of hierarchical feature representations $C_{1}$, $C_{2}$, $C_{3}$, $C_{4}$. Then, the decoder takes the last-layer feature representation $C_{4}$ as input and progressively reconstruct the input ($\hat{R}$ or $\hat{L}$). The encoder contains four stacked 3$\times$3 convolutional layers with each convolutional layer equipped with a ReLU layer for activation. The stride of all convolutional layers is set to 1. The output feature channels of each convolutional layers in the encoder are set to 16, 32, 64, 128, respectively.

Since the reflection and illumination components contain NTI degradation information in different aspects, we design tailored losses to guide the reconstruction of each component. Specifically, the losses imposed on the reflection component reconstruction include a structure loss ${\mathcal{L}}_{str}$ and a color loss ${\mathcal{L}}_{color}$, while the loss imposed on the illumination component reconstruction is a mean square error (MSE) loss ${\mathcal{L}}_{mse}$. In this way, the learned reflection feature representations will focus more on the structural and color information due to the joint guidance of ${\mathcal{L}}_{str}$ and ${\mathcal{L}}_{color}$, while the learned illumination feature representations will focus more on the luminance information.

1) Structure loss: It is well known that the HVS is highly sensitive to image structural information and low-quality NTIs will inevitably change the structural perception [53]. For this, we adopt a structural similarity (SSIM) [53] loss between the input reflection image $R$ and its corresponding reconstructed version $\hat{R}$ for encouraging the encoder to have the capacity of extracting informative structural features. The SSIM loss is defined as follows:

where $SSIM(R,\hat{R})$ computes the structural similarity score between image $R$ and image $\hat{R}$ according to the SSIM metric [53].

2) Color loss: It is a common sense that NTIs will introduce color distortions and the reflection component contains almost all color information of the scene. Therefore, a simple yet effective color loss between $R$ and $\hat{R}$ is also desired, which will encourage the encoder to have the capability of extracting effective color features. Inspired by [54], the blurring operation can remove high frequencies of an image and promote color comparison. Thus, the following color loss is introduced:

where ${R}_{B}$ and ${\hat{R}}_{B}$ are the blurred versions of $R$ and $\hat{R}$, respectively:

where $\Omega(i,j)$ is an image patch centered by the pixel at $(i,j)$ and $G({\Delta}_{i},{\Delta}_{j})$ is the 2-D Gaussian blur kernel, which can be expressed as:

where the parameters $T=0.053$, $\mu=0$, and $\sigma=3$ are set according to [54].

3) MSE loss: For the reconstruction of illumination component, we only apply a simple MSE loss which is defined by the Euclidean distance between the $L$ and $\hat{L}$:

The overall loss for our feature encoding module is $\mathcal{L}_{feat}=\mathcal{L}_{str}+\mathcal{L}_{color}+\mathcal{L}_{mse}$. Constrained by these tailored losses for self-reconstruction, the content-related features and illumination-related features can be well extracted from the reflection and illumination component, respectively.

We consider bilinear techniques to combine the reflection and illumination feature representations into an unified one. Bilinear models have shown powerful capability in modeling two-factor variations, such as style and content of images [55], location and appearance for fine-grained recognition [56], temporal and spatial aspects for video analysis [57], etc. It also has been applied to address the BIQA problem where the synthetic and authentic distortions are modeled as the two-factor variations [58]. Here, we tackle the blind NTIQE problem with a similar philosophy, where the reflection-related and illumination-related degradations are modeled as the two-factor variations.

Given an input NTI and its side output feature maps from the reflection and illumination encoders, $C_{i}^{R}$ and $C_{i}^{L}$ are both with the size of $h_{i}\times w_{i}\times d_{i}$ since the reflection and illumination encoder share the same architectures and configurations. Before performing bilinear pooling, $C_{i}^{R}$ and $C_{i}^{L}$ are separately fed into a $1\times 1$ convolutional layer to obtain their corresponding compact version with 32 channels ($\hat{C}_{i}^{R}$ and $\hat{C}_{i}^{L}$), i.e., $h_{i}\times w_{i}\times 32$. Then, bilinear pooling is performed on $C_{i}^{R}$ and $C_{i}^{L}$ as follows:

where the outer product $B_{i}$ is a vector of dimension $32\times 32$.

According to [59], bilinear representation is usually mapped from Riemannian manifold into an Euclidean space using signed square root and $\ell_{2}$ normalization [60]:

where $\odot$ means the element-wise product. Finally, the bilinear pooled feature representations over all scales are concatenated into a single vector:

Finally, $\hat{B}$ is fed into two fully connected layers for quality prediction, which outputs a scalar indicating the overall quality score. Here, we consider the $\ell_{2}$ norm as the empirical loss, which has been widely used in previous works:

where $Q_{k}$ is the ground truth subjective quality score of the $k$-th image in a mini-batch and $\hat{Q}_{k}$ is the predicted quality score by DDB-Net. It is noteworthy that bilinear pooling is a global strategy and therefore our DDB-Net can receive input images with arbitrary sizes.

Our DDB-Net is trained on the target NTI quality database by minimizing the following hybrid loss function:

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are the weights used to control the relative importance of different loss terms. The optimal weights are $\lambda_{1}=0.1$, $\lambda_{2}=0.2$, and $\lambda_{3}=0.7$, respectively. During training, all parameters are radomly initialized and we use the Adam optimization algorithm [61] with a batch size of $16$. We run $100$ epoches with a learning rate of $3\times 10^{-5}$ and use Batch normalization to stabilize the training process. All the training images are resized into $512\times 512\times 3$ before feeding into the network. The model is implemented by PyTorch [62] with a single NVIDIA GTX 2080Ti GPU. During testing, the EAI stream will not be used.

Since there is always no available pristine reference for real-world NTIs, the quality evaluation of NTIs can only be performed in a no-reference manner. Therefore, we compare the performance of the proposed DDB-Net against 20 existing BIQA methods, including 15 handcrafted feature-based BIQA methods (i.e., BLIINDS-II [9], BRISQUE [10], CurveletQA [11], DIIVINE [12], NRSL [13], NFERM [14], GM-LOG [15], GWH-GLBP [16], SSEQ [17], BIQME [18], ILNIQE [19], NIQE [20], BNBT [27], MDM [7], HOSA [23]) and five deep learning-based BIQA methods (i.e., WaDIQaM [37], DBCNN [38], TSCNN [64], VCR [65], and GraphBIQA [66]). The handcrafted feature-based BIQA methods include two types: training-based and training-free. The training-based ones commonly adopt elaborately designed features to characterize the level of deviations from statistical regularities of high-quality natural images, based on which a quality prediction function is learned via supprot vector regression (SVR) [67]. The training-free ones (i.e., ILNIQE [19] and NIQE [20]) first build a pristine statistical model from a large collection of high-quality natural images and then measure the distance between this pristine statistical model and the statistical model built on the distorted image as the estimated quality score. By contrast, the deep learning-based BIQA methods directly optimize an end-to-end function mapping from the input image to its quality score without any efforts on manual feature engineering.

Despite the above results have demonstrated the promising performance of our DDB-Net on each individual dataset, it remains unknown whether the model pretrained on one dataset can be well generalized to another. Therefore, we in this section conduct cross-dataset validations by training the model on one dataset and testing it on another one. Note that the ranges of MOS values of the two datasets are different, i.e., [0,1] for the NNID dataset and [0,5] for the EHND dataset, we linearly normalize the MOS values of the EHND dataset into the range of [0,1] to facilitate cross-dataset validation. The performance results of cross-dataset validations are shown in Table III. In the table, we only show the results of the methods that have already achieved fairly good performance (the SRCC values are higher than 0.8) in the intra-dataset validation experiments. From Table III, we can observe that the performance results obtained by training on NNID and testing on EHND are worse than the results obtained by training on EHND and testing NNID. It is expectable because the images in EHND are enhanced by different algorithms, which may suffer from extra artifacts beyond those appeared in the original raw NTIs. Despite this, our proposed DDB-Net still owns the best generalization capability among all competitors.

Besides the above two large-scale night-time image quality datasets (i.e., NNID and EHND), we are also interested in the performance of different BIQA methods on a small-scale dataset. To the best of our knowledge, except for NNID and EHND, there is no other specific dataset for NTI quality evaluation. Therefore, we pick out all the NITs from the CLIVE dataset (CLIVE-NT subset) for testing. As a result, a total number of 159 NTIs were selected to form the CLIVE-NT subset. Some examples are shown in Fig. 8. Since CLIVE-NT subset is quite small, it is not suitable for training. Thus, we use the model trained on the whole NNID dataset to test the performance on the CLIVE-NT subset. The results are shown in Table. IV. We can observe that the performance values are consistently inferior to those on the NNID dataset. It is exceptable because the characteristics of these two datasets are quite different. Despite of this, our proposed DDB-Net is still better than other compared methods by a large margin.

As we have mentioned in Eq. (19), we use $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ to control the relative importance of different loss terms. How to determine these weight values is non-trivial. Since the main task in NTIQE is to predict the quality score of an input NTI, we first set a relatively large weight to $\lambda_{3}$, i.e., $\lambda_{3}=\{0.6,0.7,0.8,0.9\}$. Then, the other two weights $\lambda_{1}$ and $\lambda_{2}$ are determined based on the constraint $\lambda_{1}+\lambda_{2}=1-\lambda_{3}$. We test the performance results of different weight values on the NNID dataset, as shown in Table V. We can find that the best SRCC, KRCC, and RMSE are obtained when $\lambda_{1}=0.1$, $\lambda_{2}=0.2$, and $\lambda_{3}=0.7$ while the best PLCC is obtained when $\lambda_{1}=0.1$, $\lambda_{2}=0.1$, and $\lambda_{3}=0.8$. Given that the PLCC metric is related with the curve fitting process, it is less important than SRCC and KRCC. Therefore, we take $\lambda_{1}=0.1$, $\lambda_{2}=0.2$, and $\lambda_{3}=0.7$ as the final weights in our implementation.

We finally compare the computational complexity of different methods including running time, parameter number, and floating point operations (FLOPs). Note that the parameter number and FLOPs are only applicable to deep learning-based methods. The computational complexity is tested on a PC with an Intel Xeon Silver 4210 CPU @ 2.20GHz and a RTX 2080Ti GPU. The results of running time, parameter number, and FLOPs are shown in Table VI. We can find that deep learning-based methods generally have faster running speed than the handcrafted feature-based methods during the testing stage. Among all methods, DBCNN has the fastest running time followed by our proposed DDB-Net which ranks the second place. Among all deep learning-based methods, our DDB-Net has the lowest number of parameters and the second-lowest number of FLOPs. Note that our DDB-Net only has 0.38M parameters, which is efficient. Overall, our proposed DDB-Net can evaluate the quality of NTIs quite efficiently.