SID-NISM: A Self-supervised Low-light Image Enhancement Framework

According to the definition of the Retinex theory, the reflectance component of an image should be invariant to the illumination component and other imaging conditions, which suggests that any pair of images with the same content should share the same reflectance regardless of their different illumination conditions. Thus inspired by the observation, a self-supervised learning image decomposition network called SID-Net is designed to split given low-light image into its reflectance, illumination and noise directly by taking as inputs the original image and its histogram equalization version. Notice that since the histogram equalization image is served as one of the paired images under different illumination condition to share the same reflectance, not the ground-truth normal-light images as in the supervised-learning methods, it doesn’t matter whether histogram equalization would produce images with unpleasant color distortion or unbalanced illumination. Besides, although theoretically capturing paired images or utilizing other image generation methods are acceptable as well, considering the practical application and the computational cost, histogram equalization image becomes the best option.

As illustrated in Fig. 1, SID-Net firstly takes as inputs the source image $S_{low}$ and its corresponding histogram equalization version $S_{he}$ to split both of them into three components according to the robust Retinex theory, then iteratively converges to the optimal decomposition maps by minimizing the designed loss function. Since it is a self-supervised network without external information from training images, the key point of SID-Net is to devise reasonable and effective loss function to guide the network to generate expected decomposition maps. The loss function of SID-Net is formulated as follows,

where $\lambda_{rc}$, $\lambda_{L}$, $\lambda_{R}$, $\lambda_{N}$ denote the coefficients to balance the reflectance similarities and the guidance of generating illumination, reflectance and noise maps. They are signi ficantly smaller than 1 to address the importance of the fidelity term $\mathcal{L}_{rec}$ in the optimization, which are 0.01, 0.1, 0.001, 0.01 respectively in experiments. Notice that the reconstruction loss $\mathcal{L}_{rec}$ and the three guidance loss terms $\mathcal{L}_{L},\mathcal{L}_{R},\mathcal{L}_{N}$ are both applied on the original low-light image $S_{low}$ and the histogram equalization image $S_{he}$ to separate them into reflectance $R_{low},R_{he}$, illumination $L_{low},L_{he}$ and noise $N_{low},N_{he}$ in a self-supervised way.

Retinex reconstruction loss. First and foremost, according to the robust Retinex theory [12], the three decomposed maps should be able to reconstruct the original image. Thus the reconstruction loss $\mathcal{L}_{rec}$ is formulated as,

It corresponds to the image formation, which could constrain the distance between estimated $R\times L+N$ and the source image $S$.

Reflectance consistency loss. As described in Sect. 3, the reflectance reflects the intrinsic property of captured objects, which should be invariant to the scene lighting and imaging conditions. That is to say, in the case of the same captured objects, i.e. image content, the reflectance maps of the low-light image and its histogram equalization version should be as close as possible. Therefore, based on the above assumption, the reflectance consistency loss $\mathcal{L}_{rc}$ is designed to constrain the reflectance similarities between $R_{low}$ and $R_{he}$.

Illumination smoothness and consistency loss. The loss function designed for the illumination map $\mathcal{L}_{L}$ is consist of two parts, the smoothness term and the consistency term, which can be expressed as,

where $\nabla$ denotes the gradient including $\nabla_{h}$ (horizontal) and $\nabla_{v}$ (vertical) and $\alpha$ denotes the coefficient balancing the strength of structure-awareness and edge-preservation which is set as 10 in experiments.

The first term is adopted to guide the illumination map to be piece-wise smooth in textural details while preserving the general structure of the original images. A common-used smoothness prior in various image restoration tasks, total variation minimization (TV) [14, 16], which minimizes the gradient of the whole image, is selected here as the fundamental of the illumination smoothness term. The second one is designed to preserve strong mutual edges while depressing weak ones. Similar to the illumination smoothness loss, it utilizes the weighted version of TV loss. Notice that since TV loss is structure-blindness, the original TV function is weighted with the gradient of the reflectance [22] and illumination maps [26] respectively to make the constructed loss be aware of the image structure and the two illumination maps be consistent with each other in terms of the image edges.

Reflectance contrast and color loss. To generate accurate reflectance map, in addition to the basic reconstruction loss, a novel reflectance contrast and color loss $\mathcal{L}_{R}$ is introduced to improve the image contrast and restore the image color as much as possible, which can be expressed as,

where $\beta$ is the gradient amplification factor which is set as 10 in experiments. $R^{H}$ and $S^{H}$ are the hue channels of the reflectance map and the source image after converting them from the RGB to the HSV color space.

For the first term, as we know, low-light images always suffer from low contrast, which often indicates smaller gradient magnitudes [12]. Hence we attempt to manipulate the gradient magnitudes of the reflectance to boost the contrast of the enhanced results by amplifying the gradient of the input image with the factor $\beta$. Notice that $\nabla\hat{S}$ is a variant of the gradient of the input image $\nabla{S}$, in which small gradients, i.e. the noise, are suppressed before amplification.

The second term is adopted to avoid color mismatch in the decomposed reflectance, which enforce the hue component of the reflectance map to be close enough to that of the source image.

Noise estimation loss. Although the reflectance contrast loss attempts to suppress noise by ignoring small gradients, noise may still appear in flat dark regions. It is necessary to guide the decomposition of the noise map by,

which constrains the overall intensity of the noise [12].

As for the detailed network configuration, SID-Net first adopts a commonly used feature extractor, which could be any classic network such as ResNet, U-Net, or even a simple CNN structure. Then, three separate $3\times 3$ convolutional layers project the image features into branches, namely reflectance $R$, illumination $L$, and noise $N$ respectively. At the end, sigmoid function is used to constrain both $R$ and $L$ in the range of [0, 1], while tanh function is adopted to simulate additive noise to make $N$ fall in [-1, 1].

After obtaining the low-light illumination map $L$ of the given image $S$ by SID-Net, a manipulation operator would be adopted to enhance $L$ as described in Sect. 3. In the previous methods [6, 17, 25], researchers got used to refine the illumination map through Gamma correction, say $\hat{L}=L^{1/\gamma}$. When $\gamma>1$, the illumination map will be brightened up. In Fig. 2(a), the function curve of Gamma transformation with $\gamma=2.2$ is shown by the red line. It can be seen that pixels whose intensity are less than 20% of the maximum illumination value, namely 0.2 in the scale of [0, 1], will be brightened to 50%, while the pixels whose original intensity is large enough ($>80\%$) will keep the themselves basically unchanged. This property of Gamma correction would definitely do favor to enhance the illumination map especially in those dark regions, but it also brings two major flaws: 1) the contrast of the original image is destroyed due to the over-brightening of dark areas; 2) the illumination level of the final enhanced image is still insufficient because the bright areas are barely changed. As shown in Fig. 2(c), the overall quality of the enhanced result is improved compared with the low-light input Fig. 2(b), but it haven’t been unable to reach the standard of ’normal-light’ in human perception.

To combat the existing issues of Gamma correction, we consider depressing the brightening of dark pixels while increasing the illumination saturation of bright pixels. More concretely, for dark pixels, it is still necessary to be brightened up but should be carried on at a relatively lower level as compared with Gamma correction. In other words, the slope of the illumination operator at pixels whose intensity are less than 0.2 should be smaller than that of Gamma correction. On the contrary, the slope at bright pixels should be larger than that of Gamma correction, which could increase the illumination levels of bright pixels so as to render the final enhanced results bright enough.

According to the above considerations, a novel manipulation operator called Nonlinear Illumination Saturation Mapping (NISM) is proposed, which is formulated as,

where $\eta$ is the control parameter as $\gamma$ in Gamma correction. The function curve of NISM with $\eta=2.2$ is depicted in the blue line in Fig. 2(a). It can be seen that the curve shape of NISM is consistent with the previous analysis about the improved curve slope. Indeed, the constructed NISM and the Gamma correction are symmetrical about $y=1-x$.

The final enhanced image is generated by recombining the decomposed reflectance $R$ and the refined illumination $\hat{L}$ by $R\times\hat{L}$. The corresponding result is shown in Fig. 2(d), which reflects that NISM could transform the decomposed illumination map of the original low-light image into a normal-light level.

In implementation $\eta$ is not fixed. It is computed by first clustering pixels of the low-light illumination map into two clusters, bright pixels and dark pixels, by KMeans. Then taking the minimum illumination value of bright pixels as threshold $T$, $\eta$ is calculated by,

which means to map the minimum illumination value of bright pixels to 0.8 under NISM.

As a self-supervised framework, SID-NISM doesn’t need any training dataset or prior information. For any given low-light image, SID-Net could split it into reflectance, illumination and noise components directly within hundred iterations, then NISM would enhance the decomposed illumination to combine with the reflectance. Besides, all the coefficients mentioned in Sect. 4 would remain the same during the experiments, which means it is not necessary to adjust the parameters for different inputs.

Evaluations are performed on real-scene images from four public datasets, DICM [11], LIME [6], LOL [22], and MEF [13]. And the proposed method SID-NISM is compared with 6 state-of-the-art low-light image enhancement algorithms, including 1) a weighted variational model for simultaneous reflectance and illumination estimation (SRIE) [4], 2)illumination map estimation method (LIME) [6], 3) joint enhancement and denoising method via sequential decomposition (JED) [17], 4) perceptually bidirectional similarity based illumination estimation (PBS) [25], 5) deep learning based Retinex decomposition (Retinex-Net) [22], and 6) unsupervised-learning based back-lit image restoration network (ExCNet) [24].

In the following subsections, both subjective and objective experimental results will be exhibited. More experiments including the intermediate decomposition results and the analysis of the effect on the object detection task can be found in the supplementary materials.

Since Retinex-based low-light image enhancement methods consider the problem as an image decomposition problem, it is necessary to show our intermediate decomposition results firstly. Fig. 3 exhibits the decomposition results of two low-light images, Totoro and toy in LOL dataset. Their final enhanced results can be found in the main paper. In general, the proposed SID-Net can decompose reasonable reflectance, illumination and noise maps for given input images in a self-supervised way successfully. The decomposed reflectance maps contain sufficient color and structure details of the captured scene. The illumination maps reflect the lighting conditions as observed in human eyes. While the noise maps extract underlying noise component in dark regions.

Fig. 4 compares the decomposition results of the proposed image decomposition network SID-Net with three state-of-the-art Retinex-based methods, LIME, JED and Retinex-Net, on Bookshelf in LOL dataset and Balloon in MEF dataset.

It can be seen that all of the four methods can decompose reasonable reflectance and illumination maps for given input images as expected. However, there also exists some differences among the four methods. In the aspect of reflectance, it is obvious that LIME can restore more color and structure details of the original objects compared with the other methods, because the reflectance of LIME is calculated by conducting element-wise division on the source image and the estimated illumination, which preserve the Retinex theory to a large extent. As for the illumination maps, SID-Net and Retinex-Net outperform other methods due to the specific structure-awareness illumination smoothness loss term, which makes it more consistent with the image structure. Besides, both as image decomposition networks, the biggest difference between SID-Net and Retinex-Net is that the proposed SID-Net is an unsupervised learning network without any prior training process, while the other one is a supervised learning network based on large-scale training dataset. As for the practical image decomposition performance, the two networks are evenly matched. Even in the reflectance maps, the results of SID-Net are clearer with more details than that of Retinex-Net. In general, as shown in Fig. 4, SID-Net can extract underlying consistent reflectance from the given input image directly, and portray the lightness and shadow of the image in the illumination map. Although without prior training, the SID-Net can achieve the same or even better performance at the image decomposition stage.

User study was conducted to perform comparisons between state-of-the-art methods in the dimensions of the overall image quality and some unpleasing artifacts through human perceptions. To evaluate the overall image quality, the subjects gave their personal feelings on the scale of 0 to 10, where 0 represents “poor”, 5 means “good”, and 10 is “excellent”. As for the artifacts, there are five artifacts to be selected from (multiple-choice). Those are black edges, blur, overexposure, gray shadow and halo as shown in Fig. 5. Once the subjects observe phenomenon in the images consisting with the description of the artifacts, the corresponding option would be checked. For all the enhanced images obtained by different methods, the subjects move the slider to give the image quality score and select specific artifacts according to their observations. The above procedure was completed on the Amazon Mechanical Turk platform by thirty workers.

The results of the user study are summarized in graphs shown in Fig. 6 and 7. In Fig. 6, a boxplot is used to exhibit the distributions of the image quality scores for different methods, in which the red solid line represents the median while the blue dotted line represents the mean score. Through observing the median, mean, and the first quartile, it is obvious that the participants showed a strong bias in preference towards our results (median: 6.0, mean: 5.51, quartile: 8.0) when compared to Retinex-Net [22] (median: 3.0, mean: 3.05, quartile: 4.0), gave higher scores when compared to SRIE [4] (median: 5.0, mean: 5.04, quartile: 7.0), JED [17] (median: 5.0, mean: 5.34, quartile: 7.0), PBS [25] (median: 5.0, mean: 5.31, quartile: 7.0) and ExCNet [24] (median: 5.0, mean: 5.41, quartile: 7.0), and had no preference when compared with LIME [6] (median: 6.0, mean: 5.67, quartile: 8.0). The results in Fig. 6 indicate that SID-NISM outperforms most of the state-of-the-art methods in terms of the overall image quality.

In Fig. 7, a stacked bar graph is adopted to illustrate the average counts of artifacts in the results of different methods, in which each color bar corresponds to the five artifacts in Fig. 5 respectively. Specifically, the average counts are computed by $M/(N*P)$, where $M$ is the total counts of artifacts in the results of each method, $N$ is the number of images, $P$ is the number of subjects. It can be seen that SID-NISM tends to introduce less artifacts (average counts: 0.6) than other methods (average counts: above 0.8) in total. Generally, SID-NISM doesn’t lead too many unexpected artifacts. SRIE [4] would introduce gray shadow into their results, while the results of LIME [6] may be over-enhanced to become overexposure. JED [17] has more blurry cases, while Retinex-Net [22] contains the most black edges in their results. These phenomenons can be also observed in the visual examples of the next subsection.

In order to facilitate readers to visually compare the results of different low-light image enhancement approaches, several examples are exhibited in Fig. 8 and  9.

In general, it is clear that SID-NISM could brighten up the dark regions of low-light images successfully. More importantly, it can also restore the strong contrast of scene content sufficiently. For example, taking candle in Fig. 8 as input, the contrast between the candle in the foreground and the glass cup in the background is clearer and stronger in the result of SID-NISM compared with other methods. Another example is the toy in Fig. 9. It is obvious that the contrast between the toy’s face and the background is stronger in our result. This property renders the results of our method seem more active and attractive, which is consistent with the user study about the image overall perceptions.

In addition, as discussed in Sect. 5.3, SID-NISM could restore the overall quality of low-light images on the premise of introducing less unexpected artifacts, including preserving clear boundary (glass cup in candle), avoiding gray shadow (the inside of the door in street) and halo (boundary of the door in street). Especially, our results contain less noise such as the sky in street and the white background in toy. Combining with the above intuitive observation and data statistics in the user study, it can be seen that our method is more robust with less artifacts, which makes our framework more competitive and practical in both people’s daily use and commercial cases.

In addition to the above subjective experiments, there are many indexes with or without reference images that can be used to evaluate the quality of the enhanced images objectively. In this section, we take use of gray entropy (GE), color entropy (CE, sum of the entropy of RGB channels), gray mean illumination (GMI), gray mean gradient (GMG), NIQE [15], PSNR, and SSIM [21] to compare different low-light image enhancement methods. Generally, the first four indexes could reflect the gap between the enhanced results and the reference high-light images from four aspects including the information in gray and color channels, the illumination degree and the image gradient magnitude. NIQE assesses the overall naturalness of the enhanced results and a lower value roughly corresponds to a higher overall naturalness. PSNR could verify the performance of denoising algorithms and a higher value usually indicates a better image quality. SSIM measures the structure similarity between the enhanced results and the reference images, which is in the range of [0, 1]. It should be noted that these indexes can only reflect the image quality in specific aspects, which may be not completely consistent with the evaluation results given by the human visual system.

The quantitative results over LOL and MEF datasets (the two datasets with reference normal-light images) are reported in Table 1. Obviously, the proposed SID-NISM could achieve better performance compared with existing state-of-the-art methods in terms of the image information, the overall quality and the noise condition. It is worth mentioning that we also compare the performance of our framework with and without $\mathcal{L}_{R}$ and $\mathcal{L}_{N}$ to quantitatively evaluate the merit brought by the two novel reflectance and noise terms in the loss function of SID-Net. It can be seen that they not only help improve the image color and gradient information of the enhanced results greatly, but also decrease the noise influence and promote the overall image quality.