NLHD: A Pixel-Level Non-Local Retinex Model for Low-Light Image Enhancement

Simplified Retinex theory assumes that we can decompose an observed image into an illumination component and a reflectance component. As a seminal work, Single Scale Retinex (SSR) [18] and Multi-Scale Retinex (MSR) [19] implement Gaussian filtering on the images, take the low-frequency results as the illumination component, and remove the filtered results from the original images to obtain reflectance component. However, these methods are prone to introduce color distortion and halo artifacts in regions with large brightness deviations. In addition, the over-smoothed illumination component results in excessive structural and texture information getting into the reflectance component so that the enhancement of the reflectance component produces an intense sharpening phenomenon.

In recent years, the Retinex model is still widely applied to the low-light image enhancement task. Multi-scale derived images Fusion (MF) [20] adjusts the illumination component by fusing multiple derivatives of its initial estimation. Low-Rank Regularized Retinex Model (LR3M) [14] suppresses the noise in the reflectance component by adding a low-rank constraint into the decomposition model but misses enough details from the illumination component. A Weighted Variational Model (WVM) [8] was proposed to estimate the reflectance and illumination simultaneously. The Joint intrinsic-extrinsic Prior model (JieP) [6] enhances the illumination component through exponential transformation and obtains ideal low-light image enhancement results. Low-Light Image Enhancement via Illumination Map Estimation (LIME) [21] estimates the brightness component of each pixel by finding the maximum value in the RGB channels. The structure prior is added to the initial illumination component to refine it into the final illumination component. Finally, the enhanced image is generated according to the illumination component. The Structure and Texture Aware Retinex model (STAR) [12] well estimates both the illumination and reflectance components for image enhancement. Because the above methods preserve part of the structure and texture information in the illumination component, the sharpening phenomenon has been alleviated.

Unlike previous Retinex-based methods, our Non-local Haar Decomposition (NLHD) method makes full use of the pixel-level non-local self-similarity prior to decompose the low-light images; it can achieve more appropriate illumination and reflectance components. Because the illumination component preserves much more structure and texture information, the enhanced image has almost no sharpening phenomenon.

Deep learning has achieved great success in many fields. Many deep learning based low-light image enhancement methods have also been developed in recent years.

Retinex Decomposition based Generative Adversarial Network (RDGAN) [10] integrates the Retinex model into a deep learning framework, splitting the Generative Adversarial Network (GAN) [33] into a Retinex decomposition network module and a fusion reinforcement network module. Deep Retinex Decomposition for Low-Light Enhancement Network (Retinex-Net) [15] combines Retinex model with semantic information perception to enhance segmented image regions. Low-light Net (LLNet) [25] achieves the low-light image enhancement based on a deep auto-encoder network. Multi-Branch Low-Light image Enhancement Network (MBLLEN) [2] extracts image features at different levels, enhances the image through multiple sub-networks, and finally generates the output image through multi-branch fusion. Kindling the Darkness (KinD) [3] added an adjustment network to adjust the illumination, which alleviates the problem of overexposure. Deep Light Enhancement Generative Adversarial Network (EnlightenGAN) [4] uses the information extracted from the input image itself to regularize the unpaired training and proposes a global-local discrimination structure, a self regularized perceptual loss fusion and attention mechanism; four non-reference loss functions are designed to satisfy the non-reference training. Because most of the above deep learning based methods combine the Retinex model, the Retinex decomposition is still achieved by convolution operation without considering non-local prior; the over-smoothed illumination component still causes sharpening.

Generally, there are many similar image blocks in an image, which may not come from the same local region. This property is usually called non-local self-similarity. Non-local self-similarity was initially proposed by the Non-local Means (NLM) [26] to solve the image denoising problem. NLM estimates each pixel by computing a weighted average of some similar pixels obtained by implementing image block-matching, so NLM is a spatial domain method. BM3D [27, 34] is a transform domain non-local method that combines the non-local idea with discrete wavelet transform (DWT) and discrete cosine transform (DCT). BM3D is a patch-level non-local method, i.e., the signal in a single image patch is still local, so it is not an utterly non-local method. In order to better apply the non-local idea to the image denoising problem, NLH [5] proposed to further search similar pixels in a similar image patch group and can achieve better image denoising results.

In the NLH method, a similar image block group is obtained by block-matching as the same as NLM and BM3D, then scan each image block to a column vector, stack all the vectors to get a 2D matrix, finally implement row-matching to obtain similar pixel groups. A separable Haar transform is implemented on each similar pixel group to achieve the signal decomposition.

In this paper, we apply the NLH idea to decompose the image into illumination and reflectance components and further achieve low-light image enhancement. The proposed image decomposition method makes full use of the pixel-level non-local self-similarity in the images, so the enhanced images by our method look much more natural than the enhanced images by most existing local filtering based Retinex decomposition methods, especially the enhanced images by our method have almost no sharpening phenomenon.

Retinex model is developed based on the human visual cognition system. The model decomposes the image into two components: illumination and reflectance, i.e.,

where $O$ is an image, $I\in\mathbb{R}^{m\times n}$ is the illumination component of $O$, representing the incident light image, namely the brightness of the light on the object surface in the image, which determines the dynamic range that the pixel value can reach in the image; $R\in\mathbb{R}^{m\times n}$ is the reflectance property image of the objects, representing the internal attribute of the image, namely the structure of the objects in the image. The basic idea of the Retinex model is to remove or reduce the influence of incident light on the image by some methods to preserve the object’s natural attribute as much as possible. Therefore, estimating or decomposing the illumination component $I$ and reflectance component $R$ is the key problem of the Retinex model.

In essence, the illumination component $I$ represents the brightness of the object surface in the image, which comes mainly from the smoother regions of the image; However, the reflectance component $R$ is generally distributed in the small structural parts such as texture in the image. Therefore, the Retinex decomposition should be attributed to better decomposing the image’s smooth regions and texture regions. Based on this prior knowledge, we propose a new Retinex decomposition method based on the pixel-level non-local Haar transform in this paper. We present the details of the proposed method in §III-B and §III-C.

Given a low-light color image $I\in\mathbb{R}^{h\times w\times c}$ in RGB color space. We implement block-matching and row-matching in RGB space to obtain similar pixel groups. We firstly compute the mean value of a reference image block in each R, G, B channel, according to the maximum mean value to decide the block-matching channel $C_{m}$. According to a certain sliding step size, a reference image block $B_{r}$ with the size of $\sqrt{n}\times\sqrt{n}$ is selected in $C_{m}$ to implement block-matching by Euclidean distance to get a similar block group with $N_{2}$ blocks, then extract the same position’s blocks in other two channels, so we can get three similar block groups. Each image block with the size of $\sqrt{n}\times\sqrt{n}$ is stretched into a column vector $V_{l}\in\mathbb{R}^{\sqrt{n}\times\sqrt{n}}(l=1,...,N_{2})$ in each channel respectively. All $V_{l}$ are stacked into a matrix $M_{b}$ with $\sqrt{n}\times\sqrt{n}$ rows and $N_{2}$ columns, we finally get three $M_{b}$ all together in three color channels respectively. In order to better mine the self-similarity in the images, we further implement row-matching in each $M_{b}$. Each row $R_{r}$ is used as the reference row to calculate the Euclidean distance with all other rows to find the most similar $N_{3}-1$ rows, put together $R_{r}$ with the most similar $N_{3}-1$ rows to construct a dimension $N_{3}\times N_{2}$ similar pixel matrix $M_{s}$. Specifically, for the i-th row as the reference row, the Euclidean distance between the i-th row and all other rows is calculated as follows,

In order to effectively decompose the image into illumination and reflectance components in the Retinex model, we implement separable Haar transform on the similar pixel group $M_{s}$, i.e., the vertical and horizontal separable lifting transformation. The transform can be represented as the following,

where $C_{h}$ is the transform spectral matrix of the Haar transform, $H_{l}$ and $H_{r}$ is the Haar transform matrix. Due to the characteristics of separable lifting Haar transform, $C_{h}(1,1)$ is the weighted average of all the pixels in the $M_{s}$, which we define as the low-frequency coefficient, we only use $C_{h}(1,1)$ to reconstruct the ideal illumination component $I_{l}$ by the inverse Haar transform. Instead, use the rest of $C_{h}-C_{h}(1,1)$ coefficients to reconstruct the reflectance component $I_{r}$ by the inverse Haar transform. The specific detail of NLHD is shown in Fig. LABEL:Decomposition.

Because the illumination component and the reflectance component have their respective characteristics, we respectively implement different enhancement operations on illumination component $I_{l}$ and reflectance component $I_{r}$. Three reflectance components $I_{r_{r}}$, $I_{r_{g}}$ and $I_{r_{b}}$ can be initially obtained by $R$, $G$ and $B$ channels respectively, one can use the minimum value fusion of these reflectance components as the final reflectance component $I_{r}$, but the following method can achieve the best image enhancement results in our experiments,

where $|\cdot|$ is the absolute value sign, and we finally get the enhanced reflectance component $I_{r}^{e}$ by simply adding 1.0 to $I_{r}$, i.e., $I_{r}^{e}=1.0+I_{r}$.

As for illumination component $I_{l}$, we also initially obtained three illumination components $I_{l_{r}}$, $I_{l_{g}}$ and $I_{l_{b}}$, we firstly get bright illumination component $I_{l}$ by maximum value fusing among the above three illumination components as the following,

We use different index ${\gamma_{1}}$ and ${\gamma_{2}}$ to achieve $I_{l}$ enhancement respectively, where ${\gamma_{1}}$ and ${\gamma_{2}}$ are both adaptively calculated as follows,

we get ${\gamma_{2}}$ by the two following situations: if $\frac{||{I_{l}(I_{l}<\theta_{1})||_{0}}}{||{I_{l}(I_{l}<\theta_{2})||_{0}}}>\theta$

else

where $||\cdot||_{0}$ means the number of non-zero elements. In this stage, we set $\alpha_{1}=0.35$, $\alpha_{2}=0.005$, $\alpha_{3}=0.05$, $\beta_{1}=0.45$, $\beta_{2}=0.61$, $\theta=0.9$, $\theta_{1}=0.05$, $\theta_{2}=0.15$. Two different enhanced $I_{l_{1}}^{e}$ and $I_{l_{2}}^{e}$ can be obtained by the following formulas,

Here, if we only use exponential transformation to enhance the illumination component, the extremely dark regions in the low-light images are generally over-enhanced, some mosaic artifacts are usually introduced, the noise is amplified as well. If we only use logarithmic transformation to enhance the illumination components, the relatively bright regions in the low-light images usually present over-exposure phenomenon. In order to solve these problems, we propose to enhance the illumination component as the following,

Because real-world low-light images present various illumination levels, better-enhanced results of some extremely dark images could not be achieved just by the above procedure. We use a further algorithm 1 to achieve balanced enhancement results on various initial illumination levels.

Finally, the illumination component and the reflectance component are applied to the Retinex model, that is,

where $I^{e}$ is the initial enhancement image, $\odot$ is the element-wise multiplication operation. The V channel in HSV color space is replaced by $I^{e}$, then HSV color space is converted to RGB color space to achieve the final RGB enhancement image.

The noise distribution in the low-light images is usually uneven between brighter regions and darker regions. The darker the regions, the noise is the stronger. So the traditional image denoising method is not entirely suitable for the low-light image noise suppression problem. The pixel-level non-local Haar transform (NLH) was initially proposed to solve the image denoising problem. However, NLH estimates noise level on the whole image, so it is unsuitable for low-light image noise estimation.

In this paper, we further develop a new local noise estimation method and combine it with the NLH method to achieve better noise suppression in the low-light image enhancement procedure. The proposed method has a little bit different from the NLH method; we use the minimum row-matching distances to estimate local noise level, further use the mean value of the similar pixel group, the local noise level, and the standard deviation of minimum row-matching distances to decide the hard-thresholding parameter value. The local noise standard deviation $\sigma$ is computed as the following,

where $d_{n}$ is the minimum row distance in each row-matching procedure referring to Eq.2, $N_{1}$ is the size of image block.

We adaptively select the hard-thresholding parameter according to the similar pixel group mean value $m_{s}$. The smaller the mean value, the larger the hard-thresholding parameter. Besides, we also use minimum row-matching distance standard deviation to adaptively decide hard-thresholding parameter; the larger the distance standard deviation value, the smaller hard-thresholding parameter to better preserve the edge details in images, such as textures and contours. The local minimum row-matching distance standard deviation can be calculated as the following,

We use parameters $N_{1}=6$, $N_{step}=5$, $N_{s}=13$, $N_{2}=16$, $N_{3}=4$ in the noise suppression procedure, where $N_{1}$ is the image block size, $N_{step}$ is the sliding step size of reference image block, $N_{s}$ is the block-matching searching neighbor radius, $N_{2}$ is the number of similar image blocks, and $N_{3}$ is the number of the rows in each similar pixel group. The hard-thresholding parameter $Thr$ is computed as the following,

where $m_{s}$ is the mean value of the similar pixel group, the parameter $k$ adjusts $\sigma_{d}$ value such that it is close to the value of $m_{s}$, $k=1000$ in our experiments. The role of this formula is that we allocate larger $Thr$ in the darker regions in the images; this corresponds to the fact that the darker the regions, the stronger the noise. Besides, the larger $\sigma_{d}$ means the richer image details, so we should allocate the smaller $Thr$ to better preserve image details. On the contrary, we should use larger $Thr$ to improve the denoising ability. Meanwhile, we also use structural hard-thresholding similar to the original NLH method. We only implement two steps iteration here and remove the Wiener filtering stage in the original NLH method. Because we use a group of low complexity parameters and the simplified procedure, the running speed of this algorithm is relatively much faster than the original NLH denoising method.

In this subsection, we propose a non-local saturation reduction method to attenuate the color deviation problem. CIELab color space can be effectively used to measure the color deviation degree; we transform denoised RGB low-light image to CIELab color space, then compute the mean value of $a$ channel and $b$ channel, the global color deviation degree $D_{c}$ can be measured as the following,

where $M_{a}$ and $M_{b}$ are the mean values of channel $a$ and channel $b$ respectively. In channel $a$, the positive values represent the red color; the negative values refer to the green color. Similarly, the positive values in channel $b$ represent the yellow color; the negative values refer to the blue color. The zero value of $D_{c}$ means that the color is well-balanced; the larger $D_{c}$ value means the stronger color deviation in the image.

We propose a pixel-level non-local color deviation correction method in this paper, input the $S$ channel of HSV space to illumination decomposition procedure, synchronously using the block-matching and row-matching results to get similar pixel group $G_{S}$ in $S$ channel, using the formula $\hat{G}_{S}=G_{S}^{\gamma}$ to reduce the saturation, where $\gamma>1$, the larger $\gamma$ value, the stronger degree of the saturation reduction. The $\gamma$ is computed as the following,

where $min_{m}=min(m_{R},m_{G},m_{B})$ is the minimum value of similar pixel group mean values $m_{R}$, $m_{G}$ and $m_{B}$ for three channels in RGB color space, $std_{m}=std(m_{R},m_{G},m_{B})$ is the standard deviation of these three mean values, $k$ is an empirical parameter used to control the saturation reduction, we take its value as 0.013 in our experiments. $\alpha$ is an upper limitation to void the over reduction of the saturation; we take its value as 4.5 in our experiments. Because the darker images or the darker regions in the images usually result in the stronger color deviation, and the smaller standard deviation among the three color channels means the stronger color deviation as well, so we use the larger $\gamma$ value to reduce the saturation in these two situations. All the block-matching and row-matching parameters are the same ones as the illumination decomposition procedure.

The proposed NLHD method can decompose images to illumination and reflectance components in Retinex model, the low frequency coefficients are used to reconstruct the illumination component, the high frequency coefficients are used to reconstruct the reflectance component. We have both 5 parameters in each reconstruction, take patch size $\sqrt{n}=11$, number of similar patches $m=16$, number of rows in row matching $N_{3}=16$, sliding step size of reference patches $N_{step}=10$ and the window size $N_{s}=23$ for searching similar patches to obtain reflectance component in each RGB channels. $\sqrt{n}=6$, $m=8$, $N_{3}=2$, $N_{step}=6$ and $N_{s}=13$ are used to obtain the illumination component in each RGB channels. We use Eq.4 and Eq.5 to achieve reflectance component $I_{r}$ and illumination component $I_{l}$ respectively.

The general Retinex algorithms assume that the illumination component is usually smooth, but that is not necessarily the case; one can see from the recent Retinex decomposition methods [6, 12] that the illumination component preserves much larger structure information, but better low-light image enhancement results can be achieved. Generally, there is no ground truth of illumination and reflectance [12], so the quantitative evaluation of the existing Retinex decomposition method has always been a difficult problem. In order to evaluate the effectiveness of the proposed NLHD method, we qualitatively compare the illumination and reflectance with some of the most advanced Retinex models, including CRM [23], SIRE [7], Weighted Variational Model (WVM) [8], Joint intrinsic-extrinsic Prior (JieP) model [6], Robust Retinex Model (RRM) [9], NPE [11], LIME [21], STAR [12], and the latest deep learning based method KinD++ [24]. Similar to these methods, we replace the decomposed illumination and reflectance component to the V channel of HSV color space and return the decomposed components to RGB space. We compare the Retinex decomposition results of the above methods in Fig. 4. We can see that the proposed NLHD method can effectively achieve the smoothing effect and preserve main structure information in the illumination component; the reflectance component can better extract the tiny details of the low-light images.

We run the codes provided by the authors or collected from the Internet of CRM [23], SIRE [7], Weighted Variational Model (WVM) [8], Joint intrinsic-extrinsic Prior (JieP) model [6], Robust Retinex Model (RRM) [9], NPE [11], LIME [21], STAR [12] and the latest deep learning based method KinD++ [24] on 35images and 200darkface datasets respectively. Table I compares Average NIQE value and average LOE value of these methods in the two datasets. One can see that our results are better than other methods. Fig. 5 and Fig. 6 respectively show an image from 35images enhancement result comparison among the proposed method and other existing methods; Fig. 7 shows an image from 200darkface enhancement result comparison among the proposed method and other existing methods. We can see from Fig. 5, Fig. 6 and Fig. 7 that the enhanced result images by the proposed method have better subjective visual quality than other methods. We also implement the proposed method on LOL dataset [1] to verify the effectiveness of extremely dark image enhancement; the objective evaluation results can be seen in Table II, the proposed method achieved much better objective evaluation results on SSIM, PSNR, and $\Delta$E. Fig. 8 shows an extreme dark low-light image, the corresponding ground truth image, and all the enhanced images by different methods; we can see from this figure that the proposed NLHD method achieved a much better enhancement result than all other methods; the proposed method effectively suppressed the noise and achieved the color deviation correction.

Comparison on Speed. Table III compares the running time of different Retinex decomposition methods on a low-light $1200\times 900$ RGB image and the image enhancement procedure. We notice that the speed of our method is not faster than all other methods; we will further refine our algorithms and programs to improve the running speed in our future work.

To further verify the effectiveness of the NLHD method, we compare the face detection results on the enhanced Dark Face dataset [13] with various low-light image enhancement methods. We choose the current excellent object detection method YOLOv5 [31] to achieve face detection on the original low-light images and enhanced images by various low-light enhancement methods, respectively. We firstly train the YOLOv5 model on the Wider Face dataset [30] and then use the Dark Face dataset to test. Since the label of the test set is not public, we randomly select 600 images from the training set of the Dark Face dataset for evaluation. These images are enhanced by various methods, including CRM [23], SIRE [7], Weighted Variational Model (WVM) [8], Joint intrinsic-extrinsic Prior (JieP) model [6], Robust Retinex Model (RRM) [9], NPE [11], LIME [21], STAR [12] and the latest deep learning based method KinD++ [24]. Fig. 10 compares the visual results of face detection between the original images and the enhanced images by the proposed NLHD method. It can be seen that the face detection results on the enhanced images can be improved to a certain extent; even many faces hidden in the darker regions can be detected. In order to more objectively compare the test results, we also depict the precision-recall (P-R) curves under 0.6 IoU threshold in Fig. 9. As shown in Fig. 9, the proposed NLHD based low-light image enhancement method effectively improved the low-light environment face detection performance by the YOLOv5 model. It can be seen that the face detection result of the proposed NLHD method achieved the best performance among all the compared methods listed in the figure; it can obtain the highest Average Precision (AP) score. So the proposed NLHD method can be used for the low-light environment object detection task.

a) Is the proposed NLHD necessary? To show the necessity of NLHD, we implement low-light image enhancement experiments by the NLHD method (w/ NLHD) and directly image enhancement by the bright channel of RGB space (wo NLHD) on the 35images dataset and the 200darkface dataset, respectively. We first get the bright channel of RGB space, the maximum pixel values in R, G, and B channels are fused to construct a bright channel (Actually, channel V in HSV space is the bright channel of RGB space), then we implement the exponential transform and the logarithmic transform respectively on the bright channel with the same parameters as the NLHD based enhancement method. We use the minimum fusion result of the enhanced bright channel to replace the V channel in HSV space to achieve the final enhanced image. We compare NLHD with the above method; the objective evaluation results in Table IV show that the performance of the NLHD is much better than that of the above method. The results show that the NLHD method can use pixel-level non-local self-similarity to achieve better Retinex decomposition, thus achieving better low-light image enhancement results.

b) Minimum fusion strategy v.s. separate exponential transform or separate logarithmic transform. The proposed exponential and logarithmic transform minimum fusion strategy can achieve better objective and subjective image enhancement results. In order to further verify the effectiveness of this strategy, we compare the three enhancement results, separate exponential transform (NLHD w/o exp), separate logarithmic transform (NLHD w/o log), and the minimum fusion (NLHD), as shown in Table V. We just use NIQE to evaluate the objective metric and use the 35images dataset here. Fig. 11 shows the comparison of the results with these three situations. We can see that the minimum fusion strategy achieved the best subjective visual image enhancement results.

c) Retinex model v.s. separate illumination or separate reflectance. We reconstruct the enhanced image by separate illumination component (NLHD w/o illumination), separate reflectance component (NLHD w/o reflectance), and the whole Retinex model based NLHD method (NLHD). We use NIQE to evaluate the objective metric and use the 35images dataset here. We can see from Table VI that the NLHD method presents the best performance, NLHD w/o illumination can achieve a close result to the NLHD method, NLHD w/o reflectance result is the worst one. The objective evaluation results correspond to the subjective visual quality, partly referring to Fig. 4.