Dual Degradation-Inspired Deep Unfolding Network for Low-Light Image Enhancement

Recently, deep low-light image models have highly outperformed the traditional enhancement models not only in performance but also in running time. As a pioneering work, LLNet [32] lightened low-light images using the denoising auto-encoder. After that, a large number of researchers [34, 61, 59, 53, 8, 76, 23, 75, 26] built the mapping relation between low-light images and normal-light images via different blocks or manners. Alternatively, some methods [55, 73, 50] attempted to decompose low-light images into reflectance and luminance maps via a data-driven deep model built on Retinex theory. Besides, more researchers [23, 17, 30, 36] have started to construct an unsupervised enhancement model to alleviate the shortage of paired images. However, above methods generally connot consider the physical imaging principle.

Deep unfolding model inherits the advantages of the interpretability of physical models and the powerful representation capability of deep models driven by a large number of images, which has shown prominent performance in many vision tasks, e.g., super resolution [14, 33, 67, 11] and compressive sensing [6, 39, 46]. Concretely speaking, deep unfolding model could optimize the iterative solver of physical models via some convolution networks in an end-to-end manner. In the field of low-light image enhancement, URetinex [56] proposed a Retinex theory-inspired unfolding model, however too many constraint terms hindered its performance. RUAS [30] and SCI [36] adopted a reference-free training mechanism to supervise the unfolding framework, limiting their performance. Unlike them that only learn a universal prior information, our method unfolds a novel dual degradation model based on space difference into an elaborated network, which learn two complementary and effective prior information to restore low-light images.

Fundamentally, given a low-light image $\mathbf{y}\in\mathbb{R}^{H\times W\times 3}$, this paper aim to recover a normal-light image $\mathbf{x}\in\mathbb{R}^{H\times W\times 3}$ from it. $H\times W\times 3$ denotes the spatial dimension of an image. According to Retinex theory, the degradation model of low-light images can be described as:

$$\mathbf{y}=\mathbf{D}\otimes\mathbf{x},$$

(1)

where $\mathbf{D}$ is the illumination layer, which in fact is a hybrid degradation operator that may include illuminance deterioration, color distortion, and detail loss  [50, 77]. Hence, we can recover $\mathbf{x}$ by minimizing the following energy function:

$$\min_{\mathbf{x}}\frac{1}{2}||\mathbf{y}-\mathbf{Dx}||^{2}_{2}+\lambda\mathcal{J}(\mathbf{x}),$$

(2)

where $||\mathbf{y}-\mathbf{Dx}||^{2}_{2}$ is the data fidelity term, $\mathcal{J}(\mathbf{x})$ represents the degradation prior term, and $\lambda$ denotes the hyperparameter weighting the significance of prior term. Due to the ill-posedness of degradation model, many hand-crafted prior items, e.g., non-local similarity  [65, 66] and low-rank prior  [9], are introduced to approximate a desired solution. However, above priors are hard to depict the universal structure of natural images. Recently, deep denoising prior  [10, 69] is proposed to characterize the image-generic skeleton in a data-driven manner, which show more competitive performance in many low-level vision tasks. Hence, this paper also adopts deep prior as the regularization constraint.

Traditional degradation models  [9, 65] generally conduct in a single image space, e.g., RGB or Y, which show similar deterioration type. However, low-light images show diverse deterioration types, leading to less effectiveness of above degradation model for their enhancement. Of note, we find the degradation specificity between luminance and chrominance spaces from low-light images  [1, 18], which is demonstrated in Fig. 3. One can see obvious degradation difference in luminance and chrominance spaces, which inspires us to design DDM to describe different deterioration process. DDM can be obtained via transforming Eq. (2):

	$\displaystyle\min_{\mathbf{x}_{\text{lum}},\mathbf{x}_{\text{chrom}}}$	$\displaystyle\frac{1}{2}||\mathbf{y}_{lum}-\mathbf{D}_{lum}\mathbf{x}_{lum}||^{2}_{2}+\lambda_{1}\mathcal{J}(\mathbf{x}_{lum})$		(3)
		$\displaystyle+\frac{1}{2}||\mathbf{y}_{chrom}-\mathbf{D}_{chrom}\mathbf{x}_{chrom}||^{2}_{2}+\lambda_{2}\mathcal{J}(\mathbf{x}_{chrom})$

where $\mathbf{D}_{lum}$ and $\mathbf{D}_{chrom}$ refer to luminance and chrominance degradation operators, $\mathbf{x}_{lum}\in\mathbb{R}^{H\times W\times 1}$ and $\mathbf{y}_{lum}\in\mathbb{R}^{H\times W\times 1}$ denote luminance component of normal-light and low-light images, $\mathbf{x}_{chrom}\in\mathbb{R}^{H\times W\times 2}$ and $\mathbf{y}_{chrom}\in\mathbb{R}^{H\times W\times 2}$ denote chrominance component of normal-light and low-light images, $\mathcal{J}(\mathbf{x}_{lum})$ and $\mathcal{J}(\mathbf{x}_{chrom})$ denote the luminance space prior and chrominance space prior, $\lambda_{1}$ and $\lambda_{2}$ are two weight parameters. More specifically, $\mathbf{x}_{lum}$, $\mathbf{y}_{lum}$, $\mathbf{x}_{chrom}$, and $\mathbf{y}_{chrom}$ can be obtained via:

	$\displaystyle\mathbf{x}_{lum},\mathbf{y}_{lum}$	$\displaystyle=\mathbf{H}_{lum}(\mathbf{x},\mathbf{y}),$		(4)
	$\displaystyle\mathbf{x}_{chrom},\mathbf{y}_{chrom}$	$\displaystyle=\mathbf{H}_{chrom}(\mathbf{x},\mathbf{y}),$

where $\mathbf{H}_{lum}$ and $\mathbf{H}_{chrom}$ signify the luminance and chrominance transform. For compactness, we omit $\mathbf{x}_{lum},\mathbf{y}_{lum}=\mathbf{H}_{lum}(\mathbf{x},\mathbf{y})$ and $\mathbf{x}_{chrom},\mathbf{y}_{chrom}=\mathbf{H}_{chrom}(\mathbf{x},\mathbf{y})$ in the following. Moreover, we can easily transform luminance and chrominance components to original RGB space via the inverse transform of $\mathbf{H}_{lum}$ and $\mathbf{H}_{chrom}$. DDM disentangles the intricate low-light degradation process into two easy independent degradation operators, which significantly reduce the modelling difficulty of prior term.

To achieve accurate normal-light images from low-light images, we design an alternate optimization solution to solve DDM. Firstly, we separate Eq. (3) into two independent objects to facilitate the optimization solution:

$$\small\min_{\mathbf{x}_{\text{lum}}}\frac{1}{2}||\mathbf{y}_{lum}-\mathbf{D}_{lum}\mathbf{x}_{lum}||^{2}_{2}+\lambda_{1}\mathcal{J}(\mathbf{x}_{lum}),$$

(5)

$$\small\min_{\mathbf{x}_{\text{chrom}}}\frac{1}{2}||\mathbf{y}_{chrom}-\mathbf{D}_{chrom}\mathbf{x}_{chrom}||^{2}_{2}+\lambda_{2}\mathcal{J}(\mathbf{x}_{chrom}),$$

(6)

where Eq. (5) represents luminance degradation model and Eq. (6) refers to chrominance degradation model. They can be parallelly updated in luminance and chrominance spaces.

Taking Eq. (5) as example, it can be solved via proximal gradient algorithm [39, 6], which has been demonstrated its effectiveness in many inverse problems. Specifically, we can alternatively deduce the following updating rule to obtain an approximate solution:

	$\displaystyle u^{k}=\mathbf{x}^{k-1}_{lum}-\rho^{k}\mathbf{D}_{lum}^{T}(\mathbf{D}_{lum}\mathbf{x}_{lum}^{k-1}-\mathbf{y}_{lum}),$		(7a)
	$\displaystyle\mathbf{x}^{k}_{lum}=Prox_{\lambda_{1}}(u^{k}),$		(7b)

where $\mathbf{x}^{k}_{lum}$ is the $k$-th updating solution, $\rho^{k}$ denotes the updating stepsize, $\mathbf{D}_{lum}^{T}$ is the transpose of $\mathbf{D}_{lum}$, $Prox_{\lambda_{1}}$ represents the proximal operator. Generally, the first item is called as the gradient descent and the second item is thought as the proximal mapping.

Similarly, chrominance degradation model is also solved by:

	$\displaystyle v^{k}=\mathbf{x}^{k-1}_{chrom}-\eta^{k}\mathbf{D}_{chrom}^{T}(\mathbf{D}_{chrom}\mathbf{x}_{chrom}^{k-1}-\mathbf{y}_{chrom}),$		(8a)
	$\displaystyle\mathbf{x}^{k}_{chrom}=Prox_{\lambda_{2}}(v^{k}).$		(8b)

Finally, we can obtain the output results during $k$-th updating process via:

$$\mathbf{x}^{k}=[\mathbf{H}^{T}_{lum}(\mathbf{x}^{k}_{lum});\mathbf{H}^{T}_{chrom}(\mathbf{x}^{k}_{chrom})],$$

(9)

where $\mathbf{H}^{T}_{lum}(\cdot)$ and $\mathbf{H}^{T}_{chrom}(\cdot)$ are the inverse transformation of luminance and chrominance transform. $[\cdot;\cdot]$ represents the space merging operation.

Based on above designed iteration optimization solution, we unfold each iterative step into corresponding modules to construct our DASUNet, which is shown in Fig. 2. Specifically, our DASUNet is composed of $k$ stages, each of which contains two parts, luminance optimization stream (LOS) and chrominance optimization stream (COS). Furthermore, we design a space aggregation module (SAM) to combine the output of each stage of COS and LOS, which can interact the complementary features learned from luminance and chrominance spaces to produce more high-quality normal-light restored images.

Luminance Optimization Stream. LOS is composed of gradient descent modules (GDM) and prior modeling modules (PMM), which are unfolded from Eq. (7). Taking $k$-th stage for example, GDM adopts low-light luminance component $\mathbf{y}_{lum}$ and the $(k-1)$-th restored result $\mathbf{x}_{lum}^{k-1}$ as inputs, which is depicted in Fig. 4(a). A non-trivial problem is the construction of the degradation operator $\mathbf{D}_{lum}$ and its transpose $\mathbf{D}_{lum}^{T}$. Inspired by [39, 6], we employ a residual convolution block to simulate $\mathbf{D}_{lum}$ and $\mathbf{D}_{lum}^{T}$. Their structure all adopt Conv-PReLU-Conv, and the channel numbers of two Convs are set to $n$ and 1. For $\rho^{k-1}$, it is a learnable parameter that is updated in each epoch and initialized to $0.5$.

Another important module is PMM, which is also thought as a denoising network in many works [10, 69]. Previous methods usually exploit convolution block, e.g., U-net and Resblock, to learn a universal image prior driven by tremendous specified-type images. However, they only extract local image prior via stacking more convolutions but cannot learn more representative long-range image structure. Recently, Transformer has been demonstrated its superiority for long-range information modeling in many vision tasks [12, 31, 68, 4]. Motivated by them, we design a novel Transformer-based PMM to model long-range structural knowledge, as shown in Fig. 5(b). Moreover, some researchers [44, 60] found the information flow among adjacent stages limits the performance of deep unfolding networks and propose a memory-augmented mechanism to facilitate the information flow across stages. Consequently, we build a high-way feature transmission path among adjacent stages without any convolution to aggravate the model efficiency, as shown in Fig. 2.

Specifically, the input is first fed into a convolution with kernel size of $3$ and a channel attention block (CAB) [72] to extract shallow features. Then, two $3\times 3$ convolutions with $n$ channels are used to aggregate the shallow features and high-way features from previous stage. Next, two Transformer blocks are used to learn long-range prior features, each of which includes a self-attention layer and a feed-forward layer. Finally, the learned features are sent into high-way features transmission path to flow into the next stage and SAM to produce the restored results of this stage. Overall, LOS can be described as:

	$\displaystyle u^{k}=\text{GDM}_{\lambda_{1}}(\mathbf{x}_{lum},\mathbf{x}^{k-1}_{lum}),$		(10a)
	$\displaystyle\mathbf{x}^{k}_{lum},\mathbf{F}^{k}_{lum}=\text{PMM}_{\lambda_{1}}(u^{k},\mathbf{F}^{k-1}_{lum}).$		(10b)

In PMM of LOS, we propose a novel luminance adjustment Transformer (LAT) to replace the Transformer block of PMM. LAT is shown in Fig. 4(e). Considering the local smoothness of brightness, we construct a local brightness difference via the pooling layer and pixel-wise subtraction. Then, we learn a brightness scale factor $\alpha$ exploiting a convolution layer to adjust the brightness. Finally, we use a convolution to obtain the final luminance prior information.

Chrominance Optimization Stream. COS has the same structure as LOS except for PMM. In COS, we customized a new self-attention layer, Wavelet decomposition Transformer, based on the characteristic of chrominance channels of low-light images, which is shown in Fig. 4(d). The input features are split into low-frequency part and high-frequency part via Wavelet filters. Low-frequency part is inputted into a Transformer block to extract long-range degradation knowledge. Since high-frequency part is sparse, it passed into a convolution block to learn local degradation prior. The design can facilitate the modeling of detail prior and reduce the computation overhead.

Similarly, COS is represented as:

	$\displaystyle v^{k}=\text{GDM}_{\lambda_{2}}(\mathbf{x}_{chrom},\mathbf{x}^{k-1}_{chrom}),$		(11a)
	$\displaystyle\mathbf{x}^{k}_{chrom},\mathbf{F}^{k}_{chrom}=\text{PMM}_{\lambda_{2}}(v^{k},\mathbf{F}^{k-1}_{chrom}).$		(11b)

Space Aggregation Module. SAM is proposed to combine luminance prior and chrominance prior based on Equation (10). It takes the outputs of LOS and COS as input to produce the enhanced normal-light image. Specifically, it firstly concatenate $\mathbf{x}^{k}_{lum}$ and $\mathbf{x}^{k}_{chrom}$. Then, a $3\times 3$ convolution with $n$ channels is used to combine them. After that, a CAB is added to emphasize the significant features. Finally, we yield the enhanced images in this stage via a $3\times 3$ convolution with $3$ channels. SAM is shown in Fig. 5(c) and can be written as below:

$$\mathbf{x}^{k}=\text{SAM}(\mathbf{x}^{k}_{lum},\mathbf{x}^{k}_{chrom}).$$

(12)

Given $N$ pairs of low/normal-light images $\{\mathbf{y}_{i},\widetilde{\mathbf{x}}_{i}\},i=[1,2,\cdots,N]$, we can train our proposed DASUNet via the following loss function:

$$\mathfrak{L}=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{k}w^{j}\mathbb{C}(\mathbf{x}_{i}^{j},\widetilde{\mathbf{x}}_{i}),$$

(13)

where $k$ is the stage number, $w^{j}$ is the weighting parameter of the $j$-th stage, $\mathbb{C}(\cdot,\cdot)$ indicates the Charbonnier loss, and $\mathbf{x}_{i}^{j}$ denotes the enhanced image of j-th stage of DASUNet. In our training, we set $w^{k}$ = 1 and other $w^{j}$ = 0.1.

Our DASUNet is conducted in PyTorch platform with an RTX 3090 GPU. The channel number $n$ for all module (without clear statement) is set to $64$ and the stage number $k$ is initialized as $4$ in our experiments. Adam optimizer with default setting ($\beta_{1}=0.9$ and $\beta_{2}=0.99$) is used as the training solver of DASUNet. The initial learning rate is specified as $2\times 10^{-4}$ and gradually decreases to $1\times 10^{-6}$ during $500$ epochs via the cosine annealing strategy. Moreover, we also use the warming up strategy ($3$ epochs) to steady the training of our DASUNet. Training samples are randomly cropped with size of $256\times 256$ and then rotated and flipped for data augmentation. For convenience, images are firstly transformed to YCbCr space for training or inference and are finally re-transformed back to RGB domain for impartial evaluation.

We evaluate the effectiveness of our DASUNet on two popular benchmarks: LOL  [55] and MIT-Adobe FiveK  [2]. LOL dataset is comprised of $500$ paired of low/normal-light images, in which $485$ pairs are used for training and the rest is used for test. MIT-Adobe FiveK dataset contains $5000$ low-light images from various real scenes. Like other methods, we use the images retouched by expert-C as reference images. The first $4500$ pairs images are set as the training set and the last $500$ pairs images are selected as test set. Three full-reference evaluators, PSNR, SSIM, and LPIPS  [71], are selected as objective metrics. More datasets, including LOL-v2 and some unpaired datasets, and their results are shown in Supplementary Material.

To evaluate the performance of our proposed DASUNet, we compare our results with seventeen classical low-light image enhancement methods which include four traditional models (AGCWD [22], SRIE [15], LIME [19], and ROPE [57]), two unsupervised methods (Zero-DCE [17] and EnlightenGAN [23]), five recent deep black-box methods (MIRNet [64], Bread [18], LLFormer [53], UHDFour [26], and LLDiffusion [52]) and four deep Retinex-based methods (RetinexNet [55], KinD [73], KinD++ [74], URetinex [56], and RetinexFormer [5]), on LOL dataset. Quantitative comparison results are summarized in Table 1. It can easily observe that our method outperforms other comparison methods in PSNR and SSIM and achieves the comparative results in LPIPS evaluator. Qualitative results are presented in Fig. 5. As can be seen, RetinexNet cannot achieve expected enhancement images and KinD produces over-smoothing restored images. URetinex shows the color deviation on the ground. LLFormer and RetinexFormer seem to be under-enhanced and noisy. However, the enhanced result by our DASUNet shows more vivid details and more natural scene light, which remove some artifacts and noise that cannot be eliminated by other methods.

Moreover, we conduct experiments on MIT-Adobe FiveK dataset to demonstrate the generalization capability of our proposed DASUNet. We compare our model with eight state-of-the-art methods including SRIE [15], LIME [19], IAT [8], SCI [36], DSN [76], DSLR [29], CSRNet [20], and MIRNet [64] in Table 2. As tabulated in Table 2, our method significantly surpasses the second-best methods, MIRNet, in terms of PSNR, SSIM, and LPIPS. To show the effectiveness of our DASUNet, visual comparisons are manifested in Fig. 6. One can observe that SRIE and DSN connot achieve favorable enhancements. CSRNet overenhances the low-light images. MIRNet encounters ring artifacts in the second row of image. Overall, our DASUNet produces impressive enhanced images.

Dual degradation model. Based on the degradation specificity between luminance and chrominance spaces, we proposed a DDM for low-light image enhancement. To demonstrate its effectiveness, we conduct some comparison experiments on various color spaces and degradation models on LOL dataset, the results of which are presented in Table 3. Single model and triple model denotes one degradation model and three degradation models on corresponding sapces. One can see from Table 3 that the design philosophy behind DDM is effective. Single model cannot consider the the degradation specificity on different spaces and triple model could lose the mutual benefits between homogeneous degradation spaces. Visual comparison of different degradation models are illustrated in Fig. 7(a). Single model introduces some visual artifacts and triple model yields some blurs, while our model produce clearer result.

Stage number. It is well-known that the stage number $k$ is essential for iteration optimization solution. We explore the performance difference of different stage numbers and experimental results are delineated in Fig. 8. When $k$ exceeds $4$, the enhancement performance reaches convergence. Consequently, the stage number in our experiments is set to $4$. Visual comparsion are shown in Fig. 7(b). One can see clearer results with stage advancement.

We report the average running time of our method and recent proposed deep learning-based methods on LOL dataset in Fig. 9. Also, we list model parameters of all test methods in Fig. 9. Although LLFormer and MIRNet can achieve noticeable performance, they are heavy in model size. URetinex and Zero-DCE show compact model design, but their performance is poorer than ours. One can see that our method achieves the best performance with an acceptable running time and a relatively economic model size. It embodies a favorable trade-off between the effectiveness and efficiency of our proposed model.

LOL-V2 dataset  [61] contains 789 pairs of normal/low-light images, in which 689 pairs are used to train the enhancement model and 100 pairs are used as testset. MEF  [35] and DICM  [25] are two unpaired datasets without reference images. They are respectively composed of 17 and 69 images, which usually are used to evaluate the generalization ability of enhancement algorithms. For unpaired datasets, we introduce a no-reference evaluator NIQE  [38] to assess their performance.

In PMM, we design new luminance adjustment Transformer (LAT) and Wavelet decomposition Transformer (WDT) based on different characteristics of different spaces. To analyze their effects, we perform a set of ablation experiments via breaking them down. As listed in Table 1, the performance will be enhanced by 1.57 dB and 1.72 dB compared to the baseline when adding LAT and WDT. Finally, LAT and WDT jointly achieve the best performance.

Besides, we study the effect of SAM and high-way feature path. As shown in Table 2, the performance will be heavily decreased when deleting SAM and high-way feature path. It demonstrates the importance of SAM and high-way feature path for space prior combination and feature flow.

We compare our results with seven methods, including SRIE  [15], LIME  [19], EnlightenGAN  [23], Zero-DCE  [17], URetinex  [56], SNRANet  [59], UHDFour  [26], RetinexFormer  [5], LLDiffusion  [52], and ACCA  [78], on LOL-V2 dataset. As shown in Table 3, we outperform other comparison methods in PSNR and SSIM. Visual comparisons are shown in Fig. 1. As shown in Fig. 1, LIME and URetinex over-enhance the low-light images. Zero-DCE produces under-enhanced results. UHDFour encounters the color cast in the first-line image. Our method yields visual-pleased results. It demonstrates the advantages of WDT and LAT.

Besides, we compare our results with SRIE  [15], Zero-DCE  [17], RetinexNet  [55], KinD  [73], MIRNet  [64], Bread  [18], and LLFormer  [53] on MEF and DICM datasets. As shown in Table 4, we obtain the best performance on two datasets in terms of NIQE. Visual comparisons are shown in Fig. 2 and 3. Retinex produces unnatural looks, and KinD achieves under-enhanced results. While MIRNet over-enhances low-light images. LLFormer produces checkboard artifacts. It can be observed that our method yields impressive enhancement images.