Geometric-Aware Low-Light Image and Video Enhancement
via Depth Guidance

In recent years, there have been notable advancements in learning-based LLIE techniques [32, 24, 34, 8, 36, 37, 20, 10, 30, 7, 29], primarily emphasizing supervised approaches due to the availability of abundant image pairs for training. For example, MIRNet [31] adopts a multi-scale architecture to effectively capture and distill information at various levels. This results in improved image quality with enhanced brightness, contrast, and details, along with a reduction in noise. Fu et al. [5] implemented illumination augmentation on a pair of images, successfully achieving self-supervised Retinex decomposition learning. This innovative approach contributes to further advancements in LLIE techniques.

Beyond LLIE, there is a rising need for LLVE solutions. Danai et al. [14] proposed a data synthesis mechanism that generates dynamic video pairs static datasets, formulating a LLVE framework. Wang et al. [19] introduced a multi-task model capable of simultaneously estimating noise and illumination, particularly effective for videos with severe noise conditions. Xu et al. [26] innovatively designed a parametric 3D filter tailored for enhancing and sharpening low-light videos, contributing to the latest developments in video enhancement techniques.

While significant progress has been achieved in both LLIE and LLVE, there remains a challenge in obtaining accurate enhancement results in certain demanding scenarios. This difficulty arises from the highly ill-posed nature of directly recovering normal-light photos from these images. Consequently, the incorporation of suitable priors becomes imperative to address these challenges effectively.

Given the ill-posed nature of LLE, the integration of suitable priors is essential to achieve the desired enhancement results. Recent methods have proposed to enhance the corresponding effects by incorporating multi-modal maps as unified priors. For instance, SKF [23] utilizes semantic maps to optimize the feature space for low-light enhancement. SMG [27] adopts a generative framework that integrates edge information, enhancing the initial appearance modeling specifically designed for low-light scenarios.

Nevertheless, the existing priors primarily operate at the 2D level, potentially falling short of fully capturing the geometric intricacies of the entire 3D scene. Recognizing the importance of incorporating 3D geometric information, which plays a pivotal role in determining illumination distribution, becomes crucial for enhancing LLE tasks. The central focus of this paper lies in exploring methods to effectively acquire and utilize 3D priors in the context of LLE.

Depth estimation has been a long-standing task, and it has recently reached new heights with the development of large models. By training on extensive datasets that encompass diverse scenes and objects, current depth estimation models (e.g., DPT [13]) exhibit remarkable zero-shot performance on various images and videos. Consequently, we suggest distilling depth information from SOTA depth prediction networks and integrating it into the low-light enhancement task.

$\mathcal{F}$ takes the input of $I_{d}$ and output $\hat{d}_{d}$, as

$$\hat{d}_{d}=\mathcal{F}(I_{l};\theta_{f}),$$

(1)

where $\theta_{f}$ is the parameters to learn. The ground truth to train $\mathcal{F}$ is obtained as the output of a pre-trained open-world depth estimator $\mathcal{D}$ (the parameter of $\mathcal{D}$ is frozen in the training stage), as $d_{n}=\mathcal{D}(I_{n})$.

We combine the extracted features from $\mathcal{G}$ and $\mathcal{F}$ using a cross-attention module $\mathcal{C}$. This process refines the features in $\mathcal{G}$ by incorporating depth information from $\mathcal{F}$, ultimately enhancing the final results, denoted as $\hat{I}_{n}$, as

$$\hat{I}_{n}=\mathcal{G}(I_{d},\mathcal{F}_{e}(I_{l});\theta_{g}),$$

(2)

where $\mathcal{F}_{e}(I_{l})$ denotes the extracted feature with $\mathcal{F}$. The objective can be represented as

$$\hat{\theta}_{g},\hat{\theta}_{f}=argmin(\mathcal{L}(\hat{I}_{n},I_{n})+\mathcal{L}(\hat{d}_{d},d_{n})).$$

(3)

In the following sections, we will provide the details of feature fusion.

Our experiments have revealed that conducting the fusion is more appropriate in the encoder part, as opposed to the decoder part, as done in SKF [23]. This choice is driven by the fact that in the decoder part, the feature discrepancy tends to increase because these features are closer to the target outputs, which belong to different domains. To elaborate, the features in the decoder of $\mathcal{G}$ represent the 2D image content, while the features in $\mathcal{F}$ capture the 3D geometrical information. Hence, it is more suitable to perform the fusion in the encoder part of both $\mathcal{G}$ and $\mathcal{F}$.

Based on the analysis provided above, we have identified both the location and the specific features to be fused within HDGFFM. In the following sections, we will elaborate on the fusion method itself.

The fusion process is carried out through our proposed cross-attention mechanism, which will be elaborated on in the upcoming section.

In this paper, our approach involves directly guiding the depth-aware features into the pathways of $\mathcal{G}$. Additionally, we integrate the results of the cross-attention computation with the outputs of the self-attention computation, based on a feature correlation measurement.

The self-attention is first conducted for $f_{g,l}$, by computing the self-similarity matrix, as

$$A_{l}=Softmax(W_{q}(f_{g,l})\times W_{k}(f_{g,l})^{T}\times\tau),$$

(5)

where $W_{q}(f_{g,l})$ and $W_{k}(f_{g,l})$ has the shape of $C\times(H\times W)$, $T$ is the transpose operation, $\tau$ is the temperature value that is a hyper-parameter, $\times$ is the matrix multiplication operation, and $A_{l}$ has the shape of $C\times C$. The output of the self-attention is obtained as

$$f^{\prime}_{g,l}=A_{l}\times W_{v}(f_{g,l}).$$

(6)

Regarding cross-attention, the similarity matrix can be computed between the image and depth-aware features, and the final output can be determined based on this computed similarity matrix. The procedure can be written as

		$\displaystyle\hat{A}_{l}=Softmax(W_{q}(f_{g,l})\times\hat{W}_{k}(\hat{f}_{d,l})^{T}\times\tau),$		(7)
		$\displaystyle f^{\prime\prime}_{g,l}=\hat{A}_{l}\times\hat{W}_{v}(\hat{f}_{d,l}).$

As both $f_{g,l}$ and $\hat{f}{d,l}$ are derived from the encoder process of $I_{d}$ and exhibit homogeneity, as discussed in Sec. 3.2, $f^{\prime}{g,l}$ and $f^{\prime\prime}{g,l}$ also exhibit small heterogeneity between them. To further minimize the disparity between $f^{\prime}{g,l}$ and $f^{\prime\prime}_{g,l}$, we introduce a method to model the correlation between these two sets of features and utilize this correlation as a weighting factor for the fusion process. The correlation is obtained as

$$w_{l}=Sigmoid(\mathcal{O}(f^{\prime}_{g,l}\oplus f^{\prime\prime}_{g,l})),$$

(8)

where $\oplus$ is the channel concatenation operation. The fusion output can be written as

$$\bar{f}_{g,l}=w_{l}\cdot f^{\prime\prime}_{g,l}+f^{\prime}_{g,l}+f_{g,l},$$

(9)

where $\cdot$ is the element-wise multiplication. Moreover, the final outputs of HDGFFM is obtained with the attention outputs and the feed-forward network, as

$$F_{g,l}=f_{g,l+1}=\mathcal{H}(\bar{f}_{g,l})+f_{g,l},$$

(10)

where $\mathcal{H}$ denotes the feed-forward network.

In Eq. 3, two objectives are considered: the restoration loss for enhancement and the depth supervision loss to obtain accurate depth priors. The restoration loss, denoted as $\mathcal{L}_{g}$, can employ the same loss terms as those used for the target model $\mathcal{G}$. On the other hand, the depth supervision loss is implemented as the $L_{2}$ distance between the predicted depth values and the ground truth depth information, as

$$\mathcal{L}_{d}=\|\hat{d}_{d}-d_{n}\|_{2}.$$

(11)

The final loss to train our strategy can be written as

$$\mathcal{L}=\mathcal{L}_{g}+\lambda\mathcal{L}_{d},$$

(12)

where $\lambda$ is the loss weight, which is robust across different target scenarios. We guarantee that our code and models will be released upon the publication of this paper.

In this section, we perform various ablation studies on the SDSD datasets, encompassing both indoor and outdoor subsets, to assess the significance of different components in our framework. Specifically, we evaluate the key components in HDGFFM through three ablated cases: (i) The location where the geometric factor is added. (ii) The method of fusion between image features and depth-aware features. (iii) Different weights for depth loss. The details of each case are elaborated below.

• •

  “Ours with Decoder”. As indicated in Sec.3.1 and Eq.2, the fusion in our framework occurs in the encoder of the target network $\mathcal{G}$. This choice is made because both the image content features and depth-aware features are extracted from the low-light input image, fostering homogeneity and enhancing fusion effects. To investigate the impact of alternative fusion points, we aim to test whether fusing image content features and depth-aware features in the decoder of $\mathcal{G}$ yields comparable results.

• •

  “Ours w/o Corre.”. As highlighted in Eq.8, we employ a correlation computation method via Sigmoid before the fusion of image content features and depth-aware features. This approach is designed to further mitigate inhomogeneity between these features. In this ablation setting, we omit the correlation computation, i.e., remove the Sigmoid from Eq.8. Consequently, the weight $w_{l}$ can have unlimited score ranges in this scenario.

• •

  “Ours with Add”. In a straightforward fusion approach, the two features are directly added without considering correlation computations. This setting neglects the incorporation of correlation computations between the features.

• •

  “Ours with $\lambda\times 5$” and “Ours with $\lambda\div 5$”. As specified in Eq. 12, there is a hyper-parameter that governs the contribution of image reconstruction loss and depth prediction loss, denoted as $\lambda$. We aim to analyze the effects of varying the value of $\lambda$ on the final results.

As depicted in Tables 3 and 4, it is evident that the results of all ablation settings are inferior to our original implementation in the comparison section. In comparing “Ours” with “Ours with Decoder”, we validate the impracticality of fusing restored image content features and depth-aware features since the former is close to the normal-light output while the latter is close to the low-light inputs. Furthermore, comparing “Ours” with “Ours w/o Corre.” and ”Ours with Add” underscores the significance of our proposed “Correlation-based Cross Attention for Fusion”. Lastly, the consistent and even improved results across different values for $\lambda$ highlight the robustness of the hyperparameter choice in our framework. This emphasizes that researchers can choose optimal $\lambda$ values for different datasets.

To evaluate the effectiveness of our proposed HDGFFM in terms of subjective evaluation, we carried out a user study with 50 participants (using online questionnaires). Our objective is to verify the subjective quality of our approach compared with baselines, and the evaluation scenarios should cover across various datasets and frameworks.

For more details, we initially select $T$ videos/images randomly from each testing dataset for a given LLIE/LLVE approach, where $T=20$ for LLIE and $T=10$ for LLVE. Our user study employs an AB-test strategy, where each participant is presented with an image/video from the corresponding baseline and an image/video from our framework. Importantly, participants are unaware of which one is the baseline, as the order of our results and baseline results is randomized during each evaluation. During each evaluation, participants compare 5 pairs for a specific method on a given dataset. They have the option to choose which one is better or indicate that the two are the same. Participants are instructed to evaluate the results based on criteria such as natural color, minimal noise, realistic illumination, and stable temporal changes (for LLVE). Each participant is provided with an example. Each participant completes 15 tasks (10 methods × 5 pairs). It takes approximately 30 minutes for a participant to finish the user study.

The results of the user study are presented in Fig. 6. Clearly, the participants consistently preferred the results of our method. This indicates that our approach significantly improves the human subjective perception of existing LLIE and LLVE frameworks.