DarkVisionNet: Low-Light Imaging via RGB-NIR Fusion
with Deep Inconsistency Prior

As previously described, the network needs to be aware of the inconsistent regions on the two inputs. We design an intuitive function to measure the inconsistency from image features. Firstly binary edge maps are extracted from each feature channel. Then the inconsistency is defined as

	$\displaystyle\mathcal{F}(edge^{C_{:}},edge^{N})$	$\displaystyle=\lambda(1-edge^{C_{:}})(1-edge^{N})$
		$\displaystyle\quad+edge^{C_{:}}\cdot edge^{N}$		(1)

where $C_{:}\in\mathbb{R}^{H\times W}$ and $N\in\mathbb{R}^{H\times W}$ denote R/G/B channel of the clean RGB image and NIR image, $edge^{C_{:}}$ and $edge^{N}$ respectively represent the binarized edge maps of $C_{:}$ and $N$, which is obtained by binarizing its mean value as a threshold after Sobel filtering.

As shown in Figure 3, $\mathcal{F}(\cdot,\cdot)$ equals to $0$ in the regions where $edge^{C_{:}}$ and $edge^{N}$ shows severe inconsistency. On the contrary, $\mathcal{F}(\cdot,\cdot)$ equals to $1$ in the regions where the structures of RGB and NIR are consistent. And in other regions, $\mathcal{F}(\cdot,\cdot)$ is set to a hyperparameter $\lambda(0<\lambda<1)$, indicating that there is no significant inconsistency. Utilising the output inconsistency map of $\mathcal{F}$, the inconsistent NIR structures can be easily suppressed by a direct multiplication.

Even though function $\mathcal{F}$ subtly describes the inconsistency between RGB and NIR images, it cannot be applied directly in extremely low light cases. As shown in Figure 4, the calculated inconsistency map contains nothing but non-informative noise when facing extremely noisy RGB image. To avoid the influence of noise in the structure inconsistency extraction, we propose the Deep Structure Extraction Module (DSEM) and Deep Inconsistency Prior (DIP), where we compute the structure inconsistency in feature space. Considering the processing flow of RGB and NIR are basically the same, we give a unified description here to keep the symbols concise.

The detailed architecture of DSEM is shown in Figure 5(a). DSEM takes multi-scale features $feat_{i}$ ($i$ represents scale) from restoration network $R$ and outputs multi-scale deep structure maps $struct_{i}$. In order for DSEM to predict high-quality deep structure maps, we introduce a clear supervision signal $struct_{i}^{gt}$ (addressed later) for DSEM and the training loss is calculated as:

$\displaystyle\mathcal{L}_{stru}=\sum_{i=1}^{3}\sum_{c=1}^{Ch_{i}}Dist(struct_{i,c},struct_{i,c}^{gt}),$

(2)

where $Ch_{i}$ is the channel number of the deep structures in the $i$th scale, $Dist$ is Dice Loss (Deng et al. 2018), $struct_{i,c}$ is the $c$th channel of the predicted deep structures in the $i$th scale and $struct_{i,c}^{gt}$ is the corresponding ground-truth. The Dice loss is given by $Dist(P,G)=(\sum_{j}^{N}p_{j}^{2}+\sum_{j}^{N}g_{j}^{2})/(2\sum_{j}^{N}p_{j}g_{j}),$ where $p_{j}$, $g_{j}$ is the value of the $j$th pixel on the predicted structure map $P$ and ground-truth $G$.

The extracted deep structures contain rich structure information and are robust to noise. With $struct_{i}$ of the noisy RGB and NIR images, we can introduce inconsistency function $\mathcal{F}$ to obtain high-quality knowledge of structure inconsistency:

$\displaystyle M^{DIP}_{i,c}$

$\displaystyle=\mathcal{F}(struct_{i,c}^{C},struct_{i,c}^{N})$

(4)

where $C\in\mathbb{R}^{H\times W\times 3}$ and $N\in\mathbb{R}^{H\times W}$ denote the noisy RGB image and NIR image, $struct_{i,c}$ is the $c$th channel of the features from the $i$th scale and $M^{DIP}_{i,c}$ is the corresponding inconsistency measurement. Since $M^{DIP}_{i,c}$ represents the structure inconsistency instead of intensity inconsistency, we apply $M^{DIP}_{i,c}$ directly to $struct_{i,c}^{N}$ instead of $feat^{N}_{i,c}$ in the form of:

$$\hat{struct_{i,c}^{N}}=M^{DIP}_{i,c}\cdot struct_{i,c}^{N}.$$

(5)

Under the guidance of the DIP, $\hat{struct_{i,c}^{N}}$ discards the structures that are inconsistent with RGB, thus empowering the deep features with prior knowledge to tackle structure inconsistency. As we will show in the experiments later, inconsistent structures in the NIR structure maps can be significantly suppressed after multiplying with $M^{DIP}_{i,c}$.

To further fuse the rich details contained in $\hat{struct_{i,c}^{N}}$ into RGB features, we designed a multi-scale fusion module as shown in Figure 5(c). As pointed out in (Jung, Zhou, and Feng 2020), denoising first and fusion later can improve the fusion quality. So we follow (Jung, Zhou, and Feng 2020) to reuse the denoised output of the restoration network $R$ set up for noisy RGB as the input of the multi-scale fusion module.

The total loss function we used is formulated as:

$$\mathcal{L}=\mathcal{L}_{rec}^{C}+\mathcal{L}_{rec}^{\hat{C}}+\mathcal{L}_{rec}^{N}+\lambda_{1}\cdot\mathcal{L}_{stru}^{C}+\lambda_{2}\cdot\mathcal{L}_{stru}^{N}$$

(6)

where $\mathcal{L}_{stru}^{C}$ and $\mathcal{L}_{stru}^{N}$ are the loss function for RGB/NIR deep structures prediction, which is described above. $\lambda_{1}$ and $\lambda_{2}$ are the corresponding coefficients and set to $1/1000$ and $1/3000$. $\mathcal{L}_{rec}^{C}$, $\mathcal{L}_{rec}^{\hat{C}}$ and $\mathcal{L}_{rec}^{N}$ represent the reconstruction loss of fused-RGB/coarse-RGB/NIR image respectively. All of them are Charbonnier loss (Charbonnier et al. 1994) in the form of:

$\displaystyle\mathcal{L}_{rec}$

$\displaystyle=\sqrt{\left\|\mathbf{X}-\mathbf{X}_{gt}\right\|^{2}+\varepsilon^{2}}$

(7)

where $\mathbf{X}$ and $\mathbf{X}_{gt}$ represent the network output and the corresponding supervision. The constant $\varepsilon$ is set to $10^{-3}$ empirically.

All experiments are conducted on a device equipped with two 2080-Ti GPUs. We train the proposed DVN from scratch in an end-to-end fashion. Batchsize is set to 16. Training images are randomly cropped in the size of 128*128, and the value range is [0, 1]. We augment the training data following MPRNet (Zamir et al. 2021), including random flipping and rotating. Adam optimizer with momentum terms (0.9, 0.999) is applied for optimization. The whole network is trained for 80 epochs, and the learning rate is gradually reduced from 2e-4 to 1e-6. $\lambda$ in function $\mathcal{F}$ is set to 0.5 for all configurations. The AutoEncoder network used to provide supervision signals for DSEM is pretrained in the same way, except that it only trained for 5 epoches and the input is clean RGB and NIR images separately. See supplementary material for more training details.

The synthesis of low-light data for training includes two steps. The first step is to reduce the average value of raw images taken under normal light to 5 (10-bit raw data). The second step is to add noise to the pseudo-dark raw images, including Gaussian noise with variance equals to $\sigma$, and Poisson noise with a level proportional to $\sigma$.

In this section, we verify that the proposed DIP is effective in handling the mentioned structure inconsistency. For comparison, we retrain a baseline, which is the same as the proposed DVN only without the DIP module. As Figure 8(a) shows, the NIR shadow of the grass still remains in the fusion result without DIP, but not in the fusion result with DIP. This directly proves that DIP can handle the structure inconsistency. Figure 8(b) shows that DIP can also deal with serious structure inconsistency caused by the misalignment between RGB-NIR images to a certain extent (this example pair cannot be aligned even after image registration). This has practical value because the problem of misalignment frequently occurs in applications. Taking into account the nature of DIP, the remaining artifacts are in line with expectations, since they are concentrated near the pixels with gradients in the RGB image.

In addition, Figure 8 also visualizes the deep structure of RGB, NIR, consistent NIR (DIP-weighted) as well as DIP Maps. It is obvious that even facing noisy input, the RGB deep structure still contains clear structures. The visual comparison between the NIR deep structure and the consistent NIR deep structure proves that the introduction of DIP can handle structure inconsistency in deep feature space.

We evaluate the effectiveness of each component in the proposed algorithm on the DVD benchmark quantitatively in this section. PSNR and SSIM are reported in Table 3. The baseline network directly fuse NIR features with RGB features (row 1 in Table 3).

Intermediate supervision $\mathcal{L}_{rec}^{\hat{C}}$ and $\mathcal{L}_{rec}^{N}$ effectively improve the performance as Table 3 (row 1 and 2) shows. This indicates the necessity of enhancing the noise suppression capability of the network for clean structure extraction.

Applying DSEM to learn deep structures without DIP can improve performance as well as Table 3 (row 1 and 3) shows. However, since the inconsistent structures are not removed, the benefits are not obvious, even if we use intermediate supervision and DSEM simultaneously as row 4 shows.

As Table 3 (row 5) shows, after introducing DIP to deal with the structure inconsistency, the network performance can be further improved by a large margin. This demonstrates the effectiveness of our proposed algorithm and the necessity to focus on the structure inconsistency problem on RGB-NIR fusion problem.