Nighttime Dehazing with a Synthetic Benchmark

Before synthesizing nighttime images, we must first understand real-world light colors. To this end, we collected over 1,300 real-world nighttime images from the internet using search terms such as “nighttime road”, ”street light”, $etc$. Some examples are shown in Figure 2. Generally, the dominant illumination is from artificial active lights. Therefore, we used them to calculate the prior distribution of light colors. First, they were resized into 100x100. Then, different scales of patches were densely sampled from them, ranging from 11x11 to 100x100 pixels. At each patch, we used MRP to estimate its color cast. The results are plotted as black scattered dots in Figure 2. Note that we assume the red channel is 1 since the light colors are biased to warm colors. At each scale, we calculated the mean (center) and the standard deviation of all samples as shown in the purple circles. We then used linear regression to fit these centers and obtain a prior equation of the light colors, $i.e.$,

Then, we shifted the line upwards and downwards according to the standard deviations to determine the boundaries, as shown by the red dotted lines; the region bounded by them covered 98.68% of samples. Next, we calculated the distribution of green values as shown in the green histograms. According to this distribution and Eq. (3), we randomly sampled light colors inside the boundaries, which are plotted as red dots and visualized in the upper left corner. They are visually realistic and mimic real-world light colors.

Different from (Zhang et al., 2017), where the illuminance is calculated from the light path distance via an exponential decay model, we follow the inverse-square law (Millerson, 2013, p. 26) and Lambert’s cosine law (Basri and Jacobs, 2003) to calculate the illuminance from the light path distance, incident light direction, and surface normal direction. Therefore, we should first reconstruct scene geometry by calculating the surface normals.

Scene reconstruction: Taking the Cityscapes dataset (Cordts et al., 2016) as an example, given a clear image $\bm{{\rm{R}}}$, semantic labels $C$, and depth map ${D}$, our target is to synthesize a realistic nighttime hazy image. First, we use SLIC (Achanta et al., 2012) to segment the semantic label into super-pixels. After calculating the world coordinates $\bm{{\rm{x_{i}}}}$ of pixels on each super-pixel $\bm{{\rm{z}}}$ based on the depth and camera parameters, a 3d plane is fitted to obtain the normal vector $\bm{{\rm{v\left(\bm{{\rm{z}}}\right)}}}$ and bias $m$ according to:

Ray simulation: Then, the illuminance reaching $\bm{{\rm{z}}}$ from the $k^{th}$ light can be calculated according to Lambert’s cosine law (Basri and Jacobs, 2003):

We use the bold $\bm{{\rm{{L_{k}}}}}$ to denote the light intensity $L_{k}$ and color $\bm{{\rm{{\eta_{k}}}}}$ together. $\beta_{l}$ is a parameter, $d\left(\bm{{\rm{z}}}\right)$ is the distance between the $k^{th}$ light source and $\bm{{\rm{z}}}$, and ${{\bm{{\rm{u_{k}}}}}\left(\bm{{\rm{z}}}\right)}$ is the incident light direction from the $k^{th}$ light to $\bm{{\rm{z}}}$. Then, the illuminance from all light sources are:

The illuminance map is further refined using the fast guided filter (He and Sun, 2015). In our experiments, we place virtual light sources along the roadsides according to the semantic labels $C$, which are 5 meters high and spaced every 30 meters. The light colors are randomly sampled as in Section 3.1.

Rendering: The haze absorbing and scattering effects rely on haze transmission, which can be calculated from the scene depth according to exponential decay model (Sakaridis et al., 2018):

where $\beta_{t}$ is the attenuation coefficient controlling the thickness of the haze. $d\left(\bm{{\rm{x}}}\right)$ is the scene depth. Finally, we render the haze effect to generate the nighttime haze image by integrating the illuminance and transmission into Eq. (1).

In this paper, we call the above method “3R” and summarize it in Algorithm 1. As can be seen from the walls, roads, and cars in Figure 3(c), the illuminance is more realistic than Figure 3(b), since 3R leverages the scene geometry and real-world light colors. The haze further reduces image contrast, especially in distant regions. It is noteworthy that we mask out the sky region due to its inaccurate depth values. 3R also generates some intermediate results such as $L$, $\bm{{\rm{\eta}}}$, $t$, which are worthy of further study, for example for learning disentangled representations.

Following (Sakaridis et al., 2018), 550 clear images were selected from Cityscapes (Cordts et al., 2016) to synthesize nighttime hazy images using 3R. We synthesized 5 images for each of them by changing the light positions and colors, resulting in a total of 2,750 images, called “Nighttime Hazy Cityscapes” (NHC). We also altered the haze density by setting $\beta_{t}$ to 0.005, 0.01, and 0.02, resulting in different datasets denoted NHC-L, NHC-M, and NHC-D, where “L”, “M”, and “D” represent light haze, medium haze, and dense haze. Further, we also modified the method in (Zhang et al., 2017) by changing the constant yellow light color with our randomly sampled real-world light colors described in Section 3.1 and synthesized images on the Middlebury (70 images) (Scharstein and Szeliski, 2003) and RESIDE (8,970 images) datasets (Li et al., 2018). Similar to NHC, we augmented the Middlebury dataset by 5 times, resulting in a total of 350 images. They are denoted NHM and NHR, respectively. The statistics of these datasets are summarized in Table 1 in Section 6.

Following (Zhang et al., 2017), we define the maximum reflectance $\bm{{\rm{M_{s}}}}$ of a daytime clear image patch $\bm{{\rm{R_{{\Omega_{s}}}}}}$ centered at $\bm{{\rm{x}}}$ as:

where $s$ is the scale of patch $\Omega_{s}$ and $\bm{{\rm{M_{s}}}}=\left[{{M_{sr}},{M_{sg}},{M_{sb}}}\right]$. In (Zhang et al., 2017), $M_{sc}$ is assumed to be close to 1 for every local patch, $i.e.$, the MRP. However, it may not hold for monochromatic areas such as lawn.

To migrate the issue, we calculate $\bm{{\rm{M_{s}}}}$ at multiple scales and determine the optimal scale for each pixel from a probability view. Specifically, we can treat the maximum reflectance as the probability of a surface patch that completely reflects incident light at all frequency range, $i.e.$,

For patches containing distinct colors or bright pixels, as shown in the blue rectangles in Figure 4(a), ${P_{s}}$ of a small patch is close to 1. For some monochromatic areas, $e.g.$, the left-most green rectangle region, it needs to enlarge the patch size to include more diverse pixels. However, a large patch means a bad localization when estimating the local color cast. To address this issue, we define an optimal scale that is sufficiently large but not necessary to be larger to obtain the highest ${P_{s}}$. Since ${P_{s}}$ is a monotonically increasing function of $s$, the optimal scale is defined as:

where $\mathcal{S}$ is the set of all scales. Figure 4(b) shows the optimal scale map of (a). The optimal scales are very small for pixels with distinct colors or bright pixels. By contrast, the optimal scale is very large for monochromatic pixels. Based on the above definition, we assume:

where $\mathcal{X}$ is the index set of all pixels. In this paper, we call it optimal-scale maximum reflectance prior (OS-MRP).

To validate this prior, we calculated the histograms of $P_{s^{*}}$ over 1,000,000 patches sampled from daytime clear images by referring to (He et al., 2009; Zhang et al., 2017). Ten scales ranging from 7x7 to 43x43 were used when calculating $P_{s^{*}}$. A fixed scale of 25x25 was used for DCP and MRP. The results are plotted in Figure 5. Compared with MRP, more patches have the maximum reflectance in our optimal-scale case, $i.e.$, a higher bar in the last range, and lower bars in the others. Further, the optimal scale is nearly uniformly distributed as shown in Figure 5(d), implying that OS-MRP does not obtain unfair advantages from larger patches over MRP.

There are two kinds of basic operation in OSFD: the patch-wise max, min, sum, and the pixel-wise sum, multiplication, division. To avoid the dense patch-wise max and min, we adopt an overlapping sliding window algorithm, reducing the complexity from $\mathcal{O}\left(r^{2}N\right)$ to $\mathcal{O}\left(N\right)$. $r$ is the patch radius. $N$ is the number of pixels. The stride of the sliding window is $r$. For the patch-wise sum, we follow (He and Sun, 2015) and use the summed-area table algorithm, reducing the complexity from $\mathcal{O}\left(r^{2}N\right)$ to $\mathcal{O}\left(N\right)$. Moreover, compared with the original guided filter (He et al., 2010), the computational complexity of the fast guided filter is reduced from $\mathcal{O}\left(N\right)$ to $\mathcal{O}\left({N\mathord{\left/{\vphantom{N{{d^{2}}}}}\right.\kern-1.2pt}{{d^{2}}}}\right)$, where $d$ is the sub-sampling ratio. Therefore, considering that $\bm{{\rm{\eta_{s}}}}$ is calculated at all $\left|S\right|$ scales, the total computational complexity of OSFD is $\mathcal{O}\left(\left|\mathcal{S}\right|N\right)$. Further, we down-sample the image $\bm{{\rm{I}}}$ by a ratio $k_{s}$ when calculating $\bm{{\rm{\eta_{s}}}}$, where ${k_{s}}\geq 1$ is the ratio of the patch radius between scale $s$ and the lowest scale. Instead of using a larger patch on $\bm{{\rm{I}}}$, we use a fixed-size patch on the down-sampled image $\bm{{\rm{I_{s}}}}$, which reduces the complexity of patch-wise max from $\mathcal{O}\left(N\right)$ to $\mathcal{O}\left({\frac{1}{{k_{s}^{2}}}N}\right)$. Although the total computational complexity is still $\mathcal{O}\left(\left|\mathcal{S}\right|N\right)$, it is faster.

In this paper, we devise a simple baseline model named ND-Net based on the deep convolutional neural network. Inspired by the recent success of the encoder-decoder structure in image restoration tasks including daytime dehazing (Zhang and Tao, 2020), we also adopt an encoder-decoder structure which consists of a MobileNet-v2 backbone as the encoder and a fully convolutional decoder. Considering that nighttime image dehazing tries to learn the inverse mapping of Eq. (1), which is less complex than large vision tasks, we use the MobileNet-v2 as the encoder backbone due to its computational efficiency and lightweight parameters. Since this part of the work is to provide a feasible learning-based solution and validate the usage of the proposed synthetic benchmark, we leave it as the future work to explore other backbones and devise specific dehazing blocks.

The decoder has five convolutional blocks and a convolutional prediction layer. Each convolutional block has two branches with the similar structure of the first block of each stage of the ResNet. One branch is for residual projection and has a convolutional layer followed by a Batch Normalization layer and a ReLU layer. The other branch has a bottleneck structure that consists of an 1*1 convolutional layer for feature dimension reduction, a 3*3 convolutional layer, and an 1*1 convolutional layer for feature dimension recovery. Each convolutional layer is followed by a Batch Normalization layer and a ReLU layer. After each block, we use bilinear interpolation to increase the feature resolution to its counterpart from each stage in the encoder and add them together via an element-wise sum. In other words, we add skip connections between corresponding encoder and decoder blocks to form a U-Net structure and leverage the encoded features at different levels and scales for decoding. The final prediction layer is an 1*1 convolutional layer without Batch Normalization and ReLU layers. We leverage the residual learning idea by adding a skip connection between the input hazy image and the prediction output via element-wise sum, making the encoder-decoder network to learn the haze residual rather than a clear image, which is more effective and easier to train.

We use the Mean Square Error (MSE) loss and the perceptual loss based on the pretrained VGG network (Simonyan and Zisserman, 2015; Johnson et al., 2016), $i.e.$,

where $\bm{{\rm{J_{GT}}}}$ is the ground truth haze-free image, ${vg{g_{l}}\left(\cdot\right)}$ denotes the feature map from the $l^{th}$ layer of the VGG network, ${\left\|\cdot\right\|_{2}}$ and ${\left\|\cdot\right\|_{1}}$ denote the L2 and L1 norms, respectively. The final training objective is a linear combination of both losses, $i.e.$,

where $\lambda$ is a loss weight.

Datasets and evaluation metrics: We used the synthetic datasets in Section 3.3 to benchmark SOTA methods and our OSFD. We also used the 150 real-world nighttime hazy images from (Zhang et al., 2017; Li et al., 2015) for subjective evaluation, denoted NHRW. We collected 1,500 real-world daytime clear images for color removal evaluation, denoted DCRW. The statistics of these datasets are listed in Table 1. PSNR, SSIM (Wang et al., 2004), and CIEDE2000 (Sharma et al., 2005) are used as the evaluation metrics.

Implementation details: We used 10 scales in OS-MRP, where the size of $\Omega_{s}$ ranged from 7x7 to 43x43. The size of $\Omega_{t}$ was set to 15x15. $\beta_{l}$ was set to 1. $\beta_{t}$ was set to 0.005, 0.1, and 0.2. $t_{0}$ was set to 0.1. The loss weight $\lambda$ was set to 0.01. OSFD was implemented in C++. 3R was implemented in MATLAB. ND-Net was implemented with PyTorch and trained on a single TITAN Tesla V100 GPU. We used the NHR dataset to train ND-Net since it contains diverse images. NHR was split into two disjoint parts, $i.e.$, 8073 images as the training set, and 897 images as the validation set. We also evaluated the model on other benchmark datasets listed in Table 1. We will release the source codes and datasets for reproducibility.

Evaluation on nighttime dehazing: We compared the proposed methods with nighttime dehazing methods including NDIM (Zhang et al., 2014), GS (Li et al., 2015), MRP and MRPF (Zhang et al., 2017) according to objective metrics and subjective inspection. The quantitative results on NHC-L, NHC-M, NHC-D, NHM, and NHR are listed in Table 2 and 3. As can be seen, with increasing haze density, it becomes difficult to remove the color cast and haze, and the performance of all the methods degrades. NDIM achieves the worst scores for all the metrics. From Figure 1(b) and Figure 6(b), we can see that NDIM tends to generate a bright dehazed image but with color artifacts and amplified noise. GS’s performance is much better than NDIM, $e.g.$, less haze left in the results. As can be seen from Figure 1(c) and Figure 6(c), GS removes the glows and generates clear images. However, it also produces artifacts around light sources and distinct edges with sharp intensity discontinuity.

The proposed OSFD achieves the best scores among all the prior-based methods, demonstrating that it can remove haze, recover details, and preserve local structures efficiently (see red rectangle regions). It outperforms MRP by a margin of about 0.05 SSIM score on NHC-L, NHC-M, and NHC-D, validating the superiority of OS-MRP over MRP. As can be seen from Figure 1(d)-(e) and Figure 6(d)-(e), MRP fails with the monochromatic road, wall, light source, and lawn regions, leading to whitish results due to the inaccurate color cast estimate biased towards the intrinsic colors of images. On the contrary, the proposed OS-MRP works well, and leads to more visually realistic results. As for the proposed CNN-based baseline ND-Net, it achieved the best scores in terms of all metrics. Note that ND-Net was trained on the NHR dataset and cross-validated on the NHC ones. These results in Table 2 confirms that the ND-Net trained on NHR has a good generalization ability. The dehazed results by ND-Net in Figure 1(f) and Figure 6(f) are more visual pleasing than others, $e.g.$, less color, illumination, and noise artifacts. Nevertheless, there is still some residual haze in the results, especially in the light source regions, which can be addressed in the future work by taking the glow effect into account. Results on the NHM and NHR datasets are summarized in Table 3. We present more visual results in the supplementary materials.

Evaluation on color cast removal: We also evaluated the performance of GS, MRP, and OSFD on color cast removal. Note that for a clear image without a color cast, it is expected to estimate a white color cast and not change the original colors in the image. Therefore, we test the above methods on the DCRW dataset to see how well they retain the colors. The results are summarized in Table 4. OSFD achieves the best performance on both datasets. It benefits from the proposed OS-MRP, which can adapt to the diverse local statistics in natural images and select the optimal scale to estimate the color cast (see the road, lawn, wall, and light source regions in Figure 1 and Figure 6). Further, it is noteworthy that all these methods employ the DCP to estimate the transmission and remove haze. As noted above, this prior fails when there is a color cast. Therefore, better color removal performance matters for the subsequent dehazing stage.

We evaluated the run-time of the different methods (Zhang et al., 2014; Li et al., 2015; Zhang et al., 2017) on a PC with an Intel CORE i7 CPU and 16 Gb memory and ND-Net on a Tesla V100 GPU. The codes are from the original authors. NDIM (Zhang et al., 2014), GS (Li et al., 2015), MRPF (Zhang et al., 2017), MRP (Zhang et al., 2017), our OSFD and ND-Net process a 512x512 image in 5.63s, 22.52s, 0.236s, 1.769s, 0.576s, and 0.0074s, respectively. Although OSFD calculates $\bm{{\rm{\eta}}}$ at 10 scales, it is 3x faster than MRP and 10x faster than NDIM and GS. The efficiency of OSFD arises from its $\mathcal{O}\left(\left|\mathcal{S}\right|N\right)$ complexity and the acceleration tricks used in Section 4.3. Since the basic operations in OSFD are paralleled, it promises to be faster using GPU acceleration. The proposed ND-Net is the fastest one and runs at 135 frames per second (FPS), benefiting from its lightweight backbone and GPU acceleration. It is promising to be used for real-time applications.

Although the synthetic nighttime hazy images produced by 3R are visually realistic, they may contain artifacts due to inaccurate depth, i.e., the car boundaries in Figure 3. This can be addressed by using effective depth filtering/completion methods. Further, the light sources are set to isotropic point lights, which can be changed to other forms of light sources, $e.g.$, directional light, and volume light. Also, we can further add the glow images of virtual light sources into the final hazy image according to Eq. (3) in (Li et al., 2015) to simulate the glow effect. OSFD may generate results with color artifacts and residual haze due to the following limitations. 1) The statistical prior may not work in large monochromatic areas or dense haze regions with different color casts. 2) The optimal scale may be inaccurate if the initial dehazing stage fails. 3) Since dehazing is disentangled from color cast removal, it may be affected by the residual color cast. For example, see the trees and light sources in the first and fourth rows and the bluish distant areas in the last row in Figure 6.

Encouraged by recent progress in daytime dehazing, we are optimistic that deep models have the potential to address these limitations as evidenced by the proposed simple baseline ND-Net. There is still a large room for future studies. 3R generates a large amount of visually realistic training images with intermediate ground truth labels including low-light images, color cast, and illuminance, which can provide auxiliary supervision to train a better model. For example, devising disentangled generative models based on the explicit imaging model and intermediate labels is also very promising.

In this part, we present more visual results synthesized by the proposed 3R (see Section 3 in the paper for more details.). First, we compare the proposed 3R with the synthetic method in (Zhang et al., 2017) in Section 8.1.1. Then, we present some controllable synthetic results by changing the hyper-parameters in 3R, $e.g.$, $\beta_{t}$ for haze density, $\eta$ for light colors, and $\beta_{l}$ for illuminance intensity in Section 8.1.2. Finally, we also present some synthetic results on the Virtual KITTI dataset (Gaidon et al., 2016) in Section 8.1.3.