Progressive Depth Learning for Single Image Dehazing

As explained in Section 1, estimating the depth information and the transmission map are closely related tasks, and both are ill-posed problems. To address this problem, depth and transmission estimation network is designed as a circular and stage-wise cascade network composed of depth estimation module and transmission reconstruction module (with depth guidance) repeatedly, as shown in Fig. 3. The loss function for the whole framework is as follows:

where $L^{k}_{d}$ and $L^{k}_{t}$ denote the loss functions for the depth and transmission estimation in the $k$-th stage respectively, $K$ is the number of the total stages for estimation, $L_{a}$ denotes the loss function for the atmosphere light estimation and $L_{Dhaze}$ denotes the loss function for the dehazed image. Each term will be introduced in the following parts.

Depth estimation module in the first stage applies a simple but effective encoder-decoder architecture proposed by (Alhashim and Wonka, 2018) for preliminary depth estimation. The encoder is the DenseNet-169 network which encodes the input into a feature vector. The decoder consists of several blocks with $2x$ bilinear up-sampling layers and convolutional layers followed by LeakyRelu to restore the final depth map at half the input resolution.

Following (Alhashim and Wonka, 2018), a weighted combination loss is applied as the loss function for depth estimation in the first stage:

where $\hat{d}_{k}(x)$ is the reciprocal of the predicted depth in the $k$-th stage for pixel $x$ and $d$ denotes the reciprocal of the ground truth depth, $n$ is the total number of the depth image. Using the reciprocal makes the training numeric more stable and alleviates the problem that loss $L_{d}$ will be larger when the ground truth depth values are bigger. $L_{depth}$ denotes the pixel-wise difference between the $\hat{d}_{k}(x)$ and the ground truth $d(x)$, L1 loss can be applied as:

The second term $L_{grad}$ measures distortions of high-frequency details. The reconstruction errors of the image gradients in the horizontal and vertical directions are denoted as $g_{h}$ and $g_{v}$ respectively:

The third term applies Structural Similarity (SSIM) which is commonly used in the depth estimations to faithfully restore the structures. $L_{SSIM}(\hat{d}_{k}(x),d(x))$ is defined as :

If the depth information is obtained, Eq. 1 indicates only exponential calculation is needed to reconstruct the transmission. However, Eq. 1 is only an approximation to the complex dehazing problem. Besides, the depth estimation would contain errors. Thus, we use a CNN model to mimic the restoration process from the hazy images with the guidance of depth estimations.

A stage-wise training strategy and the loss function in Eq. 2 is utilized. It means when in the first stage, only the architecture in the first stage is constructed and trained for the dehazing process. To avoid interference of optimization for depth estimations, transmission and atmospheric light estimation modules are trained using Eq. 2, while the depth estimation module is fixed during training.

After the model learning in the first stage is finished, the depth estimation and the transmission reconstruction modules can be further built for more stages to refine the estimation progressively. The hazy image is further fused with transmission to be fed into the next depth estimation module. Depth estimation from hazy images can be further and largely improved by transmission guidance. There are multiple benefits to build this circular and stage-wise cascade network composed of depth estimation module and transmission reconstruction module (with depth guidance) repeatedly. Firstly, depth or transmission guidance can significantly reduce the difficulties of estimating the transmission or depth. With the aid of transmission, depth estimations in the following stages can be a much easier task, and the estimation model could be much simpler than the depth estimation model in the previous stage. In turn, the same situation applies to the transmission reconstructions. Secondly, a stage-wise cascade network introduces the depth or transmission in each stage to supervise learning. The supervision in the later stages would assist the training in the previous stages. Third, with a progressively refined guidance, the estimations can be further refined.

In our implementations, considering the trade-off between the performances and the computation burdens, only two stages are constructed, and the second stage only contains depth estimation. The second depth estimation module in our implementations applies the same structures as the encoder-decoder structure in the transmissions estimation module in the first stage. In the test phase, the depth estimation by the second stage would guide the transmission estimation and restore the dehazed images. The transmission estimation module is shared for the first and second stages in the testing phase, which further reduces the parameter numbers.

The experiments prove that the progressive depth learning strategy improves the depth estimations for the hazy images and largely improves the dehazing performances, benefiting from the more accurate depth and transmission estimations. In fact, since the haze provides some cues for the depth estimations, our depth estimation model with hazy images outperforms the state-of-the-art depth estimation model with hazy-free images.

With the estimated transmission and atmospheric light, hazy image $I^{\prime}(x)$ can be computed from the ground-truth haze-free images I(x) as Eq. 1. The L1 loss between estimated hazy image $I^{\prime}(x)$ and hazy input image $I(x)$ can be utilized to self-supervise the reconstruction loss.

The NYU-depth2 dataset (Silberman et al., 2012) is a widely applied indoor dataset for dehazing comparisons. Following the common setting (Alhashim and Wonka, 2018) of depth estimation task, the NYU-depth2 dataset splits into 120K training images and 654 testing images.

Following (Ren et al., 2016), three atmospheric light conditions $A\in[0.7,1.0]$ and the scattering coefficient $\beta\in[0.5,1.5]$ are randomly sampled for each image to generate the corresponding hazy images. 6000 haze-free images are randomly selected from the training images of the NYU-depth2 dataset (Silberman et al., 2012) to generate the 18000 hazy images as our training set TrainA. The images are resized to $512\times 512$, which is the same setting as DCPDN (Zhang and Patel, 2018) and the batchsize is set as 8. Note that the compared method DCPDN (Zhang and Patel, 2018) applies four atmospheric light conditions where $A\in[0.7,1.0]$ and $\beta\in[0.4,1.6]$. For a fair comparison, we follow the (Zhang and Patel, 2018) to synthesize the testing set. The 654 images from the testing data of the NYU-depth2 dataset are applied to form $654*4$ testing images denoted as TestA. Although some test settings are not met in our training, our model performs excellently in the test set and outperforms all the compared methods in Test A. As shown in Fig.4, DCPDN and our methods are among the best to restore the dehazed images, while the AOD-Net fails to remove the hazes, DCP makes the images dark, and GCANet changes the brightness of the images. DCPDN fails in some regions such as distant places of the house in the first line of Fig.4.

To demonstrate the generalization ability of our indoor model, the model trained by the NYU-depth2 dataset is applied on the Middleburry stereo dataset (Scharstein et al., 2014). 23 images with ground truths from the Middlebury stereo dataset are synthesized to form $23*4$ images as TestB following (Zhang and Patel, 2018). The quantitative comparisons on the TestB are reported in Table 2, which demonstrates that our model exhibits excellent generalization abilities and achieves favorable results on the Middlebury stereo dataset.

KITTI (Geiger et al., 2013) and KITTI 2015 (Menze and Geiger, 2015) are real-world datasets of street views from an autonomous driving platform, and sparse ground-truth depth information is obtained with LiDAR. For the real outdoor dataset, the performance of both dehazing and depth estimations is provided to demonstrate the effectiveness of our progressive depth estimation and image dehazing strategy. The depth estimation has largely improved the transmission estimations and dehazed results of the outdoor dataset.

To train the proposed network, 6000 images are randomly selected from the training set of the KITTI raw data. In addition, 697 test images of the KITTI raw data and KITTI 2015 are used to compare our algorithm against the state-of-the-art method (Song et al., 2020). Images are resized to $1280\times 384$ resolution for the training and testing. Although depth estimation by stereo matching is easier than single image depth estimation, it is surprising that our method outperforms (Song et al., 2020) in the depth estimations and dehazing tasks.

FRIDA 3 (Caraffa and Tarel, 2014) contains 66 foggy outdoor road scenes, which is synthetically generated clear/hazy pairs of stereo images (Caraffa and Tarel 2012). Note that (Song et al., 2020) adapted the model on the FRIDA 3 dataset, our approach adapted with the same settings is reported for FRIDA 3 which achieves the best performance.

Cityscapse is also tested for a random selected 200 images in table.5.

Our method achieves the best objective evaluation performances in vast majorities of the experiments and performs especially well in the outdoor dataset. From Fig.5 and Fig.6, compared with the groundtruth images, our methods has a great advantage for the distant areas with the thick hazes, since depth estimations has constrained the transmission estimation process. Almost all the compared methods either change the brightness or leave thick hazes.Our approach preserves the details and original brightness as well as remove the heavy hazes. Outdoor hazy images have larger distances and thicker hazes for which predicting transmissions are more challenging problem. Progressive depth learning and transmission estimation lead to better depth estimation and dehazing performances.

1) Removing haze first and then estimate the scene depth for the dehazed images (‘GCANet+DenseDepth’ and ‘DCPDN +DenseDepth’).

2) Fine-tuning the state-of-the-art depth estimation models for the hazy images (‘DenseDepth-FT’).

3)Depth estimation by the state-of-the-art models for the corresponding haze-free images (‘DenseDepth+haze-free’).

4)Depth estimation by our progressive depth learning approach (‘PDLD-d1’,‘PDLD-d2’), where ‘PDLD-d1’ and ‘PDLD-d2’ are estimated results by our depth estimation module in the first and second stages respectively.

As shown in Fig.2, removing haze first and then estimating depth or fine-tuning the state-of-the-art depth estimation models only marginal improves the performance of the depth estimation.

The error metrics in Table 6 are commonly applied for the depth estimation evaluations, which are threshold accuracy (${\delta}_{i}$), average relative error (rel), square relative error ($Sq.rel$), root mean squared error (rms), average $log_{10}$ error ($log_{10}$). Please refer to (Alhashim and Wonka, 2018) for the detail calculation formula. The $\uparrow$ indicates the larger the values, the better performance, while the $\downarrow$ is to the opposite.

In Fig. 2, average pixel-wise abs errors are investigated for the pixels whose depths are within certain distances range $[d-2,d]$ in the horizontal axis in KITTI dataset. DenseDepth+hazy-free denotes the DenseDepth model performs on the hazy-free images. The DenseDepth-FT denotes the model fine-tuned for hazy images, PDLD-d1 and PDLD-d2 denote the depth estimation modules in the first and second stages, respectively.

Observed from Table 6 and Fig.2, it is obvious that haze severely disturbs the depth estimation of the existing depth estimation models trained on the hazy-free images. Fine-tuning the existing models of haze-free image depth estimation or combinations of the dehazing and depth estimation sequentially improves the situations but is far from enough. Our progressive depth learning for the dehazing has largely improved the situations. Even the estimated depth in the first stage has outperformed all the above approaches. It is surprising that our progressively depth learning approach even outperforms depth estimations by the state-of-the-art models for the corresponding hazy-free images. It is obvious that our model captures the inner relationship of depth and transmission and hazes provide some cues for depth estimations. Simply combination of dehazing and depth estimation or simple adaptation like fine-tuning provide marginal help for the situations.

In this part, the depth estimations and dehazing results by each stage are compared on KITTI dataset. The baseline denotes our backbone model with only one stage. It is clear in table 7 that our progressive learning approach largely surpasses the dehazing and depth estimation performances of baseline model. It is interesting that with our progressive learning strategies, the depth estimation module in the first stage is greatly improved. The depth estimation module in the first stage benefits from the improved transmission estimation module and more supervision is back-propagated from the progressive learning process. Depth estimation by the second stage produces more accurate results than the first stage which proves the training progressively refine the estimations. Besides, guidance by more accurate depth estimations largely improve the transmission estimation and dehazing results which clearly proves the power of exploiting the relationship between dehazing and depth estimation. The dehazing and depth estimation forms a positive feedback and the progressive depth learning and dehazing approach improves each term iteratively.

To demonstrate the effectiveness on the real applications, the proposed approach is evaluated on real word hazy images provided by the existed work (Zhang and Patel, 2018). As revealed in Fig.7, method of DCP (He et al., 2010) has the problem in the regions where the light is very bright,such as sky. AOD-Net (Li et al., 2017) and GCANet (Chen et al., 2018) seems to make some regions dark and leave some hazes. The contrasts is too large in the second row for (Zhang and Patel, 2018). The hazy removal of (Zhang and Patel, 2018) is not uniform which looks unnatural. Our approach produces visually pleasing dehazed images.