A Comprehensive Survey and Taxonomy on Single Image Dehazing Based on Deep Learning

This survey does not cover all themes of dehazing research. We focus our attention on deep learning based algorithms that employ monocular daytime images. This means that we will not discuss in detail about non-deep learning dehazing, underwater dehazing (Wang et al., 2021b, 2017; Li et al., 2016), video dehazing (Ren et al., 2018b), hyperspectral dehazing (Mehta et al., 2020), nighttime dehazing (Zhang et al., 2020a, 2017a, 2014), binocular dehazing (Pang et al., 2020), etc. Therefore, when we refer to “dehazing” in this paper, we usually mean deep learning based algorithms whose input data satisfies four conditions: single frame image, daytime, monocular, on the ground. In summary, there are three contributions to this survey as follows.

• •

  Commonly used physical models, datasets, network modules, loss functions and evaluation metrics are summarized.

• •

  A classification and introduction of supervised, semi-supervised, and unsupervised methods is presented.

• •

  According to the existing achievements and unsolved problems, the future research directions are prospected.

Section 2 introduces physical model ASM, synthetic & generated & real world datasets, loss functions, basic modules commonly used in dehazing networks and evaluation metrics for various algorithms. Section 3 provides a comprehensive discussion of supervised dehazing algorithms. A review of semi-supervised and unsupervised dehazing methods is in Section 4 and Section 5, respectively. Section 6 presents quantitative and qualitative experimental results for three categories of baseline algorithms. Section 7 discusses the open issues of dehazing research.

A critical challenge for a logical and comprehensive review of dehazing research is how to properly classify existing methods. In the classification of Table 1, there are several items that need to be pointed out as follows.

• •

  The DCP is validated on the dehazing task, and the inference process utilizes ASM. Therefore, those methods that utilize DCP are considered to be ASM-based.

• •

  This survey treat the knowledge distillation based supervised dehazing network as a supervised algorithm rather than a weakly supervised/semi-supervised algorithm.

• •

  Supervised dehazing methods using total variation loss or GAN loss are still classified as supervised.

Here we give the notational conventions for this survey. Unless otherwise specified, all symbols have the following meanings: $I(x)$ means hazy image; $J(x)$ denotes haze-free image. $t(x)$ stands for transmission map; $x$ in $I(x)$, $J(x)$ and $t(x)$ is pixel location. The subscript “$rec$” denotes “reconstructed”, such as $I_{rec}(x)$ is the reconstructed hazy image. The subscript “$pred$” refers to the prediction output. According to (Gandelsman et al., 2019), atmospheric light may be regarded as a constant $A$ or a non-uniform matrix $A(x)$. In this survey, atmospheric light is uniformly denoted as $A$. In addition, many papers give the proposed algorithm an abbreviated name, such as GFN (Ren et al., 2018a) (gated fusion network for single image dehazing). For readability, this survey uses abbreviations as references to these papers. For a small amount of papers that do not give a name for their algorithms, we designate the abbreviated name according to the title of the corresponding paper.

Haze is a natural phenomenon that can be approximately explained by ASM. McCartney (McCartney, 1976) first proposed the basic ASM to describe the principles of haze formation. Then, Narasimhan (Narasimhan and Nayar, 2003) and Nayar (Nayar and Narasimhan, 1999) extended and developed the ASM that is currently widely used. The ASM provides a reliable theoretical basis for the research of image dehazing. Its formula is

where $x$ is the pixel location and $A$ means the global atmospheric light. In different papers, $A$ may be referred to as airlight or ambient light. For ease of understanding, $A$ is noted as atmospheric light in this survey. For the dehazing methods based on ASM, $A$ is usually unknown. $I(x)$ stands for the hazy image and $J(x)$ denotes the clear scene image. For most dehazing models, $I(x)$ is the input and $J(x)$ is the desired output. The $t(x)$ means the medium transmission map which is defined as

where $\beta$ and $d(x)$ stands for the atmosphere scattering parameter and the depth of $I(x)$, respectively. Thus, the $t(x)$ is determined by $d(x)$, which can be used for the synthesis of hazy image. If the $t(x)$ and $A$ can be estimated, the haze-free image $J(x)$ can be obtained by the following formula:

The imaging principle of ASM is shown in Fig. 2. It can be seen that the light reaching the camera from the object is affected by the particles in the air. Some works use ASM to describe the formation process of haze, and the parameters included in the atmospheric scattering model are solved in an explicit or implicit way. As shown in Table 1, ASM has a profound impact on dehazing research, including supervised (Ren et al., 2016; Cai et al., 2016; Li et al., 2017a; Zhang and Patel, 2018), semi-supervised (Li et al., 2020a; Liu et al., 2021; Chen et al., 2021b), and unsupervised algorithms (Li et al., 2021a; Golts et al., 2020).

For computer vision tasks such as object detection, image segmentation, and image classification, accurate ground-truth labels can be obtained with careful annotation. However, sharp, accurate and pixel-wise labels (i.e., paired haze-free images) for hazy images in natural scenes are almost impossible to obtain. Currently, there are mainly two approaches for obtaining paired hazy and haze-free images. The first way is to obtain synthetic data with the help of ASM, such as the D-HAZY (Ancuti et al., 2016), HazeRD (Zhang et al., 2017b) and RESIDE (Li et al., 2019c). By selecting different parameters for ASM, researchers can easily obtain hazy images with different haze densities. Four components are needed to synthesize a hazy image: a clear image, a depth map $d(x)$ corresponding to the content of the clear image, atmospheric light $A$ and atmosphere scattering parameter $\beta$. Thus, we can divide the synthetic dataset into two stages. In the first stage, clear images and corresponding depth maps are needed to be collected in pairs. In order to ensure that the synthesized haze is as close as possible to the real world haze, the depth information must be sufficiently accurate. Fig. 3 shows the clear image and corresponding depth map in the NYU-Depth dataset (Silberman et al., 2012). In the second stage, atmospheric light $A$ and atmosphere scattering parameter $\beta$ are designated as fixed values or randomly selected. The commonly used D-HAZY (Ancuti et al., 2016) dataset is synthesized when both $A$ and $\beta$ are $1$. Several researches choose different $A$ and $\beta$ in order to increase the diversity of the synthesized images and thus improve the generalization ability of the trained model. For example, MSCNN (Ren et al., 2016) sets $A\in(0.7,1.0)$ and $\beta\in(0.5,1.5)$. Fig. 4 shows the corresponding hazy image when $\beta$ takes $6$ different values.

The second way is to generate the hazy image by using a haze generator, such as I-HAZE (Ancuti et al., 2018b), O-HAZE (Ancuti et al., 2018c), Dense-Haze (Ancuti et al., 2019a) and NH-HAZE (Ancuti et al., 2020a). The well-known competition New Trends in Image Restoration and Enhance (NTIRE 2018-2020) dehazing challenge  (Ancuti et al., 2018a, 2019b, 2020b) are based on these generated datasets. Fig. 5 shows four pairs of hazy and haze-free examples contained in the datasets simulated by the haze generator. The images in Fig. 5 (a) are from indoor scenes, while (b), (c) and (d) are all pictured at outdoor views. There are differences in the pattern of haze in these three outdoor datasets. The haze in (b) and (c) is evenly distributed throughout the entire image, while the haze in (d) are non-homogeneous in the whole scenes. In addition, the density of haze in (c) is significantly higher than that in (b) and (d). These datasets with different characteristics provide useful insights for the design of dehazing algorithms. For example, in order to remove the high density of haze in Dense-Haze, it is necessary to design dehazing models with stronger feature extraction and recovery capabilities.

The main advantage of synthetic and generated haze is that it alleviates the difficulty during data acquisition. However, the hazy images synthesized based on ASM or generated by haze generator cannot perfectly simulate the formation process of real world haze. Therefore, there is an inherent difference between the synthetic and real world data. Several researches have noticed the problems of artificial data and tried to construct real world datasets, such as MRFID (Liu et al., 2020a) and BeDDE (Zhao et al., 2020). However, due to the high costs and difficulties of data collection, the current real world datasets do not contain enough examples as the synthetic dataset, like RESIDE (Li et al., 2019c). To facilitate the comparison of different datasets, we summarize the characteristics of various datasets in Table 2.

CNNs are widely used in current deep learning based dehazing networks. Commonly adopted modules are standard convolution, dilated convolution, multi-scale fusion, feature pyramid, cross-layer connection and attention. Usually, multiple basic blocks are formed into a dehazing network. In order to facilitate the understanding of the principles of different dehazing algorithms, the basic blocks commonly used in network architectures are summarized as follows.

• •

  Standard convolution: It is shown that using standard convolution in a sequential connection way to build neural networks is effective. Therefore, standard convolution are often used in dehazing models (Li et al., 2017a; Ren et al., 2016; Sharma et al., 2020; Zhang et al., 2020e) together with other blocks.

• •

  Dilated convolution: Dilated convolution can increase the receptive field while keeping the size of the convolution kernel unchanged. Studies (Chen et al., 2019c; Zhang et al., 2020c; Zhang and He, 2020; Lee et al., 2020b; Yan et al., 2020) have shown that dilated convolution can improve the performance of global feature extraction. Moreover, fusing convolution layers with different dilation rates can extract features from different receptive fields.

• •

  Multi-scale fusion: CNNs with multi-scale convolution kernels have been proven effective in extracting features in a variety of visual tasks (Szegedy et al., 2015). By using convolution kernels at different scales and fusing the extracted feature together, dehazing methods (Wang et al., 2018a; Tang et al., 2019; Dudhane et al., 2019; Wang et al., 2020) have demonstrated that the fusion strategy can obtain the multi-scale details that are useful for image restoration. In the process of feature fusion, a common way is to spatially concatenate or add output features obtained by convolution kernels of different sizes.

• •

  Feature pyramid: In the research of digital image processing, the image pyramid can be used to obtain information of different resolutions. The dehazing network based on deep learning (Zhang et al., 2020e; Zhang and Patel, 2018; Zhang et al., 2018b; Singh et al., 2020; Zhao et al., 2021a; Yin et al., 2020; Chen et al., 2019a) uses this strategy in the middle layer of the network to extract multiple scales of space and channel information.

• •

  Cross-layer connection: In order to enhance the information exchange between different layers and improve the feature extraction ability of the network, cross-layer connections are often used in CNNs. There are mainly three types of cross-layer connections used in dehazing networks, which are residual connection (Zhang et al., 2020g; Qu et al., 2019; Hong et al., 2020; Liang et al., 2019; Chen et al., 2019d) proposed by ResNet (He et al., 2016), dense connection (Zhu et al., 2018; Zhang et al., 2022a; Dong et al., 2020a; Chen and Lai, 2019; Guo et al., 2019a; Li et al., 2019a) designed by DenseNet (Huang et al., 2017), and skip connection (Zhao et al., 2021a; Dudhane et al., 2019; Yang and Zhang, 2022; Lee et al., 2020b) inspired by U-Net (Ronneberger et al., 2015).

• •

  Attention in dehazing: The attention mechanism has been successfully applied in the research of natural language processing. Commonly used attention blocks in computer vision include channel attention and spatial attention. For the feature extraction and reconstruction process of 2D image, channel attention can emphasize the useful channels of the feature map. This unequal feature map processing strategy allows the model to focus more on effective feature information. The spatial attention mechanism focuses on the differences in the internal location regions of the feature map, such as the distribution of haze on the entire map. By embedding the attention module in the network, several dehazing methods (Liang et al., 2019; Chen et al., 2019d; Liu et al., 2019a; Qin et al., 2020; Yin et al., 2020; Lee et al., 2020b; Yan et al., 2020; Dong et al., 2020c; Yan et al., 2020; Zhang et al., 2020e; Yin et al., 2021; Metwaly et al., 2020; Wang et al., 2021d) have achieved excellent dehazing performance.

This section introduces the commonly adopted loss functions in supervised, semi-supervised and unsupervised dehazing models, which can be used for transmission map estimation, clear image prediction, hazy image reconstruction, atmospheric light regression, etc. Several algorithms use multiple losses in combination to obtain better dehazing performance. A detailed classification and summary of the loss functions used by different dehazing methods is presented in Table 3. In the loss function introduced below, $X$ and $Y$ denote the predicted value and ground truth value, respectively.

Due to the presence of haze, the saturation and contrast of the image are reduced, and color of the image is distorted by the uncertainty mode. To measure the difference between the dehazed image and the ground truth haze-free image, objective metrics are needed to evaluate the results obtained by various dehazing algorithms.

Most papers use Peak Signal-to-Noise Ratio (PSNR) (Huynh-Thu and Ghanbari, 2008) and SSIM (Wang et al., 2004) to evaluate the image quality after dehazing. The computation of PSNR needs to use the formula (11) to obtain the mean square error (MSE):

where $X$ and $Y$ respectively represent two images to be evaluated. $H$ and $W$ are their height and width, that is, the dimensionalities of $X$ and $Y$ should be strictly the same. The pixel position index of the image is represented by $i$ and $j$. Then, the PSNR can be obtained by logarithmic calculation as following:

where $N$ equals $8$ for $8$-bit images. SSIM is based on the correlation between human visual perception and structural information, and its formula is defined as following:

where $\mu_{x}$, $\mu_{y}$, $\sigma_{x}$ and $\sigma_{y}$ represent the mean and variance of $X$ and $Y$, respectively; $\sigma_{xy}$ is the covariance between two variables; $C_{1}$ and $C_{2}$ are constants used to ensure numerical stability.

Since haze can cause the color of scenes and objects to change, some works use CIEDE2000 (Sharma et al., 2005) as an assessment of the degree of color shift. PSNR, SSIM and CIEDE belong to full-reference evaluation metrics, which means that a clear image corresponding to a hazy image must be used as a reference. However, real world pairs of hazy and haze-free images are difficult to keep exactly the same in content. For example, the lighting and objects in the scene may change before and after the haze appears. Therefore, in order to maintain the accuracy of the evaluation process, it is necessary to synthesize the corresponding hazy image with a clear image (Min et al., 2019).

In addition to the full-reference metric PSNR, SSIM and CIEDE, the recent works (Li et al., 2020c, 2019c) utilizes the no-reference metrics SSEQ (Liu et al., 2014) and BLIINDS-II (Saad et al., 2012) to evaluate dehazed images without ground truth. No-reference metric is of crucial value for real world dehazing evaluation. Nevertheless, the evaluation of current dehazing algorithms are usually conducted on datasets with pairs of hazy and haze-free images. Since the full-reference metric is more suitable for paired datasets, it is more widely used than the no-reference metric.

According to the ASM, the dehazing process can be divided into three parts: transmission map estimation, atmospheric light prediction, and haze-free image recovery. MSCNN (Ren et al., 2016) proposes the following three steps for solving ASM: (1) use CNN to estimate the transmission map $t(x)$, (2) adopt statistical rules to predict atmospheric light $A$, and (3) solve $J(x)$ by $t(x)$ and $A$ jointly. MSCNN adopts a multi-scale convolutional model for transmission map estimation and optimizes it with L2 loss. In addition, $A$ can be obtained by selecting $0.1\%$ darkest pixels in $t(x)$ corresponding the one with the highest intensity in $I(x)$ (He et al., 2010). Thus the clear image $J(x)$ can be obtained by

Different papers may use different statistical priors to estimate $A$, but the strategies they use for dehazing are similar to MSCNN. ABC-Net (Wang et al., 2020) uses the max pooling operation to obtain maximum value from each channel of $I(x)$. SID-JPM (Huang et al., 2018) filters each channel of a RGB input image by a minimum filter kernel. Then the maximum value of each channel is used as the estimated $A$. LAPTN (Liu et al., 2018) also applies the minimum filter together with the maximum filter for the prediction of $A$. These methods generally do not require atmospheric light annotations, but need paired of hazy images and transmission maps.

Instead of using convolutional networks and statistical priors jointly to estimate the physical parameters of ASM, some works implement the prediction of physical parameters entirely through CNN. DCPDN (Zhang and Patel, 2018) estimates the transmission map through a pyramid densely connected encoder-decoder network, and uses a symmetric U-Net to predict the atmospheric light $A$. In order to improve the edge accuracy of the transmission map, DCPDN has designed a hybrid edge-preserving loss, which includes L2 loss, two-directional gradient loss, and feature edge loss.

DHD-Net (Xie et al., 2020) designs a segmentation-based haze density estimation algorithm, which can segment dense haze areas and divide the global atmosphere light $A$ candidate areas. HRGAN (Pang et al., 2018) utilizes a multi-scale fused dilated convolutional network to predict $t(x)$, and employs a single-layer convolutional model to estimate $A$. PMHLD (Chen et al., 2020) uses a patch map generator and refine the network for transmission map estimation, and utilizes VGG-16 for atmospheric light estimation. It is worth noting that if using regression training to obtain $A$, the ground truth labels are generally required.

ASM can be incorporated into CNN in a reformulated or embedded way. AOD-Net (Li et al., 2017a) founds that the end-to-end neural network can still be used to solve the ASM without directly using ground truth $t(x)$ and $A$. According to the original ASM, the expression of $J(x)$ is

AOD-Net proposes $K(x)$, which has no actual physical meaning, as an intermediate parameter describing $t(x)$ and $A$. $K(x)$ is defined as

where $b$ equals $1$. According to the ASM theory, $J(x)$ can be uniquely determined by $K(x)$ as

AOD-Net considers that the non-joint transmission map and atmospheric light prediction process may produce accumulated errors. Therefore, independent estimation of $K(x)$ can reduce the systematic error. FAMED-Net (Zhang and Tao, 2020) extends this formulation in a multi-scale framework and utilizes fully point-wise convolutions to achieve fast and accurate dehazing performance. DehazeGAN (Zhu et al., 2018) incorporates the idea of differentiable programming into the estimation process of $A$ and $t(x)$. Combined with a reformulated ASM, DehazeGAN also implements an end-to-end dehazing pipeline. PFDN (Dong and Pan, 2020) embeds ASM in the network design and proposes a feature dehazing unit, which removes haze in a well-designed feature space rather in the raw image space. It is instructive that ASM can still help the dehazing task for non-explicit parameter estimation.

Generative adversarial networks have an important impact on dehazing research. In general, supervised dehazing networks that rely on paired data can use adversarial loss as an auxiliary supervisory signal. Adversarial loss (Gui et al., 2022) can be seen as two parts: the training objective of the generator is to generate images that the discriminator considers to be real. The optimization purpose of the discriminator is to distinguish the generated image from the real image contained in the dataset as possible as it could. For dehazing task, the effect of the adversarial loss is to make the generated image closer to the real one, which is beneficial for the optimization of the haze-free $J(x)$ and transmission map $t(x)$ (Zhang et al., 2019a), as shown in Fig. 6.

Inspired by patchGAN (Isola et al., 2017), which can better preserve high-frequency information, DH-GAN (Sim et al., 2018), RI-GAN (Dudhane et al., 2019) and DehazingGAN (Zhang et al., 2020c) use $N\times N$ patches instead of a single value as the output of the discriminator. Several works explore joint training mechanisms of multiple discriminators, such as EPDN (Qu et al., 2019) and PGC-UNet (Zhao et al., 2021a). Discriminator $D_{1}$ is used to guide the generator on a fine scale, while discriminator $D_{2}$ helps the generator to produce a global realistic output on a coarse scale. In order to realize the joint training of the two discriminators, EPDN downsamples the input image of $D_{1}$ by a factor of $2$ as the input of $D_{2}$. The adversarial loss is

where the form of adversarial loss $\ell_{A}$ is the same as the single GAN. The exploration of GAN brings plug-and-play tools to supervised algorithms. Since the training of the discriminator can be done in an unsupervised manner, the quality of the dehazed images can be improved without the requirement of extra labels. However, the training of GAN sometimes suffers from instability and non-convergence, which may bring certain additional difficulties to the training of the dehazing network.

According to the scattering theory, the farther the scene is from the camera, the more aerosol particles pass through. This means that areas within a single hazy image that are farther from the camera have higher densities of haze. Therefore, LAP-Net (Li et al., 2019b) proposes that the algorithm should consider the difference in haze density inside the image. Through multi-stage joint training, LAP-Net implements an easy-to-hard model that focuses on a specific haze density level by a stage-wise loss

where $\mathcal{F}$ represents the transmission map prediction network with parameter $\theta^{s}$ in stage $s$. In the first stage, the transmission map prediction network is responsible for estimating the case with mild haze. In the second and subsequent stages, the prediction result of the previous stage and the hazy image are used as joint input for processing higher haze density.

The density of haze may be related to conditions such as temperature, wind, altitude, and humidity. Thus, the formation of haze should be space-variant and non-homogeneous. Based on this observation, HardGAN (Deng et al., 2020) argues that estimating the transmission map for dehazing may be inaccurate. By encoding the atmospheric brightness as $1\times 1\times 2$ matrix $\gamma_{i}^{G}\&\beta_{i}^{G}$ and pixel-wise spatial information as $H\times W\times 2$ matrix $\gamma_{i}^{L}\&\beta_{i}^{L}$ for $i$-th channel of the input $x$, HardGAN designs the control function of atmospheric brightness $G_{i}$ and spatial information $L_{i}$ as

where $\mu$ and $\sigma$ denote the mean and standard deviation for $x$, respectively. After obtaining $G_{i}$ and $L_{i}$, HardGAN uses a linear model to fuse them for recovering haze-free image

where $HA$ is calculated by the intermediate feature map of HardGAN via using instance normalization followed by a sigmoid layer, and $*$ denotes element-wise product.

DMMFD (Deng et al., 2019) designs a layer separation and fusion model for improving learning ability, including reformulated ASM, multiplication, addition, exponentiation and logarithmic decomposition:

where $R_{i}(x)$ stands for layers in the network; $A_{0}$ and $t_{0}(x)$ are the atmospheric light and transmission map estimated by the feature extraction network, respectively. $J_{1}{(x)}$, $J_{2}{(x)}$, $J_{3}{(x)}$, and $J_{4}{(x)}$ can be used as four independent haze-layer separation models. It is based on the assumption that the input hazy image $I(x)$ can be separated into a haze-free layer $J(x)$ and another layer $H(x)$, denoted as $I(x)=\phi(J(x),H(x))$. The final dehazing result is obtained by weighted fusion of the intermediate outputs $J_{0}{(x)}$, $J_{1}{(x)}$, $J_{2}{(x)}$, $J_{3}{(x)}$ and $J_{4}{(x)}$ by five learned attention maps $W_{0}$, $W_{1}$, $W_{2}$, $W_{3}$ and $W_{4}$ as

Through ablation studies, DMMFD has demonstrated that the fusion of multiple layers can improve the quality of the scene restoration process.

GFN (Ren et al., 2018a) proposes two observations on the influence of haze. First, under the influence of atmospheric light, the color of a hazy image may be distorted to some extent. Second, due to the existence of scattering and attenuation phenomena, the visibility of objects far away from the camera in the scene will be reduced. Therefore, GFN uses three enhancement strategies to process the original hazy image and use them as inputs to the dehazing network together. The white balanced input $I_{wb}(x)$ is obtained from the gray world assumption. The contrast enhanced input $I_{ce}(x)$ is composed of average luminance value $\widetilde{I}(x)$ and the control factor $\mu$, as following:

where $\mu=2\cdot(0.5+\widetilde{I}(x))$. By using the the nonlinear gamma correction, the $I_{gc}{(x)}$ used to enhance the visibility of the $I(x)$ can be obtained by

where $\alpha=1$, and $\gamma=2.5$. The final dehazing image $J(x)$ is determined by the combination of three inputs, where $C_{wb}$, $C_{ce}$ and $C_{gc}$ are confidences map for fusion process:

MSRL-DehazeNet (Yeh et al., 2019) decomposes the hazy image into low frequency and high frequency as the base component $I_{base}{(x)}$ and the detail component $I_{detail}{(x)}$, respectively. The basic component can be thought as the main content of the image, while the high frequency component denotes the edge and texture. Therefore, the dehazed image can be obtained by the dehazing function $D(\cdot)$ of the basic component and the enhancement function $E(\cdot)$ of the detail component. The whole process can be represented by a linear model as following:

DIDH (Shyam et al., 2021) decomposes both the input hazy image $I(x)$ and the predicted haze-free image $J(x)$ to obtain $(I(x),LF(I(x)))$, $(I(x),HF(I(x)))$, $(J(x),LF(J(x)))$ and $J(x),HF(J(x))$, where $LF(\cdot)$ and $HF(\cdot)$ denotes Gaussian and Laplacian filter, respectively. By fusing the decomposed data with the pre-decomposition data as the input of the discriminator, DIDH can improve the quality of the image generated by the adversarial training process. With the help of the discriminators $D_{LF}$ and $D_{HF}$, the dehazing network can be optimized in an adversarial way.

Compared with conventional data augmentation, such as rotation, horizontal or vertical mirror symmetry, and random cropping, the transformation and decomposition of the input is a more efficient strategy for the usage of hazy images.

Knowledge distillation (Gou et al., 2021) provides a strategy to transfer the knowledge learned by the teacher network to the student network, which has been applied in high-level computer vision tasks like object detection and image classification (Wang et al., 2019). Recent work (Hong et al., 2020) presents three challenges for applying knowledge distillation to the dehazing task. First, what kind of teacher task can help the dehazing task. Second, how the teacher network helps the dehazing network during training. Third, which similarity measure between teacher task and student task should be chosen. Fig. 7 shows the knowledge distillation strategy adopted by various dehazing algorithms. Different methods may use different numbers and locations of output features to compute feature loss.

KDDN (Hong et al., 2020) designs a process-oriented learning mechanism, where the teacher network $T$ is an auto-encoder for high-quality haze-free image reconstruction. When training the dehazing network, the teacher network assists in feature learning, and optimizes $L_{T}=||J(x)-T(J(x))||_{1}$. Therefore, the teacher task and the student task proposed by KDDN are two different tasks. In order to make full use of the feature information learned by the teacher network and help the training of the dehazing network. KDDN uses feature matching loss and haze density aware loss by a linear transformation function $g$, which are represented by (28) and (29), respectively.

where $T^{m}$ represents the $m$-th layer of the teacher network, and the corresponding $S^{n}$ represents the $n$-th layer of the student network; $\psi$ is obtained by the normalization operation. KDDN can be trained without real transmission map, replaced by the residual between hazy and haze-free images.

KTDN (Wu et al., 2020) is jointly trained using a teacher network and a dehazing network with the same structure. Through feature level loss, the prior knowledge possessed by the teacher network can be transferred to the dehazing network. SRKTDN (Chen et al., 2021a) uses ResNet18 pre-trained on ImageNet (with classification layers removed) as the teacher network, transferring many statistical experiences to the Res2Net101 encoder for dehazing. DALF (Fang et al., 2021) integrates dual adversarial training into the training process of knowledge distillation to improve the imitation ability of the student network to the teacher network. Applying knowledge distillation to dehazing networks provides a new and efficient way to introduce external prior knowledge.

The input data of the dehazing network are usually three channel color images in RGB mode. By calculating the mean square error of hazy and haze-free images in RGB space and YCrCb space on the Dense-Haze dataset, Bianco et al. (Bianco et al., 2019) find that haze shows obvious numerical differences in the two spaces. The error values for the red, green and blue channels in RGB space are very close. However, in the YCrCb space, the error value of the luminance channel is significantly larger than that of the blue and red chroma components. AIP-Net (Wang et al., 2018a) performs a similar error comparison of color space transformation on synthetic datasets and obtained the same conclusion as  (Bianco et al., 2019). Quantitative results (Wang et al., 2018a; Bianco et al., 2019) obtained by training the dehazing network in the YCrCb color space show that RGB space is not the only effective color mode for deep learning based dehazing methods. Furthermore, TheiaNet (Mehra et al., 2021) comprehensively analyzes the performance obtained by training the dehazing model in RGB, YCrCb, HSV and LAB color spaces. Experiments (Singh et al., 2020; Sheng et al., 2022; Chen et al., 2021a; Dudhane and Murala, 2019b) show that converting images from RGB space to other color spaces for model training is an effective scheme.

In the process of training a non-ASM-based supervised dehazing network, a common way is to use the hazy image as the input of the network and expect to obtain a clear image. In this process, the clear image is used as a positive example to guide the optimization of the network. By designing pairs of positive and negative examples, AECR-Net provides a new perspective for treating hazy and haze-free images. Specifically, the clear image $J(x)$ and the dehazed image $J_{pred}(x)$ are taken as a positive sample pair, while the hazy image $I(x)$ and the dehazed image $J_{pred}{(x)}$ are taken as a negative sample pair. According to contrastive learning (Khosla et al., 2020), $J_{pred}(x)$, $J(x)$ and $I(x)$ can be regarded as anchor, positive and negative, respectively. For the pre-trained model $G(\cdot)$, the dehazing loss can be regarded as the sum of the reconstruction loss and the regularization term, as following:

where $G_{i}$ represents the output feature of the $i$-th layer of the pre-trained model; $\lambda$ is the weight ratio between the image reconstruction loss and the regularization term; $w_{i}$ is weight factor for feature output; $\phi$ is the dehazing network. AECR-Net provides a universal contrastive regularization strategy for existing methods without adding extra parameters.

Dehazing methods based on deep learning usually set the optimization objective to obtain a single dehazed image. By introducing 2-dimensional latent tensors, pWAE (Kim et al., 2021) can generate different styles of dehazed images, which extends the general training purpose. pWAE proposes dehazing latent space $z_{h}$ and style latent space $z_{s}$ for dehazing, and applies the mean function $\mu(\cdot)$ and standard deviation function $\sigma(\cdot)$ to perform the transformation of the space:

A natural question is how to adjust the magnitude of the spatial mapping to control the degree of style transformation. pWAE uses linear module $z_{h}^{s}=\alpha{z_{h\to{s}}}+(1-\alpha)z_{h}$ to adjust the weight between the dehazed image and the style information. By controlling $\alpha$, different degrees of stylized dehazed images corresponding to the style image can be obtained.

DehazeFlow (Li et al., 2021b) proposes that the dehazing task itself is ill-posedness, so it is unreliable to learn a model with deterministic one-to-one mapping. With a conditional normalizing flow network, DehazeFlow can compute the conditional distribution of clear images from a given hazy image. Therefore, it can learn a one-to-many mapping and obtain different dehazing results in the inference stage. The non-deterministic output (Kim et al., 2021; Li et al., 2021b) brings interpretable flexibility to the dehazing algorithm. Since the visual evaluation criteria of individual human beings are inherently different, one-to-many dehazing mapping provides more options for photographic work.

Non-deep learning based research (Galdran et al., 2018) demonstrates the duality between Retinex and image dehazing. Apart from the already widely used ASM, recent work (Li et al., 2021c) explores the combination of Retinex model and CNN for dehazing. Retinex theory proposes that the image can be regarded as the element-wise product of the reflectance image $R$ and the illumination map $L$. Assuming the reflectance $R$ is illumination invariant, the relationship between hazy and haze-free image can be modeled by a retinex-based decomposition model as following:

where $*$ means the multiplication in an element-wise way. $L_{r}$ can be seen as absorbed and scattered light caused by haze, which is determined by the ratio of the hazy image illumination map $L_{I}$ and natural illumination map $L_{J}$.

Compared with ASM, which has been proven to be reliable, Retinex has a more compact physical form and fewer parameters to be estimated, but is not widely combined with CNN for dehazing, yet.

Rather than directly learning the mapping from hazy to haze-free image, several methods argue that the residual learning method can reduce the learning difficulty of the network. The dehazing methods based on residual learning is performed at the image level. GCANet (Chen et al., 2019c) uses the residual {$J(x)-I(x)$} between haze-free and hazy images as the optimization objective. DRL (Du and Li, 2018), SID-HL (Xiao et al., 2020) and POGAN (Du and Li, 2019) believe that the way of residual learning is related to ASM, and the relationship can be obtained by reformulated ASM:

where $r(x)=(A-J(x))(1-t(x))$ can be interpreted as a structural error term. By predicting nonlinear signal-dependent degradation $r(x)$, a clear image $J(x)=I(x)-r(x)$ can be obtained.

The dehazing methods based on CNN usually use down-sampling and upsampling for feature extraction and clear image reconstruction, and this process less considers the frequency information contained in the image. There are currently two approaches to combine frequency analysis and dehazing networks. The first is to embed the frequency decomposition function into the convolutional network, and the second is to use frequency decomposition as a constraint for loss computation. For a given image, one low-frequency component and three high-frequency components can be obtained by wavelet decomposition. Wavelet U-net  (Yang and Fu, 2019) uses discrete wavelet transform (DWT) and inverse discrete wavelet transform (IDWT) to facilitate high quality restoration of edge information in a clear image. Through the 1D scaling function $\phi(\cdot)$ and the wavelet function $\psi(\cdot)$, the 2D wavelets low-low (LL), low-high (LH), high-low (HL) and high-high (HH) are calculated as follows

where $m$ and $n$ represent the horizontal and vertical coordinates, respectively. MsGWN (Dong et al., 2020c) uses Gabor wavelet decomposition for feature extraction and reconstruction. By setting different degree values, the feature extraction module of MsGWN can obtain frequency information in different directions. EMRA-Net (Wang et al., 2021d) uses Haar wavelet decomposition as a downsampling operation instead of nearest downsampling and strided-convolution to avoid the loss of image texture details. By embedding wavelet analysis into convolutional network or loss function, recent studies (Yang et al., 2020; Dharejo et al., 2021; Fu et al., 2021) have shown that 2D wavelet transform can improve the recovery of high-frequency information in the wavelet domain. These works successfully apply wavelet theory and neural networks to dehazing tasks.

Besides, TDN (Liu et al., 2020b) introduces the Fast Fourier transform (FFT) loss as a constraint in the frequency domain. With the help of supervised training on amplitude and phase, the visual perceptual quality of images is improved without any additional computation in the inference stage.

The scattering particles in the hazy environment will affect the accuracy of the depth information collected by the LiDAR equipment. S2DNet (Hambarde and Murala, 2020) proves that high-quality depth estimation algorithms can help the dehazing task. According to ASM, it can be known that the transmission map $t(x)$ and the depth map $d(x)$ have a negative exponential relationship $t(x)=e^{-\beta{d(x)}}$. Based on this physical dependency, SDDE (Lee et al., 2020a) uses four decoders to integrate estimates of atmospheric light, clear image, transmission map, and depth map into an end-to-end training pipeline. In particular, SDDE proposes a depth-transmission consistency loss based on the observation that the standard deviation value ($std$) of the transmission map and the depth map pair should tend to zero:

where $t_{pred}{(x)}$ and $d_{pred}{(x)}$ represent the predicted transmission map and depth map, respectively. Aiming at better dehazing, TSDCN-Net  (Cheng and Zhao, 2021) designs a cascaded network for two-stage methods with depth information prediction. Quantitative experimental results (Yang and Zhang, 2022) show that this joint estimation method can improve the accuracy of dehazing task.

Existing experiments (Li et al., 2017a) show that the existence of image haze may cause various problems in detection algorithms, such as missing targets, inaccurate localizations and unconfident category prediction. Recent work (Sakaridis et al., 2018) has shown that haze also brings difficulties to semantic scene understanding. As a preprocessing module for high-level computer vision tasks, the dehazing process is usually separated from high-level computer vision tasks.

Li et al. (Li et al., 2017a) jointly optimize the object detection algorithm and AOD-Net, and the result proves that the dehazing algorithm can promote the detection task. LEAAL (Li et al., 2020c) proposes that fine-tuning the parameters of the object detector during joint training may lead to a detector biased towards the haze-free images generated by the pretrained dehazing network. Different from the fine-tuning operation of Li et al. (Li et al., 2017a), LEAAL uses object detection as an auxiliary task for dehazing and the parameters of the object detector are not fully updated during the training process.

UDnD (Zhang et al., 2020g) takes advantage of each other by jointly training a dehazing network and a dense-aware multi-domain object detector. The object detection network is trained by adopting the classification and localization terms used by Region Proposal Network and Region of Interest. The multi-task training approach used by UDnD can consider the reduced inter-domain gaps and the remained intra-domain gaps for different density levels.

Recent work has explored performing dehazing and high-level computer vision tasks simultaneously without using ASM. SDNet (Zhang et al., 2022a) combines semantic segmentation and dehazing into a unified framework in order to use semantic prior as a constraint for the optimization process. By embedding the predictions of the segmentation map into the dehazing network, SDNet performs a joint optimization of pixel-wise classification loss and regression loss. The classification loss is

where $P$ is the total number of pixels; $s_{i}$ is the class prediction at position $i$; $s^{*}$ denotes the ground truth semantic annotation.

(Zhang et al., 2022a, 2020g; Li et al., 2020c) show that segmentation and detection can be performed by embedding with dehazing networks. The way of joint dehazing task and high-level vision task may reduce the computational load to a certain extent by sharing the learned features, which can expand the goal of dehazing research.

The “end-to-end” CNN stands for the non-ASM-based supervised algorithms, which usually consist of well-designed neural networks that take a single hazy image as input and a haze-free image as output. Networks based on different ideas are adopted, which are summarized as follows.

• •

  Attention mechanism: FFA-Net (Qin et al., 2020), GridDehazeNet (Liu et al., 2019a), SAN (Liang et al., 2019), HFF (Zhang et al., 2022c).

• •

  Encoder-Decoder: CAE (Chen and Lai, 2019).

• •

  Based on dense block: 123-CEDH (Guo et al., 2019a).

• •

  U-shaped structure: DSEU (Lee et al., 2020b), MSBDN (Dong et al., 2020b).

• •

  Hierarchical network: DMHN (Das and Dutta, 2020).

• •

  Fusion with bilateral grid learning: 4kDehazing (Zheng et al., 2021).

The end-to-end dehazing networks have an important impact on the entire dehazing field, proving that numerous deep learning models are beneficial to the dehazing task.

PSD (Chen et al., 2021b) proposes a domain adaptation method that can be extended by existing models. It first uses a powerful backbone network for pre-training to obtain a base model suitable for synthetic data. Then the fine-tuning of the real domain in an unsupervised manner is applied to the well-trained network, thereby improving the generalization ability of the model to real world hazy images. To achieve fine-tuning in the real domain, PSD combines DCP loss, Bright Channel Prior (BCP) (Wang et al., 2013) loss and Contrast Limited Adaptive Histogram Equalization (CLAHE) loss together. BCP loss and CLAHE loss are

where $t_{BCP}$ represents the transmission map estimated by BCP and $t_{pred}$ is the predicted output of the network. $I_{CLAHE}$ is the hazy image reconstructed by CLAHE and other physical parameters.

During fine-tuning, the model may forget the useful knowledge it learned during the pre-training phase. Therefore, PSD proposes a feature-level constraint, which is achieved by calculating the feature map difference between the fine-tuning network and the pre-trained network. By feeding the synthetic data and real data into the fine-tuning network and the pre-trained network, respectively, four feature maps can be obtained: $F_{syn}^{tune}$, $F_{syn}^{pre}$, $F_{real}^{tune}$, and $F_{real}^{pre}$. Then, the loss for preventing knowledge forgetting can be calculated as

As shown in Fig. 8(a), supervised pre-training is performed first, and then the dehazing network is fine-tuned in an unsupervised form.

SSDT (Zhang and Li, 2021) uses the encoder-decoder network for pre-training in the form of domain translation. After pre-training, two encoders and one decoder are selected to obtain the holistic dehazing network $G(\cdot)$. Then, $G(\cdot)$ is fine-tuned with synthetic hazy images and real world hazy images. The two-stage method described above needs to ensure that the pre-training process can meet the accuracy requirements, otherwise it may bring accumulated errors to the fine-tuning process.

From the perspective of dual learning, the dehazing task and the haze generation task can be able to assist each other. Based on this assumption, DCNet (Chen et al., 2021c) proposes a dual-task cycle network that jointly utilizes labeled dataset $N$ and unlabeled dataset $M$ by dehazing network $DN(\cdot)$ and haze generation network $HGN(\cdot)$. The total loss is combine by dehazing loss $L_{DN}$ and reconstruction loss $L_{HGN}$, that is $L_{total}=L_{DN}+L_{HGN}$, as

where $\epsilon_{1}$ and $\epsilon_{2}$ are regularizing hyper-parameter; $P(I_{i}(x))$ equals $1$ when the $I_{i}(x)$ comes from the labeled dataset $N$, and equals $0$ otherwise. As shown in Fig. 8 (b), the predicted haze-free image $J_{pred}(x)$ is first obtained by the left network, and then the hazy image $I_{rec}(x)$ is reconstructed by the right network.

Liu et al. (Liu et al., 2021) use a disentangled image dehazing network (DID-Net) and a disentangled-consistency mean-teacher network (DMT-Net) to combine labeled and unlabeled data. DID-Net is responsible for disentangling the hazy image into a haze-free image, the transmission map, and the global atmospheric light. DMT-Net is used to jointly exploit the labeled synthetic data and unlabeled real world data through disentangled consistency loss. The supervised loss consists of four terms: haze-free image prediction $L_{J}^{s}$, transmission map prediction $L_{t}^{s}$, atmospheric light prediction $L_{A}^{s}$, and hazy image reconstruction $L_{rec}$: