Unpaired Overwater Image Defogging Using Prior Map Guided CycleGAN

The prior-based methods have been widely used in the past few years, via estimating the transmission maps and atmospheric light based on the statistics of natural images. Fattal et al. [18] formulated a refined image degradation model [4] that accounts for a shading factor and surface reflectance coefficients. Tan et al. [19] proposed an automated method based on two observations: clean images have a higher contrast than foggy images and the variation of airlight changes smoothly across small regions. He et al. [1] proposed a dark channel prior (DCP), which assumes that there is at least one channel in the RGB space that is close to zero in a clean natural image, to estimate transmission map and atmospheric light. Following He et al. [1], many works, such as [20][21], improved the dark channel prior for the image defogging problem. In addition, Zhu et al. [8] recover the depth information by building a linear model under the color attenuation prior, where the parameters of the linear model are learned with a supervised learning method. Fattal et al. [7] assert that pixels in local patches of images typically exhibit a one-dimensional distribution called color-lines. Similarly, Berman et al. [6] assume that a few hundred distinct colors can well approximate the colors of a clean image, and then perform image defogging based on this non-local prior. Based on the assumption that there is a linear relationship between the foggy and clean images in the minimum channel, Wang et al. [22] developed a computationally efficient defogging method. However, the accuracy of the estimated transmission map greatly affects the defogging effect. To address this problem, Liu et al. [23] reformulate the image defogging problem as a task of enhancing local visibility and global contrast that does not rely on atmospheric physical models. Although these methods show effectiveness in image defogging, they may fail when dealing with certain scenarios. For instance, dark channel prior [1] may fail for some scene objects (such as sky and white building) which are similar to the atmospheric light. Using the haze-line prior [6] for the dense fog may introduce color distortion. In comparison, we optimize the network by building the prior into the loss functions, and our data-driven model offers good generalization ability.

With the advance in CNNs and the support of large-scale datasets, data-driven methods for image defogging have drawn significant attention in recent years. Cai et al. [24] exploited an end-to-end trainable CNN to estimate the transmission map in an atmospheric scattering model for recovering the clean image. Ren et al. [25] proposed a multi-scale neural network named MSCNN for image defogging, which consists of a coarse net and a fine net to predict the transmission map. Unlike the above methods using CNN for estimating the transmission map, Li et al. [2] reformulated the atmospheric scattering model to combine the two unknown variables, i.e., transmission map and atmospheric light, into one variable. Then, they leverage a lightweight CNN to estimate this variable. Instead of using physical models, Ren et al. [26] adopted a fusion-based strategy to obtain a confidence map by integrating white balance, contrast enhancing, and gamma correction, derived from the original foggy image, and then recovered the clean image by weighting these three maps with a confidence map. Liu et al. [27] remove fog by using a grid network constructed with three modules: pre-processing, backbone, and post-processing, which can effectively alleviate the bottleneck issue and help reduce the artifacts in the final output. Qu et al. [3] proposed an Enhanced Pix2pix Dehazing Network (EPDN) which contains two enhancing blocks in the tail of decoder to refine the intermediate output. Different from other GAN-based methods, the discriminator supervises only the intermediate result of EPDN. Dong et al. [28] proposed a multi-scale and dense connected network for image defogging, where the strengthen-operate-subtract boosting strategy is incorporated in the decoder, and the dense feature fusion module is used to extract the features from non-adjacent levels and to restore the spatial information. Qin et al. [29] also adopted a fusion strategy, where they applied channel attention and pixel attention after the high-level feature for giving more weight to important features. To efficiently extract the features from the input image, Yi et al. [30] proposed a Multi-scale Topological Network (MSTN) that uses a multi-scale feature fusion module and an adaptive feature selection module to achieve progressive image defogging. However, this multi-branch network architecture increases the complexity of the network and further increases the training time.

Although the supervised learning-based methods have achieved favorable performance, they are limited to synthetic images and perform poorly on real-world images because of the inherent domain shift introduced by the synthetic training set. Hence, some semi-supervised and unsupervised methods are integrated to enhance the real-world foggy images. Engin et al. [31] introduced an enhanced CycleGAN which adopt unpaired real-world images as training set. Li et al. [13] proposed a semi-supervised network by combining a supervised learning branch with an unsupervised learning branch. Shao et al. [14] proposed a domain adaptation paradigm by incorporating two image defogging modules and an image translation module to bridge the gap between the synthetic and real domains. Golts et al. [12] exploited a dark channel prior energy function, which no longer relies on paired images to calculate the loss for the optimization of the network. Zhao et al. [32] proposed a semi-supervised two stage network, which utilized a CNN to refine the defogged image of DCP [1]. Chen et al. [33] select three physical priors to constitute a loss committee, i.e., dark channel loss, bright channel loss, and contrast limited adaptive histogram equalization loss, that guides the unsupervised training.

The above mentioned semi-supervised and unsupervised methods offer promising results on real-world images, but little attention has been paid to the image defogging of overwater scenes. Recently, Zheng et al. [34] proposed an enhanced CycleGAN to tackle the overwater image defogging problem. Nevertheless, this method simply replaces the training set of land foggy images as used in other methods with overwater images. Its defogging performance is limited as this method does not leverage the properties of the overwater scenes. To this end, our method focuses on the characteristics of overwater foggy images, and improves the defogging performance and visual quality of the defogged images.

As He et al. [1] claimed, the dark channel prior is invalid when the scene objects are similar to the atmospheric light. This is also reflected in Fig. 3 (b). It can be seen that the pixel value of the water and sky is not close to 0 in clean images.

A characteristic of the overwater scenes is that the sky and water usually occupy most of the area. In dark channels, they tend to be brighter than objects on the water because their pixel values are generally close to the atmospheric light. Based on this feature, the objects will be highlighted when the dark channel of the overwater image is inverted, as shown in Fig. 3 (c). To emphasize this feature, we try to enhance the contrast of the inverted dark channel. Fig. 3 shows the difference between the dark channel (DC), inverted dark channel (IDC), IDC after gamma correction, IDC after histogram equalization (HE), and IDC after min-max normalization (i.e. the prior map). It can be seen that though the contrast is enhanced, the gamma correction may lower the pixel value of the objects, and the histogram equalization may amplify background noise. In contrast, the min-max normalization maintains the highlight of the object while reducing noise. Fig. 4 is the intensity histogram over 4,531 inverted dark channels of an overwater image dataset [34] (details described in Sec. IV-A; the images are rescaled to 256$\times$256 for computational efficiency). It can be seen that the min-max normalization stretches the pixel values in the middle region while keeping the sky regions darker and the objects brighter, this statistic provides strong support to our proposed prior map.

Specifically, given a foggy image $I$ whose pixel value is in the range of [0, 1], its corresponding dark channel can be expressed as:

where $x_{i}$ and $y_{i}$ are pixel coordinates of $I$, $I^{c}$ denotes the $c$-th color channel of $I$, and $N(x_{i})$ denotes the local neighborhood centered at $x_{i}$. To obtain the prior map, we first remove the minimum filtering $\mathop{min}\limits_{y_{i}\in N(x_{i})}(\cdot)$ from the procedure for dark channel calculation, and invert the pixel value as follows:

where $I_{idc}$ represents the inverted dark channel. The reasons for removing the minimum filter are two. One is that it reduces the required computational resources, as performing the sliding window algorithm is time-consuming. The other is that the minimum filter may lead to unclear boundaries of the objects in the defogging results because it exploits all the surrounding pixels.

Then, the min-max normalization is performed on the $I_{idc}$ to get the prior map,

After getting the prior map, we further define the prior map guided cycle-consistency loss:

and the prior map guided GAN loss:

where $x$ and $y$ are foggy image and clean image, $I_{p}^{foggy}$ and $I_{p}^{clean}$ are the corresponding prior maps, and $||\cdot||_{1}$ is an $L_{1}$ norm. In $L_{pgcyc}$ and $L_{pgG}$, the loss of objects is increased and the loss of sky is reduced, as a result, the network is guided to recover the objects.

The details of the defogging network $G$ are shown in Fig. 5. We employ an encoder-decoder structure that includes an encoder for downscaling, a feature transformation module for enhancing the representation capability, and a decoder that consists of two UIMs for upscaling the feature maps while reducing the loss of details. In the forward propagation phase, the network learns a residual from the input image and enhance the defogging effect through the coarse-to-fine framework.

In the encoder, the first layer has 64 filters with a kernel size 7$\times$7 and a stride of 1, the second and the third layers have 128 and 256 filters, respectively, with a kernel size of 3$\times$3 and a stride of 2. The feature transformation module contains 9 residual blocks [35] to enhance the representation capability of the network. The decoder cascades two upscaling inception modules and two convolutional layers. The first convolutional layer in the decoder has 64 filters with a kernel size 3$\times$3 and a stride of 1, the second convolutional layer has 3 filters, each with a kernel size of 7$\times$7 and a stride of 1. It should be noted that the architecture of the fogging net $F$ is the same as $G$.

1) Upscaling Inception Module: As the name suggests, the UIM is inspired by the Inception module [16], it is a combination of three parallel upscaling branches: deconvolution, bilinear interpolation, and PixelShuffle [17], with their output filter banks concatenated into a single output. As shown in Fig. 5, the 1$\times$1 convolution is applied before the deconvolution and bilinear interpolation, which offers a channel-wise pooling to decrease the number of feature maps whilst retaining their salient features. Due to the bilinear interpolation being a fixed upscaling method, we cascade two 3$\times$3 convolutional layers after it to enhance its representation ability, while the other two branches cascade one 3$\times$3 convolutional layer after the upscaling layer. In the two UIMs in the decoder, the number of feature maps of the final 3$\times$3 convolutional layer of the deconvolution, bilinear interpolation, and PixelShuffle [17] are (64, 32, 32) and (32, 16, 16) respectively. After obtaining all the upscaled feature maps, we adopt the Squeeze-and-Excitation (SE) layer [36] to selectively emphasize informative features and suppress less useful ones, which consists of a global average pooling layer to generate channel-wise statistics and a bottleneck layer with two fully connected (FC) layers.

2) Long-range Residual Coarse-to-fine framework: The two complementary mechanisms: namely, the long-range residual connection and coarse-to-fine framework are adopted in our network for enhancing the defogging performance. As shown in Fig. 5, we first add a long-range residual connection between the input foggy image and the final output layer, which enables the network to preserve detailed information by learning the residual from the input foggy image. Nonetheless, this will also limit the representation capability of the network. Thus, we extend the defogging procedure into three stages, namely, the coarse, finer, and finest stage. As shown in Fig. 6, except that the coarse stage takes the same two 1/4 downscaled foggy images as input, in the finer and finest stage, respectively, the 1/2 and full-size defogged images generated by the previous stage are integrated as additional information to enhance the defogging performance. As a consequence of these two mutual compensations, the final defogged image retains the detailed information of the input image while also having a desired defogging effect.

3) Discriminator: In our network, we have two discriminators. As shown in Fig. 2, $D_{X}$ is used to distinguish between the input foggy images and the generated foggy images from the generator $F$; in the same way, $D_{Y}$ is used to discriminate between the generated clean images from $G$ and the input clean images. We adopt PatchGAN [37] discriminator as $D_{X}$ which consists of 5 convolutional layers. Each layer has the same kernel size 4$\times$4 with a stride of 2, and the number of filters is 64, 128, 256, 512, 1 from the lowest to highest.

For the discriminator $D_{Y}$, we employ the pixel level rather than the patch level discriminator, since the size of the prior map is equal to the input image, and the former produces an array of the same size as the input image, while the latter produces an array with reduced size. The discriminator $D_{Y}$ consists of 3 convolutional layers, where each layer has the same kernel size 1$\times$1 with a stride of 1, and the number of filters is 64, 128, 1 from the lowest to highest. After the output is obtained from $D_{Y}$, it is further weighted by the prior map with a high penalty for the objects.

The loss function of PG-CycleGAN includes 4 items, i.e., the prior map guided cycle-consistency loss $L_{pgcyc}$, the prior map guided GAN loss $L_{pgG}$, the cycle-consistency loss $L_{cyc}$, and the perceptual loss $L_{VGG}$.

Prior map guided cycle-consistency loss and prior map guided GAN loss are defined in Equations (5) and (6), they give higher penalties to objects on the water and lower penalties to the sky.

Cycle-consistency loss aims to minimize the objective between the foggy images $X$ and its reconstructed foggy images $F(G(x))$, which can be expressed as:

Perceptual loss is also used to enhance the textural information of the recovery image, which is based on VGG16 [38] to further constrain the generators, defined as:

where $\phi$ represents the feature maps of the second and fifth pooling layer of the VGG16 network, and $||\cdot||_{2}$ is an $L_{2}$ norm.

Overall loss function. Combining all the above losses, the whole loss function is formulated as,

where $\lambda_{i}$ ($i=1,2$) is a hyperparameter indicating the weight. In our empirical experiments, it is found that the network is sensitive to $L_{pgG}$ which can significantly improve the defogging performance after fine-tuning. Thus, we set $\lambda_{1}$ to 1 which can stabilize the training, and set $\lambda_{2}$ to a value that is slightly larger 2.

In this work, we use unpaired real-world images for training instead of the widely used paired synthetic images. Table I shows the comparison of training set sizes for different defogging methods. Unlike other methods that employ a large number of paired and unpaired images for training, our training set only contains 2,090 unpaired foggy and 2,441 clean overwater images. The used dataset is proposed by Zheng et al. [34], which also contains 188 real-world overwater foggy images for testing.

To demonstrate the applicability of PG-CycleGAN in wider contexts, two land image datasets O-HAZE [41] and NH-HAZE [42] are also employed as test sets. O-HAZE [41] contains 45 different outdoor scenes depicting the same visual content recorded in haze-free and hazy conditions, under the same illumination parameters. NH-HAZE [42] is a non-homogeneous real dataset with pairs of real hazy and corresponding haze-free images, from 55 outdoor scenes.

In our network, all the training samples are resized to 512$\times$512, and then randomly cropped to 256$\times$256 for data augmentation. During training, ADAM optimizer [43] is employed for the generators and discriminators with a learning rate of 0.0001 and batch size of 1. We adopt cosine annealing schedule where the initial learning rate is decreased to 0 after 200 epochs according to the cosine function. The network is trained on Pytorch deep learning platform with the acceleration of an Nvidia GeForce RTX 3090 GPU, and it takes about 30 hours for training.

We compare the performance of the PG-CycleGAN with several state-of-the-art defogging algorithms as mentioned earlier, and a set of experiments including qualitative and quantitative on overwater and land images is conducted.

1) Qualitative comparisons on overwater images: In this experiment, we evaluate the visual quality of the defogged images obtained by PG-CycleGAN on real-world overwater images.

Fig. 7 shows the results of different methods on overwater images. We can observe that, in general, the resulting images given by our method has higher brightness than those by DCP [1]. This is because of the inaccurate estimation of atmospheric light with the DCP method. The results of the DAD [14] often suffer from color distortion and unrealistic looking. Due to the domain gap between the synthetic and real-world foggy images, images restored by MSBDN [39] remain somewhat foggy, especially in distant areas. PSD [33] sometimes fails to handle the overwater scene, which leads to severe color shifting. Although RefineDNet [32] shows pretty good results, it tends to darken the images, and the sky regions in the 5th row of Fig. 7 are blurred. This may be due to the drawback that the RefineDNet [32] adopts the DCP [1] in the first stage to restore visibility, while DCP is difficult to restore sky regions of the foggy images. As can be seen in the 4th and 6th rows of Fig. 7, the defogged images of the self-supervised method SLAD [40] also remain somewhat foggy.

2) Qualitative comparisons on land images: In this experiment, we evaluate the visual quality of the defogged images by PG-CycleGAN on land images.

Figs. 8 and 9 show the results of different methods on real-world land images on the O-HAZE dataset. From the 4th row of Fig. 8, we can observe that with the DCP method, fog around the leaves still remains. The MSBDN, PSD, and SLAD degrade substantially when dealing with thick fog, as shown in Fig. 9, where certain amount of fog still remains. Although DAD can remove the fog well, the resulting images have severe color distortion. Due to the use of the DCP in the first defogging stage, the resulting images by RefineDNet are too dark as compared to the ground-truth.

In comparison, our network can generate cleaner images from the overwater and land foggy images. As demonstrated in the zoom-in regions in Fig. 7, e.g. the 1st and 2nd rows, the crowd and cars are well recovered due to the prior map guided losses. Similarly, as shown in the 3rd row of Fig. 7, other methods neglect the fine rope, yet our model can still effectively defog it. Furthermore, the 4th-6th rows in Fig. 7 show that our results have less remaining fog and artifacts. From Fig. 9, we can observe that the results of PG-CycleGAN are closer to the ground-truth than other methods. Considering those results together, we can see that PG-CycleGAN restores more details and obtains images that are visually clearer.

3) Quantitative comparisons on overwater images: Because the overwater images in the test set have no corresponding clean images, full-reference image quality assessment metrics are not suitable for our experiments. Recently, some non-reference image quality assessment metrics [44][45][46][47] have been proposed for quantitatively evaluating defogging performance. We adopt three non-reference evaluation criteria: FADE [44], HazDesNet [47], and BAVE [46], in which FADE and HazDesNet are designed to evaluate the defogging performance by predicting the density of the fog. Lower values of the FADE and HazDesNet denote better defogging performance. The BAVE is an evaluator of visibility enhancement that contains three indicators, which are the rate of visible edges $e$, the quality value of the contrast restoration $r$, and the normalized saturate value of pixels $\sigma$. For a defogged image, the higher values of $e$ and $r$, and the lower value of $\sigma$ denote the better quality of the recovered image.

A quantitative comparison of the test set which contains 188 real-world overwater images is reported in Table II. It can be seen that our method gives the best results in both FADE and the HazDesNet, which demonstrate that the PG-CycleGAN can remove the fog more effectively. Although PG-CycleGAN is ranked the second in terms of visible edges $e$ and the quality value of the contrast restoration $r$, and the third in terms of the indicator $\sigma$, it is not far from the best results. Combining all the performance indicators, it can be argued that our results have less fog remaining and better visual quality.

We further present the fog density map predicted by HazDesNet in Fig. 10. It can be seen that benefiting from the prior map guided losses, the fog density map of our result is closer to dark blue compared to other methods, which indicates that our method has better defogging performance.

4) Quantitative comparisons on land images: In this experiment, we adopt the full-reference image quality assessment metrics SSIM and PSNR for comprehensively evaluating the performance of our proposed PG-CycleGAN.

Table III gives the quantitative comparison on the O-HAZE and NH-HAZE datasets. We can observe that although the PG-CycleGAN is trained on overwater images, it achieves the best performance in SSIM on the NH-HAZE dataset, and ranked the second in terms of PSNR on both the O-HAZE and NH-HAZE datasets. Furthermore, as shown in Fig. 8, the values of FADE and HazDesNet of PG-CycleGAN are lower than other methods, which also demonstrates that our method leads to better quality images with less fog remaining than other methods.

Overall, the quantitative and qualitative comparisons show that our proposed method not only generates images with better visual effects on overwater images, but also offers excellent defogging performance for land images.

In order to verify the effectiveness of the LRC, UIM, and prior map guided losses, we conduct a series of ablation studies to analyze our method.

Fig. 11 illustrates the effectiveness of the LRC. After the coarse-to-fine framework is removed (as shown in Fig. 11 (b)), we can see that although learning the residual from the foggy input can retain detailed information, the results tend to be insufficient defogging. In addition, due to the limited representation power of the network, the artifacts are also introduced. When the long-range residual connection is removed (as shown in Fig. 11 (c)), the defogging results show color shift. After the long-range residual connection and coarse-to-fine framework are both removed, the results may include color shift and heavy fog, and the results of CycleGAN also show the clear color shifts. The above discussion shows that the combination of the long-range residual connection and coarse-to-fine framework can result in a good complementarity that facilitates the restoration of detailed information and provides a desirable defogging effect.

In Fig. 12 (b)-(e), the results are generated by using deconvolution, bilinear interpolation, PixelShuffle [17], and UIM as the upscaling method, respectively. It can be seen that the zoom-in regions in Fig. 12 (b), (d), and (e) contain more fog than Fig. 12 (f), and the images in Fig. 12 (c) contain color distortion whose color tends to be blue. This figure shows that PG-CycleGAN eliminates color distortion and offers better defogging effect as compared with the baseline methods.

For evaluating the effectiveness of the prior map guided losses, we conduct an ablation study involving the following three control groups: 1) not using the prior map guided cycle-consistency loss, 2) adopting PathGAN [37] discriminator which is widely used in image generation tasks instead of regular discriminator, 3) using regular discriminator but not imposing the prior map. The visual results are shown in Fig. 13, which shows that with the absence of the prior map guided cycle-consistency loss, the resulting color tends to be blue. In Fig. 13 (c), some regions are blurred and the defogging results are very artificial. Furthermore, with the absence of the prior map, the regular discriminator cannot distinguish between real and generated images, which leads to undesired green regions in the defogging results as shown in Fig. 13 (d). We can see that in Fig. 13 (e), the results of CycleGAN show insufficient defogging. On the contrary, as shown in Fig. 13 (f), the PG-CycleGAN optimized with the prior map offers better visual effects, e.g., the objects are finely recovered.

Overall, as illustrated in Figs. 11-13, the LRC restores detailed information while also providing a desirable defogging effect. The UIM integrates the strengths of three upscaling branches and generates cleaner and visually more pleasing results. The prior map guided losses enhance the recovery of the objects on the water. Table IV quantitatively compares the contribution of each part. When either part is removed, both FADE and HazDesNet have increased, i.e., the defogging performance deteriorates. This further demonstrates the effectiveness of our proposed method.

Table V gives the comparison of inference times on a 512$\times$512 image between the CycleGAN [15], the PG-CycleGAN with deconvolution instead of UIM, and the PG-CycleGAN. It can be observed that UIM does not significantly slow down the inference speed. This is benefiting from the 1x1 convolution, which reduces the dimension before upscaling feature maps.

As with previous methods, there are limitations in the proposed method. As shown in Fig. 14, when the local area in the original input image is near to white, which usually happens on white hulls, the pixel in the corresponding prior map will tend to 0. The reason is that if all the pixels of the three channels in the input possess a large value, the local region will show brightly in the dark channel, and then appear dark after the dark channel is inverted. This may lead to suboptimal solutions in network optimization, as it cannot give the objects a high penalty. The following solutions could be used to overcome this problem, i.e., refining the weights of the prior map guided losses, or adding an extra PatchGAN [37] discriminator to optimize the model.