GridFormer: Residual Dense Transformer with Grid Structure for Image Restoration in Adverse Weather Conditions

Image restoration in adverse weather conditions is the task of restoring a high-quality image under weather-related foreground degradations like rain, fog, and snow. Especially, image restoration in adverse weather conditions typically includes image deraining ba2022not ; kang2011automatic ; li2019heavy ; yang2019joint ; wang2020model , image dehazing berman2016non ; cai2016dehazenet ; ren2018gated ; liu2019griddehazenet , image desnowing ren2017video ; liu2018desnownet ; zhang2021deep , and multi-weather restoration li2020all ; valanarasu2022transweather ; ozdenizci2023restoring . The traditional model-based methods he2010single ; luo2015removing ; zhu2017joint focus on exploring appropriate weather-related priors to address the image restoration problem. However, there has been a surge in the number of data-driven methods proposed in recent years. Next, we mainly discuss these data-driven methods in detail.

Deraining: The task of removing rain streaks from images has been approached using a deep network called DerainNet, proposed by Fu et al.fu2017clearing . This approach learns the nonlinear mapping between clean and rainy detail layers. Several techniques have been proposed to improve performance, such as the recurrent context aggregation in RESCAN li2018recurrent , spatial attention in SPANet wang2019spatial , multi-stream dense architecture in DID-MDN zhang2018density , conditional GAN-based method in zhang2019image , and conditional variational deraining based on VAEs du2020conditional . Another approach to image deraining is removing raindrops. Yamashita et al.yamashita2005removal developed a stereo system to detect and remove raindrops, while You et al.you2015adherent proposed a motion-based method. Qian et al.qian2018attentive developed a raindrop removal benchmark and proposed an attentive GAN. Quan et al.quan2019deep introduced an image-to-image CNN embedded attention mechanism to recover rain-free images, and Liu et al.liu2019dual designed a dual residual network to remove raindrops. Zhang et al.zhang2021multifocal proposed a multifocal attention-based cross-scale network that employs spatial and channel attention to explore cross-scale correlations of rain streaks and background for image draining. Recent works aim to remove both streaks and raindrops from images simultaneously quan2021removing ; xiao2022image .

Dehazing: Two pioneering methods for image dehazing are DehazeNet cai2016dehazenet and MSCNN ren2016single , which first estimate the transmission map and generate haze-free images using an atmosphere scattering model narasimhan2000chromatic . AOD-Net li2017aod represents another advancement, which estimates one variable from the transmission map and atmospheric light. DCPDN zhang2018densely employs two sub-networks to estimate the transmission map and the atmospheric light, respectively. Recent works have focused on directly restoring clear images from hazy images, using attention mechanisms qin2020ffa ; zhang2020pyramid , multi-scale structures dong2020multi ; liu2019griddehazenet , GAN structures qu2019enhanced and transformers song2022vision . The network in liu2019griddehazenet is a similar method to our GridFormer. However, GridFormer significantly differs from liu2019griddehazenet in several ways. First, GridFormer is the first transformer-based method for image restoration in adverse weather conditions, whereas liu2019griddehazenet is a CNN-based method specifically designed for image dehazing. GridFormer is more general in terms of its utility. Second, in each GridFormer layer, we design a novel compact-enhanced transformer layer and integrate it in a residual dense manner. This promotes feature reuse and consequently enhances feature representation, whereas liu2019griddehazenet uses existing residual dense blocks in its network. Finally, extensive experiments demonstrate the superior performance of GridFormer compared to the method in liu2019griddehazenet .

Desnowing: In DesnowNet liu2018desnownet , translucency and residual generation modules were employed to restore image details. Li et al.li2019stacked proposed a stacked dense network with a multi-scale structure. Chen et al.chen2020jstasr introduced a desnowing method called JSTASR, which is specifically developed for size- and transparency-aware snow removal. They used a joint scale and transparency-aware adversarial loss to improve the quality of the desnowed images. Li et al.li2020all adopted the network architecture search technique to obtain excellent results. Zhang et al.zhang2021deep proposed a dense multi-scale desnowing network that incorporates learned semantic and geometric priors. More recently, some works chen2022snowformer ; zhang2022desnowformer have explored the transformer architecture and further improved the performance.

Multi-weather restoration: Beyond the above task-specific image restoration methods, recent works li2020all ; valanarasu2022transweather ; li2022all attempt to address multi-weather restoration in a single architecture. Li et al.li2020all proposed All-in-One networks with a multi-encoder and decoder structure to restore adverse multi-weather degraded images. Specifically, they adopt separate encoders for different weather degradations and resort to neural architecture search to seek the best task-specific encoder. In li2022all , All-in-one restoration network consists of a contrastive degraded encoder and a degradation-guided restoration network. Valanarasu et al.valanarasu2022transweather proposed an end-to-end multi-weather image restoration model named TransWeather that achieves high performance on multi-weather restoration. The core insights in TransWeather are the intra-path transformer block and transformer decoder with learnable weather-type embeddings. In this paper, our work aligns with this direction and focuses on designing a general model to address the multi-weather restoration problem. In addition, there are methods aimed at designing effective network architecture for image restoration. For example, MPRNet zamir2021multi and MAXIM tu2022maxim are general image restoration methods that have also been successful in addressing a range of adverse weather conditions. MIMOUNet cho2021rethinking adopts an encoder-decoder-based U-shaped network with multi-input and multi-output to achieve image deblurring. In our method, we employ the coarse-to-fine strategy to the transformer network in the grid structure for image restoration under adverse weather conditions.

Recently, vision transformers have witnessed great success in low-level image restoration. Specifically, inspired by the seminal work in vaswani2017attention , Chen et al.chen2021pre proposed an Image Processing Transformer (IPT) for general image restoration, which employs a special multi-head and multi-tail structure to adapt for the specific image restoration tasks. However, IPT requires costly pre-training on large-scale datasets. Further, SwinIR liang2021swinir and Uformer wang2022uformer modify the original Swin Transformer block and obtain good performance with relatively low computational cost. In particular, SwinIR stacks the proposed residual transformer blocks to extract deep features for image reconstruction. Uformer adopts a U-shape structure, embedding the proposed LeWin transformer blocks to predict residual images. Yao et al.yao2022dense adopted the LeWin transformer block as a basic unit and introduced the dense residual skip connection to propose a dense residual skip-connection network based on transformer called DenSformer for image denoising. Liang et al.liang2022drt proposed a recursive transformer, which first introduces a recursive local window-based self-attention structure in the network. A recent method, Restormer zamir2021restormer , which is a multi-scale hierarchical transformer architecture, has also yielded fine restoration performance on image restoration e.g., deraining. Inspired by the success of these methods, we propose a general grid framework with novel transformer blocks to restore images in adverse weather conditions. SwinIR and DenSformer are similar methods to our GridFormer. However, while SwinIR fuses Swin Transformer and convolutional layers in its residual Swin Transformer block, our GridFormer’s residual dense block more effectively enhances feature reuse. Unlike DenSformer’s dense residual transformer block, our approach is characterized by the unique compact-enhanced self-attention mechanism, local feature fusion, and local skip connections within the residual dense transformer block.

Motivation. Our motivation arises from the urgent need for techniques that restore images captured in unfavorable weather conditions. Weather-related factors, such as haze, rain, and snow, significantly impact the quality and perception of images, which in turn affects various practical applications such as surveillance, autonomous driving, and outdoor photography. The main objective of developing the proposed GridFormer is to address the persistent challenges caused by adverse weather conditions on image quality. Our goal is to create an image restoration framework that effectively handles a range of adverse weather scenarios, thereby enhancing the quality of images affected by these conditions.

Architecture. As shown in Fig. 2, GridFormer contains three paths from the weather-degraded images to the recovered ones, where each path conducts restoration at different image resolutions. In GridFormer, the higher resolution path continuously interacts dynamically with the lower resolution path in the network to remove weather degradation accurately, and the lower resolution path provides useful global information owing to larger receptive fields. Each path is composed of seven GridFormer layers. Different paths are interlinked with a down-sampling layer, an up-sampling layer, and weighted attention fusion units to compose the columns of the GridFormer. Thanks to the grid structure with three rows and seven columns, information from different resolutions can be shared effectively. Specifically, GridFormer consists of three parts: grid head (GH), grid fusion module (GFM), and grid tail (GT). We present the details of each part in the following.

Grid head. To extract initial multi-resolution features, we use a grid head architecture to process pyramid input images in parallel. Every path in the grid head consists of a feature embedding layer, achieved by $3\times 3$ convolutions, and a GridFormer layer. As shown in Fig. 2, given a weather-degraded image $\mathbf{X}_{0}$, the grid head extracts hierarchical features $\mathbf{F}=\left\{\mathbf{F}_{0},\mathbf{F}_{1},\mathbf{F}_{2}\right\}$ in different channels (i.e., $C$, $2C$, and $4C$) from pyramid images $\mathbf{X}=\left\{\mathbf{X}_{0},\mathbf{X}_{1},\mathbf{X}_{2}\right\}$ (1/2, 1/4 scales for $\mathbf{X}_{1}$ and $\mathbf{X}_{2}$). In our experiments, we use $C=48$. The grid head computation can be defined as:

where $i$ is the $i$-th network path, and $\mathrm{E}_{i}$ is the feature embedding layer. The $\downarrow$ symbol denotes the down-sampling layer, where we use a $3\times 3$ convolution with a pixel-unshuffle operation shi2016real to halve the features in the spatial dimensions while doubling the channels. $\mathrm{GFL}$ is a GridFormer layer that is mainly built from residual dense transformer blocks.

Grid fusion module. To fully integrate the hierarchical features of different rows and columns in the network, we propose a grid fusion module between the grid head and the grid tail. The structure of the proposed grid fusion module is organized into a 2D grid pattern. As illustrated in Fig. 2, the fusion module is designed in a grid-like structure of three rows and five columns. In particular, each row contains five consecutive GridFormer layers that keep the feature dimension constant. In the column axis, according to the position in the grid, we resort to the down-sampling layers or up-sampling layers to change the size of the feature maps for feature fusion. Fig. 3 (a) shows a representative grid unit in the fusion module. The GridFormer layer is a dense structure consisting of three residual dense transformer layers (RDTL) and a $1\times 1$ convolution, which will be discussed in the next subsection. The down-sampling and up-sampling layers are symmetrical and use a $3\times 3$ convolution with pixel-shuffle or pixel-unshuffle operation shi2016real to change the feature dimensions. In addition, considering that the features of different scales may not be equally important, we use a simple weighted attention fusion strategy to achieve feature fusion from the different row and column dimensions. Inspired by zheng2022t ; wang2022ultra , we first generate two trainable weights for different features, where each parameter is an $n$-dimensional vector ($n$ is the channels of feature). We add these weighted features to derive the fusion features. Grid units in the grid fusion module provide different information flows for feature fusion shown in Fig. 3 (b), which guides the network to produce better-recovered results in combination with different complementary information.

Grid tail. To further improve the quality of the recovered images, we design a grid tail module to predict multi-scale outputs. The structure of the grid tail is symmetrical to that of the grid head. Specifically, each path is composed of a GridFormer layer, a $3\times 3$ convolution, and a long skip connection for image reconstruction. The skip connection is used to transmit input information directly to the grid tail module, which maintains the color and detail of the original image. The complete process is formulated as:

where $\hat{\mathbf{X}}_{i}$ is the final result of GridFormer on the $i$-th path, $\mathrm{C}_{i}$ is a $3\times 3$ convolution, and $\hat{\mathbf{F}}_{i},i\in\{0,1,2\}$ is the output feature of the grid fusion module. To optimize the network parameters, we train GridFormer using a combination of two losses, multi-scale Charbonnier loss charbonnier1994two and perceptual loss, where the weight of perceptual loss johnson2016perceptual is set to $0.1$. Next, we detail the core component residual dense transformer block that is used to build the elemental layer of GridFormer.

Previous works huang2017densely ; zhang2018residual ; liu2019griddehazenet ; zhang2021deep ; zheng2022t have shown that using dense connections has many advantages, mitigating the vanishing gradient problem, encouraging feature reuse and enhancing information propagation. Accordingly, we propose to design the transformer with dense connections to build the basic GridFormer layers. Specifically, we propose residual dense transformer blocks (RDTB) to compose GridFormer using different settings. As illustrated in Fig. 4, RDTB contains densely connected transformer layers, local feature fusion, and local residual learning. When implementing the dense connection, we mainly incorporate three layers of residual dense transformer layers (RDTL), with the growth rate set at 16. This implies that each individual RDTL generates 16 new feature maps. These newly generated feature maps are subsequently concatenated with the feature maps received from the preceding layer. Within each RDTL, we use several compact-enhanced transformer layers (CETL) with a ReLU glorot2011deep activation function to extract features, and adopt a $1\times 1$ convolution to ensure the same number of channels for input and output features. For local feature fusion and local residual learning, we introduce a $1\times 1$ convolution and a local skip connection in RDTB to control the final output.

The direct application of transformers vaswani2017attention ; dosovitskiyimage to our grid network will lead to high computational overhead, we thus develop a cost-effective compact-enhanced attention, with the stages of sampler and compact self-attention for improving the efficiency, as well as a local enhancement stage for enhancing the local information in the transformer. Fig. 5 illustrates the detailed structure of the proposed compact-enhanced attention.

Feature sampling. We first design a sampler to produce down-sampled input tokens for the subsequent self-attention computation. The sampler is built by an average pooling layer with stride $r$. The sampler layer not only increases the receptive field to observe more information, but also enhances the invariance on the input token. In addition, the produced lower-resolution features can reduce the computation of subsequent layers. The feature sampling step is formulated as:

where $\mathbf{Z}_{in}\in\mathbb{R}^{H\times W\times C}$ represents the input token. $\mathbf{Z}\in\mathbb{R}^{\frac{H}{r}\times\frac{W}{r}\times C}$ is the output token. $\mathrm{Avg}_{r}$ indicates the average pooling operation with stride $r$. In the experiments, we empirically set $r$ as $4$, $2$, and $2$ in three rows of GridFormer layers, respectively (see Sec. 4.4).

Compact self-attention. Given a feature of dimensions $H\times W\times C$, recent low-level transformer-based methods valanarasu2022transweather ; lee2022knn ; wang2022uformer aim to explore the long-range dependence between key and query to calculate the $N\times N$ attention map ($N=H\times W$), which leads to high complexity and fails to model the global information from the channel dimension. Thus, for more efficient computation in self-attention, we resort to a different strategy. Specifically, as illustrated in Fig. 5, for an output feature $\mathbf{Z}\in\mathbb{R}^{\frac{H}{r}\times\frac{W}{r}\times C}$ from the sampler, we first implement the split operation by dividing it along the channel dimension to produce $\mathbf{z}_{1}\in\mathbb{R}^{\frac{H}{r}\times\frac{W}{r}\times\frac{C}{2}}$ and $\mathbf{z}_{2}\in\mathbb{R}^{\frac{H}{r}\times\frac{W}{r}\times\frac{C}{2}}$. We then apply a convolution layer with reshape operation on $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$, which projects $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$ into Queries ($\mathbf{q}_{1},\mathbf{q}_{2}\in\mathbb{R}^{\frac{C}{2}\times\frac{HW}{r^{2}}}$), Keys ($\mathbf{k}_{1},\mathbf{k}_{2}\in\mathbb{R}^{\frac{C}{2}\times\frac{HW}{r^{2}}}$) and Values $\mathbf{v}_{1},\mathbf{v}_{2}\in\mathbb{R}^{\frac{C}{2}\times\frac{HW}{r^{2}}}$), respectively. Inspired by existing methods petit2021u ; susladkar2023gafnet ; gu2023adafuse ; zhang2024cf , we exchange the values produced by them to perform multi-head self-attention, which can improve the interaction between $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$. Compared with the method of exchanging queries for feature interaction in cross-attention petit2021u ; zhang2024cf , our approach exchanges the values for interaction and feature fusion, finding it beneficial for better restoration performance (see Sec. 4.4). Finally, we obtain the result $\hat{\mathbf{Z}}$ by concatenating the output of the two multi-head self-attention and changing their dimensions. The proposed compact self-attention mechanism can be formulated as:

where $[\cdot]$ indicates the concatenation operation. The major computational overhead in transformers mainly arises from the self-attention (SA) layer. In contrast to recent transformer-based methods that employ spatial modeling for SA, the complexity of the key-query dot-product interaction grows quadratically with the spatial resolution of input, i.e., $O(N\times N)$. Our proposed compact self-attention addresses this by performing SA across channels instead of the spatial dimension, resulting in cross-covariance computation across channels to produce an attention map that implicitly encodes the global context. Consequently, our compact self-attention generates an attention map of size $\mathbb{R}^{C\times C}$, instead of the huge regular attention map of size $\mathbb{R}^{N\times N}$. Thus, our compact self-attention successfully reduces complexity.

Local enhancement. As shown in Fig. 5, we add a local feature enhancement stage in the tail of compact self-attention. This stage consists of a deconvolution operation, sometimes referred to as a “transposed convolution,” with a deconvolution for local feature propagation and a $1\times 1$ convolution for local fusion:

where $\mathbf{Z}_{out}$ is the final output. $\mathrm{Conv_{1\times 1}}$ and $\mathrm{Deconv}$ are $1\times 1$ convolution and deconvolution layers respectively.

Inspired by existing works yin2023multiscale ; valanarasu2022transweather ; ye2022perceiving ; jiang2020decomposition ; li2022two ; yu2022frequency ; ali2023vision ; hsu2023wavelet ; qiao2023dual , we use a loss function combining the Charbonnier loss charbonnier1994two and the perceptual loss wang2018esrgan to train our GridFormer. We regard the Charbonnier loss as a pixel-wise loss, which is used between the recovered images and the ground truth images at each scale, and the perceptual loss is used to help our model produce visually pleasing results. The Charbonnier loss is defined as:

where $\hat{\mathbf{X}}_{k}$ and $\mathbf{I}_{k}$ refer to the restored image and ground-truth image respectively, and $k$ represents the index of the image scale level in our GridFormer. The constant $\varepsilon$ is empirically set to $10^{-3}$. For the perceptual loss, following previous work wang2018esrgan , we adopt a pre-trained VGG19 simonyan2014very to extract the perceptual features from the $Conv5\_4$ layer of VGG19, and then use the $L_{1}$ loss function to compute the difference between the perceptual features of the restored images and their corresponding ground truths. This effective perceptual loss focuses on capturing high-level semantic information, resulting in sharper edges and visually appealing outcomes, all while ensuring computational efficiency wang2018esrgan . Specifically, the perceptual loss is as follows:

where $C$, $H$, and $W$ denote the dimensions of the feature map obtained from the $Conv5\_4$ layer of the pretrained VGGNet $\phi$.

The final loss function $\mathcal{L}$ to train our proposed GridFormer is shown as follows:

where $\mathcal{L}_{\text{char}}$ denotes the Charbonnier loss, $\mathcal{L}_{\text{per}}$ is the perceptual loss. $\alpha$ is a hyper-parameter that is used to balance these two losses. In our experiments, it is empirically set to $0.1$.

While HRNet wang2020deep , HRFormer NEURIPS2021_3bbfdde8 , and RevCol cai2022reversible utilize a grid-like structure, they diverge from our GridFormer. First, GridFormer captures multi-scale features directly from the pixel level, in contrast to HRNet and HRFormer which perform multi-scale feature extraction at the feature layer level, and RevCol, which does not incorporate a multi-scale mechanism. Second, GridFormer integrates a new self-attention mechanism to enhance the fusion of multi-scale features more effectively. This approach sets it apart from HRNet, HRFormer, and RevCol, which do not employ compact self-attention in their feature fusion processes. Third, our network is intricately designed for image restoration under adverse weather conditions, striving to produce images of superior quality. Unlike HRNet, HRFormer, and RevCol, which are not specifically engineered for this challenge, our network architecture is uniquely suited to tackle the complexities inherent in this task.

We evaluate GridFormer on several image restoration tasks under severe weather conditions.

Datasets. For image dehazing, the first setting uses ITS li2018benchmarking to train the model and test it on indoor SOTS li2018benchmarking . Another setting is training and testing on Haze4K liu2021synthetic covering both indoor and outdoor scenes. Desnowing is evaluated on Snow100K liu2018desnownet . RainDrop qian2018attentive is used for raindrop removal, and Outdoor-Rain li2019heavy is used for image deraining and dehazing. For multi-weather restoration, we train the model on a combination of images degraded in adverse weather conditions similar to ozdenizci2023restoring . Table 1 lists the datasets used for the different tasks. In the following, we introduce the dataset and experimental details for specific tasks for image restoration in adverse weather conditions.

Image dehazing. Following liu2019griddehazenet ; qin2020ffa ; song2022vision ; tu2022maxim , we conduct our experiments on RESIDE li2018benchmarking and Haze4K liu2021synthetic datasets. Specifically, for the RESIDE dataset, we adopt Indoor Training Set (ITS) to train the model and test the model on the indoor set of the SOTS dataset. ITS contains $13,990$ indoor pair images and the indoor set of the SOTS dataset includes $500$ indoor pair images. For the Haze4K dataset, we follow the previous work ye2021perceiving . The Haze4K dataset contains $3,000$ haze and haze-free image pairs for training and $1,000$ for testing. The Haze4K dataset is more challenging, which considers both indoor and outdoor scenes.

Image desnowing. For this task, we use the popular Snow100K dataset liu2018desnownet for training and evaluating the proposed method. Snow100K contains $50,000$ training and $50,000$ testing images. The testing set has three sub-sets i.e., Snow100K-S/M/L, which refers to different snowflake sizes (light/mid/heavy). The Snow100K-S, Snow100K-M, and Snow100K-L have $16611$, $16588$, and $16801$ image pairs, respectively. In our experiment, we keep the same setup of ozdenizci2023restoring . Specifically, we use the training set to train our model and evaluate the proposed method on Snow100K-S and Snow100K-L.

Raindrop removal. Consistent with previous works qian2018attentive ; li2022all ; valanarasu2022transweather ; ozdenizci2023restoring , we adopt a representative RainDrop dataset qian2018attentive for raindrop removal. The RainDrop dataset includes $861$ synthetic raindrop training images and $58$ images for testing.

Image deraining and dehazing. For this task, we train our GridFormer with Outdoor-Rain dataset li2019heavy , which considers dense synthetic rain streaks and provides realistic scene views. Therefore, this dataset is designed to solve the problem of image deraining and dehazing. It consists of $9,000$ images for training and $750$ for testing.

Multi-weather restoration. Following the previous works li2022all ; valanarasu2022transweather ; ozdenizci2023restoring , we use a mixed dataset called All-weather, in which the training set contains $18,069$ images sampled from Snow100K liu2018desnownet , Raindrop qian2018attentive , and Outdoor-Rain li2019heavy . We use the Snow100k-S/L test sets to evaluate the model’s performance for the image desnowing task. In addition, we adopt the testing sets of the RainDrop dataset and Outdoor-Rain dataset to evaluate the model’s performance for the raindrop removal task and image deraining & dehazing task, respectively.

Implementation details. We implemented GridFormer in PyTorch, using the AdamW optimizer loshchilovdecoupled with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The learning rate is set to $3\times 10^{-4}$ and decreased to $10^{-6}$ using the cosine annealing decay strategy loshchilov2016sgdr . For each task, we train the model with different iterations and patch sizes. At training time we use random horizontal and vertical flips for data augmentation. Following the setup in xiao2022image ; valanarasu2022transweather ; tu2022maxim , we evaluate the performance by PSNR and SSIM calculated in RGB space for image dehazing, and on the Y channel for other tasks.

Comparison methods. The comparison methods for the image dehazing task are traditional method DCP he2010single , CNN-based methods DehazeNet cai2016dehazenet , MSCNN ren2016single , AOD-Net li2017aod , GFN ren2016single , GCANet chen2019gated , GridDehazeNet liu2019griddehazenet , MSBDN dong2020multi , PFDN dong2020physics , FFA-Net qin2020ffa , and AECR-Net wu2021contrastive , and recent transformer-based methods DehazeF-B song2022vision , and MAXIM-2S tu2022maxim . For the task of image desnowing, the comparison methods are SPANet wang2019spatial , JSTASR chen2020jstasr , RESCAN li2018recurrent , DesnowNet liu2018desnownet , DDMSNet zhang2021deep , SnowDiff₆₄ ozdenizci2023restoring , and SnowDiff₁₂₈ ozdenizci2023restoring . As for the raindrop removal task, the comparison methods are pix2pix isola2017image , DuRN liu2019dual , RaindropAttn quan2019deep , AttentiveGAN qian2018attentive , IDT xiao2022image , RainDropDiff₆₄ ozdenizci2023restoring , and RainDropDiff₁₂₈ ozdenizci2023restoring . The comparison methods for the image deraining and dehazing task are CycleGAN zhu2017unpaired , pix2pix isola2017image , HRGAN li2019heavy , PCNet jiang2021rain , MPRNet zamir2021multi , RainHazeDiff₆₄ ozdenizci2023restoring , and RainHazeDiff₁₂₈ ozdenizci2023restoring . Finally, the comparison methods for the multi-weather restoration task are All-in-One li2020all , TransWeather valanarasu2022transweather , Restormer zamir2021restormer , WeatherDiff₆₄ ozdenizci2023restoring , and WeatherDiff₁₂₈ ozdenizci2023restoring .

Dehazing results. We perform image dehazing on different datasets to evaluate the performance of GridFormer. We compare the performance of GridFormer with various methods, including traditional prior-based methods, CNN-based methods, and recent transformer-based methods. Table 2 shows the quantitative results in terms of PSNR and SSIM. It shows that GridFormer achieves the best performance on the indoor subset of SOTS regarding all metrics. In particular, GridFormer obtains a significant gain of $4.23$ dB in PSNR compared to the second-best method MAXIM-2S tu2022maxim .

We further compare the performance on the more challenging Haze4K dataset, which includes more realistic images from both indoor and outdoor scenarios. GridFormer obtains the best performance in terms of all metrics on this dataset as well. Fig. 6 provides a visual comparison for the SOTS indoor dataset. The recovered images by GridFormer contain finer details and are closer to the ground truth.

Desnowing results. We evaluate the desnowing performance on the public Snow100K dataset liu2018desnownet . The test set is divided into three subsets according to the particle size: Snow100K-S, Snow100K-M, and Snow100K-L. We select Snow100K-S and Snow100K-L for testing. Table 3 shows the quantitative results. On the Snow100K-S subset, GridFormer outperforms the diffusion-based method SnowDiff₆₄ ozdenizci2023restoring by $2.3$ dB and by $0.0072$ in terms of PSNR and SSIM. As for the most difficult Snow100K-L subset, GridFormer still gains an improvement of $2.66$ dB and $0.0195$ in terms of PSNR and SSIM compared to the second-best method SnowDiff₆₄. Fig. 7 and Fig. 8 provide the visual comparisons, showing that GridFormer is effective in removing image corruption due to snow while producing perceptually pleasing results.

RainDrop removal results. In Table 4 we present the quantitative results for raindrop removal on the RainDrop dataset. For an extensive comparison, we compare GridFormer with seven different methods: pix2pix isola2017image , DuRN liu2019dual , RaindropAttn quan2019deep , AttentiveGAN qian2018attentive , IDT xiao2022image , RainDropDiff₆₄ ozdenizci2023restoring , and RainDropDiff₁₂₈ ozdenizci2023restoring . The results show that GridFormer is competitive. Specifically, GridFormer achieves the best performance in terms of PSNR and achieves almost the same level of performance as the state-of-the-art method RainDropDiff₆₄ ozdenizci2023restoring in terms of SSIM with a difference of $0.0022$. A visual comparison of the results on RainDrop is provided in Fig. 9. It shows that our method can remove raindrops successfully and generate realistic images.

Deraining and dehazing results. For the image deraining and dehazing task, we conduct experiments on the Outdoor-Rain dataset li2019heavy . This dataset has $9,000$ pairs of images for training, and $750$ pairs for testing, where degraded images are synthesized considering the rain and haze scenes simultaneously. The comparisons between GridFormer and other state-of-the-art methods are reported in Table 5. GridFormer outperforms other competitors in terms of PSNR, and ranks second place regarding SSIM. More specifically, GridFormer achieves $0.11$ dB and $0.46$ dB improvement in terms of PSNR when compared to RainHazeDiff₆₄ ozdenizci2023restoring and MPRNet zamir2021multi . Fig. 10 shows the visual comparison, indicating that GridFormer can handle haze and rainfall scenarios well at the same time, and generate vivid results.

Multi-weather restoration results. We further explore the potential of GridFormer for multi-weather restoration. Specifically, we first train our model on the mixed dataset sampled from Snow100K liu2018desnownet , Raindrop qian2018attentive , and Outdoor-Rain li2019heavy datasets. Then, we evaluate our model on the Snow100k-S/L test sets, the RainDrop test dataset, and the Outdoor-Rain test dataset. We choose four representative multi-weather restoration methods for comparison: All-in-One network li2020all is a CNN-based method, TransWeather is based on transformers, WeatherDiff₆₄ and WeatherDiff₁₂₈ are diffusion models. Table 3, 4, and 5 summarize the quantitative results. GridFormer achieves the best performance in all weather conditions. We present visual comparisons in Fig. 11. Images produced by GridFormer exhibit fewer artifacts and are closer to ground truth compared to other methods. In an additional experiment, we set $C=32$ in the grid head to construct a tiny variant of the network called GridFormer-S for comparison. The results show that our methods achieve competitive results with less complexity and parameters, see Table 5.

Cross-dataset evaluation. To further verify the models’ performance across different datasets, we conduct a cross-dataset evaluation on different SOTA methods. To be specific, the models (i.e., TransWeather, Restormer, WeatherDiff₆₄, WeatherDiff₁₂₈, GridFormer-S, and GridFormer) are trained on the All-weather dataset, and then directly applied to the specific deraining datasets Rain100L yang2017deep and Test100 zhang2019image for testing. Experimental results in Table 6 show that our GridFormer-S and GridFormer outperform other approaches.

Performance in real-world scenarios. To further verify the effectiveness of the proposed method in real-world scenarios, we conduct a qualitative comparison experiment on the real-world hazy image from the Internet. The comparison result is shown in Fig. 12. Compared with current state-of-the-art methods, our method effectively removes the haze and produces a clear result. The result shows that our method outperforms the current methods in real-world scenarios.

Efficiency comparison. We also analyze the efficiency of our models. Table 7 displays the comparison results of different methods in terms of the memory consumption and inference time for $512\times 512$ resolution. Specifically, we choose top three SOTA methods (i.e., Restormer, WeatherDiff₆₄, and WeatherDiff₁₂₈) for comparison. Compared with other SOTA methods, our GridFormer-S exhibits the highest inference time. In addition, GridFormer-S and GridFormer are competitive in terms of memory consumption.

Image restoration in adverse weather conditions can enhance the image content, which can be easily incorporated into other high-level vision tasks. As a result, we investigate the potential of GridFormer in improving the performance of object detection, image segmentation, and image caption algorithms when dealing with adverse weather scenes. In the case of object detection, we consider both synthetic and real-world images. Fig. 13 shows detection results, where we use Google Vision API for object detection. We observe that haze, rain, and snow greatly reduce the detection accuracy, that is, increased missed detection, higher false detection, and lower detection confidence. In contrast, the detection accuracy and confidence of the images recovered by GridFormer show significant improvement over those of weather-degraded images. Fig. 14 showcases the segmentation results, utilizing the state-of-the-art Segment Anything Model kirillov2023segment for image segmentation. Our restoration results demonstrate an enhancement in segmentation accuracies, indicating that GridFormer effectively facilitates subsequent segmentation performance. Lastly, Fig. 15 presents the image caption results using the BLIP model li2022blip . These results show that the BLIP model can generate detailed captions by utilizing our restoration results, further validating the effectiveness of our GridFormer.

We conduct extensive ablation studies to verify the proposed compact-enhanced self-attention, residual dense transformer block, grid structure, and used loss functions. Specifically, we conduct ablation studies on the tasks of raindrop removal and dehazing to analyze the performance of GridFormer. For each model, we train it for $2\times 10^{5}$ iterations using a batch size of $12$ on the RainDrop dataset qian2018attentive and the ITS dataset li2018benchmarking respectively. Subsequently, we assess the performance of each model on the testing sets of the RainDrop dataset and the testing SOTS-Indoor dataset. The detailed results are presented as follows.

A. Compact-enhanced self-attention. We verify the impact of feature sampling, channel split, and local enhancement operations in compact-enhanced self-attention. Table 8 shows the comparison results. After applying feature sampling (FS) and channel split (CS) operations respectively, the model achieves $0.89$ dB and $1.16$ dB improvements in the RainDrop dataset ($0.92$ dB and $2.70$ dB improvements in the SOTS-Indoor dataset), while the computational complexity is significantly reduced. Using local enhancement (LE) operations, the performance gains on the RainDrop and SOTS-Indoor datasets are $0.59$ dB and $1.33$ dB respectively. The ablation study results suggest the effectiveness of these operations. We also conduct an additional ablation study on the RainDrop dataset to verify the effectiveness of exchanging the Values for feature interaction and fusion in our compact-enhanced self-attention. Specifically, we focus on exchanging only the Queries between $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$ to investigate its impact on performance. The results of this experiment demonstrate that exchanging the Queries results in a PSNR value of $31.81$, which is inferior to the outcome achieved by exchanging the Values ($32.57$). These ablation results show the effectiveness of the Value exchange in enhancing interaction and feature fusion, thereby contributing to improved restoration performance. Furthermore, in our model, the choice of the step parameter $r$, as described in formula 3, indeed impacts the model’s computational complexity and performance. Thus, we evaluate the effect of different settings of $r$ in the GridFormer. Table 9 shows that our model with the $[4,2,2]$ setting achieves a better trade-off between computation cost and performance.

B. Residual dense transformer block. To demonstrate the effectiveness of the proposed residual dense transformer block, we conduct ablation studies by considering the following three factors: (1) dense connections (DC), (2) local fusion with $1\times 1$ convolution (LF), and (3) local skip connection (LSC). Specifically, we analyze the different models by progressively adding these components. The results are shown in Table 10. We observe that each component improves the performance, where dense connections contribute the most.

C. Exploring different configurations in the grid structure of GridFormer. To comprehensively understand the impact of GridFormer’s grid structure, we have conducted ablation experiments involving variations in the number of rows and columns. Each row within our GridFormer framework corresponds to a distinct scale, while the columns in the grid fusion module act as conduits that facilitate the exchange of information across diverse scales. This grid structure profoundly influences the information interchange that occurs among the grid units within the Grid Fusion module. In our study, we have systematically altered the number of rows, ranging from $1$ to $4$, while maintaining columns at values of $3$, $4$, $5$, and $6$. The results with different configurations are shown in Table 11. By increasing $r$ and $c$, the performance is improved, and the overhead gradually becomes complex. The model performance achieves its maximum for $r=3$ and $c=6$. Thus, we select these values in our final model.

D. Other GridFormer components. The skip connection from input images and the perceptual loss also contribute to improving the performance. Without the skip connection from the input image, the PSNR value would decrease from $32.57$ dB to $31.85$ dB on the testing set of the RainDrop dataset. Training GridFormer without the perceptual loss results in a PSNR of $32.72$ dB on the testing set of the RainDrop dataset.