Structural Residual Learning
for Single Image Rain Removal

Garg and Nayar [23, 24] initially discussed the visual effects of raindrops on imaging systems and proposed a video deraining method, by capturing dynamics of raindrops through a space-time correlation model and describing the photometry of rain via a motion blur model. Later, researchers made more investigations on exploring physical properties of rain streaks for video deraining, like temporal-chromatic [25, 26] and spatio-temporal frequency characteristics [27, 28].

In the past years, more intrinsic prior structures of rain streaks and background scenes of rainy videos have been analyzed and formulated for deraining algorithm design. For example, Chen et al. [29] investigated the non-local similarity and repeatability of rain streaks and provided a low rank model. To remove rain and snow from a video, Kim et al. [30] adopted a low rank matrix completion strategy. To handle heavy rain streaks and dynamic scenes, Ren et al. [31] decomposed rain streaks into two classes: sparse ones and dense ones, and detected them with different models. Recently, Wei et al. [32] used stochastic manner to encode rain streaks as a patch based mixture of Gaussian. Li et al.[21] further explored the prior structure of rain streaks and described that they have the repetitive local patterns and multi-scale characteristics. Based on the observation, the authors proposed a multi-scale convolutional sparse coding model that achieves the state-of-the-art deraining performance when the background layer in a video is also finely extracted.

Very recently, deep learning techniques have achieved great success in various low-level vision tasks, such as image denoising [33, 34], image super-resolution [35, 36, 37], and image deblurring [38, 39]. Recent years have also witnessed the development of deep learning in the deraining task [40]. To remove rains from videos, Liu et al. [41] provided a hybrid rain model and constructed a multi-task learning network architecture that successively accomplished rain degradation classification, rain removal, and background restoration. Further, the authors developed a dynamic routing residue recurrent network [42] for handling dynamically detected rainy videos. Compared with these video deraining methods, the single image deraining task is much more challenging in practice due to the lack of temporal information.

Against this single image deraining task, Xu et al. [6] considered the chromatic property of rain streaks and achieved a coarse rain-free image via a guided filter. Kim et al. [43] analyzed the geometric property of rains to detect rain streak regions, and then reconstructed the derained result by executing nonlocal means filtering on the detected region.

Later on, researchers resorted to utilizing domain knowledge to encode rains for helping the deraining task [44, 45]. For example, based on morphological component analysis, Fu et al. [46] considered single image deraining as a signal decomposition problem. Then the authors executed a bilateral filter, dictionary learning, and sparse coding to acquire LFP and HFP for obtaining the derained result. Afterwards, Luo et al. [10] adopted a screen blend model and proposed to utilize high discriminative codes over a learned dictionary to sparsely approximate rain and background layers. To represent multiple orientations and scales of rain streaks, Li et al. [9] designed GMM based patch prior. By employing the sparsity and gradient statistics of rain and background layers, Zhu et al. [20] formulated three regularization terms to progressively extract rain streaks. Afterwards, Gu et al. [11] utilized analysis sparse representation to represent image large-scale structures and synthesis sparse representation to describe image fine-scale textures. Meanwhile, Zhang et al. [19] proposed to learn a set of sparsity based and low rank based convolutional filters for illustrating background and rain layers, respectively. Albeit achieving good performance on certain scenarios, these prior-model-based methods need inevitable time-consuming iterative inferences and always could not finely fit practical diverse rain shapes in real rainy images due to relatively simple while subjective prior assumptions.

More recently, some CNN based DL methods have been raised for the task [16, 47, 18]. Fu et al. [12] first designed the DerainNet to predict clean image. To make training process easier, the authors [13] further proposed DDN to remove rain content in HFP. Later, Zhang et al. [47] proposed a conditional generative adversarial deraining network. Considering the complicatedness of rain streaks, the authors [48] further incorporated a residual-aware classifier process and designed a rain density-aware multistream dense network. Furthermore, Yang et al. [14] reformulated the commonly-used rain model and developed a multi-task architecture to jointly learn binary rain streak map, appearance of rain streaks, and clean background. To achieve better visual quality, the authors further introduced a detail preservation step [15]. Very recently, by repeatedly unfolding a shallow ResNet with a recurrent layer, Ren et al. [17] presented a simple baseline network, PReNet. To make the network better reflect the prior knowledge of rain structures, some beneficial attempts have also been made. For example, Mu et al. [49] proposed to implicitly describe prior structures by data-dependent networks and then formulate them into bi-layer optimization iterations. To alleviate the hard-to-collect-training-sample and overfitting-to-training-sample issues, Wei et al. [50] formulate rain layer prior as GMM and train the backbone, DDN, in a semi-supervised manner. Albeit good performance, these DL methods embed these beneficial prior rain knowledge inside the network architecture [49] or on the loss formulation for training the network [50]. The high flexibility of the nonlinear network mapping might output unexpected mappings deviated from practical rain shapes especially in the testing stages with non-seen rain configurations in training samples. Their generalization performance could thus possibly be negatively influenced. This is the main motivation of this study, to make network extract rain layer, possibly complying with the intrinsic rain structures the previous researches have been explored, from a rainy image.

An input rainy image is denoted as $\mathbf{O}\in\mathbb{R}^{H\times W}$, where $H$ and $W$ denote its height and width, respectively. The commonly-used rainy image model is:

where $\mathbf{B}$ and $\mathbf{R}$ represent the background layer and the rain layer of the image, respectively. The goal of DL based single image derainers is to design rational network architectures to learn non-linear mapping functions from an input rainy image $\mathbf{O}$ to its background layer $\mathbf{B}$ or the residual rain layer $\mathbf{R}$.

Our aim is to construct a network architecture with its output residual capable of sufficiently representing prior rain structures explored by previous investigations, so as to rectify a rational and interpretable network prediction even for those non-seen and complicated rainy image inputs. Specifically, three known priors have been considered in this network design task. The first is the local-repetitive-pattern prior as proposed in [11] and  [19]. That is, rain streaks always repeatedly appear at different locations over a rainy image with similar local patterns like shape, thickness, and direction. The second is the coefficient-sparsity prior, which describes the fact that the rain streaks are always sparsely scattered over an image [10, 20]. The third is the multi-scale prior [21], which refers to the phenomenon that rain streaks in different distances appear with different sizes in a rainy image since they are pictured from different distances by cameras. Based on this, the single image rainy model (1) can be more elaborately formulated as:

where $\mathbf{R}_{s}$, $\mathbf{R}_{m}$, and $\mathbf{R}_{l}$ denote three different scales of rain layer separation, respectively.

We then introduce how to construct a deep network with carefully designed structural residual output finely conveying all the aforementioned prior knowledge of rain streaks.

In this section, we construct a single image deraining network to learn structural residual output. As shown in Fig. 2, the proposed network, called structural residual network (SRNet), mainly consists of three parallel sub-networks with similar structures but different dilated factors (DF), to extract rain layer at different scales. The rain-removed result $\widetilde{\mathbf{B}}$ is estimated by subtracting the entire residual rain layer $\widetilde{\mathbf{R}}$ from the input $\mathbf{O}$. The design details are described as follows.

As the sensitivity of the human visual system depends on local luminance, contrast, and structure, this can be quantitatively measured by the structure similarity (SSIM) index [54]. The larger the SSIM value is, the better the quality of the image is. To train the SRNet, similar to [17], we simply adopt the negative SSIM as the objective function, written as:

We use PyTorch[52] to implement the proposed SRNet, based on a PC equipped with an Intel (R) Core(TM) i7-8700K at 3.70GHZ and one Nvidia GeForce GTX 1080Ti GPU. Since we mainly care about the effectiveness of the network architecture on rain removal, instead of relying on some extra tricks such as designing complicated loss functions and tuning hyper-parameters, we directly adopt the parameter settings of the latest baseline network–PReNet [17]. Specifically, the patch size is 100$\times$100 and the Adam optimizer[55] with the batch size of 18 is used. The initial learning rate is $1\times$$10^{-3}$ and divided by 5 after reaching 30, 50, and 80 epochs. The total epoch is 100. It is worth mentioning that for all datasets in subsequent experiments, these parameter settings are the same. This would show the favorable robustness and generality of our method. Besides, to balance the trade-off between running time and deraining performance on synthetic and real datasets, we simply select depth $T=2$ and width $N=64$ as the default setting in our experiments, where the depth $T$ denotes the times of symmetrical downsampling/upsampling operations, namely, the number of the module Resblock+MaxPooing (Resblock+MaxUnpooing) in Fig. 2(b).

Synthetic Datasets. We adopt four benchmark datasets: Rain100L [15], Rain100H [15], Rain1400 [13], and Rain12 [9]. Rain100L has only one type of rain streaks and consists of 200 image pairs for training and 100 ones for evaluation. With five types of rain streak directions, Rain100H is more challenging and contains 1800 image pairs for training and 100 ones for testing. Rain1400 contains 14000 rainy images synthesized from 1000 clean images with 14 kinds of rain streaks, where 12600 rainy images are used as training samples and 1400 ones as testing samples. Like [17], the trained model for Rain100L is utilized to evaluate Rain12 that only includes 12 image pairs. For SIRR, we adopt the real 147 rainy images [50] as unsupervised training samples.

Performance Metrics. Since the groundtruths of synthetic datasets are available, we provide quantitative comparisons based on two commonly used metrics: peak-signal-to-noise ratio (PSNR) [56] and SSIM. As the human visual system is sensitive to the Y channel of a color image in YCbCr space, we compute PSNR/SSIM based on the luminance channel, and the Matlab computation codes are released by  [15].

Evaluation on Rain Removal. From the derained results shown in Fig. 6, it is easy to see that traditional model-based DSC, GMM, and JCAS methods leave distinct rain streaks, and DL-based Clear, DDN, and SIRR also leave evident rain marks. Besides, apart from RESCAN, PReNet, and the proposed SRNet, other competing methods blur image textures obviously. From the comparison of extracted rain layers, it is observed that the result by the SRNet covers less texture. Fig. 7 further compares the deraining results on another test image with diverse rain patterns. As displayed, among all comparison methods, SRNet more evidently removes rain streaks and restores background image.

Evaluation on Detail Preservation. We select two more difficult samples from Rain100H and Rain1400, respectively, to evaluate the deraining ability of different methods. As depicted in Fig. 8 and Fig. 9, complicated rain patterns adversely degrade the rain removal performance of most methods. However, the proposed SRNet still has advantages over rain removal and detail preservation, and achieves best PSNR and SSIM scores.

Tables I reports the quantitative comparison results of all competing methods on synthetic datasets with diverse, complicated, and different rain types. It is clear that due to the strong fitting ability of CNN, the deraining performance of DL based methods generally outperforms those by conventional prior-model-based methods. Comparatively, the proposed SRNet achieves the best average PSNR and SSIM values on Rain1400 with 14 kinds of rain types. This reflects the better robustness and generality of SRNet.

Real Datasets and Performance Metrics. For real application, evaluating the generalization ability of a single image derainer is the key point. In this section, we evaluate the deraining performance of all competing methods on real rainy images, based on two real datasets from [18] and  [50], respectively. The former is denoted as SPA-Data and contains 1000 image pairs for evaluation, and the other is called Internet-Data and consists of 147 rainy images without groundtruths. For the experiments on SPA-Data, we compute PSNR and SSIM based on the Y channel. While for Internet-Data, we can only provide visual comparison. Specifically, for each one among the DL-based comparison methods, we can obtain three pre-trained models based on the three synthetic datasets: Rain100L, Rain100H, and Rain1400, respectively. For each method, we select the model with the best PSNR/SSIM on SPA-Data to predict the generalization results for real rainy images from SPA-Data. While for Internet-Data, like previous works, the rain-removed result is the one with the best visual quality among the predicted results by the three models.

Evaluation on SPA-Data. Fig. 10 shows the derained results of all comparison methods on a real rainy image from SPA-Data. As seen, when dealing with the images with complex rain shapes, traditional model based DSC, GMM, and JCAS do not work well, which can be explained by that the adopted specific prior assumptions cannot always accurately represent the complicated rain distribution. DL based competing methods also leave obvious rain streaks and even blur image details, such as PReNet. Nevertheless, as compared with other methods, the proposed SRNet achieve better generalization performance.

To objectively evaluate the generalization ability of these methods, we provide the quantitative rain removal performance comparison on SPA-Data listed in Table II. As reported, the most competitive baseline methods on synthetic datasets, such as RESCAN and JORDER_E, cannot perform as well as the traditional model-based JCAS on the real dataset, possibly due to the overfitting-the-training-samples issue. It is worth mentioning that in this case, JCAS obtains a good SSIM as the method utilizes the complementary of ASR and SSR to finely describe texture structures. Although SPANet has relatively inferior deraining effect on synthetic datasets, its generalization performance is competitive. Observing the comparison results presented in Table I and Table II, it is easy to see that among all comparison methods, the proposed SRNet is a competing single image derainer as it achieves satisfactory rain removal performance on both synthetic and real datasets. This confirms the effectiveness of structural residual design of our network architecture and the fine generalization ability of SRNet.

Evaluation on Internet-Data. Furthermore, we select another two hard samples from Internet-Data with complicated rain distribution. Fig. 11 and Fig. 12 demonstrate the corresponding generalization comparison results. From these figures, similar to the experiment analysis on SPA-Data, traditional model-based methods always leave severe rains, and other DL based methods still have obvious rain marks and blur image details. Comparatively, the proposed SRNet has evident superior performance in rain removal and detail preservation.

Here we select several rainy images with different rain shapes from aforementioned datasets to further evaluate the effectiveness of the proposed SRNet on extracting rain layer. As shown in Fig. 13, when dealing with complicated rainy images with diverse rain streaks (light/heavy density, short/long shape, different directions), the proposed method can finely extract rain layer and preserve background details well.

The Effect of Training Datasets. Based on some typical real rainy images, Fig. 14 shows the derained results predicted by SRNet trained on the three synthesized datasets, including Rain100L, Rain100H, and Rain1400. For each input rainy image, we use a box to indicate the best recovery result visually. It can be observed that the model trained on Rain100L performs well in the case of light/thin rain streaks; the trained model on Rain100H is applicable to heavy/accumulated rain types, and for Rain1400, it is suitable for long and thin ones.

Network Modules Analysis. Here we quantitatively analyze the importance of different modules of the proposed SRNet, as listed in Table III. We train all the variants on Rain100L and compare their generalization ability on SPA-Data. $B_{a}$, $B_{b}$, $B_{c}$, and $B_{d}$ denote four variants of only using the small scale branch (DF=1) in Fig. 2(a). $B_{a}$ and $B_{b}$ are basic Resnet-like networks, only consisting of Resblocks. $B_{c}$ and $B_{d}$ include symmetrical downsampling and unsampling layers following the corresponding Resblocks. $B_{f}$ is the default SRNet. $B_{e}$ is a variant that SRNet adopts weight sharing among three parallel branches, which shows that the SRNet can notably reduce network parameters by weight sharing without unsubstantial degradation in generalization performance. It is worth mentioning that even $B_{d}$ only adopts small scale, it outperforms most other competing methods on real deraining performance as shown in Table II, which corroborates the rationality of the proposed encoder-decoder network. The versus between every two variants and the corresponding functioning module is concluded as Table IV.

Finally, we evaluate the deraining ability of the SRNet on real rainy videos without groundtruth, by comparing with current state-of-the-art video deraining methods, including the traditional model-based Garg et al. [24], Kim et al. [30], Jiang et al. [57], Ren et al. [31], Wei et al. [32], Li et al. [21], and the DL-based Liu et al. [41].

Fig. 15 and Fig. 16 present the visual comparison results based on two real rainy frames: one is captured by surveillance systems on a street with complex moving objects, and the other is obtained at night. From Fig. 15, it is easily seen that the model-based video deraining methods, including Garg et al.’s, Kim et al.’s, Jiang et al.’s, Wei et al.’s, and Li et al.’s, cause different degrees of artifacts at the location of the moving car, and the DL based Liu et al.’s method leaves obvious rain streaks. Even not utilizing temporal information among multi-frames, the proposed single image deraining method, SRNet, still performs well on removing rain streaks and preserving background textures. From Fig. 16, SRNet has the ability to detect obvious rain streaks. Although some model-based video deraining methods can remove more rain streaks than SRNet, they need many frames and thus have unfavorable real-time performance. It is noteworthy that as compared with the DL-based Liu et al.’s video deraining method, SRNet can also preserve relatively more image details, like trunks.