Physical Model Guided Deep Image Deraining

As shown in Fig. 1, the proposed network consists of three sub-networks: rain streaks network, rain-free network, and guide-learning network. The first two sub-networks have the same structures that are both encoder-decoder. And in order to learn better spatial contextual information to further guide to restore clear image, the estimated rain streaks and rain-free image are cascaded to input the guide-learning network with multi-stream dilation convolution to further refine the deraining results. Moreover, to restrain the rain streaks network and rain-free network to generate better according results, the add between estimated rain streaks and rain-free images is restrained via $L_{1}$ norm according to rainy physical model 1. Furthermore, MSRB is designed to acquire multi-scale information by combining the multi-scale operations and residual block.

Multi-scale features have been widely leveraged in many computer vision systems, such as face-alignment [20], semantic segmentation [21], depth estimation [22] and single image super-resolution [23]. Combining features at different scales can result in a better representation of an object and its surrounding context. Therefore, multi-scale residual block (MSRB) is proposed that is the concatenation between different scales of feature maps and the residual block, as shown in Fig. 2.

We describe the MSRB mathematically: Firstly, we utilize $Pooling$ operation with different size of kernels and strides to obtain the multi-scale features:

$$y_{i}=Pooling_{i}(x),i=1,2,4.$$

(2)

where $Pooling_{i}$ denotes $Pooling$ operation with $i\times i$ kernel and stride.
Lastly, all the scales are fused and feed into three convolution layers then add the original input signal $x$ to learn the residual:

$$z=H(Cat[Up_{1}(y_{1}),\cdots,Up_{4}(y_{4})])+x.$$

(3)

where $Up_{i}$ denotes $i\times$ Upsampling operation and $Cat$ denotes concatenation operation at the channel dimension. $H$ denotes a series of operations that consist of two $3\times 3$ and one $1\times 1$ convolution operations. The MSRB can learn features with different scales and all different features are fused to learn the primary feature.

We use $L_{1}$-norm as the loss function.

For the rain streaks network and rain-free network:

$$L_{rain}=\lVert\widetilde{\bm{R}}-{\bm{R}}\rVert_{1},$$

(4)

$$L_{rain-free}=\lVert\widetilde{\bm{B}}-{\bm{B}}\rVert_{1},$$

(5)

where $\widetilde{\bm{R}}$, $\widetilde{\bm{B}}$ denote the estimated rain streaks layer and clean background image, ${\bm{R}}$ and ${\bm{B}}$ denote the ground truth of rain streaks and rain-free image. For guide-learning network:

$$L_{guide}=\lVert\widehat{\bm{B}}-{\bm{B}}\rVert_{1},$$

(6)

where $\widehat{\bm{B}}$ denote the output of guide-learning network, i.e. the final estimated rain-free image.

Moreover, we compute the $L_{1}$-norm of the input rainy image $\bm{O}$ and the sum of $\widetilde{\bm{R}}$, $\widetilde{\bm{B}}$ in order to constrain the solution space of rain streaks and rain-free network according to the rainy physical model 1:

$$L_{p}=\lVert\widetilde{\bm{B}}+\widetilde{\bm{R}}-\bm{O}\rVert_{1},$$

(7)

So the overall loss function is defined as:

$$L=L_{guide}+\alpha L_{rain}+\beta L_{rain-free}+\gamma L_{p},$$

(8)

where $\alpha,\beta,\gamma$ are constant.

Synthetic Datasets

. We carry out experiments to evaluate the performance of our method on three synthetic datasets: Rain100H, Rain100L, and Rain1200, which all have various rain streaks with different sizes, shapes, and directions. There are 1800 image pairs for training and 200 image pairs for testing in Rain100H and Rain100L. In Rain1200, 12000 images are for training and 1200 images for testing. We choose Rain100H as our analysis dataset.

Real-world Testing Images

. We also evaluate the performance of our method on real-world images, which are provided by Zhang et al. [16] and Yang et al. [7]. In these images, they have different rain components from orientation to density.

Training Settings

. In the training process, we randomly crop each training image pairs to $160\times 160$ patch pairs. The batch size is chosen as 64. For each convolution layer, we use leaky-ReLU with $\alpha=0.2$ as the activation function except for the last layer. We use ADAM algorithm [24] to optimize our network. The initial learning rate is $5\times 10^{-4}$, and is updated twice by a rate of $1/10$ at 1200 and 1600 epochs and the total epoch is 2000. $\alpha,\beta$ and $\gamma$ are set as 0.5, 0.5 and 0.001, respectively. Our entire network is trained on 8 Nvidia GTX 1080Ti GPUs based on PyTorch.

Evaluation Criterions

. We use peak signal to noise ratio (PSNR) and structure similarity (SSIM) to evaluate the quality of the recovered results in comparison with ground truth images. PSNR and SSIM are only computed for synthetic datasets, because not only the estimated rain-free images are needed, but also corresponding ground truth images during the computing process. For real-world images, they can only be evaluated by visual comparisons.

Tab. 1 shows quantitatively comparative results between our method and six state-of-the-art deraining methods on Rain100H, Rain100L and Rain1200. There are two conventional methods: DSC [1] (ICCV15) and LP [4] (CVPR16), and four deep learning-based methods: DDN [6] (CVPR17), JORDER [7] (CVPR17), RESCAN [17] (ECCV18) and PReNet [19] (CVPR19). As we can see that our proposed method outperforms these state-of-the-art approaches on the three datasets.

We also show several challenging synthetic examples for visual comparisons in Fig. 3. As the prior based methods are obviously worse than deep learning-based methods according to Tab. 1, we only compare the visual performances with deep learning methods. The first column in Fig. 3 are synthetic images that are severely degraded by rain streaks. Fig 3 (b) and Fig. 3 (c) are the results of DDN [6] and JORDER [7]. Obviously, they both fail to recover an acceptable clean image. Fig. 3 (d) and Fig. 3 (e) are the results of RESCAN [17] and PReNet [19], which have unpleasing artifacts in the masked boxes. As shown in Fig. 3 (f), our results generate best deraining results no matter in quantitatively or visually.

To evaluate the robustness of our method on real-world images, we also provide two examples on real-world rainy datasets in Fig. 4. For the first example, our method generates the clearest and cleanest result, while the other methods remain some obvious artifacts or rain streaks. For the second example, the other methods get unpleasing artifacts in the masked box, while our approach generates better clear results. We provide more examples on both synthetic and real-world datasets in our supplemental materials.

Exploring the effectiveness of multi-scale manner and the restraint of physical model in our network is meaningful. So we design some experiments with different combinations of the proposed network components, such as three sub-networks, multi-scale structure, multi-stream dilation convolution, and $L_{p}$. Tab. 2 shows the comparative results and W and W/O mean whether using the multi-scale structure or not. We can observe that the multi-scale manner boosts deraining performance on all models. This illustrates that our designed multi-scale manner is meaningful. Furthermore, Tab. 3 compares the effectiveness of multi-stream dilation convolution and physical model constraint $L_{p}$. Fig. 5 provides the outputs of three sub-networks on two real-world rainy images. As we can see, the cropped patches in Fig. 5 (d) perform better than Fig. 5 (c) in detail and texture information, which demonstrates that the guide-learning network is effective in our proposed network.

• •

  M_1: Only rain streaks network.

• •

  M_2: Only rain-free network.

• •

  M_3: Only input the estimated rain-free image to guide-learning network.

• •

  M_4: (Default) the input is the concatenation of the estimated rain streaks and rain-free image to the guide-learning network.

• •

  R_1: Our proposed network without multi-stream dilation convolution.

• •

  R_2: Our proposed network without $L_{p}$.

• •

  R_3: Our proposed network with multi-stream dilation convolution and $L_{p}$, i.e. our proposed final network.

$L_{p}$ is the $L_{1}$-norm of subtraction between the rainy image and the direct sum of the estimated two images from the first two sub-networks.