Light Field Raindrop Removal via 4D Re-sampling

Due to the lack of texture details in occluded areas, the single image raindrop removal problem is much more challenging. The progress of this task has been stagnated until the rapid progress of deep learning in recent years.

[3] used three convolutional layers to remove raindrops and generated outputs with relatively poor performance. [12] proposed a generative adversarial network for raindrop removal, where the generative network produces the attention map via an attentive-recurrent network and applies this raindrop attention mask to generate a raindrop-free image through a contextual auto-encoder. [14] proposed a network, in which the shape-driven attention module exploits the physical shape properties of raindrops, including closedness and roundness, and the channel attention refines the features relevant to the background layer or the raindrop layer. [16] designed a uncertainty guided multi-scale attention network which benefits from the exploration of the blur-level of raindrops, and utilized several basic modules to explore the inherent correlations of the similar raindrop patterns across scales.

Since there are no reference algorithms in LFRR, we briefly present some approaches in related LF based fields in this subsection.

The LF is a 4D data with unique structural characteristics. A hot and general research topic is to design appropriate modules to extract effective information from the LF. [23] proposed a spatial-angular separable convolution module, in which two convolution layers are separately and serially extract features in spatial or angular dimensions. [20] designed an interaction mechanism to incorporate decoupled spatial and angular information. The angular feature in this model is extracted by a no-padding convolution layer which contains the global angular information. [17] devised a module named Multi-Dimension Fusion Block (MDFB) which utilizes four parallel convolution layers to extract features from sub-aperture images, micro-lens images, horizontal and vertical epipolar images (EPIs) respectively. The MDFB is also adopted in our model for LF feature extraction.

The LF spatial super-resolution (LFSSR) task has many similarities with LFRR, and LFSSR approaches are conveniently adjusted to solve the LFRR task. Firstly, both LFRR and LFSSR process all views of the LF, while many typical LF tasks only deal with the central view of the LF, such as saliency detection [8], depth estimation [1] and occlusion removal [26]. Secondly, they both are low-level image restoration tasks and aim to enhance the quality of the input LF. Thirdly, they both need effective complementary information and structural characteristics existed within and between views of the LF. Besides, existing LFSSR approaches up-sample features and LFs by spatial interpolation, transposed convolution or pixel shuffle. By removing up-sampling operations, most LFSSR approaches can be utilized to handle the LFRR task.

The goal of the LFRR is to restore the background area obscured by raindrops in each view of the LF. The input LF with raindrops is represented as $I(u,v,x,y)\in\mathbb{R}^{U\times V\times X\times Y\times 3}$, where $(U,V)$ and $(X,Y)$ are angular and spatial resolutions, respectively, and the number $3$ refers to the RGB color channels. The overall architecture is divided into two parts: the re-sampling module (RSM) and the refinement module (RM), as shown in Fig.3.

For each pixel in $I$, the RSM first predicts re-sampling positions, and then carries out the proposed 4D interpolation on $I$ according to re-sampling positions to obtain the sampled LF which is also the initial raindrop removal output $O_{i}$:

The RM extracts features from $I$ and $O_{i}$ and predicts a residual map $R$ to refine $O_{i}$:

The final raindrop removal output $O_{f}$ is the summation of $O_{i}$ and $R$:

where $R$, $O_{i}$ and $O_{f}$ have the same size with $I$.

There are two weaknesses existed in the re-sampling strategy. On one hand, it is difficult for pixel re-sampling strategy to recover the region completely polluted by raindrops at all views. On the other hand, since the re-sampling position is fractional, the 4D interpolation will inevitably bring slight pixel errors, e.g., pixels in clean backgrounds are slightly affected by surrounding pixels.

The RM is a module strongly coupled with the RSM for addressing shortcomings of the RSM. As shown in the bottom part of Fig.3, the RM first extracts features from $I$ and $O_{i}$, and then generates a residual map $R$ to refine $O_{i}$. The raindrop LF $I$ is introduced to the RM for ensuring information integrity.

Specifically, $I$ and $O_{i}$ are first separately fed to a weight-shared 2D convolution layer which works on spatial dimensions to generate the initial feature $F^{0}_{I}$ and $F^{0}_{O_{i}}$. Considering that the difference between $I$ and $O_{i}$ reflects on the areas of raindrops and wrong re-sampled backgrounds, the difference is beneficial for identifying the areas to be refined. Therefore, we propose a novel Difference Guided Encode Block (DGEB) to focus more on areas where need improvement by leveraging the guidance of the difference between $I$ and $O_{i}$.

As shown in the gray box of the Fig.3, the first DGEB takes the pair of $F^{0}_{I}$ and $F^{0}_{O_{i}}$ as inputs to achieve interaction. Firstly, $F^{0}_{I}$ and $F^{0}_{O_{i}}$ are separately encoded by a weight-shared MDFB, and the generated features are called the encoded $F^{0}_{I}$ and the encoded $F^{0}_{O_{i}}$. Then, the difference feature $F^{1}_{diff}$ are calculated by the element-wise subtraction between the encoded $F^{0}_{I}$ and the encoded $F^{0}_{O_{i}}$. Next, the difference feature $F^{1}_{diff}$ are fed to two $1\times 1$ convolution layers to produce two single-channel attention maps. Finally, these two attention maps are separately fused with the encoded $F^{0}_{I}$ and the encoded $F^{0}_{O_{i}}$ by element-wise multiplication and summation. The outputs of the first DGEB are denoted as $F^{1}_{I}$ and $F^{1}_{O_{i}}$. In RM, we cascade $N$ DGEBs for feature extraction and interaction, i.e., the outputs of a DGEB form the inputs of its subsequent DGEB, and the final output features are represented as $F^{N}_{I}$ and $F^{N}_{O_{i}}$. In summary, the DGEB can be formulated as

where $F^{k}_{I}$ and $F^{k}_{O_{i}}$ represent the output features corresponding to $I$ and $O_{i}$ of the $k^{th}$ DGEB, respectively, $H^{(k)}$ represent the MDEB in the $k^{th}$ DGEB.

The outputs $F^{N}_{I}$ and $F^{N}_{O_{i}}$ are then concatenated along the channel dimension. Inspired by [5], the Squeeze-and-Excitation Block (SEB) is adopted here for better information fusion. Suppose the input feature is $F$, The calculation process of the SEB is:

where $AvgPool$ is the global spatial average pooling, $Linear$ refers to a full-connection layer, and $\sigma$ is the sigmoid function. The residual map $R$ is produced by 2D spatial convolution layers from the fused features. The final raindrop removal output $O_{f}$ is the summation of $O_{i}$ and $R$.

Besides the widely used Mean Absolute Error (MAE), we utilize the SSIM function [21] and EPI gradient loss [7] as additional evaluation metrics for the generated LF. These three loss functions are denoted as $L_{MAE}$, $L_{SSIM}$ and $L_{EPI}$. The combination of the MAE loss function and SSIM loss function can preserve the per-pixel similarity as well as the local structure. The EPI gradient loss is beneficial for enhancing view consistency of the output LF.

Given the raindrop-free ground truth $Y$ and the model’s output $\hat{Y}$, the learning Objective of $\hat{Y}$ is defined as:

where $\alpha$ and $\beta$ denote the hyper parameters used to balance losses, which are set to 0.1 and 1 in the experiment.

The whole training process of the proposed model is divided into two stage.

In the first stage, both $O_{i}$ and $O_{f}$ are supervised by the raindrop-free ground truth $Y$. The supervision of $O_{i}$ is necessary during the initial training, because it enables the model to effectively search the raindrop-free backgrounds in $I$. The learning objective of this stage is

where the weight hyper parameter $\lambda$ is set to 0.5 in the experiment. When the model converges, the training enters the second stage.

In the second stage, we remove the supervision of $O_{i}$ to make the model pay attention to learning the prediction of the final result $O_{f}$. The learning objective of the second stage is

When the model converges again, the training ends. The second training stage brings slight performance improvement (about 0.15dB PSNR).

We compare the proposed model with six 2D image rain removal methods: AttGAN [12], RSECAN [10], SDA [14], PIDN [15], DualRes [11] and MSPIR [24]. Among them, RSECAN, PIDN and MSPIR are rain streak removal algorithms. Because these three algorithms do not model the physical properties of rain streaks, but make improvements in the design of the convolutional networks, all of them can be transferred to the raindrop removal task.

For a more convincing comparison, four LF spatial super resolution methods are adjusted for LFRR task which are SAS [23], MDFN [17], InterNet [20], MEGNet [25]. As mentioned in Section 2.2, algorithms in LF spatial super resolution domain reconstruct all views of the LF by extracting complementary information between different views, which has many similarities to solve the LFRR problem. Besides, these four methods are convenient to be adjusted to handle LFRR problem by removing the up-sampling operations.

We train and validate models on the proposed LFRR dataset. For 2D methods, views in LFs are separated, e.g., a pair of LFs with $7\times 7$ angular resolutions is equivalent to $49$ 2D image pairs. We augment training data for 2D methods by random cropping, flipping, rotating and resizing. The popular $4$ LF super resolution methods are carefully adjusted by removing their up-sampling operations. Apart from selecting appropriate initial learning rate, we maintain the original configuration of algorithms involved in comparisons. The quantitative evaluation is done in terms of two metrics: PSNR and SSIM calculated on the luminance (Y channel of the YCbCr space).

The proposed model is implemented by PyTorch. All experiments are run on one NVIDIA GeForce GTX A4000. The designed model has $M=5$ MDFBs and $N=5$ DGEBs. Each MDFB or DGEB has the same number of input and output channel $c=64$. In training, we keep the angular size $7\times 7$ and randomly crop patches of spatial size $96\times 96$ as inputs. To relieve the over-fitting problem on the premise of protecting the structural characteristics of LFs, we augment the training set by random global flipping and rotating for 4D approaches. The proposed model is trained by Adam [9] optimizer with batch size of 1. The initial learning rate is $3e^{-4}$, and the number of epochs is set to $4000$. We use cosine annealing strategy to adjust learning rate during the training, and set the final minimum learning rate to 0. The total training time for the proposed model is approximately 2 days.

See Fig. 5 for the visualization of some results of five top ranking methods.

In image restoration quality, 2D methods generate significant artifacts and wrong textures, and even poorly affect raindrop-free backgrounds, which can be attributed to the limited scale of training set and the lack of details in the occluded background. For other 4D methods, the restored background areas are not smooth and natural enough. These methods rely on convolutional neural network to implicitly predict the residual between input and output, which is unstable. By comparisons, our model first removes raindrops by directly utilizing useful pixel information in raindrop LFs, which reduces the dependence on and the difficulty of residual learning. In Fig.5, our results show the clearest and most real details in the restored backgrounds.

Because the LFRR is to deal with all views of the LF, the view consistency is also an important measurement. In Fig.5, we show horizontal EPIs corresponding to the dotted lines for view consistency comparisons. While the EPIs of other methods contain noisy artifacts, our EPIs have the most consistent lines compared with ground truth.

The quantitative comparisons are shown in Table.1. By comparison, our model achieves the best performance in both PSNR and SSIM with light-weight parameters. It is noted that compared with the second best method InterNet, our model generates better results with only one twentieth of its learnable parameters.

According to the Table.1, the overall performance of 2D methods is not as good as that of 4D methods. 2D models benefit for more available data sets and higher upper limitation of model parameters than 4D models. However, when training in the current small-scale training set, these 2D methods have two obvious disadvantages. Firstly, because texture details covered by raindrops in single image are completely lost, when restoring the occluded background areas, 2D methods rely more on the inpainting patterns learned from the training data, while the small-scale training set provides little inpainting patterns. Secondly, compared with the LF based methods, due to the lack of scene structure characteristics in single image, 2D methods are weak in the design of the learning objective.

It can be seen that SDA performs worse than other 2D approaches. The method SDA introduces the edge map as additional input, and builds module to exploit the physical shape properties of raindrops, including closedness and roundness. However, since the camera focuses on the background when shooting, raindrop areas in our dataset have severe blur and grid effects, which brings great trouble to SDA. Besides, some scenes in our dataset contain rain stripes that are also hard for SDA to deal with.

Compared with 2D inputs, the LF with unique structural characteristics provides abundant complementary information, which enables 4D approaches more dependent on the input LF rather than the inpainting patterns learned from the training set. Therefore, 4D methods can achieve great raindrop removal performance by training on a small-scale dataset.

Apart from the 4D re-sampling in both spatial and angular dimensions mentioned above, re-sampling can be separately conducted in the spatial or angular dimensions. Since the re-sampling module and the refinement module are strongly coupled, we conduct two ablation experiments with the refinement module to evaluate re-sampling patterns:

• •

  2D spatial re-sampling with the refinement module.

• •

  2D angular re-sampling with the refinement module.

In both cases, the model predicts 2D location offset, then performs the bilinear interpolation on $I$ in corresponding dimensions according to the predicted re-sampling positions. The results are listed in the first two lines of Table 2.

It is hard to recover the occluded background areas by spatial re-sampling due to the lack of texture details. The useless spatial re-sampling also hinders the subsequent processing. As shown in the first row of Table.2, the performance of the model using spatial re-sampling is the worst.

When re-sampling only on angular dimensions, the model can obtain effective information from other views to supplement background details. However, due to the disparity and occlusion existed in backgrounds, angular re-sampling strategy is difficult to accurately obtain the details of the occlusion boundaries in backgrounds. By comparison, 4D re-sampling can alleviate this problem by providing a wider search space. In the second row of Table.2, the model using angular re-sampling performs better than the spatial re-sampling model and worse than the 4D re-sampling model, which not only verifies the effectiveness of the angular re-sampling, but also confirms the superiority of 4D re-sampling. It is worth mentioning that the proposed model using angular re-sampling already performs better than all approaches involved in comparisons.

As mentioned in Section 4.3, the RM is designed to solve the shortcomings of the re-sampling strategy. In this subsection, we conduct two ablation experiments to separately verify the effectiveness of the RM and the DGEB:

• •

  Abandon the refinement module.

• •

  Replace the DGEB in the refinement module with the MDFB.

For the former ablation experiment, we enlarge the channel number of the RSM to $80$ to ensure the similar parameter quantities compared with the complete model. The corresponding results are listed in the third and forth lines of the Table 2. Without the RM, the model’s performance decreases a lot, e.g., the PSNR decline of $1.29$dB is observed, which illustrates that the RM is essential and significant. The comparison between the forth line and the fifth line show that the DGEB brings slight PSNR and SSIM improvements.

In this subsection, we discuss more about the performance of each module.

In Fig. 6, we show a visual instance of the 4D re-sampling procedure. The 4D re-sampling procedure illustrates that points of raindrop areas can re-sample the pixels with less raindrop pollution in other views. According to the offset list, 4D re-samplings in raindrop area are dominated by the offset in angular domains, while the spatial offset expands the sampling range on the basis of angular offset, which is accord with our analysis about the spatial and angular re-sampling in Section 6.1. The re-sampled result show that the re-sampling module directly utilizes the complementary pixel information between different views to produce pretty initial raindrop removal result.

In Fig.7, more intermediate results produced by our model are visualized. We can see that the residual map $R$ generated by the refinement module plays an auxiliary role to make $O_{i}$ more natural by providing some high-frequency information, which illustrates that the 4D re-sampling strategy reduces the dependence on and the difficulty of implicit residual learning.