Blur More To Deblur Better: Multi-Blur2Deblur For Efficient Video Deblurring

The unsharp masking [10, 17] is a classical image sharpening technique to enhance high-frequency components in an image by utilizing the difference between the input image and its blurred (or unsharp) image using Gaussian filtering (we denote this more blurred image). This difference emphasizes (or masks) high-frequency components such as edges that were degraded by Gaussian filtering (or more blurring). In this work, we extend this idea of unsharp masking to the case of video deblurring.

Instead of Gaussian blurring, we propose to generate more blurred images for video deblurring using sharp images with (1). For the blurred image $B^{n}_{t}$, the more blurred video frames for the same reference frame $S[nt]$ will be $B^{n+2}_{t}$, $B^{n+4}_{t}$ and $B^{n+6}_{t}$ that have long exposures to encode more motion information over wide acquisition times as compared to $B^{n}_{t}$. As illustrated in Figure 3, the difference between the input image and a more blurred image encodes the information for sharp region with small motion and blurred region with large motion effectively.

Note that more blurred images can be available during training, but not during testing. Thus, more blurred images are ideal and must be predicted during testing. In addition, for other unsharp masking operations ($e.g.$, scaling), we propose to resort to the power of DNNs (see next Section).

We empirically validated our conjecture on more blurred images with deblurring. We performed a single image deblurring with the original input image $B^{11}_{t}$ and/or with ideal more blurred images $B^{13}_{t}$, $B^{15}_{t}$ and so on from the GoPro dataset [13]. We denote the set of input images $B^{11}_{t},B^{13}_{t},\ldots,B^{19}_{t}$ by $B^{\{11,13,\ldots,19\}}_{t}$. A modified U-Net [18] was trained with the input blurred image and/or ideal more blurred images for the same ground truth sharp frame. Single image deblurring results with the ideal more blurred image sets are summarized in Table 1. Set 1 corresponds to a usual single image deblurring and Sets 2 to 5 correspond to our image deblurring with ideal more blurred images with long exposures.

As compared to the original performance with the input image $B^{11}_{t}$, deblurring with additional more blurred images yielded significantly improved performance. As the more blurred images with longer exposures are used for deblurring, the better performances in PSNR were observed until $n=17$ where dramatic performance boost of 3.33 dB was achieved as compared to the original deblurring with the input with $n=11$. However, after $n=17$, more information from more blurred images did not help to improve deblurring performance. We chose to use 3 more blurred images.

Ideal more blurred images significantly improved the performance of deblurring as investigated in Table 1, but they are available only during training, not during testing. We conjecture that these more blurred images corresponding to the $n+6$ high-speed video frames can be predicted from the input image $B_{t}^{n}$ and its neighboring images $B_{t-1}^{n}$, $B_{t+1}^{n}$ that are corresponding to $3n$ high-speed video frames. Thus, the goal of our proposed MBRNN is to progressively predict the multi-blurring images ${B}^{\{n+2,n+4,n+6\}}_{t}$ (available during training) from $B_{\{t-1,t,t+1\}}^{n}$ (available during training and testing). Our MBRNN is responsible for the MB step of our proposed MB2D.

MBRNN model: A modified U-Net [18] was used as the baseline network for our proposed MBRNN for essentially progressive blurring that is similar to the network in [16] for progressive deblurring. Thus, our proposed MBRNN to yield more blurred images is modeled as:

where $\mathrm{Net_{B}}$ is the baseline network, $\mathrm{\hat{B}}^{n+2k}_{t}$ is the estimated more blurred image at the $k$th iteration ($k=1,2,3$), and $\mathrm{F}^{i}$ is connecting recurrent feature map from the encoder of MBRNN and to the decoder of MBRNN. Figure 4(a) illustrates how the model of MBRNN works to progressively append small amount of motion blurs to the center (or reference) frame by exploiting the estimated more blurred image and intermediate feature map from the previous iteration. Figure 5 shows examples of estimated more blurred images from MBRNN such that small motion blurs (b, c, d) are progressively added to the center frame (a), emphasizing fast moving objects. This phenomenon is also observed quantitatively as decreased spectral densities (e) that is the exact opposite to progressive deblurring in [16].

MBRNN loss: We trained our proposed MBRNN using a simple L1 function: Given $\mathrm{N}$ multi-blurring images $\mathrm{B}^{n+2k}_{t}$ as ground truth that were synthesized from high-speed video frames, the loss function for MBRNN is:

where $t$ is an index and $\mathrm{\hat{B}}^{n+2i}_{t}$ is the output of MBRNN.

Our proposed MB2D consists of the MB step and the D step for more blurring and deblurring, respectively. It is possible to use MBRNN (MB step) and existing video deblurring methods (D step) for improved performance (see next Section for results). While this approach connects the MB and D steps at image level, we propose to connect these two steps at feature levels for further performance boost.

MSDR model: Figure 4(b) illustrates the schematic of the D step using the predicted more blurred images and connecting recurrent feature map from MBRNN. Our proposed Multi-Scale video Deblurring network with connecting Recurrent feature map (MSDR) exploits the information from MBRNN, the predicted more blurred images $\hat{B}^{\{n+2,n+4,n+6\}}_{t}$ and the connecting recurrent feature map $F^{(s)}$, in the multi-scale framework for deblurring with down-sampling. Our proposed MSDR with connecting recurrent feature map is modeled as:

where $s$ is a down-scaling index ($s=1,2,3$ for $\times 1,\times 2,\times 4$ downsamplings, respectively), $\mathrm{\hat{B}}^{{\bf m},(s)}_{t}$ is the set of $\mathrm{\hat{B}}^{n+2,(s)}_{t}$, $\mathrm{\hat{B}}^{n+4,(s)}_{t}$, $\mathrm{\hat{B}}^{n+6,(s)}_{t}$ such that $\mathrm{\hat{B}}^{n+2,(s)}_{t}$ is the downsampled image of $\mathrm{\hat{B}}^{n+2}_{t}$ with the scale index $s$, $\mathrm{\hat{I}}^{(s)}$ is a restored sharp image with the scale index $s$, $\mathrm{F}^{(s)}$ denotes the downsampled connecting recurrent feature map from MBRNN, and $\mathrm{Net_{D}}$ is the modified baseline network from $\mathrm{Net_{B}}$ for for multi-scale framework (see supplemental for details), and $Up$ is a bilinear up-sampling. Our MSDR iteratively restores the latent sharp image from down-scales to the original scale.

Connecting recurrent feature map: As illustrated in Figure 4, the recurrent feature maps $\mathrm{F}$ are extracted from the decoders of MBRNN at every iteration and then concatenated before feeding into our MSDR. We call the concatenated feature maps “connecting recurrent feature map (CRFM)” that bridges between MBRNN and MSDR at feature levels for the best possible deblurring performance of our proposed MB2D concept. It seems that CRFM carries more information on blurs around the edges of moving objects than the more blurred images have, leading to the further performance improvement of video deblurring (see 0.64 dB improvement in PSNR via our proposed CRFM as shown in Table 2).

We performed the ablation studies to demonstrate the effectiveness of the proposed MB2D approach for several components: the number of input frames (NIF), pre-processing (Pre-proc) such as optical flow (OF) or our MBRNN, and connecting recurrent feature map (CRFM). We chose the modified U-Net as a deblurring network without multi-scale (MS) or multi-temporal (MT) frameworks. The results are summarized in Table 2.

Table 2(a),(b) correspond to the cases of single image deblurring (with 1 input frame) and of video deblurring (with 3 input frames), respectively, showing the improved performance of (b) over (a) by using more information from neighboring frames. Table 2(b),(c),(d) correspond to different pre-processing methods for video deblurring: Using optical flow for aligned adjacent frames (c) with significantly increased memory requirement (12.0M) improved the deblurring performance by 0.08dB over (b) without pre-processing. However, our proposed MBRNN itself (d) improved the performance by 0.82dB with modestly increased memory requirement (5.2M). Table 2(e) shows additional performance improvement by 0.57dB over (d) when using CRFM from MBRNN for deblurring without increasing network parameters much.

We also performed additional studies on the number of network parameters for (b). As observed in Table 2(f),(g), increasing the number of network parameters improved deblurring performance, but our proposed MB2D (especially with CRFM) yielded better performances than the deblurring networks with similar numbers of parameters. Thus, our proposed MB2D yielded improved performance not simply because of increased network size, but because of our proposed approach with more blurred images.

Our MB2D concept can be used with various single image / video deblurring approaches such as One-Stage (OS) [12, 11, 1], Multi-Temporal (MT) [16] and Multi-Scale (MS) [13, 23, 6] methods where the MS and MT approaches achieve high performance with small number of parameters due to parameter sharing over scales or iterations. We performed another ablation study on them with our proposed MBRNN pre-processing and the results are summarized in Table 3.

Table 3(h),(k),(n) show the results of the baseline methods for OS, MT and MS approaches with the same DNN. In this case, MT approach yielded the best performance among all three approaches by up to 0.66dB. However, with our proposed MBRNN, MS approach yielded the best performance among all approaches by up to 1.08dB as shown in Table 3(i),(l),(o). Further performance improvements were observed with our proposed CRFM from MBRNN and MS approach with them yielded the best performance among all methods by up to 1.54dB. As argued in [16], MS approach seems to destroy high frequency details for good deblurring by using down-sampled images / features, but it seems that our proposed MBRNN and CRFM compensate for the degraded details to yield excellent performance for video deblurring. Thus, we chose MS for the D step of our MB2D.

To validate the necessity of adjacent frames to provide motion information for generating more blurred images, we performed an ablation study with 1 frame (without neighboring frames) and 3 frames (with neighboring frames) for MBRNN. Note that a single frame is generated with $n$ high-speed video frames while 3 frames cover $3n$ high-speed video frames that is usually wider than the more blurred images from $n+2$ to $n+6$ high-speed video frames. The performance results are summarized in Table 4, showing significant performance differences between the case with a single frame for MBRNN and the case of using neighboring frames. The largest difference was observed for the more blurred image with $n+6$ high-speed video frames and the smallest difference was still significant by 1.9dB. Note that MBRNN is computationally efficient (0.02sec) as compared to optical flow (0.076sec) [22] that we used for Su [20] instead of a MATLAB implementation for it.

Our proposed MB2D is an alternative approach to exploit neighboring frames and its MBRNN can be applied to other existing video deblurring methods with mild modifications such as multi-channel inputs with the predicted more blurred images and with CRFM. Here, we investigate the feasibility of using our proposed MB2D (especially MBRNN) along with other existing SOTA methods: Tao [23], Zhang [26], Su [20] and Pan [15]. We implemented our MB2D versions of them, called Tao^∗, Zhang^∗, Su^∗ and Pan^∗, respectively. Note that for Tao [23], Zhang [26], single image deblurring methods were converted into video deblurring methods with our MB2D by adding MBRNN. For Su [20], Pan [15], we replaced the optical flow network (PWC-Net [22]) with our MBRNN, resulting in decreased network parameter size by 6.8M. Table 5 shows that our proposed MB2D substantially improved the deblurring performance with existing methods by at least 0.52dB up to 1.6dB for all 4 existing methods.

We performed our proposed MB2D with MBRNN and MSDR on two popular benchmark video deblurring datasets: GoPro test dataset [13] and Su test dataset [20]. We use the reported quantitative performances in PSNR (dB) and SSIM of the SOTA methods for Pan [15], Zhong [29] and Nah [14]. Parameter size and run time results were measured using publicly available codes.

Tables 6 and 7 show that our proposed MB2D significantly outperformed all existing SOTA methods for video deblurring in terms of PSNR and SSIM on the GoPro and Su test datasets, respectively. Note that compared to the most recent SOTA method, Pan [15], our proposed MB2D with MBRNN and MSDR yielded substantially outperformed it with more than 3 times less network parameters (5.42M vs. 16.19M) and more than 6 times faster computation time (0.27 second vs. 1.73 second).

Figures 6 and  7 show the examples of video deblurring using a few SOTA methods or using our proposed MB2D on the GoPro and Su test datasets, respectively. Our proposed MB2D seems to better deblur for fast moving objects or people than other existing methods such as Kim [9], STFAN [30], and Pan [15], due to our efficient more blurring based neighboring video frames exploitation.