DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation
with Flow-Guided Attentive Correlation and Recursive Boosting

$t$-Alignment. To fully exploit the bidirectional flows ($f_{01}$, $f_{10}$) extracted from four blurry inputs, the intermediate flows $f_{0t}$ (or $f_{1t}$) from time 0 (or 1) to time t are linearly approximated as $f_{0t}=t\cdot f_{01}$ (or $f_{1t}=(1-t)\cdot f_{10}$). Then we apply the complementary flow reversal (CFR) [44] for $f_{0t}$ and $f_{1t}$ to finally approximate $f_{t0}$ and $f_{t1}$. Finally, we obtain t-aligned feature $F_{t}$ by applying the backward warping operation ($W_{b}$) [17] for features $F_{0}^{\prime}$, $F_{1}^{\prime}$ followed by a blending operation with the occlusion map. This is called feature-flow-based warping and blending (FWB), which is depicted by the green box in Fig. 3 (a). The t-aligned feature $F_{t}$ is computed as follows:

where $\bar{o}_{t0}=\sigma(o_{t0})$ and $\bar{o}_{t1}=1-\bar{o}_{t0}$, and $\sigma$ is a sigmoid activation function.

FAC-FB Module. Since the pixel information is spread over the blurry input frames due to the accumulation of light [14, 16, 49] as in Eq. 1, we propose a novel FAC-FB module that can effectively bolster the source feature $F_{0}^{\prime}$ (or $F_{1}^{\prime}$) by extracting the useful information in the feature-domain from its counterpart feature $F_{1}^{\prime}$ (or $F_{0}^{\prime}$) in guidance of self-induced flow $f_{01}$ (or $f_{10}$). The FAC-FB module in Fig. 3 (b) first encodes the two feature maps ($F_{0}$, $F_{1}$) by passing the outputs ($F_{0}^{\prime}$, $F_{1}^{\prime}$) of the FF-RDB module through its five residual blocks (ResB’s). The cascade ($\mathbf{ResB}^{\times 5}$) of the five ResB’s is shared for $F_{0}^{\prime}$ and $F_{1}^{\prime}$.

Refine Module. After the FAC-FB Module in Fig. 3 (a), $F_{0}^{b}$, $F_{1}^{b}$, $f_{t0}$, $f_{t1}$ and $o_{t0}$ are refined via the U-Net-based [39] Refine Module (RM) as $[F_{0}^{r},F_{1}^{r},f_{t0}^{r},f_{t1}^{r},o_{t0}^{r}]=\mathrm{RM}(\mathbf{Agg}^{1})+[F_{0}^{b},F_{1}^{b},f_{t0},f_{t1},o_{t0}]$ where $\mathbf{Agg}^{1}$ is the aggregation of $[F_{0}^{b},F_{t},F_{1}^{b},f_{t0},f_{t1},o_{t0},f_{01},f_{10}]$ in the concatenated form. Then, we get the refined feature $F_{t}^{r}$ at time $t$ by $F_{t}^{r}=\mathrm{FWB}(F_{0}^{r},F_{1}^{r},f_{t0}^{r},f_{t1}^{r},o_{t0}^{r})$ as similar to Eq. 2. Here, we define a composite symbol at time t by the combination of two feature-flows and occlusion map logit as $\mathbf{f_{F}}\equiv[f_{t0}^{r},f_{t1}^{r},o_{t0}^{r}]$ to be used in recursive boosting.

Decoder \@slowromancapi@ ($D_{1}$). $D_{1}$ is composed of $\mathbf{ResB^{\times 5}}$ and it is intentionally designed to have a function: to decode a feature $F_{j}$ at a time index $j$ to a sharp frame $S_{j}$. $D_{1}$ is shared for all the three features ($F_{0}^{r},F_{t}^{r},F_{1}^{r}$). It should be noted that $D_{1}$ decodes $F_{0}^{r},F_{t}^{r}$ and $F_{1}^{r}$ into sharp frames $S_{0}^{r},S_{t}^{r}$ and $S_{1}^{r}$, respectively, which would be applied by L1 reconstruction loss ($L_{D_{1}}^{r}$) (Eq. 9). It is reminded that the architecture from the input layer to $D_{1}$ constitutes our baseline version, called DeMFI-Net_bs. Although DeMFI-Net_bs outperforms the previous SOTA methods, its extension with recursive boosting, called DeMFI-Net_rb, can further improve the performance.

Booster Module. Booster Module iteratively updates $\mathbf{f_{P}}$ to perform PWB for $S_{0}^{r},S_{1}^{r}$ obtained from DeMFI-Net_bs. The Booster Module is composed of Mixer and GRU-based Booster (GB), and it first takes a recurrent hidden state ($F_{i-1}^{rec}$) and $\mathbf{f_{P}}^{i-1}$ at $i$-$th$ recursive boosting as well as an aggregation of several components in the form of $\mathbf{Agg}^{2}=[S_{0}^{r},S_{t}^{r},S_{1}^{r},B_{-1},B_{0},B_{1},B_{2},f_{01},f_{10},\mathbf{f_{F}}]$ as an input to yield two outputs of $F_{i}^{rec}$ and $\mathbf{\Delta}_{i-1}$ that is added on $\mathbf{f_{P}}^{i-1}$. Note that $\mathbf{f_{P}^{0}}=\mathbf{f_{F}}$ and $\mathbf{Agg}^{2}$ is not related to $i$-$th$ recursive boosting. The updating process is given as follows:

where the initial feature $F_{0}^{rec}$ is obtained as a 64-channel feature via channel reduction for $\mathrm{Conv_{1}}([F_{0}^{r},F_{t}^{r},F_{1}^{r}])$ of 192 channels. More details are provided for the Mixer and the updating process of GB in Appendices.

Decoder \@slowromancapii@ ($D_{2}$). $D_{2}$ in Fig. 3 (c) is composed of $\mathbf{ResB^{\times 5}}$. It fully exploits abundant information of $\mathbf{Agg}_{i}^{3}=[S_{0}^{r},S_{t}^{r,i},S_{1}^{r},B_{-1},B_{0},B_{1},B_{2},f_{01},f_{10},\mathbf{f_{F}},\mathbf{f_{P}}^{i},F_{i}^{rec}]$ to finally generate the refined outputs $[S_{0}^{i},S_{t}^{i},S_{1}^{i}]=D_{2}(\mathbf{Agg}_{i}^{3})+[S_{0}^{r},S_{t}^{r,i},S_{1}^{r}]$ via residual learning, where $S_{t}^{r,i}=\mathrm{PWB}(S_{0}^{r},S_{1}^{r},\mathbf{f_{P}}^{i})$ is operated by only using the updated $\mathbf{f_{P}}^{i}$ after the $i$-$th$ recursive boosting to enforce the flows to be better boosted.

Loss Functions. The total loss function $\mathcal{L}_{total}$ is given as:

where $GT_{j}$ and $N_{trn}$ denote the ground-truth sharp frame at time $j$ and total numbers of recursive boosting for training, respectively. We denote DeMFI-Net_rb($N_{trn}$, $N_{tst}$) as DeMFI-Net_rb that is trained with $N_{trn}$ and is tested by $N_{tst}$ recursive boosting. The second term in the right-hand side of Eq. 8 is called as a recursive boosting loss. It should be noted that DeMFI-Net_rb is jointly trained with the architecture of DeMFI-Net_bs in an end-to-end manner using Eq. 8 without any complex learning schedule. Note that DeMFI-Net_bs is trained with only Eq. 9 from the scratch.

Training Dataset. To train our network, we use Adobe240 dataset [45] which contains 120 videos of 1,280$\times$720 @ 240fps. We follow a blurry formation setting of [40, 41, 13] by averaging 11 consecutive frames at a stride of 8 frames over time to synthesize blurry frames captured by a long exposure, which finally generates blurry frames of 30fps with $K=8$ and $\tau=5$ in Eq. 1. The resulting blurry frames are downsized to 640$\times$352 as done in [40, 41, 13].

Training Strategy. Each training sample is composed of four consecutive blurry input frames ($B_{-1}$, $B_{0}$, $B_{1}$, $B_{2}$) and three sharp-target frames ($GT_{0},GT_{t},GT_{1}$) where $t$ is randomly determined in multiple of $1/8$ with $0<t<1$. The filter weights of the DeMFI-Net are initialized by the Xavier method [11] and the mini-batch size is set to 2. DeMFI-Net is trained with a total of 420K iterations (7,500 epochs) by using the Adam optimizer [23] with the initial learning rate set to $10^{-4}$ and reduced by a factor of 2 at the 3,750-, 6,250- and 7,250-$th$ epochs. The total numbers of recursive boosting are empirically set to $N_{trn}=5$ for training and $N_{tst}=3$ for testing. We construct each training sample on the fly by randomly cropping a $256\times 256$-sized patch from blurry and clean frames, and it is randomly flipped in both spatial and temporal directions for data augmentation. Training takes about five days for DeMFI-Net_bs and two weeks for DeMFI-Net_rb by using a single GPU with PyTorch in an NVIDIA DGX™ platform.

Test Dataset. We use three datasets for evaluation: (i) Adobe240 dataset [45], (ii) YouTube240 dataset and (iii) GoPro (HD) dataset (CC BY 4.0 license) [29] that contains large dynamic object motions and camera shakes. For the YouTube240, we directly selected 60 YouTube videos of 1,280$\times$720 at 240fps by considering to include extreme scenes captured by diverse devices. Then they were resized to 640$\times$352 as done in [40, 41, 13]. The Adobe240 contains 8 videos of 1,280$\times$720 resolution at 240 fps and was also resized to 640$\times$352, which is totally composed of 1,303 blurry input frames. On the other hand, the GoPro has 11 videos with total 1,500 blurry input frames but we used the original size of 1,280$\times$720 for an extended evaluation in larger-sized resolution. All test datasets are also temporally downsampled to 30 fps with the blurring as [40, 41, 13].

Quantitative Comparison. Table 1 shows the quantitative performance comparisons for the previous SOTA methods including the cascades of deblurring and VFI methods with the Adobe240, in terms of deblurring and CFI ($\times$2). Most results of the previous methods in Table 1 are brought from [40, 41, 13], except those of UTI-VFI (pretrained, newly tested), UTI-VFI* (retrained, newly tested) and DeMFI-Nets (ours). Please note that all runtimes (R_t) in Table 1 were measured for 640$\times$352-sized frames in the setting of [40, 41] with one NVIDIA RTX™ GPU. As shown in Table 1, our proposed DeMFI-Net_bs and DeMFI-Net_rb clearly outperform all the previous methods with large margins in both deblurring and CFI performances, and the number of model parameters (#P) for our methods are the second- and third-smallest with smaller R_t compared to PRF. In particular, DeMFI-Net_rb(5,3) outperforms ALANET by 1dB and 0.0093 in terms of PSNR and SSIM, respectively for average performances of deblurring and CFI, and especially by average 1.51dB and 0.0124 for center-interpolated frames attributed to our warping-based framework with self-induced flows. Furthermore, even our DeMFI-Net_bs is superior to all previous methods which are dedicatedly trained for CFI.

Qualitative Comparison. Figs. 4 and 5 show the visual comparisons of deblurring and VFI performances on YouTube240 and GoPro datasets, respectively. As shown, the blurriness is easily visible between $B_{0}$ and $B_{1}$, which is challenging for VFI. Our DeMFI-Nets show better generalized performances for the extreme scenes (Fig. 4) and larger-sized videos (Fig. 5), also in terms of temporal consistency. Due to page limits, more visual comparisons with larger sizes are provided in Appendices for all three test datasets. Also the results of deblurring and MFI ($\times$8) of all the SOTA methods are publicly available at https://github.com/JihyongOh/DeMFI. Please note that it is laborious but worth to get results for the SOTA methods in terms of MFI ($\times$8).

FAC. By comparing the method (f) to (d) and (c) to (a) in Table 3, it is noticed that FAC can effectively improve the overall joint performances in the both cases without and with RB by taking little more runtime (+0.06s) and small number of additional parameters (+0.09M). Fig. 6 qualitatively shows the effect of FAC for DeMFI-Net_rb(1,1) (f). Brighter positions with green boxes in the rightmost column indicate important regions $E_{1}$ after passing Eq. 3 and Conv₁. The green boxes show blurrier patches that are more attentive in the counterpart feature based on $f_{10}$ to reinforce the source feature $F_{1}$ complementally. On the other hand, the less focused regions such as backgrounds with less blurs are relatively have smaller $E$ after FAC. In summary, FAC bolsters the source feature by complementing the important regions with blurs in the counterpart feature pointed by flow-guidance. We also show the effectiveness of FAC without flow guidance when trained with $f=0$. As shown in Table 3, we obtained the performance higher than without FAC but lower than with FAC by flow-guidance, as expected. Therefore, we conclude that FAC works very effectively under the self-induced flow guidance to bolster the center features to improve the performance of the joint task.

Recursive Boosting. By comparing the method (d) to (a), (e) to (b) and (f) to (c) in Table 3, it can be known that the RB consistently yields improved final joint results. Fig. 7 shows that $\mathbf{f_{F}}$ and $\mathbf{f_{P}}$ have a similar tendency in flow characteristics. Furthermore, the $\mathbf{f_{P}}$ updated from $\mathbf{f_{F}}$ seems sharper to perform PWB in pixel domain, which may help our divide-and-conquer approach effectively handles the joint task based on warping operation. It is noted that our weakest variant (a) (w/o both RB and FAC) even outperformed the second-best joint method (UTI-VFI*) as shown in Table 2, 3 on the both Adobe240 and YouTube240.

# of Recursive Boosting $N$. To inspect the relationship between $N_{trn}$ and $N_{tst}$ for RB, we train the three variants of DeMFI-Net_rb for $N_{trn}=1,3,5$ as shown in Table 4. Since the weight parameters in RB are shared for each recursive boosting, all the variants have same #P=7.41M and each column in Table 4 has same runtime $R_{t}$. The performances are generally boosted by increasing $N_{trn}$, where each recursion is attributed to the recursive boosting loss that enforces the recursively updated flows $\mathbf{f_{P}}^{i}$ to better focus on synthesis $S_{t}^{r,i}$ via the PWB. It should be noted that the overall performances are better when $N_{tst}\leq N_{trn}$, while they are dropped otherwise. So, we can properly regulate $N_{tst}$ by considering $R_{t}$ or computational constraints, even though the training with $N_{trn}$ is once over. That is, under the same runtime constraint of each $R_{t}$ as in the column when testing, we can also select the model trained with larger $N_{trn}$ to generate better results. On the other hand, we found out that further increasing $N_{trn}$ does not bring additional benefits due to saturated performance of DeMFI-Net_rb.