A Constrained Deformable Convolutional Network for Efficient Single Image Dynamic Scene Blind Deblurring with Spatially-Variant Motion Blur Kernels Estimation

Agrawal et al.[36] pointed out that the reason why deconvolution is a highly ill-posed problem is that there are many null values in the point spread function (PSF) frequency domain space. By changing the exposure time of each frame in the video, a series of frames were used to fill the null value in the PSF frequency domain space of the blurred frame, so that the motion blur of the blurred frame became a reversible process, so as to solve the problem of deblurring objects moving at a uniform speed. Many researchers thought that an image blurred by camera shake can be viewed as the result of integrating all intermediate images the camera “sees” along the trajectory of the camera shake, therefore the so-called projective motion path blur (PMPB) model was proposed to model the spatially-variant blur. Gupta et al.[6] proposed a PMPB-based motion density function (MDF) to describe the exposure time spent on the three-dimensional motion trajectory of the camera, which was used to estimate the spatially-variant motion blur kernels. Tai et al.[7] regarded the blurred image as the integration of a series of clear scenes, which went through a sequence of planar projective transformations, and proposed a PMPB-based RL algorithm, which can incorporate many regularization priors to improve the deblurred results. Harmeling et al.[8] proposed a space-variant blind deblurring method based on filter flow by studying the type of camera jitter, and designed an experimental device to record the space-variant PSF corresponding to the blur while taking the blurred image. Hirsch et al.[9] combined with the PMPB model and the Efficient Filter Flow (EFF), and proposed an efficient algorithm that can deal with non-uniform blur caused by camera shake. Whyte et al.[10] assumed that the camera rotation was the only significant source of camera shake blur, and proposed a PMPB-based parameterized geometric model to remove the non-uniform camera rotation blur. Xu et al.[37] proposed a hierarchical estimation framework based on region trees to estimate the blur kernel step by step, and redesigned a spatially varying PSF estimation algorithm based on shock filtering invariance for non-uniform deblurring. Hu et al.[11] thought that non-uniform blur was not only caused by camera shake, but also by the change of scene depth. Therefore, a method of simultaneously estimating scene depth and removing non-uniform blur was proposed. Sheng et al.[12] proposed a PMPB-based depth-aware motion blur model with a given depth image. The authors used a PatchMatch-based depth filling method to fix the empty holes in the depth image. The Deblurring and depth filling were performed iteratively to refine the results. Bai et al.[38] observed that a coarse enough image down-sampled from a blurry observation was very close to a low-resolution version of the latent sharp image. Based on this observation, the authors proposed a coarse-to-fine progressive single-image blind deblurring algorithm. Ulyanov et al.[28] showed that an elaborate UNet is sufficient to capture the statistics prior of a single image for the low-level tasks learning. Inspired by[28], Ren et al.[39] proposed a self-supervised blind deblurring method for a single uniform blurred image, which combined deep models with the maximum a posterior (MAP), and constructed two generative networks for the latent image restoration and the blur kernel estimation, respectively.

From above discussions we can see that most optimization-based methods[36, 6, 7, 8, 9, 10, 37, 11, 12, 38, 39] can only handle either small-size camera shake without the movement of objects or the rigid object motion without camera shake. Although the PMPB model can model spatially-variant motion blurs very well, it suffers from tremendous computational cost and hence can only model small-size 3-dimensional camera shake[6, 7, 8, 10, 11, 12]. So, the optimization based methods are not suitable for the real-word complex dynamic scene deblurring problem, which contains camera shake, multiple rigid or non-rigid objects motion, and different scene depths simultaneously.

Lately, impressive progress has been made in SIDSBD by using deep-learning-based single image blind deblurring methods. Xu et al.[40] proposed an end-to-end CNN-based non-blind deblurring network to learn the deconvolution operation for the disk and motion blurs. Ramakrishnan et al.[13] proposed a generative advantageous network (GAN) whose generator consisted of the global jump links and dense connections. Ramakrishnan et al. obtained the restoration image from the inputted blurred image directly, and without the blur kernel estimation. Nah et al.[14] designed a multi-scale end-to-end deblurring network,and proposed an improved residual block for SIDSBD. Again, there was no blur kernel estimation. The DeblurGAN model proposed by kupyn et al.[15] greatly improved the values of the self-similarity measure (SSIM) and subjective visual effects through well-designed advantageous loss and content loss. Tao et al.[16] proposed a scale recurrent network (SRN), which applied the ResBlock to the encoder-decoder module, and restored sharp images with different resolutions gradually. Zhang et al.[17] proposed a spatially-variant recurrent neural network (RNN) for spatially-variant blurs, in which different weights were learned for different pixels. Inspired by the spatial pyramid matching (SPM), Zhang et al.[18] proposed a deep multi-patch hierarchical network (DMPHN) to achieve end-to-end non-uniform deblurring. The DMPHN made inputs at different levels have the same spatial resolution, therefore, the residual manner could be introduced between levels. Gao et al.[19] proposed a nested skip connection structure to replace the conventional residual connection, and a parameter selection sharing strategy between different scales for SIDSBD. Cai et al.[20] thought that appropriate image priors and regularization terms could improve the deblurring performance. Therefore, they inserted an extreme channel prior into the CNN-based blind deblurring network, and proposed an extreme channel prior embedded network (ECPeNet) for the SIDSBD. Yuan et al.[21] introduced the blur kernel estimation into deep-learning-based SIDSBD, and proposed a spatially variant deconvolution network (SVDN). In their proposed SVDN, the deformable convolution was first used to learn the sampling points of the blur kernels, and then the optical flows between the blurred image and it’s nearby frames were used to guide the learning of the sampling points. Zamir et al.[22] combined the characteristics of the encoder-decoder network with the single scale network, and designed a multi-stage network structure, in which not only the attention mechanism was introduced into each stage of the network, but also an information exchange strategy between different stages was proposed. Cho et al.[23] proposed a multi-input multi-output encoder-decoder structure and an asymmetric feature fusion strategy to fuse multi-scale features effectively. Purohit et al.[24] thought that different regions of the blurred image had different degrees of degradation, so they first designed a positioning network to identify the degraded regions, and then the learned degradation features were used to guide the recovery network for adaptive deblurring. Wang et al.[25] combined the UNet and the Transformer, and proposed a general u-shaped transformer for image restoration. In their proposed network, a block-based self-attention and a learnable multi-scale recovery modulator were proposed to capture the local features and estimate the modulation parameter of each window, respectively. In order to reduce the computational overhead of the Transformer on low-level visual tasks, Zamir et al.[26] proposed a multi-dconv head transposed attention (MDTA) module, which calculated attention in the channel dimension rather than the pixel dimension. In addition, a gated-dconv feed-forward network (GDFN) was proposed to capture the local information of images

Except for the SIDSBD, recently, the deep-learning-based video blind deblurring (DLVBD) methods have also been greatly developed and achieved significant improvement in video motion blind deblurring. Su et al.[41] proposed an end-to-end video deblurring network based on the encoder-decoder architecture, and collected real-world motion blurred video datasets using high frame rate cameras. Chen et al.[32] used a optical flow network to estimate the optical flows between consecutive frames, and the estimated optical flows were used to estimate the spatially-variant motion blur kernels. The estimated blur kernels were used to fine-tune the proposed deblurring network by using a reblurring self-supervised loss, which could boost the deblurring performance of the proposed deblurring network. Zhang et al.[33] proved that, in a motion blurred video, the pixel-wise blur kernel can be represented by the pixel-wise optical flow. Therefore, Zhang et al. used the optical flows to model spatially-variant motion blur kernels, which were used to generate the weights of the RNNs for video blind deblurring. Bai et al.[34] estimated the blur kernel form a low-resolution and uniform blurred video directly, and used the estimated blur kernel to achieve self-supervised-based high-resolution video restoration. Wang et al.[35] first estimated the average magnitude of optical flows from several clear consecutive frames, then the estimated average magnitude was used to learn the pixel-wise motion blur level of each motion blurred frame. Then, Wang et al. utilized the learned motion blur levels as guidance for effective deep video deblurring.

From above discussions we can see that, on the one hand, most existing deep-learning-based single image dynamic scene blind deblurring (SIDSBD) methods usually design deep networks to directly remove the spatially-variant motion blurs from one inputted motion blurred image, without blur kernels estimation[13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]. On the other hand, the estimation of the spatially-variant motion blur kernels and the corresponding reblurring training strategy have been widely applied to the deep-learning-based video blind deblurring (DLVBD) methods and have proved their effectiveness[32, 33, 34, 35]. Therefore we wonder that whether the estimation of spatially-variant motion blur kernels and the corresponding reblurring training strategy are still good for the SIDSBD? So, in this paper, we mainly focus on the research of the estimation of the spatially-variant motion blur kernels and the corresponding reblurring training strategy for SIDSBD.

Our proposed MSML-MIMO encoder-decoder architecture is illustrated in Fig. 1,where $B_{ijk},S_{i1k},R_{i2k}(i\in{\{1,2,3\}},j\in{\{1,2\}},k\in{\{1,2,3\}})$ represent the blur image, the recoverd image and the residual image at different scales, different levels, and different orders, respectively. In detail, $B_{ijk},S_{i1k},R_{i2k}(i\in{\{1,2,3\}},j\in{\{1,2\}},k\in{\{1,2,3\}})$ denote the observed blurred image input into the $k-th$ encoder block (EnBlock) at the $j-th$ level of the $i-th$ scale, the restoration image inferred by the $k-th$ decoder block (DeBlock) at the 1st level of the $i-th$ scale, and the residual image generated by the $k-th$ DeBlock at the 2nd level of the $i-th$ scale, respectively. $B_{1jk},B_{2jk}$ and $B_{3jk}$ are a sequence of blurry images downsampled from the observed original full resolution blurred image at different scales. $EB_{ijk}$, $DB_{ijk}$ and $CDCR_{ij}$ denote the $k-th$ EnBlock at the $j-th$ level of the $i-th$ scale, the $k-th$ DeBlock at the $j-th$ level of the $i-th$ scale, and the CDCR module at the $j-th$ level of the $i-th$ scale, respectively.

As shown in the Fig. 1, our proposed MSML-MIMO encoder-decoder architecture includes three scales, in which each scale consists of two levels, and each level contains three EnBlocks, three DeBlocks, and a CDCR module. Except the $EB_{ij1}$, the first convolution of $EB_{ij2}$ and $EB_{ij3}$ is a strided convolution with a stride of $2$ for downsampling. And the stride of the transposed convolution in $DB_{ij1}$ and $DB_{ij2}$ is $2$ for upsampling. For the $DB_{i21}$ and $DB_{i22}$, two additional $3\times 3$ convolution layers with the output channel of $3$ are used to obtain the intermediate residual images with different scales respectively. And, for the $DB_{i11}$ and $DB_{i12}$, two additional $3\times 3$ convolution layers with the output channel of $3$ are used to obtain the intermediate restoration images with different scales respectively.

As illustrated in Fig. 1, the deblurring process of CDCN starts at the second/down level of the third scale. At the second level of each scale, the inputs of $EB_{321}$, $EB_{221}$ and $EB_{121}$ are $EB_{321}^{in}=B_{321}\copyright B_{321}$, $EB_{221}^{in}=B_{221}\copyright S_{313}^{\uparrow}$ and $EB_{121}^{in}=B_{121}\copyright S_{213}^{\uparrow}$, respectively. While, the input of $EB_{32k}$ includes the output of the $EB_{32(k-1)}$ and the $B_{32k}(k\in{\{2,3\}})$, and the input of $EB_{i2k}$ includes the output of the $EB_{i2(k-1)}$ and the concatenation of the $B_{i2k}$ and the $S_{(i+1)1(4-k)}^{\uparrow}(i\in{\{1,2\}},k\in{\{2,3\}})$. At the first/up level of each scale, the input of $EB_{i11}$ is $EB_{i11}^{in}=B_{i11}\oplus R_{i23}$. While, the input of $EB_{i1k}$ includes the output of the $EB_{i1(k-1)}$ and the addition of the $B_{i1k}$ and the $R_{i2(4-k)}(i\in{\{1,2\}},k\in{\{2,3\}})$. Where $EB_{ijk}^{in}$ and $S_{ijk}^{\uparrow}$ denote the input of the $EB_{ijk}$ and the upsampled version of the $S_{ijk}$ (the green arrow in Fig. 1) respectively.

For the DeBlocks, the input of the $DB_{ij1}$ is the $CDCR_{ij}$. And the input of the $DB_{ijk}(k\in{\{2,3\}})$ is the concatenation of the $DB_{ij(k-1)}^{out}$, $EB_{ij1}^{out}$, $EB_{ij2}^{out}$, and $EB_{ij3}^{out}$, by using the resize operation (the earthy yellow arrow in Fig. 1) which can be formulated as:

where,$EB_{ijk}^{out}$ and $DE_{ijk}^{out}$ denote the outputs of the $EB_{ijk}$ and $DE_{ijk}$ respectively. The CDCR module is used to estimate the spatially-variant motion blur kernels and predict the corresponding inverse kernels and output the deblurred features. At the second level of each scale, the input of the $CDCR_{i2}$ is the $EB_{i23}^{out}$. And At the first/up level of each scale, the input of the $CDCR_{i1}$ is $CDCR_{i1}^{in}=EB_{i13}^{out}\oplus CDCR_{i2}^{out}$, where $CDCR_{ij}^{in}$ and $CDCR_{ij}^{out}$ denote the input of the $CDCR_{ij}$ module and the output of the $CDCR_{ij}$ module respectively.

From above analyses and Fig. 1 we can see that, first, our MSML-MIMO encoder-decoder architecture can conduct the residual between levels. However, beyond our early work in [27], our proposed MSML-MIMO encoder-decoder architecture conducts the residual manner between levels not only once but three times, which are between three DeBlocks of the second level and the corresponding three EnBlocks of the first level within the same scale. Second, similar to the rich residual connects between levels, for each scale, we conduct the intermediate supervision not only once but apply multiple intermediate supervisions to all DeBlocks of the first level of each scale. And except for the scale $1$, all intermediate supervision results in previous scale will be used to guide the image restoration at this scale. Finally, different from most conventional coarse-to-fine image blind deblurring networks, which fused different depths of features only between EnBlocks and DeBlocks with the same spatial resolution, our proposed MSML-MIMO encoder-decoder architecture fuses features from different spatial resolutions within each level: the second and third DeBlocks of each level take the outputs of all EnBlocks within the same level as the inputs and merge different resolution features using the concatenation operation and convolutional layers. Therefore, because of the utilization and fusion of more information flows and informative features, our proposed CDCN possesses more powerful features extraction ability, which can facilitate accurate estimation of spatially-variant motion blur kernels and higher quality image restoration for SIDSBD..

As illustrated in Fig. 2, Our proposed CDCR strategy consists of a CDCR module and a PMPB-based reblurring loss function, which is responsible for learning the sampling points of spatially-variant motion blur kernels and predicting the sampling points of their corresponding inverse kernels from one single motion blurred image. In this subsection, we will discuss the CDCR module and the PMPB-based reblurring loss function in detail.

As shown in Fig. 2, the proposed CDCR module is a small CNN, which consists of one SoftMax layer, two PReLU layers, three regular convolutional layers and one deformable convolution. On the one hand, the deformable convolution has been proved that it can make the spatial sampling locations focus on the interested image content efficiently[42, 43]. On the other hand, the PMPB model, which takes a motion blurred image as the result of integrating all intermediate images the camera “sees” along the trajectory of the relative motion, is one of the best motion blur models for modeling spatially-variant motion blurs. Therefore, inspired by the deformable convolution and the PMPB model, in our proposed CDCR module, we first use a regular convolution to learn the spatial sampling locations and the corresponding weights of the trajectory of the relative motion for each pixel, which is similar to the estimation of the motion density function of the camera shake in the PMPB model. The learning of the spatial sampling locations and the corresponding weights can be formulated as:

where,$f_{in}$ denotes the input features. For the $CDCR_{i2}$, $f_{in}$ is the blurred features extracted by the EnBlocks, i.e. the $EB_{i23}^{out}$. For the $CDCR_{i1}$,$f_{in}=EB_{i13}^{out}+CDCR_{i2}^{out}$. $Conv_{1}()$ denotes the regular convolution operation with $3N$ output channels, where $N$ denotes the number of the sampling points of the blur kernel. $Sep()$ denotes the separation operation, which divides $3N$ channels into $2N$ and $N$ for $offset_{bk}$ and $weight_{bk}$, respectively. And $offset_{bk}=[offset_{bk,1},offset_{bk,2}...offset_{bk,N}],offset_{bk,n}=(offset_{bk,n,x},offset_{bk,n,y}),n\in{\{1,2,...N\}},$ where $offset_{bk,n,x}$ and $offset_{bk,n,y}$ denote the $x$-coordinate and the $y$-coordinate of the $n-th$ sampling point of the motion blur kernel for each pixel, respectively. Therefore, $offset_{bk}$ is the spatial locations of $N$ sampling points, which approximates the trajectory of the relative motion of each pixel (i.e. the shape of the motion blur kernel). And $weight_{bk}=[weight_{bk,1},weight_{bk,2}...weight_{bk,N}]$, where $weight_{bk,n}$ denotes the fraction of the exposure time (FET) spent at $n-th$ sampling point of the motion blur kernel for each pixel. Based on the principle of the PMPB model, the $SoftMax$ operation is applied to the $weight_{bk}$ for energy conservation. Then, we propose a PMPB-based reblurring loss function to constrain the accuracy of the $offset_{bk}$ and $weight_{bk}$:

where, $(B_{ij3})^{m}$ denotes the $m-th$ observed blurred image at the $3-th$ EnBlock of the $j-th$ level of the $i-th$ scale and $(B_{reblur_{ij3}})^{m}=\sum_{n=1}^{N}warp((offset_{bk,n})_{ij3}^{m},(S_{GT_{i11}})^{m})\bigotimes(weight_{bk,n})_{ij3}^{m}$ denotes the $m-th$ reconstructed blurred image using the PMPB model at the $3-th$ EnBlock of the $j-th$ level of the $i-th$ scale. And, $(S_{GT_{i11}})^{m}$ is the $m-th$ ground truth sharp image at the $1-th$ DeBlock of the $1-th$ level of the $i-th$ scale. $(offset_{bk,n})_{ij3}^{m}$ and $(weight_{bk,n})_{ij3}^{m}$ denote the learned $offset_{bk,n}$ and $weight_{bk,n}$ for $(B_{ij3})^{m}$. $||\quad||_{2}^{2}$ and $\bigotimes$ denote the $L2$ norm and the element-wise multiplication respectively, and $warp()$ denotes the warp operation by using the bilinear interpolation.

From Equations (5) we can see that, our proposed PMPB-based reblurring loss function can constrain the solution spaces of the learned $offset_{bk}$ and $weight_{bk}$, which makes the learned sampling points fit the trajectory of the relative motion of each pixel well and achieves accurate spatially-variant motion blur kernels estimation from a single motion blurred image.

Then, after the learning of spatially-variant motion blur kernels, we apply two PReLU layers and two regular convolutional layers to the $offset_{bk}$ and $weight_{bk}$ for predicting the inverse kernels, which can be formulated as:

where, $Pre()$ denotes the PReLU activation function and $\copyright$ denotes the concatenation operation. $Conv_{3}()$ denotes the regular convolution operation with $3M$ output channels, where $M$ denotes the number of the sampling points for the inverse kernel. Again, $Sep()$ divides $3M$ channels into $2M$ and $M$ for $offset_{inbk}$ and $weight_{inbk}$, respectively. Similar to $offset_{bk}$, $offset_{inbk}$ is the spatial locations of $M$ sampling points of the inverse kernel for each pixel, which approximates the shape of the inverse kernel. And similar to $weight_{bk}$, $weight_{inbk}$ is the weights of $M$ sampling points of the inverse kernel for each pixel but without the $SoftMax$ operation.

Finally, the predicted inverse kernels are directly applied to $f_{in}$ for generating the deblurred features by using one deformable convolution.

The loss function of our CDCN consists of the multi-scale content mean square error loss, the multi-scale frequency reconstruction loss and the PMPB-based reblurring loss:

where, $(S_{i1k})^{m}$ and $(S_{GT_{i1k}})^{m}$ denote the $m-th$ recovered image and the $m-th$ ground truth sharp image at the $k-th$ DeBlock of the $1-th$ level of the $i-th$ scale, respectively. $\lambda$ is set to $0.1$ in our experiments. For the parameters of our proposed CDCN, in this paper, we propose a inter-scale parameter sharing scheme: the parameters between the levels, which have the same multi-patch model, are shared. And that, the parameters between the levels, which belong to the same scale are independent.

The multi-input multi-output strategy and the fusion of different resolution features are also introduced in MIMO-UNet for SIDSBD. However, there are two main differences between MIMO-UNet and our CDCN. The first difference is the network architecture. In MIMO-UNet, Cho et al.[23] adopt a traditional single-scale coarse-to-fine strategy, where a input blurred image is encoded and decoded only once, and the output multiple deblurred images cannot be used to guide image restoration of other scales. While, our CDCN consists of three scales, where each scale contains two levels, therefore, a current-scale blurred image will be encoded and decoded twice. And, because of the multi-scale and multi-level architecture, in our CDCN, all intermediate deblurred results can be used to guide the image restoration at the next scale. The second difference is the estimation of the spatially-variant motion blur kernels. In MIMO-UNet, Cho et al.[23] directly remove the spatially-variant motion blurs from one inputted motion blurred image, without blur kernels estimation. Compared with MIMO-UNet, our CDCN can accurately estimate spatially-variant motion blur kernels from only one single motion blurred image without the optical flow.

Another related work to our CDCN is SVDN, where the deformable convolution is utilized to learn the sampling points of the blur kernels. Although our CDCN also adopts the deformable convolution, there are three main differences between SVDN and our CDCN. First, for the learning of the sampling points, SVDN uses the bi-directional optical flows to approximate the blur kernels, and makes the spatial distribution of the deformable sampling points close to the optical flows by using a distance-based loss function. Therefore, SVDN is still essentially a multi-frame-based blur kernel estimation method. On the contrary, our CDCN learns the sampling points of spatially-variant motion blur kernels from only one inputted motion blurred image by using a regular convolution without any optical flow. Therefore, our CDCN is essentially a single-image-based blur kernel estimation method, which is suitable for the SIDSBD well. Second, compared with SVDN, our CDCN proposes a PMPB-based reblurring loss function to constrain the accuracy of the learned sampling points, which makes the learned sampling points fit the trajectory of the relative motion of each pixel well and achieves accurate spatially-variant motion blur kernels estimation from a single motion blurred image. Third, for the deconvolution operation, SVDN uses two deformable convolutions to get the deblurred features. However, our CDCN can not only generate deblurred features by using only one deformable convolution, but also can achieve better deblurring performance for SIDSBD.

In this paper, we train our model on the GOPRO[14] dataset and then test it on the GOPRO and HIDE[44] test datasets respectively. For the GoPro dataset, we use 2103 image pairs for training and the remaining 1111 pairs for testing, which is same as [14]. For the HIDE dataset, we use 2025 pairs for testing, which is same as [44].

For training of our CDCN, the Adam[45] with $\beta_{1}=0.9$,$\beta_{2}=0.999$ and $\varepsilon=1e-8$ is used as the optimizer to optimize our network for $2000$ epochs which are sufficient for convergence. The learning rate is initially set to $1e-4$ and decreased by the factor of $0.5$ at every $200$ epochs. For every training iteration, we randomly sample eight images. $N$ and $M$ are set to $3$ and $7$ respectively. The number of residual blocks in each EnBlock and each DeBlock is 8. And unless otherwise specified, the entire network uses the $3\times 3$ convolution kernel for all other convolutional layers by default. For the quantitative metrics, we use the peak signal-to-noise ratio (PSNR) and the self-similarity measure (SSIM) to evaluate the performance of our method quantitatively. For data augmentation, each patch was horizontally flipped with a probability of $0.5$.

As we discussed above, the main contributions of our proposed CDCN are: the PMPB-based reblurring loss function for constraining the convergence of the sampling points of spatially-variant motion blur kernels, a CDCR module for predicting the inverse kernels and generating deblurred features, and an MSML-MIMO architecture for more powerful features extraction ability. So, in this subsection, we will individually test these three parts to better understand the effectiveness of each part via the experiments.

1) Evaluation of the PMPB-based reblurring loss: As shown in Equations (5), the proposed PMPB-based reblurring loss function can constrain the solution spaces of the $offset_{bk}$ and $weight_{bk}$, and make the learned sampling points fit the trajectory of the relative motion of each pixel well. In order to verify the effectiveness of the PMPB-based reblurring loss, we compare our proposed CDCN with a model named CDCN-NoPMPBReBlur, which removes Equations (5) from the proposed CDCN (i.e. without any constraint on the $offset_{bk}$ and $weight_{bk}$). In other words, the values of $offset_{bk}$ and $weight_{bk}$ will be completely self-learned. The CDCN-NoPMPBReBlur model shares the same framework as Fig. 2 and is trained on the same GoPro training dataset as the proposed CDCN model. Table I shows that without the PMPB-based reblurring constraint, the PSNR and SSIM of the CDCN-NoPMPBReBlur model reduce by 0.51dB and 0.006, respectively.

Fig. 3 shows the comparisons of the distribution of sampling points with and without the PMPB-based reblurring constraint. On the one hand, from Fig. 3b, we can see that, the light and shadow on the ground provides clear information of the relative movement trajectory, and our proposed PMPB-based reblurring loss can constrain the learned sampling points fit the trajectory of the relative motion of each pixel very well: the shape of the red points is very similar to that of the light and shadow (Please see the Fig. 3a, 3b and 3c and the zoomed in regions). On the other hand, from Fig. 3c we can see that, our proposed PMPB-based reblurring loss can also make the sampling points change with the level of the blur: different blurred regions have different sampling point distributions, which are consist with the degrees of the blurs (Please see the left region, the middle region and the right region of Fig. 3c and the zoomed in region). However, the CDCN-NoPMPBReBlur model can not learn any information about the spatially-variant motion blur kernel (Please see the Figs. 3d, 3e and 3f, and the zoomed in regions)

2) Evaluation of the CDCR module: As shown in Fig. 2, we use two PReLU layers and two regular convolutional layers to predict the inverse kernels from the learned $offset_{bk}$ and $weight_{bk}$, and then the predicted inverse kernels are used to generate the deblurred features via a deformable convolution. To demonstrate the effectiveness of the CDCR module, we compare our proposed CDCN with a model named CDCN-NoCDCR, which removes $Conv_{2}()$ , $Conv_{3}()$ , two PReLU layers, and the final deformable convolution from the CDCR module, and only keep the $Conv_{1}()$ and the $SoftMax$ for calculating PMPB-based reblurring loss. Therefore, in the CDCN-NoCDCR model, $DB_{i21}^{in}=EB_{i23}^{out}$ and $DB_{i11}^{in}=EB_{i23}^{out}\oplus EB_{i13}^{out}$. The CDCN-NoCDCR model is trained on the same GoPro training dataset as the proposed CDCN model. As can be seen from Table I, without the prediction of the inverse kernels, the PSNR and SSIM of the CDCN-NoCDCR model are reduced by 0.72dB and 0.01, respectively.

3) Evaluation of the MSML-MIMO architecture: To demonstrate the advantage of our MSML-MIMO architecture, we compare our proposed CSCN with $2$ baseline models: the CDCN-1level model, where each scale only contains one level, and the CDCN-NoMIMO, where only $B_{i11}$ and $B_{i21}$ are considered as the inputs of the $i-th$ scale, and only $S_{i13}$ is considered as the output of the $i-th$ scale. The models CDCN-1level and CDCN-NoMIMO are trained on the same GoPro training dataset as the proposed CDCN model respectively. From Table I we can see that, on the one hand, compared with our CDCN model, CDCN-1level model can not conduct the residual operation between levels, and the values of PSNR and SSIM are reduced by 0.61dB and 0.007, respectively. On the other hand, without more input information and intermediate restoration results, the PSNR and SSIM of the CDCN-NoMIMO model are reduced by 0.41dB and 0.004, respectively.

Fig. 4 illustrates the subjective visual evaluation results of the models CDCN-NoPMPBReBlur, CDCN-NoCDCR, CDCN-1level, CDCN-NoMIMO and our proposed CDCN. We could found that our proposed CDCN can obtain higher quality restoration image, which has sharper and clearer edges. Table I and Fig. 4 demonstrate that all contributions play important roles in estimating accurate blur kernels and recovering of high quality images.

For the synthetic benchmark datasets comparisons, we use two synthetic datasets: the GoPro[14] and the HIDE[44], and compare our method with fourteen state-of-the-art SIDSBD methods (Xu et al.[5], DeblurGAN[15], Nah et al.[14], Zhang et al.[17], DeblurGAN-v2[46], SRN[16], Gao et al.[19], DBGAN[29], MT-RNN[30], DMPHN[18], MSCAN[27], Suin et al.[31], SPAIR[24] , MIMO-UNet+[23]). For fair comparison, the models [15], [14], [17], [46], [16], [19], [29], [30], [18], [27],[31], [24] and [23] are trained on the 2103 GoPro training image pairs. Because the method [5] is an optimization-based method, we use the executable program provided by Xu et al. for the comparison. Table II shows the values of the mean PSNR and the mean SSIM of all models on the 1111 GoPro testing image pairs and 2025 HIDE testing image pairs. From Table II we can see that our CDCN significantly outperforms all the methods and could attain the highest mean PSNR value and mean SSIM value on both the GoPro testing images and the HIDE testing images. So, in summary, our CDCN can produce better deblurring results than the state-of-the-art SIDSBD methods in terms of the quantitative metrics.

Because of the space limitation, here, we only use two GoPro testing images (Fig. 5 and Fig. 6) to illustrate the qualitative evaluation comparisons of the models [5] , [15], [14], [16], [19], [20], [27], [23] and our CDCN model. From Fig. 5 and Fig. 6 we can see that, the deblurred images by models [5] , [15], [14], [16], [19], [20], [27], [23] suffer from one or more of different degrees flaws: the blur, the distortion and the deformation, respectively. By contrast, our CDCN can obtain the highest quality restoration images: can not only remove various artifacts effectively, but also can recover sharper edges and more details. Please see the Figs. 5b-5i, and the Figs. 6b-6i, and the corresponding zoomed in regions. Table II, Fig. 5 and Fig. 6 demonstrate the superiority of our method on the synthetic benchmark datasets. More experimental results for synthetic benchmark datasets can be available at https://github.com/wuyang1002431655/CDCN.

With the exception of the synthetic blurred images, we also use the real blurred images to further demonstrate the effectiveness of our method. For real blurred images, we compare our method with the methods [16], [19] and [23], and all methods are still trained on the 2103 GoPro training image pairs. Fig. 7 and Fig. 8 illustrate the qualitative evaluation comparisons of the models [16], [19], [23] and our CDCN model on two real blurred images. From Fig. 7 and Fig. 8 we can see that, methods [16], [19] and [23] can not recover the real blurred image well, and there are too much flaws: the ringing artifacts, the blur, the distortion and the deformation, in the deblurred images. By contrast, our CDCN model can obtain the higher quality restoration images: less blur, less deformation, hardly distortion and ringing artifacts, and sharper edges and more details Please see the Figs. 7b-7e and the Figs. 8b-8e, and the corresponding zoomed in regions. Figs. 7 and 8 demonstrate that, for the real blurred images, our method still can achieve higher quality restoration results. More experimental results for real blurred images can be available at https://github.com/wuyang1002431655/CDCN.