Gyroscope-Assisted Motion Deblurring Network

Existing methods are broadly divided into direct simulation and physical acquisition. For the first one, the seminal work by Sun et al. synthesizes motion blur by convolving the entire image with a fixed kernel Sun et al. (2013). However, it does not consider the variation of the blur kernel for each pixel. Rim et al. propose using gyroscope data to generate realistic blur fields Mustaniemi et al. (2019). Although spatial transform convolution is performed, it ignores the relationship between IMU and blur trajectories. Thus, it cannot generate blurry images that closely resemble natural environments.

For the second one, the images are usually acquired using high-speed cameras. For instance, Levin et al. collects blurred photos by capturing the projected images on a wall while shaking the camera Levin et al. (2009). Though it better approximates real-world blur, the background and the blurry images are not pixel-aligned. Moreover, its efficiency and quantity are limited for network training. The GoPro dataset Nah et al. (2017) is currently the most extensively utilized in deblurring research, created by capturing high-speed video and synthesizing blurred images (via clear frame average). Some real-world blur datasets (with blurred and sharp images) Rim et al. (2020) were captured using dual cameras simultaneously. Nevertheless, these datasets comprise images from a single scene, potentially limiting the generalization of network training.

Unlike existing approaches, we use the camera IMU data to synthesize blur trajectories and extract large-scale and pixel-aligned blur images from backgrounds. Thus, our proposed strategy is more efficient, and the generated training triplets have higher quality in pixel alignment, close to real-world blur and diversity.

Image deblurring is a long-standing problem in computer vision. Earlier deblurring methods focused primarily on uniform deblurring Cho and Lee (2009); Fergus et al. (2006). However, these methods often fall short as real-world images typically exhibit non-uniform blurring, with varying blur kernels across different image regions. In the deep learning era, numerous CNN-based blind deblurring approaches have been proposed. For instance, the generative adversarial network DeblurGAN Kupyn et al. (2018) uses blurred and sharp image pairs to train a conditional GAN for deblurring. SRN Tao et al. (2018) employs recurrent networks to extract features between images at multiple scales. Similarly, MIMO-UNet Cho et al. (2021) extracts image features at various scales and merges these features to complete the deblurring task. However, these blind deblurring algorithms often fail to achieve satisfactory results due to the lack of blur-related prior on region, intensity, and motion trajectory. To overcome this problem, we feed a heatmap (converted from IMU data) into the network to guide deblurring. At the same time, we found a slight error between the blur prior data generated by the IMU and the ground truth. These errors are proportional to the size of the image pixels and are negligible on coarse-grained images. Therefore, we use the image pyramid network structure to learn blur trajectories and blur image features. Increasing the proportion of coarse-grained features can enhance the image deblurring effect. Experiments show that our method can effectively improve deblurring accuracy, achieving a state-of-the-art performance.

Motion blur is mainly due to the relative motion between the camera and the scene within a limited exposure time. Suppose there is a three-dimensional space point $A$ in the camera lens. When the camera moves, the blur trajectory of point $A$ on the image is superimposed by its imaging position within the exposure time. In this stage, there is a mapping relationship between the point $A$ on the image and the pose change of the camera. At the same time, the camera pose at each moment is obtained based on the IMU data. So, we can derive the blur trajectory of point $A$ through the IMU and use the blur trajectory to synthesize the corresponding blurred image. Table 3 summarizes all the variables and their descriptions.

Next, we separately discuss the effects of camera translation and camera rotation on the transformation of the imaging position of point $A$ on the image. When the camera only moves in translation, there is the following equation:

where $\boldsymbol{T}$ is the camera motion’s translation matrix, $\boldsymbol{K}$ is the camera’s intrinsic matrix, and $Z_{t}$ is the depth of point $A$ from the camera at time $t$. Since the camera’s exposure time is generally in the range of tens of milliseconds, the translational displacement of the camera is minimal during this period. We can assume that $Z_{t1}=Z_{t2}$, and calculate the pixel displacement of point $A$, ($\boldsymbol{a}_{t2}-\boldsymbol{a}_{t1}$) as $\boldsymbol{KT}/Z$. This equation shows that as the point $A$ is located at a deeper position ($Z$ becomes larger, $\boldsymbol{K}$ and $\boldsymbol{T}$ remain constant), the effect of camera translation on pixel motion becomes smaller. A real-world analogy is when driving a car and looking at an obstacle on the side of the road - the closer this obstacle is to the vehicle, the faster it moves.

Therefore, when the camera captures distant objects, we can ignore the effect of camera translation on image motion blur. Regarding camera rotation, when shooting, all light passes through the centre of the camera lens. Inspired by this phenomenon, we use spherical polar coordinates to represent points in space. This alternative representation facilitates problem-solving. The rotation of the camera consists of Pitch, Roll and Yaw. Since the camera exposure time is very short, we can approximate the blur trajectory during this period as the vector sum of the blur trajectories generated by each movement. The following discussion explores motion blur caused by three types of motion.

In yaw-induced motion blur, the spatial coordinate representation is depicted in Fig. 4. The $XYZ$ coordinates can be expressed as follows:

where $r,\theta,\phi$ are marked in Fig. 4 (a). Combining Eq. 2 and camera imaging principles, the pixel coordinates of the spatial point mapped onto the image are:

It’s evident from the above Eq. 3 that the variable $r$ cancels out. When capturing images, the distance $r$ between the target object and the centre of the camera lens does not influence the projected pixel position. From the spherical coordinate system, $\phi$ remains constant as the camera undergoes yaw. Only $\theta$ changes, and this change can be determined through the camera’s gyroscope readings. For a given pixel point $A$ (known $U$, $V$), we can derive the corresponding $\phi$ and $\theta$ values under various blur conditions using Eq. 3 and use the gyroscope to acquire $\Delta\theta$.

Due to space limitations, the remaining two ambiguous conditions are discussed in detail in the appendix. Using Eq. 4, we can calculate the difference in a blur for each pixel and stack these calculations to calculate the projected point of the target point on the new image. Connecting these projected points is the approximate blur trajectory of point $A$. This process needs to iteratively update the latest projection point coordinates and gyroscope information from the moment the camera exposure starts until the end of the direction. Our experiments show that the gyroscope data of our device corresponds to six sets of data in each image frame. The pixel origin for each pixel can be computed by computing the corresponding blur trajectory node along seven possible trajectories. These seven nodes connect to form a curve that outlines the trajectory. However, the accuracy of blur trajectories is inherently limited by the frequency of gyroscope data acquisition. Increased accuracy requires higher frequency gyroscope data collection.

After calculating the pixel trajectories of the four corners of the image, the projective transformation can be performed on the entire clear image, and the images collected by the camera at each moment of the exposure time can be calculated. A blurred image close to the natural environment can be obtained by superimposing these images. The imaging principle of the camera involves collecting photons during the exposure time. This process can be integrated using the following formula:

where $m$ is the number of gyroscope-acquired measurements during the camera’s exposure time, and the blur intensity varies across different stages. $I_{ij}(x)$ denotes the image after the $i$-th perspective transformation of the original image in the $j$-th stage. It defaults to the first frame’s clear image at the start of the exposure. For clarity, we select four points on the map as original points in the first frame and calculate the blur trajectories of these points using the gyroscope. We compute the corresponding perspective transformation matrix by selecting corresponding points on the four blur trajectories and derive the new instantaneous clear image $I_{ij}(x)$. $l_{jk}$ represents the pixel length of the blur trajectory corresponding to the $k$-th pixel in the $j$-th stage. To optimize the blur effect, we set the length of the longest blur trajectory as $n_{j}$. We synthesize a blurred vision by superimposing these images, as depicted in Fig. 5.

To assess the effectiveness of our proposed synthetic strategy, we conduct two comparisons: (1) a comparison of blurred trajectories with those of natural images and (2) a comparison of trained deblurring networks with existing datasets.

For the first one, we collect the starting point positions and corresponding IMU data of all blurred trajectories and use the method in this paper to predict the end point of the blurred trajectory. Comparing the expected blur track endpoints with those obtained from actual laser images (BlurTrack) allows us to quantify the difference in blurring effects. Our approach demonstrates an average pixel error of only two pixels compared to the exact blurred trajectory, indicating the high accuracy of the dataset. For the second one, we compare it with established datasets. We use this approach to synthesize new blurred datasets from clear images from the GoPro and RealBlur datasets. And trained models using the synthesized and original datasets. Subsequently, we evaluated their performance using SRN-DeblurNet Tao et al. (2018). Table 2 presents a comparative performance analysis. Our synthetic dataset outperforms the original dataset, and notably, this is compared using synthetic and accurate data, which confirms the efficacy of our artificial blur dataset. Researchers can use this method to quickly generate large amounts of data sets, significantly reducing the difficulty of data set collection.

We conduct experiments on IMU-blur and analyze the impact of blur heatmaps and deblurring networks on the deblurring effect, respectively. To conduct experiments more fairly, we use MIMO-UNet Cho et al. (2021) as the baseline method and improve it so that it can input blur heat maps. We call the improved MIMO-UNet Cho et al. (2021) as MIMO-Pro. First, we compared the effects of MIMO-Pro and MIMO-UNet Cho et al. (2021). All parameters were set according to the original MIMO-UNet Cho et al. (2021). The results are shown in the Table 3. GAMD improves PSNR by 6.93 dB. We then compared the network without adding blur heat maps with MIMO-Unet Cho et al. (2021) and found that the PSNR increased by 1.73 dB. Experiments show that GAMD mainly improves the performance of the deblurring network by inputting additional blur information.

We evaluated the GAMD against state-of-the-art deblurring networks, including SRN Tao et al. (2018), DeblurGAN-v2 Kupyn et al. (2018), and MIMO-UNet Cho et al. (2021). GAMD exhibited superior performance, achieving a PSNR of 32.48 and an SSIM of 0.9014, significantly surpassing existing deblurring methods (Table 4). Among traditional networks, MIMO-UNet Cho et al. (2021) showcased the best performance. Fig. 8 provides visual examples of the IMU-blur test dataset, comparing the deblurred images produced by GAMD (our method) against MIMO-UNet Cho et al. (2021), DeblurGAN-v2 Kupyn et al. (2018), and SRN Tao et al. (2018). The presented results underscore the efficacy of our approach, particularly in mitigating the challenges of motion blur posed by significant camera shakes.

The proposed deblurring network may still struggle with significant camera roll and offset angles. For instance, Fig. 9 presents some failure cases with our GAMD. We can clearly observe the blurry effects, even after deblurring with our proposed GAMD. Nonetheless, GAMD and IMU-blur stand out for their superior deblurring performance, making them a contribution to the field.