Deep network for rolling shutter rectification

Most present-day cameras are equipped with CMOS sensors due to advantages such as slimmer readout circuitry, lower cost, and higher frame rate over their CCD counterparts. While capturing an image, CMOS sensor array is exposed to a scene in a sequentially row-wise manner. The flip side is that, in the presence of camera motion, the inter-row delay leads to undesirable geometric effects, also known as rolling shutter (RS) distortions. This is because rows of the RS image do not necessarily sense the same camera motion. A prominent effect of RS distortion is the manifestation of straight lines as curves which call for correction of RS effect also termed as RS rectification. Rectification of RS distortion involves finding the camera motion for every row of RS image (row motions). Each row of the RS image is then warped using estimated row motions by taking one of the rows as reference. More than aesthetic appeal, the implications of RS rectification are critical for vision-tasks such as image registration, structure from motion (SFM), etc which perform scene inference based on geometric attributes in the captured images.

Multi-frame RS rectification methods use videos [10, 21, 7, 2] and estimate the motion across the frames using point-correspondences. The inter-frame motion helps with estimating the row motion of each RS frame, aiding the warping process to obtain distortion-free frames. Different algorithms are proposed for RS deblurring [20, 13], RS super-resolution [16], RS registration [26], and change detection [15]. The works in [25, 30] have addressed the problem of depth-aware RS rectification and again rely on multiple input frames. A differential SFM based framework is employed in [30] to perform RS rectification of input images captured by a slow-moving camera. [25] can handle the additional effects of occlusion that arise while capturing images using a fast-moving RS camera. Some methods have used external sensor information such as a gyroscope [4, 5, 14] to stabilize RS distortion in videos. Moreover, these methods are strongly constrained by the availability as well as reliability of external sensor data.

The afore-mentioned methods are data greedy and time-consuming except [2]. Moreover, they are rendered unusable when only a single image is available. [19, 17, 9] rely on straight lines becoming curves as a prominent effect to correct RS distortions. However, these methods are tailored for scenes that consist predominantly of straight lines and hence fail to generalize to natural images where actual curves are present in the 3D world. Moreover, they require knowledge of intrinsic camera parameters for RS rectification.

In this paper, we address the problem of single image RS rectification using a deep neural network. The prior work to use a deep network for RS rectification for 2D scenes is [18] wherein a neural network is trained using RS images as input, and ground truth distortion (i.e., motion) parameters as the target. Given an RS image during inference, the trained neural network predicts motion parameters corresponding to a set of key rows which is then followed by interpolation for all rows. A main drawback of this approach is that it restricts the solution space of estimated camera parameters to the ground truth parameters used during training. Moreover, arriving at the rectified image is challenging since the association between the estimated motion parameters and the pixel position of ground truth global shutter (GS) image is unknown. [18] attempts to solve this problem using an optimization framework as a complex post-processing step.

Recent findings in image restoration advocate that end-to-end training performs better than decoupled or piece-wise training such as in image deblurring [12, 24], ghost imaging [27], hyperspectral imaging [1] and image super-resolution [11, 6]. As also reiterated in [29], a fisheye distortion rectification network, regressing for ground truth distortion parameters and then rectifying the distorted image gives sub-optimal performance compared to an end-to-end approach for the clean image. To this end, we propose a simple and elegant end-to-end deep network which uses ground truth image to guide the rectification process during training. RS rectification is done in a single step during inference.

Rolling shutter distortion due to row-wise exposure of sensor array depends on the relative motion between camera and scene. Fig. LABEL:rollingShutter shows a scene captured using RS camera under different camera trajectories i.e., different values of $[r_{x},r_{y},r_{z},t_{x},t_{y},t_{z}]$ where $t_{\phi}$ and $r_{\phi}$, $\phi\in\{x,y,z\}$, indicate translations along and rotations about $\phi$ axis, respectively. As observed in the figure and as also stated in [18], the effect of $t_{y},t_{z}$ and $r_{x}$ on RS distortion is negligible as compared to the effect of $t_{x},r_{y}$ and $r_{z}$. Moreover, the effect of $r_{y}$ can be approximated by $t_{x}$ for large focal length and when the movement of camera towards or away from scene is minimal. Hence, it suffices to consider only $t_{x}$ and $r_{z}$ to be essentially responsible for RS image formation.

The GS image coordinates $(x_{\mbox{gs}},y_{\mbox{gs}})$ are related to RS image coordinates $(x_{\mbox{rs}},y_{\mbox{rs}})$ by

where $r_{z}(x_{\mbox{rs}})$ and $t_{x}(x_{\mbox{rs}})$ are the rotation and translation motion experienced by the $x_{\mbox{rs}}^{th}$ row of RS image. The GS-RS image pairs required for training our neural network are synthesized using Eq 1. Given a GS image and the rotation and translation motion for every row of the RS image, the RS image can be generated using either source-to-target (S-T) or target-to-source (T-S) mapping with GS coordinates as source and RS coordinates as target. Since the motion parameters are associated with RS coordinates, S-T mapping is not employed for RS image generation. In T-S mapping, each pixel location of target RS image is multiplied with corresponding warping matrix formed by the motion parameters to yield source GS pixel location. The intensity at the resultant GS pixel coordinate is found by using bilinear interpolation and then copied to the RS pixel location.

Given an RS image and motion parameters for each row of RS image, the RS observation can be rectified akin to the process of RS image formation except that the source is now the RS image while target is the RS rectified image. In S-T mapping, each pixel location of RS image along with its row wise camera motion is substituted in Eq. (1) to get RS rectified (target) pixel location. However, there is a possibility that some of the pixels in the target RS rectified image may go unfilled leaving holes in the resultant RS image. In T-S mapping, for every pixel location of RS rectified image (target), the same set of equations (i.e., Eq. (1)) can be used to solve for RS image (source) coordinates provided the camera motion acting on RS rectified coordinates is known.

Our main objective is to find a direct mapping from the input RS image to the target RS rectified image. This requires estimation of row-wise camera motion parameters, and the correspondence between estimated motion parameters and target pixel locations. We achieve the above in a principled manner as follows. We propose to use image features of input RS image for estimating camera motion parameters and devise a mapping function to relate estimated motion parameters with pixel locations of target image.

Our network architecture (Fig. 1) consists of five basic modules: motion block, trajectory module, row block, RS regeneration module and RS rectification module. The motion block predicts camera motion parameters ($t_{x}$ and $r_{z}$) for each row of the input RS image. The trajectory module ensures that the estimated camera motion parameters follow a smooth and continuous curve as we traverse the rows in the RS image, in compliance with real-life camera trajectories. For each pixel location in the target image, the corresponding camera motion is found using the row block. The output of row block and trajectory module are used by the RS rectification module to warp the input RS image to get the RS rectified image. For faster convergence during training and to better-condition the optimisation process, we also employ an RS regeneration module which takes motion parameters from the trajectory module and warps the GS image to estimate the given (input) RS image. A detailed discussion of each of the module follows.

Motion block This consists of a base block followed by $t_{x}$(translation) and $r_{z}$(rotation) blocks, respectively. The base block extracts features from the input RS image which are then used to find row wise translation and rotation motion of the input image. Thus, the motion block takes input color image of size $r\times r\times 3$ and outputs two 1D vectors of length $r$ indicating rotation and translation motion parameters for each row of the input RS image. The base block is designed using three convolutional layers. Both translation and rotation blocks, which take the output of the base network, are designed using three convolutional layers followed by two fully connected (FC) layers. The final FC layer is of dimension [$r$,1] reflecting the motion for every row of the input RS image. Each convolutional layer is followed by batch normalization.

Row block As discussed in the second section, every pixel coordinate in the GS (target) image is substituted in Eq. (1) along with the corresponding camera motion to get the RS (source) image coordinate. However, camera motion acting on each GS pixel to form the RS image is known only to the extent that it is one of the motions experienced by the rows of RS image. Specifically, for a given input RS image, all the pixels in a row stem from a single camera motion and the motion will generally be different for each row. In contrast, pixels in a row of the GS image need not be influenced by a single motion. The ambiguity of which motion to associate to a pixel in target image was addressed in [18] as a post-processing step, which is a complicated exercise and implicitly constrains the estimated motion parameters.

We propose to use a deep network to solve this issue thus rendering our network end-to-end. The row block takes an image of size $r\times r\times 3$ and outputs a matrix of dimension $r\times r$, with each location indicating which row motion of RS image must be considered from the estimated motion parameters for rectification. In real camera trajectory, camera motion is typically smooth. Consequently, the output of row block at each coordinate location can be expected to be close to its corresponding row number. Hence, for both stability and faster convergence, we learn to solve only for the residual (motivated by [3, 6]). The residual image is an $r\times r$ matrix and is the output of the row block. This image is added to another matrix (say $A$) which is initialized with its own row number i.e., $A(i,j)=i$ for the reason stated above. The resultant matrix values indicate which camera motion is to be considered from the estimated motion parameters for rectification of RS image. From now, we refer to output of row block as the sum of learnt residual with a matrix initialized with its row number at each coordinate position. The row block consists of five convolutional layers with each layer followed by batch normalization and an activation function. Use of three layers for base block and a total of 6 convolution layers for motion block is partly motivated by [18] (it uses five convolution layers for motion estimation). Since, the objective of row block (finding residual for) is comparatively less complex compared to motion block we used only 3 layers.

This section is arranged as follows: (i) dataset generation, (ii) implementation details, (iii) competing methods, (iv) quantitative analysis, and (v) visual results.