Neural Global Shutter: Learn to Restore Video
from a Rolling Shutter Camera with Global Reset Feature

We use an encoder ${E}_{\theta}$ to extract the low-dimensional representations $\{\bm{f}_{t}^{i}\}_{t=1}^{S}$ for a given video segment $\{\bm{x}_{t}\}_{t=1}^{S}$. Our encoder ${E}_{\theta}$ operates on each frame respectively. Two unique designs power its ability to address spatial-varying distortions. First, we encode each pixel’s exposure duration (EE) and feed them along with the input frame into ${E}_{\theta}$. The input $\bm{x}_{t}$, therefore, has four channels. This design comes from our observation that the RSGR distortion gradually changes and is relevant to the exposure duration. Second, we integrate the spatial attention (SA) mechanism [7] into the encoder ${E}_{\theta}$ for producing spatially selective feature maps, further enabling us to embed the position information. These two components enable the encoder to be adaptive to the exposure duration.

Similar to most of existing video deblur methods use temporal information as an essential cue, we leverage both the long-term and short-term temporal information from the input video segment $\{\bm{x}_{t}\}_{t=1}^{S}$. Specifically, we aggregate long-term temporal information $\{\bm{f}_{t}^{i}\}_{t=1}^{S}$ with two RNNs bidirectionally. We first perform forward information aggregation, with $t$ increases from 1 to $S$, the output at time $t$ is $\bm{f}_{t}^{a}=F_{a}([\bm{f}_{t}^{i},\bm{h}_{t-1}^{a}])$, with $\bm{h}_{t}^{a}=H_{a}(\bm{f}_{t}^{a})$, where $\bm{h}_{t}^{a}$ is the hidden state, $[\cdot,\cdot]$ denotes concatenation operation along the channel axis²²2We set the length of input video segment $S$ to 8.. We instantiate $F_{a}$ and $H_{a}$ with residual blocks (RBs) [6] and residual dense blocks (RDBs) [39]. The initial hidden state $\bm{h}_{0}^{a}$ is set to 0. Given the outputs from the forward aggregator, we then perform backward information aggregation, with $t$ decreases from $S$ to 1, the output at time $t$ is $\bm{f}_{t}^{b}=F_{b}([\bm{f}_{t}^{a},\bm{h}_{t+1}^{b}])$, with $\bm{h}_{t}^{b}=H_{b}(\bm{f}_{t}^{b})$. Likewise, $F_{b}$ and $H_{b}$ comprise RBs and RDBs. The initial hidden state $\bm{h}_{S+1}^{b}$ is set to 0. We also tried to initialize $\bm{h}_{S+1}^{b}$ with $\bm{h}_{S}^{a}$, but the results are unsatisfactory. We propagate bidirectionally as we find that using only one-directional temporal information will unbalance different frames. It is even worse than without using long-term temporal information.

We have incorporated the long-term temporal information, while the short-term (local) temporal information is also essential. We use a sliding window to traverse the video segment. In the window, we implicitly align the center frame feature $\bm{f}_{t}^{b}$ and its neighboring frame features $\{\bm{f}_{t\pm k}^{b}\}_{k=1}^{K}$ using a deformable convolutional network (DCN) [4]. Given the aligned neighboring features and the center feature, we concatenate them along the channel axis. We use the channel attention (CA) to learn to weight them and a convolutional layer (Conv) to learn to fuse them³³3We set the length of the window to 3, its stride to 1, and $K$ to 1..

Taking a refined feature map $\bm{f}_{t}^{o}$ as input, the decoder ${D}_{\theta}$ renders a clean GS image ${\bm{y}}_{t}$. To capture details, we let our network to learn the difference $\Delta\bm{x}_{t}$ between the generated image ${\bm{y}}_{t}$ and the original image $\bm{x}_{t}$ with a global residual connection [6]: ${\bm{y}}_{t}=\bm{x}_{t}+\Delta\bm{x}_{t}$.

We train our network in a supervised fashion using the target GS videos as the direct supervisory signal. Specifically, we compute the differences between the rendered video frames $\{{\bm{y}}_{t}\}_{t=1}^{T}$ and its GS counterparts $\{\bar{\bm{y}}_{t}\}_{t=1}^{T}$. But since GS videos often suffer from noises, we use the perceptual loss [9] and SSIM loss [36] together to suppress the negative effects caused by noisy supervision:

$${L}=\lambda\sum_{l}\lVert{\psi_{l}(\bar{\bm{y}}_{t})-\psi_{l}({\bm{y}}_{t})}\rVert_{1}+(1-\lambda)\phi({\bm{y}}_{t},\bar{\bm{y}}_{t})\,,$$

(1)

where $\psi_{l}$ is the feature extracted from $l$-th layer of a pre-trained VGG-19 network [29]; $\phi(\cdot,\cdot)\in[0,1]$ is 1 minus a differentiable version of the SSIM [31]. The parameter $\lambda$ controls the balance between perceptual and SSIM components, which is dynamically set. All of the errors resulting from $T$ frames are summarized together.

We use our contributed synthetic and real-world datasets for the following experiments.

• •

  Real-world dataset. Using the proposed optic system, we collect the first RSGR dataset and the corresponding GS ground truth. Our dataset consists of 79 video sequences captured under real scenes. Each sequence consists of 300 consecutive frames with $640\!\times\!640$ spatial resolution. The exposure time of the GS camera is 1ms, which is the same as the exposure time of the first scanline of the RSGR camera. Since the exposure time is short, the GS videos are sometimes slightly noisy. We split the dataset into a training set with 27 sequences and testing sets with 52 sequences. The testing sets have two parts. The smaller one Set-I with 3 sequences share similar imaging conditions with the training set, while the larger one Set-II with 49 sequences have worse imaging conditions, where GS videos have noises. We use Set-I as validation set. Note that prevalent datasets used for RS correction and GS deblur either are synthesized [32, 16], having significant gaps from real datasets or have no ground truth [44], which are hard for developing and evaluating data-driven algorithms. Unlike them, we capture real scenes with aligned ground truth (GT). We deliver them to the community to facilitate subsequent researches.

• •

  Synthetic dataset. We synthesize 25 video sequences for each of the GS, RS, and RSGR exposures. Each video has 29 frames with $512\!\times\!512$ resolution. The scan directions are top-to-bottom ($\downarrow$). The first scanlines of a corresponding GS, RS, and RSGR frame are exposed simultaneously and with the same duration. During synthesizing RSGR videos, we use a parameter $\xi$ to determine the ratio between readout time and the first scanline’s exposure duration. We synthesize RSGR videos with 8 different $\xi$ for both training and testing.

Our optic system offers the convenience of using captured GS videos as GT to assess different algorithms. The Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index Measure (SSIM) [36] are used as metrics. We calculate them with a single video frame. Considering the distortion is spatial-varying, we also divide the video frame into parts for evaluation.

Since this problem is unexplored, we compare our algorithm with several closely related algorithms from four different categories: 1) unsupervised GS image-based deblur algorithm that uses adversarial training to get rid of the paired training data: deblurGANv2 [11]; 2) supervised GS image-based deblur algorithm that integrates multi-scale reception fields: SRN [33]; 3) supervised GS video-based deblur algorithms that leverages the temporal information by fusing neighboring frames or with an RNN: STRCNN [8], DBN [32], IFIRNN [20], and ESTRNN [41]; and 4) supervised RS correction or deblur methods that end-to-end estimate motion and compensate distortion: DSUR [16] and JCD [42].

Table 1 reports the qualitative results. We observe that all algorithms except for deblurGANv2 improve the original videos. It demonstrates that supervised image-to-image translation algorithms can remove the RSGR distortion. But, as verified by Liu et al. [16], they cannot correct the RS distortions since a rectified pixel in the virtual GS image might lie far away from its corresponding pixel in the input RS image. The RS distortion correction requires accurately estimated motion. Moreover, our algorithm achieves the best performance. It consistently outperforms others with a large margin on all evaluation metrics. Especially, opposite to other algorithms that tend to improve only high-quality inputs captured without motion, textureless frames, etc., our algorithm improves not only the high-quality inputs but also the low-quality counterparts (see supplemental material). We suppose it is because the competing algorithms target RS or GS inputs with the same exposure duration for every pixel. Differently, our RSGR video has gradually increased exposure duration along the scan direction, leading to spatially-varying blur and brightness distortion. Our tailored algorithm incorporates a spatial-aware encoder and dual temporal information aggregators to achieve the best performance for correcting the RSGR distortion.

Figure 3 presents qualitative results. Similar to quantitative results, we find that our algorithm renders visually pleasing results that are better than other algorithms. It corrects mixed spatial-varying blur and brightness without introducing additional distortions. Remarkably, thanks to our carefully selected loss function, our rendered videos are noise-free even though there are noises in the supervisory signal. Unfortunately, other algorithms not for this problem have sub-optimal performance. They fail to remove the distortions perfectly like us and even bring some artifacts, e.g., geometric deformations, color distortions, and noises. For example, deblurGANv2 introduces noises due to adversarial training with noisy supervision. STRCNN and DBN introduce color distortions. IFIRNN and JCD yield geometric deformations owing to the large spatial-varying blur.

We have illustrated that our tailored architecture is superior in RSGR correction. But, it should be noted that besides the architecture, there also exists another factor accounting for this success: the paired training data, which gives our model sufficient supervision to learn the task-specific knowledge. We seek to figure out if others can replace the knowledge. First, we compare RSGR correction using our end-to-end learned knowledge with that borrowed from other tasks (models with pre-trained weights) directly, including RS correction [16], RS deblur [42], GS motion deblur (MT-deblur) [11], and GS out-of-focus deblur (OF-deblur) [13]. From the results in Figure 4, we observe that although using pre-trained knowledge directly relaxes the requirement for a paired dataset, they cannot deal with the spatial-varying brightness and blur. Especially, the spatial-varying blurs lead to annoying geometric distortions. The results confirm our required knowledge is different from other tasks. Second, we use handcraft knowledge instead of learning-based knowledge to correct the spatial-varying brightness distortion. The result is frustrating. Third, we train two unsupervised methods, i.e., CycleGAN [43] and deblurGANv2 [11] without paired data. The results verify our required knowledge cannot be replaced by unsupervised knowledge. Accordingly, we build the optic system and paired data with incredible efforts.

We argue that our solution is better than the RS solution. 1) Formulation. With the global reset feature, we convert the RS rectification problem into a deblur-like one, which can be solved with image-to-image translation algorithms while the RS rectification cannot. 2) Effectiveness. The results in Figure 5 show that our RSGR solution outperforms the RS solution. Especially, our solution is good at dealing with significant motion (lower neighboring PSNR/SSIM). We also find that our solution goes ahead of the RS solution with a large margin when $\xi$ is small, indicating a large exposure duration. Although our performance decays with the increase of $\xi$, we are still superior to the RS solution. Considering it is not always to require a large ratio $\xi$ unless we need to capture high-speed videos, our solution is possible to replace RS solution in actual video capture. 3) Efficiency. In addition to accuracy, our solution without explicit motion estimation is also more efficient than existing RS solutions. Classical two-frame-based RS correction methods often require a few minutes. Zhuang et al. [44] processing a $640\!\times\!480$ resolution frame requires 400 seconds, DSUR requires 0.43 seconds, JCD requires 0.83 seconds, but we only need 0.04 seconds⁴⁴4We conduct all run-time comparisons of DL-based methods, i.e. DSUR, JCD, and ours on the same machine with NVIDIA Tesla V100 GPU and Intel(R) [email protected]. The run-time result of Zhuang et al. [44] borrowed from DSUR is based on an Intel Core i7-7700K CPU.. Considering its effectiveness and efficiency, we believe our RSGR solution can replace RS solutions.

We conduct ablation experiments on our network architecture by implementing 9 different variants to explore the effectiveness of each module. From the results in Table 2, we have three main findings. First, removing any of the components of our method will weaken the performance. The result verifies that all components are necessary. Second, using spatial-aware modules (T6) has better performance than using only the temporal counterparts (S3). We argue that this is because the power of the temporal information aggregators relies on clean feature maps produced by spatial-aware encoders. With the spatial-aware design, the aggregators would gather distorted feature maps and lose their effectiveness. Third, we surprisingly find that using only one path of the long-term aggregator (T1) is worse than using full (ours) and without using the long-term temporal aggregator (T2). We assume this is because our algorithm uses long-term and short-term temporal information together. When we gather the long-term information from one direction, the information imbalance in a video sequence will trouble the short-term aggregation. Consequently, our long-term and short-term aggregators gather information from forward and backward directions and yield the best performance.

We also verify the effectiveness of different loss functions in Figure 6, including the perceptual loss [9], the gradient loss [18], the Charbonnier loss [3] and the SSIM loss [36]. We find that all losses other than the SSIM loss result in different artifacts. It is because there exist noises in the supervisory signal. This phenomenon also appears in the adversarial loss. Besides, the perceptual loss has the best performance in structure restoration since it operates on the feature level. Therefore, we combine the SSIM loss and the perceptual loss, leading to the best results.