Real-World Video for Zoom Enhancement based on Spatio-Temporal Coupling

The past few years have witnessed great success in applying deep learning to enhance the SR quality [23, 24, 42, 34, 8, 30, 18, 22, 23, 37, 7, 13, 39]. However, most existing SR methods are typically evaluated with simulated datasets, learning digital zoom in practical scenarios by these methods would be less effective and results in a significant deterioration on the SR performance.

Lately, the realistic single-frame datasets based methods are proposed by a few of the studies for zoom quality enhancement. Cai et al. [3] presented a new single-frame benchmark and a Laplacian pyramid based kernel predication network (LP-KPN) to handle the real-world scenes. Chen et al. [5] investigated the feasibility by alleviating the intrinsic tradeoff between resolution and field-of-view from the perspective of camera lenses. Zhang et al. [48] adopted a contextual bilateral loss (CoBi) to deal with the misalignment issue between paired LH and HR images, and used a high-bit raw dataset, SR-RAW, to improve the input data quality. The benefits of using raw as input over RGB is proven in [48] as well. Joze et al. [16] released a realistic dataset named ImagePairs for image SR and image quality enhancement. Besides, NTIRE 2019 [2] organized a challenge to facilitate the development of real image SR. The provided image pairs in the challenge are registered beforehand by a sophisticated image registration algorithm, thus high-quality SR results can be generated by using carefully designed deep learning architectures[6, 21, 49]. Considering that very accurate subpixel registration is difficult in realistic raw data, as well as the misaligned feature would undermine the feature extraction capability of deep learning model, instead of using single-frame pairs, to adopt temporal correlation in adjacent multi-frame clips would be a promising strategy to enhance the limited spatial information. However, to our best knowledge, there exists no work for learning-based zoom by multi-frame pairs, and the closest one is [38], which supplants the need for demosaicing in a camera pipeline by merging a burst of raw images. In this paper, we address the challenge of introducing both spatial and temporal information via real-world video dataset for learning-based zoom quality enhancement.

The inter-frame spatial and temporal information has been exploited by many recent simulated data based video SR studies [35, 12, 13, 36, 45, 1, 41, 32, 28]. The methods can be grouped into time-based, space-based, and spatio-temporal based. A representative study of time-based way is presented by Shi et al. [40], which formulated a convolutional long short-term memory (ConvLSTM) architecture to reserve information from the last frame. Regarding space-based way, the aim of which is trying to merge temporal information in a parallel manner, studies such as VSRnet [17] and DUFVSR [14] achieved the goal by direct fusion architecture and 3D convolutional neural networks (CNN), respectively. Besides, for spatio-temporal way, Yi et al. [45] proposed a novel progressive fusion network by incorporating a series of progressive fusion residual blocks (PFRBs), and Caballero et al. [1] adopted spatio-temporal networks and motion compensation, and Wang [36] introduced the enhanced deformable convolutional networks, while Wang [35] applied optical flow estimation.

However, due to the spatial misalignment issues in realistic video pairs, the aforementioned methods which mainly focus on network architecture optimization have not been able to effectively couple spatio-temporal information via pixel-wise loss functions. Instead of further optimizing the network architecture, we propose to investigate the feasibility of achieving spatio-temporal coupling in the perspective of loss function.

As shown in Figure 2, the optical system consists of one beam splitter, two global shutter cameras, two identical zoom lenses with different focal lengths, and a signal synchronizer to keep time synchronization. When capturing videos, the incoming light is first divided into two perpendicular lights by a 45^∘ beam splitter, and then pass through a long and a short focal-length camera equipped with RGGB Bayer sensor, respectively.

Here, two FLIR GS3-U3-15S5C cameras and two RICOH FL-CC6Z1218-VG manual zoom lenses are adopted to collect 4X upscale ratio video pairs. The focal length is set to 18mm in the branch of LR videos with larger field-of-view (FoV), and it is set to 72mm in the branch of HR videos with smaller FoV. Although we focus on the investigation of 4X data, which is common in video SR, our capture system can be used to capture up to 6x paired data without modifications.

For camera settings, we choose 15fps frame rate to enhance spatial-temporal correlation, 2mm shutter speed to avoid obvious blur on fast-moving objects, and a relative small aperture size to alleviate the influence of depth-of-field difference. With the proposed imaging system, 84 pairs in multiple scenes, each containing 200 frames with 1384 $\times$ 1032 resolution, are captured from different street spots. We take 16-bit LR raw and 8-bit HR sRGB as the input and ground truth for zoom learning, respectively. It should be noted that the camera has a 14-bit ADC and the affiliated Flycapture2 software converts the original 14-bit raw into 16-bit by linear scaling. The 8-bit HR sRGB images are generated by the in-built ISP of Flycapture2.

In terms of geometric alignment, we first match the FoV of each paired LR and HR frame in VideoRAW based on the predefined focal lengths. Some examples are shown in Figure 3. Since the videos taken at different focal lengths suffer from different lens distortions and perspective effects, as well as the subtle shift between light splitting paths, the misalignment is inevitable during data capturing. To address this issue, we employ homography transformation to warp the HR image based on the paired LR. Then, to match the size of LR frames based on the target zoom ratio, a scale offset is applied to HR frames. After that, we randomly crop consecutive frame patches from the paired videos for 4X SR training. Although obvious misalignment can be alleviated by the preprocessing step, as shown in Figure 4 A and B (GT: HR₀), nontrivial misalignment between paired LR and HR imagery is still unavoidable. We usually observe 10-40 pixels shift in a processed pair depending on the scene geometry. We attribute this shift to various physical effects in optical zooming, such as perspective distortion, rather than the temporal synchronization, since the two cameras are rigorously synchronized within sub-microsecond accuracy.

In terms of photometric alignment, including image brightness and color white balance, we capture the dark background and estimate white balance ratios between raw and RGB images for the blue and red channels. For all 16-bit raw images, we first subtract the black level, and then multiply the two ratios, so as to approximate the white balance of the RGB images. This correction allows to minimize the color and brightness difference between the pair, and helps to highlight the aforementioned discrepancies caused by the zooming effect.

We propose a unified framework Spatio-Temporal Coupling Loss (STCL), as shown in Figure 5, which is extensible to different existing deep learning architectures for digital zoom quality enhancement. The challenge lies in the design of the constraint for spatio-temporal coupling via realistic video datasets when paired LR and HR₀ are misaligned, and establishing precise correspondences among LR and adjacent frames HR_t is difficult. To obtain high-quality outputs, we solve the difficulties by (1) spatial alignment and fusion, and (2) temporal fusion and aggregation, both at the high-dimension feature level.

Concretely, we design a spatial constraint by exploring the correlation among LR, HR₀, and expected SR. Considering in realistic scenarios when capturing a closer view of far-away subjects with greater details, the position relations among the features in zoom-in contents should rely on short-focal length LR, while the others such as edges, texture, and color are referenced based on long-focal length HR₀. Thus, the proposed spatial constraint consists of two components. The one aligns the feature position among SR and LR in a lower scale with a position constraint kernel, while the other one is responsible for fusing the features from the paired HR₀ frame into SR in higher scale.

In terms of temporal constraint, we propose to compare the feature distribution of SR and adjacent frames rather than just comparing the appearance. Since different frames are not equally informative to the reconstruction, the weighted constraint is designed by considering the correlation between the features of HR₀ and adjacent frames. Thus, the temporal constraint is able to guide the feature extraction and aggregation from consecutive frames for the effective feature fusion and compensation. Finally, the spatio-temporal coupling can be achieved in the zoom task by the given realistic video pairs and the integration of spatial and temporal constraints.

Our objective function is formulated as

where $Loss_{s}$ and $Loss_{t}$ refer to spatial and temporal constraint, respectively. By effectively coupling spatial and temporal information from realistic video datasets, the STCL could achieve zoom quality enhancement. To the best of our knowledge, our approach proposed in this paper is the first attempt in this direction.

To align SR and LR, we first downsample SR into ${LR}^{\prime}$ to match the size of LR in a lower scale. Instead of using pixel-to-pixel loss like $L_{1}$ and $L_{2}$, we align LR and ${LR}^{\prime}$ at the feature level based on the features extracted by $[conv3\_2,conv4\_1]$ in pretrained VGG19 ($\Phi_{1}$) [31]. Then we introduce spatial awareness into CX by Gaussian kernel to constraint the spatial distance between the two similar features. Our $Loss_{a}$ can be defined as

where $L$ and ${L}^{\prime}$ are the feature space in LR and ${LR}^{\prime}$, respectively, and ${D}_{l_{i},{l}^{\prime}_{j}}$ denominates the cosine distance between feature $l_{i}$ in $L$ and ${l}^{\prime}_{j}$ in ${L}^{\prime}$. The kernel $\kappa$ can be formulated as

where ${D}^{\prime}_{l_{i},{l}^{\prime}_{j}}=\left\|(x_{i},y_{i})-(x_{j},y_{j})\right\|_{2}$ denominates the spatial coordinate distance between feature $l_{i}$ and ${l}^{\prime}_{j}$. Here, we select $\mu=0$ and $\sigma=2$. By adopting proposed $Loss_{a}$, similar feature pairs between LR and ${LR}^{\prime}$ can be aligned spatially.

Regarding ${Loss}_{r}$, since CX can be viewed as an approximation to KL divergence, and is designed for comparing images that are not spatially aligned [25], we directly apply it to perform statistical constraint between feature distributions as

where $H_{0}$ and $S$ refer to the feature space generated by $[conv1\_2,conv2\_2,andconv3\_2]$ in VGG19 ($\Phi_{2}$). Thus, ${Loss}_{s}={Loss}_{a}(L,{L}^{\prime})+{Loss}_{r}(H_{0},S)$ can align feature from LR and fusion feature from $H_{0}$ spatially.

Then, the temporal loss is defined by aggregating all the compensation losses as

In this paper, since our current dataset (15fps) mainly covers city views with pedestrians and moving vehicles under the speed of 45km/h, we choose T = 1 (3-frame clip) and $w_{\pm}$ = 0.1.

For comparison, we choose a few representative loss functions used for SR methods based on spatial and temporal concern: CX [26], which performed by comparing statistic feature distributions without considering both spatial and temporal correlations; $L_{2}$ [23], the most widely used pixel-wise spatial constraint applied in many state-of-the-art SR approaches; CoBi [48], an effective loss used in realistic SR for spatial constraint. All the baselines are integrated into SRResNet architecture for re-training on the proposed training dataset.

To evaluate our method as well as the baselines, evaluation metrics, including pixel-based PSNR, structure-based SSIM, and learning-based LPIPS, are adopted. Unlike the case of PSNR and SSIM, the lower score of LPIPS indicates better image quality.

The relative performances of different methods over testing data are listed in Table 1. In general, the proposed method outperforms others in terms of most evaluation metrics and scenes, while CX, enforcing constraints on feature distribution only, gets the worst in learning based methods. Specifically, both our method and CoBi are better than $L_{2}$ in PSRN. The $L_{2}$ loss function usually achieves particularly high PSNR in SR than others due to the pixel-to-pixel mapping via MSE. However, in realistic case where LR and HR are misaligned, such pixel-wise mapping will bring incorrectness and noise to learning, which yields the lower PSNR performance. It indicates that the pixel-wise loss cannot perform effective mapping between LR and misaligned HR. In perspective of LPIPS, since the pixel-wise optimization often lacks high frequency content, it results in perceptually unsatisfying results with overly smooth textures. Our model and CoBi still perform much better than $L_{2}$ in all scenes. Besides, by adopting spatio-temporal coupling, our model shows better performance than CoBi in all cases. Such results verify the effectiveness of introducing temporal components in zoom quality enhancement.

Qualitative comparison of our model against baselines in all three scenes is shown in Figure 6. Within these scenes, moving vehicles and pedestrians are presented. Since the direct Bicubic upsampling from LR (the $2^{nd}$ column) brings very blurry outlook, and the CX (the $4^{th}$ column) leads strong artifacts that caused by the inappropriate feature matching, we mainly focus on the comparison between ’weak-spatial’ $L_{2}$, ’spatial’ CoBi, and our ’spatiao-temporal’.

In scene 1, we focus on the comparison of character (the $1^{st}$ row) and vegetation (the $2^{nd}$ row). Due to the weak-spatial mapping, $L_{2}$ results in very blurry for the character. Although CoBi can generate clear character as ours, for the high frequency texture back on the wall, our method achieves sharper edges and finer textures. As for vegetation, our method yields the most consistent visual result without any artifacts. In scene 2, it appears that our method can super-resolve zebra crossing (the $3^{th}$ row) with higher quality, while CoBi is too pale with limited contrast. Regarding the number plate on the moving vehicle (the $4^{th}$ row), although all the results generated by baselines and ours are not very clear, which we think is caused by high noise and signal loss in original LR image, our method still generates better results. In scene 3, our method results in very clear appearance for distant guideboard (the $5^{th}$ row) and wall (the $6^{th}$ row), while $L_{2}$ is too blurry to see the details and CoBi yields additional unnatural ’stripe’ artifacts on the vegetation. All of the above qualitative results demonstrate the effectiveness of our method on the realistic SR in different scenarios.

To further investigate the effectiveness of the proposed spatio-temporal coupling method, we conduct an ablation study using two variants: one with temporal compensation provided by multi-frame videos and the other without. The relative quantitative comparison is shown in Table 2.

It reveals that with the help of spatio-temporal coupling, the reconstruction quality on multiple metrics can be improved, which consolidates the value of temporal compensation. In Particular, for SSIM, it leads to about 4.4% (0.8152 vs. 0.7811) improvement on average.

To evaluate the generalizability of the proposed method, we further integrate our framework into more existing deep learning architectures. In this paper, EDSR [24], DCSCN [42], and FEQE [34] are adopted. For comparison, the ’Ori’ refers to the weak-spatial loss function applied in the original paper. The comparison results are generated from the average performance of three scenes, as shown in Table 3, which reveals the generalization of the proposed method.

Moreover, to demonstrate the proposed method has a favorable capability in terms of video zoom, we evaluate the perceptual quality of the generated video through blind testing. In each inquiry, we present the participants with three videos (200 frames per video) taken from different scenes. At every frame, ground truth image and corresponding images generated from baseline models (CX, $L_{2}$, CoBi) and ours are organized side by side. The location information of both ’Ours’ and the ’Baseline’ are not provided. The participants are asked to pick up the one that is more close to the ground truth video. In the experiment, the responses from 60 valid participants are collected and listed in Table 4. Since some occasional but noticeable artifacts in CoBi might severely influence subjective evaluation, especially in #2, videos generated by ours achieve a significantly higher preference rate under blind pairwise human judgment.