AIM 2020 Challenge on
Video Temporal Super-Resolution

Method. The XPixel team has proposed the Enhanced Quadratic Video Interpolation (EQVI) [20] method. The algorithm is built upon the AIM 2019 VTSR Challenge winning approach, QVI [38, 19], with three main components: rectified quadratic flow prediction (RQFP), residual contextual synthesis network (RCSN), and multi-scale fusion network (MS-Fusion). To ease the overall optimization procedure and verify the performance gain of each component, four-stage training, and fine-tuning strategies are adopted. First, the baseline QVI model is trained with a ScopeFlow [3] flow estimation method instead of the PWC-Net [36]. Then, the model is fine-tuned with an additional residual contextual synthesis network. In the third stage, the rectified quadratic flow prediction is adopted to improve the model with fine-tuning all the modules except the optical flow estimation network. Finally, all the modules are assembled into the multi-scale fusion network and fine-tuned.

Residual contextual synthesis network. Inspired by [27], a residual contextual synthesis network (RCSN) is designed to warp and exploit the contextual information in high-dimension feature space. Specifically, the $conv1$ layer of the pre-trained ResNet18 [14] is adopted to capture the contextual information of input frame $I_{0}$ and $I_{1}$. Then we apply back warping on the features with $f_{0\rightarrow t}$ to attain pre-warped features. The edge information of $I_{0}$ and $I_{1}$ is also extracted and warped to preserve and leverage more structural information. For simplicity, we calculate the gradient of each channel of the input frame as edge information. Afterward, we feed the warped images, edges, and features into a small network to synthesize a residual map $\hat{R_{t}}$. The refined output is obtained by $\hat{I}_{t}^{refined}=\hat{I}_{t}+\hat{R}_{t}$. Fig. 5 shows the overall organization of the network.

Rectified quadratic flow prediction. Given four input consecutive frames $I_{-1}$, $I_{0}$, $I_{1}$ and $I_{2}$, the goal of video temporal super-resolution is to interpolate an intermediate frame $I_{t}$, where $t\in(0,1)$. Following the quadratic model [38], a least square estimation method is utilized to improve the accuracy of quadratic frame interpolation further. Unlike the original QVI [38], all four frames (or three equations) are used for the quadratic model. If the motion of the input four frames basically conforms to the uniform acceleration model, the following equations should hold or approximately hold:

where $f_{0\rightarrow t}$ denotes the displacement of the pixel from frame 0 to frame $t$, $v_{0}$ represents the velocity at frame 0, and $a$ is the acceleration of the quadratic motion model. The equations above can be transformed into a matrix form:

where the solution $x^{*}=[v_{0}^{*}\quad a^{*}]^{T}$ can be derived as $x^{*}=[A^{T}A]^{-1}A^{T}b$. Then, the intermediate flow could be formulated as $f_{0\rightarrow t}=v_{0}^{*}t+\frac{1}{2}a^{*}t^{2}$. When the motion of input four frames approximately fits the quadratic assumption, the LSE solution can make a better estimation of $f_{0\rightarrow t}$. As the real scenes are usually complex, several simple rules are adopted to discriminate whether the motion satisfies the quadratic assumption. For any pair picked from $I_{-1}$, $I_{1}$ and $I_{2}$, an acceleration $a_{i}$ is calculated as:

Multi-scale fusion network. Finally, a novel multi-scale fusion network is proposed to capture various motions at a different level and further boost the performance. This fusion network can be regarded as a learnable augmentation process at different resolutions. Once we obtain a well-trained interpolation model from previous stages, we feed the model with one input sequence and its downsampled counterpart. Then a fusion network is trained to predict a pixel-wise weighted map $M$ to fuse the results.

where $Q$ represents the well-trained QVI model, $Down(.)$ and $Up(.)$ denote the downsampling and upsampling operations, respectively. Then, the final output interpolated frame is formulated as follows, as illustrated in Fig. 7:

The proposed method is implemented under Python 3.6 and PyTorch 1.2 environments. The training requires about 5 days using 4 RTX 2080 Ti GPUs.

Method. KAIST-VICLAB team proposes the quadratic video frame interpolation with a multi-frame synthesis network. Inspired by the quadratic video interpolation [38] and the context-aware synthesis for video interpolation [27], the proposed approach leverages these types of flow-based method. The REDS_VTSR dataset [26] is composed of high-resolution frames with much more complex motion than typical frame interpolation dataset [39]. Since the optical flow is vulnerable to complex motion or occlusion, the proposed method attempts to handle the vulnerability by a nonlinear synthesis method with more information from neighbor frames. Therefore, the proposed algorithm first estimates the optical flows between the intermediate frame and the neighboring frames using four adjacent frames. The method for estimating the optical flows follows the same method for estimating the optical flows of the intermediate frame by a quadratic method based on the PWC-net [36] in [38]. Unlike the QVI [38] method, the proposed method utilizes all the four neighboring frames for the synthesis network and incorporates a nonlinear synthesis network into the last part of our total network to better adapt to the frames containing complex motion or occlusion.

Details. The proposed model is composed of the residual dense network [44], and the inputs of the synthesis network are not only the feature maps obtained by estimating the optical flows using four neighboring frames, but also the warped neighboring frames. Also, the output of the synthesis network is the corresponding intermediate frame. Motivated by TOFlow [39], the PWC-net for the optical flow estimation is fine-tuned on the REDS_VTSR dataset. More details about the proposed network are illustrated in Fig. 8. The intermediate frames are optimized using the Laplacian loss [27] on three different scales and VGG loss. The method is developed under Python 3.6 and PyTorch 1.4 environments, using one TITAN Xp GPU. The training takes about 3 days.

Method. BOE-IOT-AIBD team has proposed the Multi Scale Quadratic Interpolation (MSQI) approach for the VTSR task. The model is trained on the REDS_VTSR dataset [26] of three different scales. At each scale, the proposed MSQI employs PWC-Net [36] to extract optical flows and further refines the flow continuously through three modules: quadratic acceleration, flow reverse, and U-Net [35] refine module. Finally, a synthesis module interpolates the output frame by warping in-between inputs and refined flows. During the inference, a post-processing method is applied to grind the interpolated frame.

Details. Fig. 9 illustrates an overview of the proposed MSQI method. Firstly, the MSQI model has trained on sequences from the REDS_VTSR dataset [26], using $\times 4$ lower resolution ($320\times 180$) and frame rate (15fps) for quick convergence. Then, the model is fine-tuned on sequences of higher resolution ($640\times 360$). In the last stage, the MSQI model is trained using full-size images ($1280\times 720$) with a 15fps frame rate. During the inference time, a smoothing function is adopted for post-processing. Also, quantization noise is injected during the training and inference phase. The proposed method also adds the quantization noise to input frames at the inference phase and employs a smooth function to grind the interpolated frame. The method adopts the pre-trained off-the-shelf optical flow algorithm, PWC-Net [36], from the SenseSlowMo [19, 38] model. Compared to the baseline SenseSlowMo [19, 38] architecture, the proposed MSQI converges better and focuses on multiple resolutions. Python 3.7 and PyTorch environments are used on a V100 GPU server with a total of 128GB VRAM.

Method. TTI team has proposed a temporal super-resolution method that copes large motions by reducing an input sequence’s spatial resolution. The approach is inspired by STARnet [11] model. With the idea that space and time are related, STARnet jointly optimizes three tasks, i.e., spatial, temporal, and spatio-temporal super-resolution.

Details. Fig. 10 illustrates the concept of the STARnet. The model takes two LR frames $I^{lr}_{t}$ and $I^{lr}_{t+1}$, with bidirectional dense motion flows of these frames $F_{t\rightarrow t+1}$ and $F_{t+1\rightarrow t}$ as input. The output consists of an in-between LR $I^{lr}_{t+n}$ and three SR frames $I^{sr}_{t}$, $I^{sr}_{t+n}$, and $I^{sr}_{t+1}$, where $n\in[0,1]$ denotes the temporal interpolation rate. While the STARnet model can be fine-tuned only for temporal super-resolution as proposed in [11], the TTI team has employed the original spatio-temporal framework. This is because large motions observed in the REDS [25] dataset make it difficult to estimate accurate optical flows, which take an essential role for the VTSR task. Therefore, the optical flows on the LR domain might support VTSR of HR frames in the REDS sequences. In the proposed strategy, two given frames are resized to LR frames $I^{lr}_{t}$ and $I^{lr}_{t+1}$, and then fed into the network shown in Fig. 10 in order to acquire $I^{sr}_{t+n}$ in the original resolution. For $\times 4$ temporal super-resolution, $n$ is set to $[0.25,0.5,0.75]$. To realize those three temporal interpolation rates with one network, input flow maps $F_{t\rightarrow t+1}$ and $F_{t+1\rightarrow t}$ are scaled according to $n$ as follows:

where $\hat{F}_{t\rightarrow t+1}$ and $\hat{F}_{t+1\rightarrow t}$ denote the re-scaled flow maps that are used as input, respectively. For higher accuracy, two changes are made to the original STARnet model. First, a deformable convolution [7] is added at the end of T- and ST-SR networks to explicitly deal with object motions. Second, the PWC-Net [36] is used to get flow maps, which show better performance than the baseline.