MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution

SRCNN [4] is the first method that employs a deep convolutional neural network in the image super-resolution task. It has inspired several following methods [5, 17, 34, 18, 21, 36, 10, 48, 49]. For example, Kim et al. [17] proposed a residual learning strategy using a 20-layer depth network, which yielded significant accuracy improvement. Instead of applying commonly used bicubic interpolation, Shi et al. [34] designed a sub-pixel convolution network to effectively upsample low-resolution input. This operation reduces computational complexity and enables a real-time network. Taking advantage of high-quality large image datasets, more networks, such as DBPN [10], RCAN [48], and RDN [49], further improve the performance of SISR.

Video super-resolution takes multiple frames into consideration. Based on the way to aggregate temporal information, previous methods can be roughly grouped into three categories.

The first group of methods process video sequences without any explicit alignment. For example, methods of [19, 22] utilize 3D convolutions to directly extract features from multiple frames. Although this approach is simple, the computational cost is typically high. Jo et al. [15] proposed dynamic upsampling filters to avoid explicit motion compensation. However, it stands the chance of ignoring informative details of neighboring frames. Noise in the misaligned regions can also be harmful.

The second line [23, 26, 16, 2, 25, 37, 32, 11] is to use optical flow to compensate motion between frames. Methods of [23, 16] first obtain optical flow using classical algorithms and then build a network for high-resolution image reconstruction. Caballero et al. [2] integrated these two steps into a single framework and trained it in an end-to-end way. Tao et al. [37] further proposed sub-pixel motion compensation to reveal more details. No matter if optical flow is predicted independently, this category of methods needs to handle two relatively separated tasks. Besides, the estimated optical flow critically affects the quality of reconstruction. Because optical flow itself is a challenging task especially for large-motion scenes, the resulting accuracy cannot be guaranteed.

The last line [38, 41] conducts deformable convolution networks [3] to accomplish video super-resolution. For example, EDVR proposed in [41] extracts and aligns features at multiple levels, and achieves reasonable performance. The deformable network is however sensitive to the input patterns, and may give rise to noticeable reconstruction artifacts due to unreasonable offsets.

Camera or object motion between neighboring frames has its pros and cons. On the one hand, large motion needs to be eliminated to build correspondence among similar content. On the other hand, accuracy of small motion (at sub-pixel level) is very important, which is the source to draw details. Inspired by the work of [30, 35], we design a hierarchical correspondence aggregation strategy to handle large and subtle motion simultaneously.

As shown in Figure 3, given two neighboring LR images ${\textbf{I}}^{L}_{t-1}$ and ${\textbf{I}}^{L}_{t}$, we first encode them into lower resolutions (level $l=0$ to $l=2$). Then, the aggregation starts in the high-level (low-resolution) stage (i.e., from $\overline{{\textbf{F}}}^{l=2}_{t-1}$) compensating large motion, while progressively moving up to low-level/high-resolution stages (i.e., to $\overline{{\textbf{F}}}^{l=0}_{t-1}$) for subtle sub-pixel shift. Different from many methods [2, 37, 32] that directly regress flow fields in image space, our module functions in the feature space. It is more stable and robust to noise [35].

The aggregation unit in Figure 3 is detailed in Figure 4. A patch-based matching strategy is used since it naturally contains structural information. As aforementioned in Figure 1(b), one-to-one mapping may not be able to capture true correspondence between frames. We thus aggregate multiple candidates to obtain sufficient context information as in Figure 4.

In details, we first locally select top-$K$ most similar feature patches, and then utilize a pixel-adaptive aggregation scheme to fuse them into one pixel to avoid boundary problems. Taking aligning ${\textbf{F}}^{l}_{t-1}$ to ${\textbf{F}}^{l}_{t}$ as an example, given an image patch ${\textbf{f}}^{\,l}_{t}$ (represented as a feature vector) in ${\textbf{F}}^{l}_{t}$, we first find its nearest neighbors on ${\textbf{F}}^{l}_{t-1}$. For efficiency, we define a local search area satisfying $\left|{\textbf{p}}_{t}-{\textbf{p}}_{t-1}\right|\leq{\textbf{d}}$, where ${\textbf{p}}_{t}$ is the position vector of ${\textbf{f}}^{\,l}_{t}$. d means the maximum displacement. We use correlation as a distance measure as in Flownet [6]. For ${\textbf{f}}^{\,l}_{t-1}$ and ${\textbf{f}}^{\,l}_{t}$, their correlation is computed as the normalized inner product of

After calculating correlations, we select top-$K$ most correlated patches (i.e., ${\bar{\textbf{f}}}^{\,l}_{t-1,1},{\bar{\textbf{f}}}^{\,l}_{t-1,2},\dots,{\bar{\textbf{f}}}^{\,l}_{t-1,K}$) in a descending order from ${\textbf{F}}^{l}_{t-1}$, and concatenate and aggregate them as

where $Aggr$ is implemented as convolution layers. Instead of assigning equal weights (e.g., $\frac{1}{9}$ when the patch size is 3), we design a pixel-adaptive aggregation strategy to enable varying aggregation patterns in different locations. The weight map is obtained by concatenating ${\textbf{F}}^{l}_{t-1}$ and ${\textbf{F}}^{l}_{t}$ and going through a convolution layer, which has a size of $H\times W\times s^{2}$ when the patch size is $s\times s$. The adaptive weight map takes the form of

As shown in Figure 4, the final value at position ${\textbf{p}}_{t}$ on the aligned neighboring frame $\bar{\textbf{F}}^{l}_{t-1}$ is obtained as

After repeating the above steps for $2N$ times, we obtain a set of aligned neighboring feature maps $\{{\bar{\textbf{F}}}^{0}_{t-N},\dots,{\bar{\textbf{F}}}^{0}_{t-1},{\bar{\textbf{F}}}^{0}_{t+1},\dots,{\bar{\textbf{F}}}^{0}_{t+N}\}$. To handle all frames at the same feature level, as shown in Figure 2, we employ an additional TM-CAM, which performs self-aggregation with ${\textbf{I}}^{L}_{t}$ as the input and produces ${\bar{\textbf{F}}}^{0}_{t}$. Finally, all these feature maps are fused into a double-spatial-sized feature map by a convolution and depth-to-space operation (PixelShuffle), which is to keep sub-pixel details.

Similar patterns exist widely in natural images that provide abundant texture information. Self-similarity [20, 29, 9, 44, 47] can help detail recovery. In this part, we design a cross-scale aggregation strategy to capture nonlocal correspondences across different feature resolutions, as illustrated in Figure 5.

To distinguish from Sec. 3.1, we use $\textbf{M}^{s}_{t}$ to denote feature maps at time $t$ with scale level $s$. We first downsample the input feature maps $\textbf{M}^{0}_{t}$ and obtain a feature pyramid as

where $AvgPool$ is the average pooling with stride 2. Given a query patch ${\textbf{m}}^{0}_{t}$ in $\textbf{M}^{0}_{t}$ centered at position ${\textbf{p}}_{t}$, we implement a non-local search on other three scales to obtain

where ${\tilde{\textbf{m}}}^{s}_{t}$ denotes the nearest neighbor (the most correlated patch) of ${\textbf{m}}^{0}_{t}$ in ${\textbf{M}}^{s}_{t}$. Before merging, a self-attention module [41] is applied to determine whether the information is useful or not. Finally, the aggregated feature $\bar{\textbf{m}}^{0}_{t}$ at position ${\textbf{p}}_{t}$ is calculated as

where $Att$ is the attention unit and $Aggr$ is implemented as convolution layers. Our results presented in Sec. 4.3.2 demonstrate that our designed module reveals more details.

Usually, reconstructed high-resolution images produced by VSR methods suffer from jagged edges. To alleviate this problem, we propose an edge-aware loss to produce better refined edges. First, an edge detector is used to extract edge information of ground-truth HR images. Then, the detected edge areas are weighted more in loss calculation, enforcing the network to pay more attention to these areas during learning.

In this paper, we choose the Laplacian filter as the edge detector. Given the ground-truth $\textbf{I}^{H}_{t}$, the edge map $\textbf{I}^{E}_{t}$ is obtained from the detector and the binary mask value at ${\textbf{p}}_{t}$ is represented as

where $\delta$ is a predefined threshold. Suppose the size of a high-resolution image is $H\times W$. The edge mask is also a $H\times W$ map filled with binary values. The areas marked as edges are 1 while others being 0.

During training, we adopt the Charbonnier Loss, which is defined as

where $\hat{\textbf{I}}^{H}_{t}$ is the predicted high-resolution result, and $\epsilon$ is a small constant. The final loss is formulated as

where $\lambda$ is a coefficient to balance the two terms and $\circ$ is element-wise multiplication.

REDS [27] is a realistic and dynamic scene dataset published in NTIRE 2019 challenge. There are a total of 30K images extracted from 300 video sequences. The training, validation and test subsets contain 240, 30 and 30 sequences, respectively. Each sequence has equally 100 images with resolution 720 $\times$ 1280. Similar to that of [41], we merge the training and validation parts and divide the data into new training (with 266 sequences) and testing (with 4 sequences) datasets. The new testing part contains the $000$, $011$, $015$ and $020$ sequences.

Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) [42] are used as metrics in our experiments.

Network Settings  The network takes 5 (or 7) consecutive frames as input. In feature extraction and reconstruction modules, 5 and 40 (20 for 7 frames) residual blocks [12] are implemented respectively with channel size 128. In Figure 3, the patch size is 3 and the maximum displacements are set to $\{3,5,7\}$ from low to high resolutions. The $K$ value is set to 4. In the cross-scale aggregation module, we define patch size as 1 and fuse information from 4 scales as shown in Figure 5. After reconstruction, both the height and width of images are quadrupled.

To demonstrate the effectiveness of our proposed method, we conduct experiments for each individual design. For convenience, we adopt a lightweight setting in this section. The channel size of network is set to 64 and the reconstruction module contains 10 residual blocks. Meanwhile, the amount of training iterations is reduced to 200K.

We compare our proposed multi-correspondence aggregation network (MuCAN) with previous state-of-the-arts including TOFlow [43], DUF [15], RBPN [10], and EDVR [41] on REDS [27], Vimeo-90K [43] and Vid4 [24] datasets. The quantitative results in Tables 3 and 4 are extracted from the original publications. Especially, original EDVR [41] is initialized with a well-trained model. For fairness, we use author-released code to train EDVR without pretraining.

On the REDS dataset, results are shown in Table 3. It is clear that our method outperforms all other methods by at least 0.17dB. For Vimeo-90K, the results are reported in Table 4. Our MuCAN method works better than DUF [15] with nearly 1.2dB enhancement on RGB channels. Meanwhile, it brings 0.25dB improvement on the Y channel compared with RBPN [10]. All these results verify the effectiveness of our method. Besides, performance on Vid4 is reported in the supplementary file. Several examples are visualized in Figure 9.

To evaluate the generality of our method, we apply our model trained on the REDS dataset to test video frames in the wild. In addition, we test EDVR with the author-released model¹¹1https://github.com/xinntao/EDVR on the REDS dataset. Some visual results are shown in Figure 10. We remark that EDVR may generate visual artifacts in some cases due to the variance of data distributions between training and testing. In contrast, our MuCAN performs well in the real world setting and shows its decent generality.