Group-based Bi-Directional Recurrent Wavelet Neural Networks
for Video Super-Resolution

As an earliest work of deep learning-based SISR, Dong et al. [4] proposed a super-resolution convolutional neural network (SRCNN). The SRCNN is a relatively shallow network. Kim et al. [14] later developed a deeper network with residual learning called very deep super-resolution (VDSR). After that, an efficient sub-pixel convolutional neural network (ESPCN) proposed by Shi et al. [27] to reduce computational complexity with keeping a deeper network. Also, Ledig et al. [18] proposed a super-resolution using a generative adversarial network (SRGAN), which was a model focusing to high frequency details. Based on [18], Lim et al. [21] proposed an enhanced deep super-resolution network (EDSR), which modified the residual module by removing batch normalization. Recently, much deeper CNN, including residual dense network (RDN) [34], DBPN [9], RCAN [33] were then introduced. They outperformed previous networks by a substantial margin.

In this section, we introduce the overall system of our group-based bi-directional recurrent wavelet neural networks (GBR-WNN). The proposed GBR-WNN consists of two core components: a temporal modeling framework, namely, group-based bi-directional RNN (GBR) and a feature extraction module which is called temporal wavelet attention (TWA).

The proposed network is illustrated in Figure 1. Given ${2N+1}$ consecutive LR input frames ${LR_{input}=\{LR_{t-N},...,LR_{t},...,LR_{t+N}\}}$, we denote the central LR frame ${LR_{t}}$ as the target frame and the other frames as neighboring frames with size of ${H\times W}$, where ${H}$ is height and ${W}$ is width. The goal of VSR is to estimate a HR target frame ${SR_{t}}$, which is close to the ground truth frame ${HR_{t}}$ with size of ${\small{s}H\times\small{s}W}$, where ${s}$ is scaling factor.

In the GBR framework, the availability of temporal information is checked by exploiting the previous and future hidden states within the scope of a group of pictures (GOP). The input LR frames are fed to the TWA module to extract elaborate spatio-temporal features based on 2D discrete wavelet transform (DWT) [24]. The output features after the TWA module pass through a reconstruction and upsampling module. Our reconstruction module is designed with several 2D CNN layers and 2D residual blocks wherein no batch normalization units in [21]. The predicted HR residual frame is obtained by adopting the depth-to-space transformation in [27]. Finally, the HR estimated frame ${SR_{t}}$ is obtained by adding the predicted residual frame to a direct upsampled LR frame ${LR_{t}}$.

In case of a sequence containing large motion, the far away neighboring frames are likely to have the missing details of the target frame. Therefore, if we utilize a wide range of motion, we are able to have good chance to improve the performance in VSR. In order to manage the hidden states of neighboring frames as well as the LR inputs of those dynamically, we propose the GBR temporal modeling framework.

The structure of the GPR framework is shown in Figure 2. We define the ${(2N+1)}$ LR input frames as the group of pictures (GOP). In a GOP, each of the LR frames ${\{LR_{t-N},...,LR_{t},...,LR_{t+N}\}}$ becomes an current target LR frame ${LR_{curr}}$ in SR order. For effective structure to estimate the central HR target frame ${SR_{t}}$, the SR order is gradually directed toward the center. As the previous and future hidden states, and SR frames, we apply the features of the frames already predicted at the current point. The time steps of the previous and future neighboring features are illustrated in Figure 2, where ${L0}$, ${L1}$, and ${N}$ are previous picture list, future picture list, and none, respectively. At the time step ${t-3}$ for the first target frame, the previous LR frame ${LR_{l0}}$ and the previous and future estimations ${H_{l0}}$, ${H_{l1}}$, ${SR_{l0}}$, and ${SR_{l1}}$ are initialized with zero. At the time step ${t+3}$ for the last target frame, the future LR frame ${LR_{l1}}$ and the future estimations ${H_{l1}}$ and ${SR_{l1}}$ are initialized with zero.

At every time step, the GBR-WNN uses following equations to estimate output SR frame at the current point. The TWA module takes three LR frames ${LR_{l0}}$, ${LR_{curr}}$, and ${LR_{l1}}$ as input to produce the temporal wavelet feature ${TWF_{curr}}$:

where ${f_{twa}(\cdot)}$ and ${[\cdot,\cdot,\cdot]}$ denote the mapping function for TWA module and concatenation, respectively. We adopt the space-to-depth [27] transformation to match the shape of the previous and future SR frames ${SR_{l0}}$ and ${SR_{l1}}$ to LR frames with scaling factor ${s}$:

where ${f_{s\rightarrow d}(\cdot)}$, ${SR_{l0}^{s\rightarrow d}}$, and ${SR_{l1}^{s\rightarrow d}}$ denote the space-to-depth mapping function, the reshaped previous and future SR frames, respectively. Then, to further enhance the feature, concatenated feature with temporal wavelet feature, previous and future hidden states and SR frames are fed into reconstruction module. This process can be represented as

where ${f_{rec}(\cdot)}$ denotes the mapping function of reconstruction module. ${H_{curr}}$ represents the extracted current hidden state. For the upsampling process, the depth-to-space mapping function ${f_{d\rightarrow s}(\cdot)}$ is applied to hidden state ${H_{curr}}$ with scaling factor ${s}$. Lastly, output SR frame ${SR_{curr}}$ of each time step is obtained by global residual learning using the estimated HR residual frame and the upsampled LR target frame:

where ${f_{up}^{\times s}(\cdot)}$ denotes the bilinear upsampling function with scaling factor ${s}$. When the frame order of the current target frame is smaller than the frame order of next target frame, the current hidden state ${H_{curr}}$ becomes the previous hidden state ${H_{l0}}$ and the current SR frame ${SR_{curr}}$ becomes the previous SR frame ${SR_{l0}}$. In contrast, if the frame order of the current target frame is bigger than the frame order of next target frame, the current hidden state ${H_{curr}}$ becomes the future hidden state ${H_{l1}}$ and the current SR frame ${SR_{curr}}$ becomes the future SR frame ${SR_{l1}}$.

Extracting fine feature of both intra-frame spatial and inter-frame temporal information is very important to improve the quality of output SR frame. To obtain the spatio-temporal feature, we design the TWA module based on the 2D Haar DWT and attention mechanism, as illustrated in Figure 3. The key role of the TWA is generating the temporal attention map and wavelet attention map. By utilizing temporal attention map, we can restore the missing feature from the neighboring frames with different degrees of motion information. Meanwhile, we can strengthen the edge and texture details by applying the spatial attention map.

Each of the three LR frames are fed into the CNN embedding layer to increase the number of features:

where ${f_{emb}(\cdot)}$ denotes the embedding function. The embedded LR features of three LR frames are ${E_{l0}}$, ${E_{curr}}$, and ${E_{l1}}$, respectively. The temporal attention features ${TAtt_{l0}}$, ${TAtt_{curr}}$, and ${TAtt_{l1}}$ for three LR frames can be calculated as

where ${\boldsymbol{\cdot}}$ and ${sigmoid(\cdot)}$ denote the dot product and the sigmoid activation function. After that, the temporal attention maps are then multiplied to the original LR frames:

where ${\odot}$ denotes the element-wise multiplication. The weighted temporal embedded features ${\tilde{E}_{l0}}$, ${\tilde{E}_{curr}}$, and ${\tilde{E}_{l1}}$ are then concatenated. Then, the current temporal feature ${TF_{curr}}$ can be represented as

where ${f_{fusion}(\cdot)}$ denotes the fusion process function based on several CNN layers.

For extracting the spatial feature, we use the 2D Haar DWT. An example of the 2D Haar DWT is shown in Figure 4. An image can be decomposed into four sub-band images using four 2D DWT filters. The low-pass filter ${f_{LL}}$ means approximation of image and high-pass filters ${f_{HL},\;f_{LH},\;and\;f_{HH}}$ mean vertical, horizontal, and diagonal edge of image, respectively. The DWT filters used in this paper are defined as

The estimated temporal feature ${TF_{curr}}$ can be decomposed into four sub-band components by using 2D DWT. The wavelet attention map ${WAtt_{curr}}$ can be generated by the upsampling function and sigmoid activation function. This process can be expressed as

Because the wavelet attention map ${WAtt_{curr}}$ is composed of four components, the size of map is ${H\times W\times nF\times 4}$, where ${nF}$ means number of features. Finally, by multiplying temporal feature ${TF_{curr}}$ and each DWT component of wavelet attention map ${[WAtt_{curr}^{LL},WAtt_{curr}^{HL},WAtt_{curr}^{LH},WAtt_{curr}^{HH}]}$, the temporal wavelet feature ${TWF_{curr}}$ can be achieved as

In this paper, we used the Vimeo-90K [32] dataset which is a large and diverse data set with high-quality frames and a range of motion types for training. The resolution of each sample in the Vimeo-90K dataset is ${448\times 256}$. For training, we used 3 channel patches of size ${64\times 64}$ as inputs. We augment the training data with random horizontal flips and ${90^{\circ}}$ rotations. We evaluated our methods on the Vid4 [23] dataset. The Vid4 dataset consists of the four test sequences, walk, foliage, city, and calendar, commonly reported in recent methods.

In all our experiments, the scaling factor ${s}$ of SR was set to 4. For evaluation, we use peak signal-to-noise ratio (PSNR) to test the quality of each frame. The overall PSNR values of each video clip were then calculated by aggregating PSNRs over all frames in a video clip. Finally, the overall PSNR values of whole Vid4 dataset were then calculated by averaging PSNRs of video clips.

For our GBR-WNN, the network takes seven consecutive frames ${(i.e.,\;N=3)}$ as inputs. Also, the number of feature in each residual block was set to 128. To train our network, we used Charbonnier penalty function [17] for loss function:

where ${\varepsilon}$ set to ${1\times 10^{-3}}$. We used Adam optimizer [15] and initially set learning rate to ${4\times 10^{-4}}$. The number of iterations was set to 600K.

The proposed GBR-WNN was implemented in PyTorch on a PC with 8 NVIDIA Tesla V100 16GB GPUs. We trained with setting the size of global mini-batch to 128, which means that the size of mini-batch for each GPU was set to 16.

We compare the proposed GBR-WNN with Bicubic and several state-of-the-art methods including all kinds of temporal modeling frameworks, (1) 2D CNN: the optical flows for VSR (SOF-VSR) [30] and the wavelet attention embedding network (WAEN) [2], (2) 3D CNN: the wide-activated 3D convolutional network for video restoration (WDVR) [5], and (3) RNN: the recurrent back-projection network (RBPN) [10] and the frame-recurrent VSR (FRVSR) [26] on Vid4 dataset.

For SOF-VSR [30] and RBPN [10], we used a provided pre-trained model to produce their results. For WAEN [2] and WDVR [5], we trained the model using Vimeo-90K dataset. We produced the result of FRVSR [26] using the super-resolved Vid4 result images provided by the authors because the source code and the pre-trained model were not opened. In this paper, the first two frames and the last two frames are not used for overall performance evaluation because the FRVSR [26] provides excluding these four frames.

The quantitative results in term of PSNR on Y (luminance) and RGB channels are shown in Table 1 and Table 2, respectively. We also show the number of parameters for each model. In our experiments, we evaluated our GBR-WNN with different number of residual blocks (RBs) in the reconstruction module. The large size model consisting of 30 RBs (GBR-WNN-L), medium size model with 20 RBs (GBR-WNN-M) and small size model with 10 RBs (GBR-WNN-S) were tested. The results show the tendency that the model with increased RBs performs better.

Comparing with other methods, our GBR-WNN-L shows the best performance and GBR-WNN-M is in the second place on average in both Y and RGB channels. Even though RBPN [10] needs 0.9M additional parameters, our GBR-WNN-L outperforms RBPN [10] in term of PSNR. Furthermore, when comparing our small size model GBR-WNN-S with RBPN, our method has better average performances.

Although FRVSR [26] has better performances on City and Foliage clips in both Y and RGB channels, the gap between our GBR-WNN-L and FRVSR [26] is small in this clips. On the contrary, the proposed GBR-WNN-L outperforms FRVSR by a large margin on Calendar and Walk clips. Comparing with WAEN using DWT-based feature extractor, our methods achieves better performance. It means that the proposed GBR framework is more effective in estimating the sophisticated temporal feature than general 2D CNN temporal modeling framework.

The qualitative results on Vid4 dataset for Calendar and City clips are presented in Figure 5. For result of each method, the upper row means zoomed a visual area and the lower row means temporal profile. For the temporal profiles, we used 144th line for Calendar clip and 550th line for City clip in the entire frame of the sequences. Our GBR-WNN-L recovers more accurate textures with more smooth temporal transition compared to the existing methods.

For a more detailed analysis of the proposed GBR-WNN, Table 3 shows the results as adopted our GBR framework and TWA module in our method. From this result, we can explain that combination of two proposed core components produces better performance than using a single component. By comparing Test Model2 to Test Model1, we can verify that the TWA module exploits useful information in VSR problem. When comparing GBR-WNN-M with Test Model2 and Test Model3, we can see that the GBR framework is very efficient and beneficial in dealing with the sequential data.