3DAttGAN: A 3D Attention-based Generative Adversarial Network for Joint Space-Time Video Super-Resolution

Video SSR improves the quality of low-resolution videos by upscaling them to higher resolutions. The methods based on deep learning use external references to construct models while simultaneously approximating complex nonlinear functions. SRCNN [8], one of the pioneering convolutional neural networks (CNN), is a simple three-layer end-to-end model. Video SSR methods take advantage of the temporal correlations by merging spatial information over adjacent frames and estimate the inter-frame motion to obtain references. Unlike single image super-resolution, VSR can exploit motion information between consecutive frames to recover lost high-frequency details in the current frame.

Tian et al. [9] proposed a temporal deformable alignment network (TDAN) that can align the reference image and supporting frames without requiring optical flow estimation. Wang et al. [10] proposed a video super-resolution network, EDVR, that uses enhanced deformable convolutions. EDVR comprises a pyramid, cascading, and deformable (PCD) alignment module to effectively address the motions, as well as a temporal and spatial attention (TSA) fusion module that emphasizes crucial features for subsequent restoration. Chan et al. [11] proposed a concise pipeline for VSR called BasicVSR, which leverages bidirectional propagation to enhance information gathering, and uses a method based on optical flow to estimate the correspondence between adjacent frames for accurate feature alignment. Building on SRGAN [12], which uses a modified version of the VGG [13] network as the discriminator, ESRGAN [14] incorporates a residual-in-residual dense block and uses a relativistic average GAN (RaGAN) [15] to evaluate the authenticity of images. VSRResNet [16] relies on a discriminator architecture during GAN training and uses two regularizers: a feature-space loss and a pixel-space loss. For video super-resolution, temporally coherent GAN (TecoGAN) [17] was the first method to introduce a spatiotemporal discriminator that takes into account temporal and spatial coherence. TecoGAN is based on frame recurrent video super resolution (FRVSR) [18], which uses the HR estimate from the preceding frame for super-resolving the next frame. Wen et al. [19] proposed an end-to-end deep convolutional network for video super-resolution that dynamically generates spatially adaptive filters to improve temporal alignment. Yi et al. [20] introduced a progressive fusion network for video super-resolution that effectively fuses temporal information, incorporates multi-scale structures, hybrid convolutions, and non-local operations, improving both performance and complexity. Li et al. [21] presented a novel approach for efficient video resolution upscaling by leveraging spatial-temporal information to divide videos into chunks, reducing model size and storage requirements while achieving real-time video super-resolution with high quality. Qiu et al. [22] introduced Frequency-Transformer for Video Super-Resolution (FTVSR++). The method operates in a combined space-time-frequency domain, distinguishes real visual texture from artifacts, and incorporates dual frequency attention. Meng et al. [23] introduced an approach to enhance video super-resolution by leveraging similar patches across distant frames through long-term cross-scale aggregation. The method is adaptable as post-processing for any super-resolution technique.

Video TSR mitigates motion artifacts and reduces judder by interpolating frames between existing ones, proving valuable for slow-motion effects creation and enhancing overall video playback quality. Video TSR is similar to video frame interpolation (VFI). The objective is to predict one or multiple frames between two input frames. While traditional methods typically use dense optical flow interpolation, recent methods often use DNNs for middle frame prediction.

Liu et al.. [24] proposed a deep voxel flow (DVF) network, which combines the advantages of classic flow-based methods with those of learning-based methods. Bao et al. [25] introduced a network that uses an adaptive warping layer to process video data through a combination of optical flow-based warping and learned interpolation kernel-based sampling of input frames. Xue et al. [26] presented ToFlow, a tailored flow representation for video processing tasks, achieved through a neural network incorporating motion estimation and video processing components. Bao et al. [27] introduced a VFI approach that generates more realistic intermediate flows by prioritizing closer objects over distant ones using a depth-aware flow projection layer. The cycle consistency network proposed by Liu et al. [28] measures the quality of synthesized frames by evaluating their ability to reconstruct the input frames accurately, thus increasing their reliability. Park et al. [29] proposed a VFI method that identifies and exploits the position and strength of complicated motion to generate intermediate frames. Lee et al. [30] proposed a method called “adaptive collaboration of flows (AdaCoF)” that can generate a target pixel by referring to a variable number of pixels from any position. Kong et al. [31] introduced a Progressive Motion Context Refine Network for efficient frame interpolation. The network predicts motion fields and image context jointly, simplifying the task by reusing existing textures from adjacent input frames. It achieves favorable results with a reduced model size and running time. Liu et al. [32] proposed a novel trajectory-aware transformer for video frame interpolation. The method formulates the warped features with inconsistent motions as query tokens and uses relevant regions in a motion trajectory from two original consecutive frames as keys and values. Park et al. [33] introduced BiFormer, a 4K video frame interpolator, which uses a bilateral transformer to predict global motion fields, refines them using blockwise bilateral cost volumes, and synthesizes intermediate frames. Plack et al. [34] presented a novel transformer-based video frame interpolation network that estimates both the interpolated frame and its expected error. The method improves visual quality, provides error maps for identifying problematic frames, and supports partial rendering passes to enhance frame quality while reducing processing time. Zhu et al. [35] presented MFNet, a frame interpolation network that focuses on motion regions and uses methods such as adaptive motion region separation, fine-grained approximation of intermediate streams, and lightweight bi-directional optical stream fusion.

Video STSR enhances both spatial and temporal resolutions. Video STSR is a very challenging inverse problem as LR frames lack detailed texture and motion information. The common approach to STSR for videos is to alternate between SSR and TSR. However, this approach leads to excessive complexity. A better approach is to generate videos that have both HR and high-frame-rate simultaneously. However, existing methods do not fully exploit the temporal and spatial characteristics across video frames.

Shechtman et al. [36] proposed a method that creates an STSR video of a scene by combining information from a number of LR videos of the same scene. The method is based on modeling each LR video as a blurred version of the HR video in space and time. Mudenagudi et al. [37] formulated video STSR as a reconstruction problem in a Markov random field-maximum a posteriori framework and used graph-cut optimization to solve the problem. Takeda et al. [38] exploited MASK regression, a method that uses local spatial orientations and motion vectors to construct adaptive filters at every position of interest. Shahar et al. [39] combined information from multiple space-time patches to super resolve input videos. Haris et al. [40] developed a CNN that super-resolves video jointly in space and time, incorporating direct lateral connections between multiple resolutions to present multi-scale features during training. Kang et al. [41] proposed a weighting scheme for efficient video processing, where all input frames are fused without the need for explicit motion compensation. Dutta et al. [42] used quadratic modeling for LR interpolation and reused flowmaps and blending masks for both LR and HR synthesis. Xiang et al. [43] proposed a deformable feature interpolation network to capture local temporal characteristics when interpolating LR frame features. They also introduced a deformable ConvLSTM to align and gather time information, leading to improved utilization of global temporal contexts. Xu et al. [44] introduced a method that uses a temporal modulation block (TMB) to control feature interpolation and a locally-temporal feature comparison (LFC) module to capture time-based information in video processing. Zhang et al. [45] proposed cross-frame transformers instead of traditional convolutions divided the input feature sequence into query, key, and value matrices,along with a multi-level residual reconstruction module. This approach allows for the use of the maximum similarity and similarity coefficient matrices obtained though the cross-frame transformer.

Attention has a significant impact on human perception [46, 47]. For example, humans can selectively focus on important visual features using partial glimpses for more efficient processing of visual structures [48]. The attention mechanism in a neural network is similar in essence to human visual attention, as it aims to select the most critical information for a given task from a vast amount of data.

The convolutional block attention module (CBAM) [49] is a lightweight and effective module that can be directly applied to feed-forward convolutional neural networks. The module comprises two components: channel attention and spatial attention. CBAM generates attention maps on channel and spatial dimensions from the input feature map. It then uses these maps to adaptively refine the features via element-wise multiplication.

To fully exploit temporal and spatial information, we propose a 3D channel-spatial attention mechanism (3D CSA), as shown in Fig. 3. Traditional attention mechanisms are designed for 2D convolutional neural networks and cannot be directly applied to 3D convolutions due to inconsistent dimensions. 3D convolutional networks have a depth dimension, and changes in depth should be considered when extracting spatial and depth features.

For the features of an intermediate 3D convolution layer $\textit{{F}}_{3D}\in\mathbf{R}^{W\times H\times D\times C}$, 3D CSA deduces the channel attention feature map $\textit{{M}}_{c_{3\mathrm{D}}}\in\mathbf{R}^{1\times 1\times 1\times C}$ and the spatial attention feature map $\textit{{M}}_{s_{3\mathrm{D}}}\in\mathbf{R}^{1\times H\times W\times C}$ in sequence.

The 3D CSA channel attention module identifies the channels that play a significant role in the final results in the fused 3D network, i.e., it selects the key features for prediction. The process is illustrated in Fig. 4.

First, the input features $\textit{{F}}_{3D}$ are separately processed by maximum pooling and average pooling based on width $W$, height $H$ and depth $D$. The 3D convolutional features are computed by element-wise addition. Next, a sigmoid activation is applied. By multiplying the generated channel feature map $\textit{{M}}_{c_{3\mathrm{D}}}$ with the input feature $\textit{{F}}_{3D}$, the final channel feature $\textit{{F}}_{3\mathrm{D}}^{\prime}$ is generated as

where $G(\cdot)$ and $P(\cdot)$ are 3D mean pooling and 3D maximum pooling operations, respectively, and $\sigma$ is a sigmoid operation.

The 3D spatial attention model of 3D CSA focuses on the pixels in the image that have a significant impact in prediction. The attention feature extraction process is shown in Fig. 5. The spatial attention module takes the feature $\textit{{F}}_{3\mathrm{D}}^{\prime}$ of the channel attention module as input and applies channel-based 3D maximum pooling and 3D average pooling operations. Next, the two extracted features are merged. Then, the dimension is reduced by the convolution operation, and the spatial attention feature map is generated through a sigmoid function. That is,

where $f$ denotes the convolution operation.

Finally, the output feature map and the input $\textit{{F}}_{3\mathrm{D}}^{\prime}$ are multiplied pointwise to generate the final feature $\textit{{F}}_{3\mathrm{D}}^{\prime\prime}$. The overall process can be expressed as

where $\otimes$ represents pointwise multiplication.

Generative adversarial networks (GANs) [50] are highly efficient models that can learn complex probability distributions from a given prior distribution. GANs were initially developed for the purpose of generating images [50]. Since then, GANs have been used for many generative tasks, including audio synthesis, 3D modeling, and image-to-image translation tasks. The exceptional generative capability of the GANs has been used to generate high-quality images for various reconstruction tasks, e.g., [51, 52, 53].

GANs and Variational Autoencoders (VAEs) are two commonly used generative models in image and video processing. GANs are better suited for tasks that demand extremely high-quality details, whereas VAEs are more suitable for tasks that emphasize stability and interpretability. GANs operate by training a generator and a discriminator network to produce high-quality images or videos. They excel in super-resolution tasks, yielding sharper images by capturing fine textures and structures. On the other hand, VAEs may produce blurry results in super-resolution tasks, as they tend to generate smoother images with less focus on details. VAEs typically impose higher quality requirements on input images or videos, potentially necessitating more training data and complex network architectures to achieve high-quality super-resolution results. Our objective is to obtain higher quality videos, including fine details and textures. Therefore, we have opted for GANs instead of VAEs.

Our goal is to improve the resolution of LR video frames. Fig. 6 shows the histogram distribution of various images. The distribution of the same image is roughly the same across different resolutions, while the distribution between different images varies widely. When we super-resolve a video, the SR video and the original video are in similar domains, which facilitates convergence of the neural network. In the proposed STSR task, GANs can automatically learn the data distribution through generative adversarial training, enabling the generation of content that closely matches the input data.

The proposed network was trained using the training-subset of Vimeo-90K dataset [56] which includes more than 60000 videos of size 7 (number of frames) $\times$ 448 (spatial width) $\times$ 256 (spatial height). The Vid4 dataset [57] and the Vimeo-90K [56] testing dataset were used for testing. For training, we first generated LR videos by spatial and temporal subsampling. Specifically, the spatial resolution was down-sampled to 112$\times$64 through bi-cubic interpolation, while the even frames were deleted. To evaluate the quality of the generated full-resolution videos, we used the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) [58]. As the proposed STSR method can generate high spatial-resolution (SSR) and high frame-rate (TSR) videos simultaneously, the SSR and TSR results are also presented.

The proposed model was developed on the PyTorch platform and trained with a 2080Ti GPU. In the implementation, we randomly cropped the videos for training to patches of size 32$\times$32$\times$4 and used ground truth video patches of size (32$\times$4)$\times$(32$\times$4)$\times$7 as the labels.

As benchmarks for STSR methods, we used the state-of-the-art (SOTA) STSR methods STARnet [40], STVUN [41], and TMNet [44], as well as a combination of SOTA SSR and TSR methods. If the source code of the SOTA method was available, we used it; otherwise, we implemented the method ourselves. In this way, all comparison methods were trained and tested in the same way as our method. For these combinations, we used DBPN [59], TDAN [9], BasicVSR [11], LatticeNet [61] and SwinIR [62] for SSR, and ToFlow [26], DAIN [27], CAIN [64] and EDSC [65] for TSR. Table I shows the results for Vimeo-90K. We can see that the combination of SSR+TSR performs better than TSR+SSR. We can also see that the proposed method had both the highest PSNR (31.68 dB on average) and SSIM (0.945).

To further verify the effectiveness of proposed method, we tested it on the REDS dataset [63]. The experimental results are shown in Table II. As can be seen from the table, the proposed method consistently maintained the best performance. To evaluate the performance under at motion conditions, we divided, as in [26], the Vimeo-90K dataset into three groups based on the level of motion: fast, medium, and slow. These groups comprise 1225, 4977, and 1613 video clips, respectively. The gains were largest in the fast motion group (0.94 dB on average), see Table III.

Our method allows us to achieve SSR in two ways. The first one (named Ours in Table IV) is to consider SSR as a byproduct of STSR. The second one (named Ours-SSR in Table IV) is to convert our STSR method into an SSR one by keeping the same temporal resolution and modifying only the spatial resolution in Convtranspose3d. Table IV compares these two approaches to five SOTA methods (DBPN [59], RBPN [60], TDAN [9], BasicVSR [11]), and SwinIR [62]. The results show that the first approach did not give competitive results. However, the second approach achieved better results than the current SOTA SSR methods.

Similarly, keeping the same spatial resolution and modifying only the temporal resolution in Convtranspose3d (named Ours-TSR in Table V) allowed us to outperform five TSR methods (DVF [24], ToFlow [26], MEMC Net [25], DAIN [27], and EDSC [65]), see Table V.

To synthesize high-quality video frames, SSR and TSR networks often require very complex reconstruction blocks. Therefore, the two-stage STSR networks have a huge number of parameters, leading to a high computational cost. Table VI compares the number of network parameters and processing time of various methods. Since our method consists of two networks (a generator and a discriminator), it includes more parameters than methods like STARnet that consist of a single network. However, because the generator is simpler than these networks, and our method only uses the generator in the testing phase, its running time is shorter.

Finally, Fig. 12 compares the visual results. The proposed method can provide better visual quality, especially for videos with rich textures and fast motion.