Reversing the Damage: A QP-Aware Transformer-Diffusion Approach for 8K Video Restoration under Codec Compression

8K video offers exceptional resolution, contrast, and motion quality, but it demands significant data and computational power for transmitting and coding [46]. With an estimated 15% of global electricity consumption attributed to information and communication technology (ICT) by 2040 [24], and video traffic accounting for 82% of global Internet traffic in 2022, efficient storage and transmission are increasingly crucial, particularly in bandwidth-limited scenarios prevalent in certain regions and demographics [17]. Video codecs offer a solution by compressing video data, but this often introduces visual artifacts like blockiness, blurring, or ringing, due to the lossy nature of compression algorithms [8]. A key parameter controlling this trade-off is the Quantization Parameter (QP), which determines the level of quantization applied to transform coefficients in the video data [42] (Fig. 1).

With the help of Video Restoration we can address this issue by recovering high-quality video sequences from their degraded compressed counterparts [30]. Video restoration, particularly for heavily compressed videos, is a highly challenging and ill-posed problem due to the inherent trade-off between compression and quality [50]. This process requires a range of techniques, including denoising [35] to remove unwanted artifacts, deblurring [34] to sharpen the frame, super-resolution [3, 19, 20] to enhance details, and crucially reducing compression artifacts – all aimed at recovering lost visual information. This task becomes even more complex when considering high-resolution videos like 8K, where the sheer volume of data exacerbates the challenges of artifact removal and quality restoration.

Our proposed model, DiQP (Fig. 2), not only introduces a novel approach to reducing video compression artifacts, but also, according to our research, is the first model specifically designed and trained for 8K videos. DiQP uniquely reverses the codec side effects by using Denoising Diffusion [15, 33]. While modern codecs such as AV1 and HEVC utilize adaptive QPs, we focus on fixed QPs to ensure robustness across varying compression levels. Unlike previous methods that add artificial noise [49], DiQP directly addresses complex compression artifacts by leveraging the inherent noise introduced during compression. DiQP features a U-shaped hierarchical network inspired by [41], with skip connections and enhanced windowed self-attention for capturing long-range dependencies in high-resolution videos while preserving local context. Look Ahead and Look Around modules further enhance temporal coherence and global awareness, while LOST embedding effectively incorporates conditional data. This combination of components allows DiQP to effectively reverse compression degradation, and significantly improve video quality restoration; particularly for 8K content.

Video Diffusion Models’ applications contain a wide scope of video analysis tasks [45], including video generation [14, 23, 32], and video editing [48, 26]. The methodologies for these tasks share similarities, often formulating the problems as Diffusion generation tasks or utilizing the potent controlled generation capabilities of Diffusion models for downstream tasks [45]. In video enhancement and restoration, CaDM [49] introduces a new approach to video streaming, reducing bitrates while improving video restoration quality compared to existing methods. This is achieved by reducing frame resolution and color depth during encoding, and then utilizing a Diffusion-based restoration process at the decoder that is aware of these encoding conditions. LDMVFI [6] marks a advancement in video frame interpolation by utilizing a conditional latent Diffusion model. It features an autoencoding network tailored for video frame interpolation, incorporating efficient self-attention mechanisms and deformable kernel-based synthesis for superior performance. VIDM [4] leverages a pre-trained Latent Diffusion Model (LDM) [28] to tackle video in-painting, demonstrating the adaptability of this tool. By providing a mask for first-person perspective videos, VIDM harnesses the image completion capabilities of LDM to generate seamless in-painted videos.

Video Transformers have found applications in various domains due to their ability to model long-range dependencies efficiently [9, 21, 11]. These applications showcase the versatility and effectiveness of Video Transformers in various video processing tasks [31]. In the restoration task, VRT [19], a new Video Restoration Transformer, allows for parallel processing of long video sequences and models long-range dependencies for video restoration. VRT jointly extracts, aligns, and fuses features at multiple scales using a novel mutual attention mechanism, achieving great performance in various video restoration tasks. RVRT [20], a recurrent video restoration transformer, combines the strengths of parallel and recurrent methods for efficient and effective video restoration. It processes video clips jointly, utilizes a larger hidden state to alleviate information loss, and introduces a novel guided deformable attention mechanism for accurate video clip alignment.

Video Restoration has gained significant attention in recent years. FTVSR [27] uses frequency-based patch representations and attention mechanisms to address the challenges of compressed video restoration. This approach preserves high-frequency details and leverages low-frequency information to guide high-frequency texture generation, effectively reducing compression artifacts. BasicVSR++ [3] improves video super-resolution using two main techniques: second-order grid propagation, which allows for more flexible information flow and aggregation across frames, and flow-guided deformable alignment, which utilizes optical flow to refine feature alignment across misaligned frames. These enhancements lead to better utilization of spatio-temporal information and improve overall performance. CAVSR [38] is designed to enhance video super-resolution specifically for compressed videos. It incorporates a compression encoder to assess compression levels in frames, using metadata such as frame type and motion vectors. This information is then used to modulate a base VSR model, enabling adaptive handling of various compression levels. The model further utilizes metadata like residual maps for accurate frame alignment, enhancing the bidirectional recurrent network’s performance. In addition to the aforementioned multi-frame-based models, VCISR [37] introduces an approach for the blind single image super-resolution (SISR) task that focuses on enhancing single-frame input affected by video compression artifacts, relying solely on spatial information.

Before delving into the specifics of our work, it is important to discuss the inspirations behind this idea. Deep generative models, such as Denoising Diffusion Probabilistic Models (DDPMs) [15], offer a compelling alternative to GANs [12, 7] without the need for adversarial training, careful optimization, or the risk of missing parts of the data distribution. DDPMs achieve this by training denoising models to progressively transform Gaussian noise into images through a Markov chain process, providing a stable and effective generative approach. While DDPMs produce high-quality images through a lengthy generative process, this method requires numerous iterations, making it significantly slower than GANs. For instance, generating 50,000 images of size 256 × 256 can take nearly 1,000 hours on a Nvidia 2080 Ti GPU, which becomes increasingly impractical for larger resolutions [33]. To address this efficiency gap, Denoising Diffusion Implicit Models (DDIMs) [33] were introduced as a more efficient alternative to DDPMs. DDIMs generalize the forward Diffusion process from the Markovian framework used in DDPMs to non-Markovian processes. This allows for the creation of ”short” generative Markov chains that can produce high-quality samples in far fewer steps. Cold Diffusion [2] further explores the boundaries of Diffusion models by eliminating the reliance on Gaussian noise or randomness altogether. Instead of using noise, it leverages arbitrary image transformations and degradations, training a restoration network to reverse these transformations. This approach challenges the traditional theoretical frameworks of Diffusion models and opens the door to new types of generative models with distinct properties compared to conventional methods. Moreover, the use of Gaussian noise schedules not only prevents Stable Diffusion models [28] from generating images with mean brightness greater or less than 0 (on a scale of -1 to 1), but also proves to be an overextension of model’s capacity. This is particularly true for restoration tasks, where the model must remove both artificially added Gaussian noise and existing artifacts [39]. Since compressed frames are not a natural intermediate step in the vanilla Diffusion process, the restoration process does not need to start from pure noise, nor does it require a large number of inference steps or a large model size—advantages that are critical for real-world applications. With all this in mind, we present DiQP as a proof of concept for using Denoising Diffusion to directly address the complex artifacts introduced by video compression in 8K resolution without adding artificial Gaussian noise [49].

Big Picture: DiQP consists of 4 different parts which are explained in depth in this section. But first let’s revisit the problem. Let $F_{\text{raw}}\in\mathbb{R}^{T\times H\times W\times C}$, be a sequence of raw target frames without added artifacts and distortions. $T,H,W,C$ are the frame number, height, width and channel number, respectively. Now if we consider $CODEC_{\text{Decode}}(CODEC_{\text{Encode}}(F_{\text{raw}},QP))=F_{\text{qp}}=F_{\text{raw}}+\text{noise}_{\text{qp}}$, our model aims to predict the $\text{noise}_{\text{qp}}$ as accurately as possible. Therefore, we formulate our model as $DiQP(F_{\text{iqp}},QP,Z)=\text{noise}_{\text{qp}}^{\prime}$, where $F_{\text{iqp}}$ is a randomly selected window from the original 8K frame, calculated by the Hadamard product of $F_{\text{qp}}$ and a random binary mask $M^{h\times w}$ : $F_{\text{iqp}}=F_{\text{qp}}\circ M$. The reconstructed output frame sequence $F_{\text{res}}$ is then obtained as $F_{\text{res}}=F_{\text{iqp}}+\text{noise}_{\text{qp}}^{\prime}$. Additional input Z includes both conditional inputs and the inputs specifically for the Look Ahead and Look Around modules, which are discussed in detail later in this section.

For a fair comparison with existing methods, we use the commonly adopted Charbonnier loss [5] between the reconstructed frame sequence and the ground truth or raw sequence, defined as:

and $\epsilon=10^{-3}$ is a constant in all the experiments.

DiQP is a novel Transformer-Diffusion model for 8K video restoration; specifically addressing the complex artifacts introduced by codec compression. By viewing the restoration process itself as a Deonising Diffusion model and leveraging the Quantization Parameter (QP) as the Diffusion step, we successfully applied this powerful framework to the challenging task of video restoration. Our approach demonstrates superior performance in restoring high-resolution videos from heavily compressed sources. The experimental results highlight the effectiveness of our model in recovering fine details and improving overall visual quality compared to other existing models. For future work, we plan to work on a more compatible LOST embedding since it is currently designed only for 8K resolution, and adapting it for other resolutions is challenging.

In this section, we delve deeper into the quantitative and qualitative results obtained from our model. Fig. S1 showcases frame 179 from sequence 26 of the SEPE8K dataset, along with our model’s output. To better illustrate our model’s performance, we highlight three specific crops from this frame: 1) the in-focus ”bag” crop, demonstrating the model’s ability to handle sharp details, 2) the out-of-focus ”hat” crop with its intricate winter hat texture, testing the model’s performance on challenging patterns, and 3) the out-of-focus ”jacket” crop, further evaluating the model’s ability to reconstruct detailed textures in less-than-ideal conditions. Our findings highlight that our model consistently delivers high-quality results in both in-focus and out-of-focus scenarios, effectively handling various textures and sharpness levels, unlike other models that struggle in such diverse situations.

To validate our model’s robustness, we evaluated its performance under a range of compression levels (QPs). This included not only the most challenging highest QP scenario but also lower QPs to ensure comprehensive learning. As detailed in Tables S1, S2, S3, and S4, our model consistently demonstrates strong performance across various compression levels, from the least to the most compressed videos. Intuitively, our DiQP exhibits a notably stronger performance advantage at lower QPs, with a margin of approximately 5 dB in the SEPE8K dataset and around 2 dB in the UVG dataset. This shows a superior ability to leverage the additional information available in less compressed videos.