Joint Reference Frame Synthesis and Post Filter Enhancement for Versatile Video Coding

The optical flow estimation module is illustrated in Fig.2(b). To achieve feature alignment between two input frames, it’s essential to estimate the optical flows $F_{0\to 1}$ and $F_{1\to 0}$ between them. Additionally, for interpolating an intermediate frame from two input frames, estimation of the optical flows $F_{t\to 0}$ and $F_{t\to 1}$ from the intermediate moment $t$ is required. Inspired by [35], which proposes an effective and precise optical flow reusing strategy, we adopt the same strategy to reduce the complexity involved in optical flow estimation.

To begin with, we utilize the IFRNet[36] as the optical flow estimator due to its fast inference speed. Since IFRNet requires inputs in RGB format, we first upsample the U and V channels of each input frame in YUV420 format and then perform a color space transformation to convert them into RGB format. The reformatted input frames are then fed into IFRNet, which estimates the optical flows $F_{t\to 0}$ and $F_{t\to 1}$. This process can be summarized as follows:

where $\Theta$ represents the intermediate optical flow estimator. Next, following the approach described in [35], we adopt a similar strategy to generate the optical flows $F_{0\to 1}$ and $F_{1\to 0}$ based on the previously estimated optical flows $F_{t\to 0}$ and $F_{t\to 1}$. This can be expressed as:

where $\Phi$ represents the optical flow reusing strategy from [35]. At this stage, all the optical flows have been estimated. These resulting optical flows will be utilized for alignment in the bidirectional recurrence, which will be explained in detail in the subsequent section.

Inspired by [37], we downsample the input frames and the optical flows to reduce the computational cost. For each input frame in YUV420 format, the Y component undergoes a PixelUnshuffle operation[38] with a scale factor of 2, followed by a concatenation operation with the U and V components to generate a six-channel tensor. The optical flows, detailed in Section III-B1, are downsampled accordingly. These designs enable most of the computations to be performed in the low-resolution feature space, thereby greatly enhancing efficiency. It’s worth noting that, instead of downsampling the input frames to estimate optical flows as mentioned in [37], performing high-resolution optical flow estimation and then downsampling optical flows will yield more accurate low-resolution optical flows.

In the backward branch, information from the succeeding state is transmitted to both the preceding and intermediate states. A weight-shared residual refinement module $\mathcal{R}^{B}$ is utilized to refine features at each state. The feature extraction process of the succeeding state is described as:

where $\mathcal{E}^{B}$ represents the weight-shared feature extraction module and $\mathcal{C}$ denotes the channel concatenation operation. To align the input channel of $\mathcal{E}^{B}$ in each state, the tensor $Zero$ filled with zeros is treated as a placeholder. Then, the preceding state receives knowledge from the succeeding feature:

where $\mathcal{W}$ denotes the flow warping operation. Next, the knowledge from the succeeding state is propagated to the intermediate state as prior knowledge for the frame synthesis, represented as:

Up to now, via backward propagation, we acquire three features in distinct states, which are intended for further refinement within the forward branch.

The forward branch shares the same idea with the backward branch. Both the succeeding and intermediate states will receive knowledge from the preceding state. The specific processes at three different states are as follows:

where $\mathcal{R}^{F}$ represents the weight-shared residual refinement module in the forward branch. So far, we have extracted features in three states. A weight-shared convolution layer was employed for each state for reconstruction, yielding a six-channel tensor. To get the output frame in YUV420 format, the first four channels of the reconstructed feature undergo a PixelShuffle operation[38] with a scale factor of $2$ to obtain the Y component, and the last two channels are U and V components, respectively.

As depicted in Fig.2(a), STENet comprises two pipelines: the enhancement pipeline and the synthesis pipeline. These pipelines can operate jointly or independently based on routing selection. The enhancement pipeline takes two input frames $I_{0}$ and $I_{1}$ as input to enhance each other, resulting in two enhanced frames $I^{Enh}_{0}$ and $I^{Enh}_{1}$:

where the $Enh$ represents the enhancement pipeline. Meanwhile, the synthesis pipeline takes two input frames $I_{0}$ and $I_{1}$ for frame synthesis, producing an intermediate synthesized frame $I^{Syn}_{t}$:

where the $Syn$ denotes the synthesis pipeline. Additionally, the enhancement pipeline can be utilized for PFE, the synthesis pipeline for RFS, and the entire STENet for JISE. Further details are provided in the subsequent section.

To effectively coordinate RFS, PFE, and JISE under RA configuration, we introduce the STEW. As depicted in Fig.3(a), each STEW contains 8 consecutive frames. Within the current STEW, we determine whether to execute RFS, PFE, or JISE based on the POC $p$ of the current to-be-coded frame. The specific mode decision process is as follows:

where $\mathcal{S}$ denotes RFS, $\mathcal{E}$ represents PFE, and $\mathcal{J}$ indicates JISE.

In each STEW, the processes of RFS, PFE, and JISE are executed alternately, their execution sequence is constrained by the encoding order of the RA configuration. As illustrated in Fig.3(a), we annotate their execution sequence using symbols. For a specific symbol, the superscript denotes its order, while the subscript indicates whether it is RFS, PFE, or JISE. $\mathcal{S}^{1}$ synthesizes an additional reference frame for the $4^{th}$ frame, followed by $\mathcal{S}^{2}$, which synthesizes an additional reference frame for the $2^{nd}$ frame. Subsequently, $\mathcal{J}^{3}$ synthesizes an additional reference frame for the $1^{st}$ frame and enhances the $0^{th}$ and $2^{nd}$ frames. Finally, $\mathcal{S}^{4}$ synthesizes an additional reference frame for the $3^{rd}$ frame, and $\mathcal{E}^{5}$ enhances the $1^{st}$ and $3^{rd}$ frames. Further operations are not described in detail. In each STEW, the latter 7 frames will reference the synthesized reference frames generated by RFS or JISE, and all frames will be enhanced by PFE or JISE. Once the last frame is decoded and enhanced, the STEW will slide forward by 8 POC distances.

DPB stores the reconstructed frames during encoding. Specific reconstructed frames are selected from DPB to be included in the RPL as reference frames. For bi-directional inter prediction, two separate RPLs are established in different directions for the to-be-coded frame. The encoder searches the best-matching blocks in reference frames as predictions of the current to-be-coded CU. Reference frames with more similar content to the to-be-coded frame can lead to smaller residual coding consumption in the bitstream. To enhance the bi-directional inter prediction in VVC, the JVET CTC [39] specifies RA configuration, which employs a hierarchical coding structure with 32 frames in a group of pictures (GOP).

Within each STEW, RFS synthesizes a virtual reference frame to enhance bi-directional inter prediction under RA configuration. As illustrated in Fig.3(a), the frames within a GOP are divided into 6 temporal layers in RA configuration, allowing the utilization of previously encoded frames in lower layers as references for the to-be-coded frames in higher layers. As shown in Fig.3(b), RFS selects two-sided reconstructed frames $I_{p-d}^{Rec}$ and $I_{p+d}^{Rec}$ as inputs to synthesize a virtual reference frame $I_{p}^{Syn}$ through the STENet’s synthesis pipeline, represented as:

where $Syn$ represents STENet’s synthesis pipeline. Considering the degradation of frame synthesis performance over long temporal distances, the POC distance $d$ should be less than 4. In current STEW, the relationship between the POC distance $d$ and the to-be-coded frame’s POC $p$ is arranged as:

As for the other frames, in which $(p\bmod 8)\in\{1,5\}$, we will provide detailed explanations in Section III-C4.

The synthesized frame is inserted into two RPLs (RPL0 and RPL1) of the current to-be-coded frame and treated as additional reference frames in two RPLs. The indexes of the synthesized frame in RPL0 and RPL1 are set to 1. Notably, the synthesized reference frame is utilized like other reference frames, thus ensuring that the encoder upholds consistent ME and MC procedures during inter prediction. RFS operates in both the encoding and decoding processes, eliminating the necessity to convey additional information into the bitstream.

In block-based hybrid coding schemes, reconstructed frames often suffer from block artifacts, ringing artifacts, and over-smoothing. Those artifacts will degrade the perceptual quality of the reconstructed video. In most block-based hybrid coding schemes, several in-loop filters operate inside the decoder to alleviate artifacts in filtered frames. On the other hand, post-processing filters operate out of the encoding and decoding loop. This means that post-processing filters can be integrated into any video standard without any codec modifications. However, most post-processing filters only use information from the current to-be-coded frame, ignoring potentially useful information from neighboring frames.

Within each STEW, PFE filters the reconstructed frames to mitigate their artifacts and distortions. PFE simultaneously improves two input frames, enabling each frame to leverage its own spatial information and the temporal information from another frame to improve its quality. As shown in Figure 3(c), PFE takes two reconstructed frames $I^{Rec}_{p-d}$ and $I^{Rec}_{p}$ as inputs and uses the STENet’s enhancement pipeline to improve them. If the POC distance $d$ between the two reconstructed frames is either too short or too long, the reconstructed frame will not be able to effectively get complementary information from other frames. Therefore, we set the POC distance $d$ between the two reconstructed frames to be 2. The process can be described as follows:

where $Enh$ represents STENet’s enhancement pipeline, $I^{Enh}_{p-d}$ and $I^{Enh}_{p}$ are two enhanced frames, and $p$ is the POC of the current to-be-coded frame. In other words, the $1^{st}$ and $3^{rd}$ reconstructed frames are filtered simultaneously, while the $5^{th}$ and $7^{th}$ reconstructed frames are filtered simultaneously. As for the other frames in the current STEW, we will provide detailed explanations in Section III-C4. The enhanced frames are considered as the final output frames and are not used as reference frames.

If the reconstructed frames exhibit minimal artifacts, PFE may degrade the quality of the enhanced frames. Hence, we offer the flexibility to enable or disable PFE at the block level. We utilize PSNR metrics to assess the similarity between the original, filtered, and reconstructed blocks. If the original block closely resembles the filtered block more than the reconstructed block, we activate PFE for that block. In contrast, we will turn off the PFE in this block. Switch flags indicating the PFE status are subsequently embedded into the bitstream and transmitted to the decoder.

In our design, the PFE and RFS are continuous in some positions. For instance, once the $0^{th}$ and $2^{nd}$ frames in the current STEW are decoded, PFE can utilize STENet’s enhancement pipeline to filter them. After PFE, RFS can utilize STENet’s synthesis pipeline to synthesize a virtual reference frame for the $1^{st}$ frame. This inspires us to introduce the JISE, in which RFS and PFE are executed jointly. Instead of utilizing RFS and PFE independently, JISE can reduce network inference complexity by reusing features between two pipelines of STENet.

For the first three frames in the current STEW, we can input the $0^{th}$ and $2^{nd}$ reconstructed frames into STENet to filter them and synthesize an intermediate frame. The synthesized frame is treated as the virtual reference frame for the $1^{st}$ frame. As for the $4^{th}$, $5^{th}$, and $6^{th}$ frames in the current STEW, the process is the same. The process can be described as follows:

where $STENet$ represents the STENet. The enhanced frames $I^{Enh}_{p-1}$ and $I^{Enh}_{p+1}$ are treated as final output frames after determining whether to keep enhanced blocks or not at the block level. The synthesized frame $I^{Syn}_{p}$ is treated as a virtual reference frame, which will be inserted into two RPLs of the current to-be-coded frame $I_{p}$.

The synthesis and enhancement pipelines reuse lots of modules. When RFS and PFE are executed independently, those reused modules will be inferred twice. In contrast, those reused modules will infer only once within JISE. That is why the proposed JISE can reduce network inference complexity.

During network inference, large input sizes can lead to excessive memory consumption, especially for $3840\times 2160$ (4K) resolution inputs. To solve this problem, we divide the input frames into blocks and work on them separately. However, using small blocks can limit the amount of spatial and temporal information captured, leading to a notable drop in STENet performance. Therefore, we use different block sizes for different resolutions. For JVET NNVC common test condition (CTC) class D[40] input frames, we use $240\times 240$ blocks with an extra padding of $8$. For input frames from other CTC classes, we use $480\times 480$ blocks with an extra padding of $16$. It is worth mentioning that the activation or deactivation decisions for the PFE are also made at the block level as previously described.

After PFE, the enhanced frames are temporarily stored in a buffer and released as output frames once all frames within the current STEW are enhanced. The temporary buffer ensures that the inputs of RFS must be the reconstructed frames without being enhanced by PFE.

Without this design, frames enhanced by PFE will be utilized for subsequent RFS. For example, as depicted in Fig.3(a), following the $\mathcal{J}^{3}$ process, the $2^{nd}$ frame is enhanced. This enhanced $2^{nd}$ frame is subsequently utilized as input during the $\mathcal{S}^{4}$ process, potentially degrading RFS performance. Specifically, during training, STENet takes reconstructed frames with compression artifacts as inputs. However, during inference, if the inputs of STENet are enhanced frames with suppressed artifacts, the quality of the synthesized frame may decline. Moreover, the synthesized frame may not serve as an effective reference for the to-be-coded frame.

We adopt the Vimeo-90k triplet dataset[41] for training. The Vimeo-90k dataset consists of 73,171 3-frame sequences with a fixed resolution of $448\times 256$. During inference, the inputs for STENet consist of two reconstructed frames containing compression artifacts, while the outputs consist of an intermediate synthesized frame and two enhanced frames with suppressed artifacts. Compression artifacts are introduced into the training data to simulate real-world inference scenarios. Specifically, we compress the first and third frames in each triplet using the all intra (AI) configuration, while employing three consecutive original frames as ground-truth frames. The quantization parameter (QP) for each triplet is randomly selected from 22, 27, 32, 37, and 42.

The STENet is trained using PyTorch on four Nvidia Geforce RTX 3060 GPUs. We use a Charbonnier penalty function[42] as the loss function and supervise all three output frames of STENet simultaneously:

where $I^{Ste}$ is the output frames of STENet and $I^{GT}$ denotes ground-truth frames. $\epsilon$ is set to $10^{-3}$ as a constant. The loss function is optimized through the Adam optimizer with ${\beta}_{1}=0.9$ and ${\beta}_{2}=0.999$. Before training, the STENet loads the pre-trained weights of IFRNet. The initial learning rates of IFRNet and other parts are set to $2.5\times 10^{-5}$ and $1\times 10^{-4}$ respectively and decay to $10^{-7}$ with one cosine annealing[43]. For data augmentation, two input frames are randomly cropped into $128\times 128$ patches, and their order is randomly swapped. The batch size is 48, and the training lasts 300 epochs.

RFS generates a virtual reference frame resembling the to-be-coded frame, effectively reducing residuals. We conduct a “tool-off” test to assess the significance of RFS, where RFS is disabled, and the proposed method serves as the anchor. The performance comparison, detailed in Table IV(a), reveals performance losses of 6.46%/12.47%/12.07% across three components when RFS is disabled. Simultaneously, as indicated in Table V(a), encoding and decoding times are reduced to 66% and 33%, respectively. RFS is the primary driver of performance improvement in our methodology and constitutes a substantial portion of the total runtime.

PFE operates out of the encoding and decoding loops, impacting only the final reconstructed frames. PFE will alleviate artifacts introduced by block partitioning and quantization. To evaluate the impact of PFE, we turn it off during inference, with the proposed method serving as the anchor. As depicted in Table IV(b), the absence of PFE leads to a noticeable decline in the performance of 1.26%/5.24%/4.87%. The more significant performance drop in chroma components implies that PFE is particularly effective in mitigating artifacts within chroma components. As illustrated in Table V(b), encoding time and decoding time are reduced to 91% and 85%, respectively. This indicates that PFE does not significantly contribute to the overall complexity of our approach.

As described in Section IV-A2, the synthesis and enhancement pipelines are jointly trained within a single STENet, significantly simplifying the training cost. To investigate the impact of joint training on performance, we separately trained two STENets. For the first STENet, we only supervise the synthesized frame (second frame) and maintain all training settings unchanged. For the second STENet, we supervise the enhanced frames (first and third frames) and keep the original training settings.

Regarding network inference, the first STENet’s synthesis pipeline is utilized in RFS, while the second STENet’s enhancement pipeline is utilized in PFE. As for JISE, the single STENet is replaced by a cascade of two pipelines. As shown in Table IV, training two separate STENets for RFS and PFE resulted in a performance decrease of 0.44%/1.74%/1.10%. It indicates that the joint training strategy not only simplifies training costs but also enhances performance in the proposed method.