DCNGAN: A DEFORMABLE CONVOLUTION-BASED GAN WITH QP ADAPTATION FOR PERCEPTUAL QUALITY ENHANCEMENT OF COMPRESSED VIDEO

The generator $G\left(\mathbf{y}_{t}\right)$ further consists of two modules: the frame alignment module $\mathbf{z}_{t}=A\left(\mathbf{y}_{t}\right)$ and the quality enhancement module $\hat{\mathbf{x}}_{t}=E\left(\mathbf{z}_{t}\right)$. In order to leverage temporal information, the former takes previous, current and next frames and aligns them using a deformable convolution. The output of the deformable convolution is a single representation $\mathbf{z}_{t}$ which already integrates information from the three frames. The key to the correct alignment is the computation of the corresponding offsets, which are predicted by a network based on U-net [13]. The number of channels of offsets is determined by the number of the input frames (i.e. three in our case), the number of spatial dimensions (i.e. two) and the size of the convolutional kernels (i.e. $3\times 3$ in our case). This approach is more efficient than aligning them using optical flows (generally conducted in a pairwise manner).

The quality enhancement module is based on an encoder-decoder structure with nine residual blocks. To avoid storing multiple sets of model parameters for enhancing videos compressed at different QPs, encoded QP information $\mathbf{q}$ is embedded into each residual block to make the network modulated by QP values. Specifically, $\mathbf{q}$ is represented using one-hot encoding [11] and fed into a fully connected layer. Softplus is selected as the activation function since it ensures positive outputs which we found beneficial in training. Finally, channel-wise multiplication is performed between the feature maps which are the output of the first convolution layer before ReLU in the residual block and the encoded QP information.

We use a patch discriminator [14] which outputs the probability of each patch being real or fake. It is implemented in a fully convolutional fashion, and the final output is the average real/fake classification probability across patches.

Our model optimizes the following adversarial loss [15] (hereinafter we omit the subindex $t$ for simplicity)

$${L_{gan}}\left({G,D}\right)={\rm{\mathbb{E}}}_{\left({\mathbf{y},\mathbf{q}}\right)}\left[{{{\left({D\left({G\left(\mathbf{y},\mathbf{q}\right)}\right)-1}\right)}^{2}}}\right]+{\rm{\mathbb{E}}}_{\mathbf{x}}\left[{D{{\left(\mathbf{x}\right)}^{2}}}\right],$$

(1)

where $\mathbf{y}$ are sequences of three consecutive decoded frames and $\mathbf{q}$ is the corresponding QP value, and $\mathbf{x}$ are raw frames, both obtained from a video dataset. The generator is trained to minimize the value of Eq. (1) while the discriminator is trained to maximize it.

We use a perceptual loss based on VGG features [16] that enforces that the features of a given pair $\left(\mathbf{y},\mathbf{q}\right)$ match the features of the corresponding target raw frame $\mathbf{x}$.

$${L_{vgg}}\left(G\right)={\rm{\mathbb{E}}_{\left({{\bf{y}},{\bf{q}},{\bf{x}}}\right)}}\sum\limits_{i{\rm{=1}}}^{{N_{f}}}{\frac{{\rm{1}}}{{{M_{i}}}}\sum\limits_{j{\rm{=1}}}^{{M_{i}}}{{{\left\|{f_{j}^{i}\left({\bf{x}}\right){\rm{-}}f_{j}^{i}\left({G\left({{\bf{y}},{\bf{q}}}\right)}\right)}\right\|}_{\rm{1}}}}},$$

(2)

where ${f^{i}_{j}}\left(\cdot\right)$ represents the $j$-th spatial element of the output tensor of the $i$-th layer from a pre-trained VGG-19 model.

Similarly, we also enforce matching the features of enhanced and of raw images in the discriminator.

$${L_{fm}}\left({G,D}\right)={\rm{\mathbb{E}}_{\left({{\bf{y}},{\bf{q}},{\bf{x}}}\right)}}\sum\limits_{i{\rm{=1}}}^{{N_{g}}}{\frac{{\rm{1}}}{{M_{i}^{g}}}\sum\limits_{j{\rm{=1}}}^{M_{i}^{g}}{{{\left\|{g_{j}^{i}\left({\bf{x}}\right){\rm{-}}g_{j}^{i}\left({G\left({{\bf{y}},{\bf{q}}}\right)}\right)}\right\|}_{\rm{1}}}}},$$

(3)

where ${g^{i}_{j}}\left(\cdot\right)$ represents the $j$-th spatial element of the output tensor of the $i$-th layer selected from the discriminator.

Finally, during training we optimize

$${\underset{G}{\operatorname{min}}}\ {\underset{D}{\operatorname{max}}}\ \left({L_{gan}}\left({G,D}\right)+{L_{vgg}}\left(G\right)+{L_{fm}}\left({G,D}\right)\right).$$

(4)

We employ 106 sequences collected by [4] for training and the Joint Collaborative Team on Video Coding (JCT-VC) standard test sequences for testing. All sequences are compressed by H.265/HEVC software HM16.5 under Low Delay P configuration at QPs 22, 27, 32 and 37. Note that the 106 raw sequences and their compressed versions are randomly cropped into $128\times 128$ clips as training samples. The training dataset includes mixed and shuffled samples at the four QPs.

We employ Adam optimizer with ${\beta_{1}}=0.9$, ${\beta_{2}}=0.999$ and ${\epsilon}={10^{-8}}$. Batch size is set to be 32. Learning rate remains unchanged at ${10^{-4}}$. For a fair comparison with previous work, we only enhance the luminance component.

We compare the proposed DCNGAN with state-of-the-art video quality enhancement networks in [4] (MFQE 2.0), [5] (STDF), [9] (MW-GAN) ¹¹1For fair comparison, the performance of MW-GAN shown in Table 1 is from their published paper since our retrained model has not achieved the same good performance and their pretrained models at the four QPs have not been published. and [10] (VPE-GAN). LPIPS [17] and DISTS [18] are employed to quantitatively evaluate the perceptual quality of enhanced videos. For better illustration, LPIPS and DISTS of “Compressed” (i.e. the input videos) are also shown in Table 1. Smaller values indicate better perceptual quality. It should be noted that we only train one model for testing sequences compressed at four different QPs, while the other networks need to train four models since their proposed network cannot achieve QP adaptation. As shown in Table 1, the proposed DCNGAN achieves the advanced performance and alleviates memory requirements at the enhancement stage.

Besides, three examples are randomly selected to qualitatively illustrate the performance of the proposed DCNGAN compared with the other video quality enhancement networks. Specifically, as shown in Fig. 2, MFQE 2.0 and STDF, which were designed to improve the PSNR of compressed videos, still tend to produce blurred results and penalize the perceptual quality. As for VPE-GAN, it alleviates blurring to some extent, but sometimes generates artifacts (e.g. the artifacts on the horse and the wall in Fig. 2). These artifacts also result in degraded perceptual quality, while the proposed DCNGAN can generate more realistic high-frequency details to combat with blurring and greatly improve the perceptual quality of compressed videos.

To evaluate the ability of QP adaptation of the proposed DCNGAN, we separately train four models at the four QPs, represented by “Trained_QP22”, “Trained_QP27”, “Trained_QP32” and “Trained_QP37”, and compare these four models with the single model trained to adapt to various QPs, represented by “Trained_4QPs”. The performance comparison is shown in Fig. 3, where LPIPS is averaged on all sequences in Table 1.

Overall, the model trained at a certain QP achieves the best performance tested at that QP because it has fully learned the characteristics of videos compressed at the corresponding QP in the training process, while the model can hardly achieve satisfactory performance in enhancing the perceptual quality of videos compressed at other QPs due to different characteristics. By feeding encoded QP information ²²2HM 16.5 compresses videos with a small QP variation. We have tried to feed QP of each frame to the network, but haven’t see benefit of retraining with small delta QP. Hence, we only feed the QP of I frame to the network. into the proposed DCNGAN, the model “Trained_4QPs” can be employed to enhance the perceptual quality of compressed videos at various QPs with advanced performance.

An alternative to deformable convolutions to align frame information is optical flow. For comparison, we replace deformable convolutions with optical flows estimated by a pre-trained SpyNet model [19], in order to achieve frame alignments. The other modules of the proposed DCNGAN (i.e., the quality enhancement with QP adaptation module, the discriminator and the loss functions) are kept unchanged. Finally, the average LPIPS performance and runtime of two approaches are compared. As shown in Fig. 4, the network using deformable convolutions is much faster and achieves better performance than that using optical flows, which highlights the accuracy and efficiency of deformable convolutions.