Beyond GFVC: A Progressive Face Video Compression Framework with Adaptive Visual Tokens

As shown in Figure 1, the proposed PFVC framework is mainly grounded on the adaptive visual tokens, thus exploring exceptional capabilities in the trade-offs between reconstruction robustness and bandwidth intelligence. At the encoder side, the input video signal can be classified as the key-reference frame $K$ (e.g., the first frame of this video) and subsequent inter frames $I_{l}\left(1\leq l\leq n,l\in Z\right)$. The key-reference frame $K$ is intra-coded by an off-the-shelf VVC encoder [2], thus providing the rich texture reference for the subsequent frame reconstruction. Afterwards, the VVC-reconstructed key-reference frame $\hat{K}$ and these remaining inter frames $I_{l}$ are all characterized into a series of adaptive visual tokens in a progressive manner. Then, the network bandwidth scenario will further determine which granularity of visual tokens can be compressed. Then, these determined tokens are coded using inter prediction, quantization and entropy coding, where a context-adaptive arithmetic coder [15, 16] is employed to achieve optimal compression performance.

The proposed PFVC decoder aims at using the decoded adaptive visual tokens to reconstruct face video at different quality levels. In particular, the off-the-shelf VVC decoder is employed to reconstruct the key-reference frame $\hat{K}$. In addition, the adaptive tokens of inter frames can be obtained by context-based entropy decoding and difference compensation. These decoded visual tokens are developed to infer the temporal motion evolution and transformed into pixel-wise dense motion fields, where their motion quality can be gradually improved with better token granularity. Finally, the powerful generative model can facilitate to reconstruct the face video with the guidance of the decoded key-reference frames and these inferred dense motion fields.

Thanks to the adaptive visual tokens, the proposed PFVC framework enjoys several benefits that are not available to other existing GFVC algorithms. First, our proposed PFVC can characterize high-dimensional visual face signal into compact tokens with different granularities, ensuring greater feature adaptability in a learnable manner. In addition, different from the existing GFVC algorithms, which can only control the RD trade-offs by adjusting Quantization Parameter (QP) levels of the key-reference frame, our proposed PFVC can also achieve coding flexibility and bandwidth intelligence by transmitting visual tokens of different granularities. Furthermore, finer-granularity tokens can improve the face reconstruction robustness and stability, facilitating to solve the inaccurate motion estimation and unreliable generation problems that sometimes plague the existing GFVC pipelines due to compact representations. Finally, our proposed PFVC only needs one model to output visual features of different granularities and achieve face reconstruction of different qualities instead of using multiple GFVC models, which greatly expands the application possibilities.

Herein, we will introduce all key techniques in the proposed PFVC framework, including adaptive visual token representation, implicit motion field evolution and GAN-based signal reconstruction. These designed schemes have greatly empowered the inference capability of the PFVC towards promising RD performance and bandwidth intelligence.

Adaptive Sparse Tokens Representation. These high-dimensional visual signal $\hat{K}$ or $I_{l}$ can be transformed to adaptive sparse tokens with different granularities via a specifically-designed token extractor. In particular, these face frames first execute down-sampling operation $\phi\left(\cdot\right)$ through a scale factor $s$, and then fed into the feature network $\mathbb{N}_{f}\left(\cdot\right)$ that is mainly composed of a U-Net architecture and a series of convolution/normalization operations to obtain motion features $\mathcal{F}_{motion}$ with the size of $1\times 16\times 16$. The motion features entailing accurate head posture and facial expressions can be learned in an unsupervised manner,

where $X$ could represent the VVC-reconstructed key-reference frame $\hat{K}$ or the inter frames $I_{l}$.

Afterwards, these learned motion features are further fed into a token network $\mathbb{N}_{t}\left(\cdot\right)$ that is essentially an encoder-decoder translator with three Fully-Connected (FC) layers. Specifically, at the encoder part $\mathbb{N}_{te}\left(\cdot\right)$ of this token network, the 2-D motion features are first flatten into a 1-D tokens with the dimension of 256, and then are sequentially sampled into three other 1-D tokens via FC layers with the dimensions of 144, 64 and 16. It should be noted that the number of FC layers and the specific dimension of each FC layer can be randomly set and herein we just provide an implementation example. The produced adaptive token list $\mathcal{L}_{tokens}$ is denoted as,

where the dimension of adaptive tokens $\mathcal{L}_{tokens}$ is 256, 144, 64 and 16. In other words, depending on the given bandwidth environment, different-granularities visual tokens can be further coded into bitstream and actualize variable bitrate. After decoding these tokens $\mathcal{\hat{L}}_{tokens}$, they are input into the decoder part $\mathbb{N}_{td}\left(\cdot\right)$ of the token network for the corresponding reconstruction of motion features $\mathcal{\hat{F}}_{motion}$,

Overall, the proposed token network can realize feature sparsity and produce visual tokens of different granularities, thus providing great possibilities to explore compression scalability.

Implicit Motion Field Evolution. Given the decoded key-reference frame $\hat{K}$, the extracted motion features $\mathcal{F}_{motion}^{\hat{K}}$ from $\hat{K}$ and the decoded motion features $\mathcal{\hat{F}}_{motion}^{I_{l}}$ from $I_{l}$, we design an implicit motion field evolution scheme to achieve effective motion transformations from feature to pixel-wise fields. Different from CFTE [17]/CTTR [5] that both use the sparse-to-dense motion estimation strategy to generate a coarse deformed frame and then transform this frame for motion fields, our proposed scheme can directly use finer-granularity feature to infer motion fields via neural network learning. In particular, an up-sampled learning network $\varphi\left(\cdot\right)$ is employed to perform scale transformation for $\mathcal{F}_{motion}^{\hat{K}}$ and $\mathcal{\hat{F}}_{motion}^{I_{l}}$. In addition, difference between these scaled features is calculated,

As an implicit motion change information, $Diff_{\left\langle I,K\right\rangle}$ is treated as the input together with the down-sampled $\hat{K}$ into the motion prediction network $\mathbb{N}_{m}\left(\cdot\right)$, aiming at estimating pixel-wise dense motion map $\mathcal{M}^{I_{l}}_{dense}$ and occlusion map $\mathcal{M}^{I_{l}}_{occlusion}$ for face signal reconstruction. The process can be formulated as follows,

where $P_{1}\left(\cdot\right)$ and $P_{2}\left(\cdot\right)$ represent two different predicted operation.

GAN-based Signal Reconstruction. Analogous to the existing GFVC generator designs, we also adopt the flow-warped GAN module $\mathbb{N}_{GAN}\left(\cdot\right)$ to reconstruct realistic face signal $\hat{I}_{l}$. In particular, $\hat{K}$ is first warped with $M^{l}_{dense}$ for motion guidance and then the transformed features are further marked with $\mathcal{M}^{I_{l}}_{occlusion}$ for occlusion impainting,

where $f_{w}$ and $\odot$ describe the back-warping and the Hadamard product operations.

To better facilitate the proposed PFVC to be compatible with different token granularities and achieve high-quality motion transformations within the same model, we propose a new progressive model training strategy with the granular probability guidance. In particular, assuming that the PFVC model needs to be trained for $N$ epochs and can process $M$ granularities $G=\left\{G_{1},G_{2},\dots,G_{M}\right\}$, the overall model training will be divided into $S=\left\{S_{1},S_{2},\dots,S_{\frac{M}{N+1}}\right\},\frac{M}{N+1}\in Z$ stages. In each stage, the PFVC model will be assigned how to learn different token granularities with the given probabilistic ratios $P=\left\{P_{1},P_{2},\dots,P_{M}\right\}$. It should be mentioned that in each stage, the given probabilistic ratios for all granularities should be summed to 1. More importantly, to prevent the overall model capacity from being directly reduced by the coarse-grained tokens, the proposed progressive strategy prioritizes training of fine granularity in the early training stages and progressively introduce coarse-grained tokens in the subsequent stages. Besides, during the $S_{1}\sim S_{\frac{M-N-1}{N+1}}$ stages, the maximum probability will be assigned to the corresponding newly-introduced token granularity to improve the PFVC model’s acceptance of this granularity. As for the $S_{\frac{M}{N+1}}$ stage, the probability of each granularity will be set at the same ratio for stabilizing model training.

In addition to the proposed progressive training strategy, the perceptual loss $\mathcal{L}_{per}$, adversarial loss $\mathcal{L}_{adv}$ and feature matching loss $\mathcal{L}_{fea}$ are adopted to supervise all PFVC stages in an end-to-end manner. The overall training loss can be described as,

where $\mathcal{L}_{per}$ and $\mathcal{L}_{adv}$ are used to ensure the fidelity and naturalness of the reconstructed frames $\hat{I}_{l}$, and $\mathcal{L}_{fea}$ is introduced to stabilize the the GAN model training. In addition, the hyper-parameters are set as follows: $\lambda_{per}=10$, $\lambda_{adv}=1$ and $\lambda_{fea}=10$.

Implementation Details. We implement our proposed PFVC framework with Pytorch libraries, and train this model with the popular VoxCeleb training dataset [18] on 2 NVIDIA TESLA A100 GPUs. During the model training phase, the Adam optimizer is used with the parameters $\beta_{1}$ = 0.5 & $\beta_{2}$= 0.999, and the learning rate is set at 0.0002 for model convergence. In addition to the synchronized BatchNorm initialization and the data repeating strategy, we also employ the progressive training strategy as shown in Table 1.

Compared Algorithms. To demonstrate the effectiveness of the proposed PFVC, we compare the state-of-the-art video coding standard VVC [2] and two representative GFVC algorithm (CFTE [17] and FOMM [10]). In particular, the testing dataset adopted in [7] includes 50 talking face videos from the VoxCeleb testing dataset [18], where each sequence contains 250 frames with the resolution of 256$\times$256. The specific compared algorithm settings are provided as follows,

• •

  VVC anchor: is configured with the Low-Delay-Bidirectional (LDB) mode in VTM reference software 22.2, where the Quantization Parameters (QPs) are set at 32/35/37/40/42/45/47/52 and the coded signal format is YUV 4:2:0.

• •

  CFTE/FOMM: follows experimental testings specified in the JVET GFVC AhG test conditions [19]. In particular, the key-reference frame is converted to the YUV 4:2:0 format and compressed via the VTM reference software 22.2 with the QPs of 2/12/22/32/42/52. The subsequent inter frames can be represented into compact parameters and then compressed via a context-adaptive arithmetic coder.

• •

  PFVC (Proposed): Its test settings are basically the same as CFTE/FOMM except the feature representation of inter frames. Specifically, the proposed PFVC can characterize the inter frames into 4 different token granularities with the dimension of 16/64/144/256. Herein, the convex hull is performed on $6\times 4=24$ RD points for a wider bitrate range and the optimal performance.

Evaluation Measures. Herein, we adopt 4 learning-based quality measures and 2 traditional quality measures to evaluate the quality of reconstructed face video, including DISTS [20], LPIPS [21], FVD [22], MANIQA [23], PSNR [24] and SSIM [25]. These 6 adopted measures mainly involve perceptual-level, temporal consistency, no-reference GAN-based and pixel-level evaluations, where smaller scores of DISTS/LPIPS/FVD and higher scores of MANIQA/PSNR/SSIM indicate better perceived quality. Therefore, Figure 4 uses 1-DISTS, 1- LPIPS and 1-FVD in order to present more coherent RD curves. In addition to reconstruction quality, the coding bitrate (i.e., kbps) is also provided to further evaluate the compression performance.

Rate-Distortion Performance. Figure 4 shows the RD performance of the VVC codec, FOMM, CFTE and our proposed PFVC scheme in terms of DISTS, LPIPS, FVD, MANIQA, PSNR and SSIM. It can be seen that the proposed PFVC can outperform the VVC codec for DISTS/LPIPS/FVD/MANIQA quality evaluations at overall bitrate ranges and for PSNR/SSIM quality evaluations at relatively low bit rate ranges. In addition, compared with the CFTE/FOMM anchors, the proposed PFVC can achieve the highest performance for all metrics and effectively improve the reconstruction quality with similar bit consumption.

As for pixel-level measurement, the proposed PFVC and other GFVC algorithms cannot achieve any advantages compared with the VVC codec. This is largely due to the fact that generative compression algorithms aim at reconstructing face signal in perceptual-level feature domain instead of pixel-level domain, which cannot adapt to the pixel-level evaluations (PSNR/SSIM) [26]. More importantly, by introducing finer token granularity, the proposed PFVC can gradually improve the GFVC ability in pixel-level reconstruction. Although the performance can be improved, the adopted token granularities are still insufficient for the motion information supplementation, resulting in the saturation of its pixel-level reconstruction quality after a certain rate point.

Subjective Quality. Figure 3 provides visual examples to verify that the proposed PFVC framework can reduce some annoying visual artifacts or recover more missing details via different-granularities visual token representations. For examples, with the provision of finer token granularity, it is possible to infer the hand details that originally do not exist in the key-reference frame. In addition, the finer token granularity can stabilize the background generation and facilitate higher face fidelity such as accurate mouth movement and eye direction.

As illustrated in Figure 4, the proposed PFVC algorithm is able to reconstruct face frames with high signal fidelity and accurate facial motion. In particular, at very low bit rates, the VVC reconstructed results exhibit blocking artifacts, while the proposed PFVC still can reconstruct vivid face signal. Besides, the facial motions in reconstructed video quality using FOMM and CFTE are not accurate, while the proposed PFVC can infer more precise head posture and facial expression.