Task-Aware Encoder Control for Deep Video Compression

Previous video codecs are designed to remove spatial-temporal redundancies effectively. Traditional video coding standards, including H.264/AVC [43], H.265/HEVC [34], and H.266/VVC [2], significantly increase the compression efficiency of images and videos. Recently, a number of learning-based video codecs [26, 25, 15, 19, 30, 36] have been proposed, designed based on residual coding or conditional coding, achieved better and better pixel-wise signal quality metrics, e.g., PSNR and MS-SSIM [42], which mainly serve the human visual experience. Recently, there are also some generative video coding methods [46, 28] that mainly consider visual comfort and perceptual quality.

To facilitate the efficiency of machine vision tasks, early research works were devoted to extracting visual features from signals. Early standards, such as CDVA [8] and CDVS [7], suggest the pre-extraction and transportation of image keypoints to facilitate image indexing or retrieval tasks. With the development of deep learning, some studies began to explore connections between compression and downstream tasks, some studies [3, 39, 1, 13, 21, 41, 45, 11, 4, 48] also focus on the joint optimization of image compression and downstream machine vision tasks by introducing a rate-distortion optimization strategy guided by downstream tasks or by adding a task-specific feature encoding stream. For instance, Torfason et al. [39] introduced a method for executing image understanding tasks, such as classification and segmentation, on compressed outputs from learning-based image compression techniques. Furthermore, Lu et al. [27] enhanced traditional codecs with a preprocessing step, thereby improving codec performance for downstream vision tasks. The field has also witnessed the emergence of self-supervised representation learning methods aimed at deriving compact semantic representations. Dubois et al. [9] presented a theoretical framework suggesting that the distortion term in the lossy rate-distortion trade-off for image classification could be approximated by a contrastive learning objective. Feng et al. [10] proposed a method to learn a unified feature representation for AI tasks from unlabeled data. These methodologies necessitate fine-tuning the downstream models to adapt to the learned features.

Despite advancements in image coding for machine applications, reconstructing high-fidelity video from extracted features remains a formidable challenge due to the critical inter-frame reference relationships. Addressing the requirements of both machine and human vision, Tian et al. [37] introduced a self-supervised edge representation as a semantic intermediary, which preserves the video’s semantic structure. Furthermore, Lin et al. [22] developed a scalable video coding framework tailored for machines, segregating semantic features for machine analysis and human viewing into separate bitstreams.

Most existing approaches [38, 22, 37]necessitate the use of distinct encoders and decoders for various tasks, which complicates and burdens the coding pipelines. Our methodology advances the “Coding for Machine” paradigm by focusing on the encoder side. We present a versatile framework capable of accommodating both human and machine vision demands without the need to modify the pre-trained decoders of existing video codecs.

There are also amounts of works performing video analysis tasks [24, 40, 18], such as image recognition, action recognition [44, 32] and multiple object tracking (MOT) [17, 6], in the compressed video domain. However, these methods focus on developing video analysis models that better leverage the partially decoded video stream, such as the motion vector. In contrast, our work focuses on the coding procedure, specifically the encoding procedure.

Our coding framework is illustrated in Fig. 2 (a). Let $\mathcal{X}=\{\boldsymbol{x}_{1},\cdots,\boldsymbol{x}_{T}\}$ represent the video sequence. The framework categorizes the $P$ frames into two types: original $P$ frames and machine vision-specific $P_{m}$ frames. The categorization of frame $\boldsymbol{x}_{t}$ is determined by the GoP Selection Network, where a value of $1$ signifies a $P$ frame and $0$ a $P_{m}$ frame. The $P_{m}$ frames, designed to reduce the bitrate while preserving critical information for future vision tasks, are generated from the preceding $P$ frames using Dynamic Vision Mode Prediction (DVMP). To maintain the quality of the decoded video sequence, each $P$ frame is computed from the previous $P$ frame with superior visual quality, rather than from a $P_{m}$ frame. In scenarios requiring coding for downstream vision tasks, a machine-centric control applies the GoP selection and DVMP, altering the encoding structure to ”$I,P_{m},P,\cdots$”. Conversely, for human viewing, a human-centric control restores the encoding structure to its original form, ”$I,P,P,P,\cdots$”.

In our coding framework, we aim to minimize the video sequence bitrate. Initial analyses of residual video codecs reveal that they allocate a substantial portion of the bitrate—exceeding 80% in high bitrate configurations—to encode and transmit residual information. This approach ensures high-quality visuals for human perception, as measured by metrics such as PSNR or MS-SSIM [42]. However, this method introduces significant redundancy, especially since downstream tasks like video object detection or tracking predominantly concentrate on specific regions of interest rather than the entirety of the frame. Consequently, it is crucial for our study to develop an approach that reduces this redundancy in the coding bitstream.

Inspired by Hu et al. [14], who developed a mode prediction technique specifically designed for the human visual system within a slimmable encoder and decoder, we introduce the Dynamic Vision Mode Prediction (DVMP) module. This innovative module autonomously selects the optimal coding mode and decides whether each feature element should be encoded and transmitted, as illustrated in Fig. 3.

To elucidate the functionality of our proposed Dynamic Vision Mode Prediction (DVMP) module, we utilize the encoded residual feature $m_{t}$ as an illustrative example. Assume $m_{t}$ possesses dimensions $c\times h\times w$ , indicating $c$ channels each with a spatial dimension of $h\times w$. The hyperprior network then predicts the mean and variance for each element, resulting in dimensions of $2c\times h\times w$. The mode prediction network’s architecture, detailed in the upper branch, consists of two convolution layers and three ResBlocks. To render the “mode prediction” process differentiable, we employ the Gumbel Softmax technique during training to ascertain the skip mode for each encoded residual feature element. This module generates a binary mask for the feature map $m_{t}$, where a “1” denotes elements imperative for machine vision that require entropy coding, and a ”0” signifies elements either irrelevant for machine vision or accurately predictable by the hyperprior network, thus streamlining the entropy coding process and bitrate reduction. Note that the showed architecture is not suitble for prior models with autoregressive components, like which in DCVC [19]. However, we can extend the module into an autoregressive way to enable the support for DCVC. The detailed structure is shown in the supplementary material.

To optimize the DVMP module, the loss function can be given by:

where $R$ denotes the coding bitrate and $\mathcal{L}_{t}$ denotes the loss of downstream vision task, respectively. $\lambda_{m}$ is a hyper-parameter used to control the trade-off.

Obtaining the $P_{m}$ frames, we firstly use a hand-crafted structure in a GoP, where the $P_{m}$ frames and $P$ frames are arranged alternately (referred as “DivGoP” and shown in Fig. 4). It is observed that this arrangement can bring a certan degree of bitrate saving in detection, since the $P_{m}$ frames cost much less bitrate than $P$ frames while maintaining the motion and machine vision information. Experimental findings, as depicted in the “FVC DivGoP” curve within Fig. 5, demonstrate that the ”$I,P_{m},P,\cdots$” structure markedly achieve about 18.6% bitrate saving.

The “DivGoP” configuration, while methodically structured, may not be universally optimal due to the variability in motion and object information across different videos, which could necessitate distinct GoP structures. For instance, videos characterized by rapid target movement might demand a higher frequency of $P$ frames to preserve the quality of reconstruction, as an excess of $P_{m}$ frames may lead to degradation that negatively affects subsequent frames. Conversely, in scenarios where the camera remains static or the target movement is minimal, the introduction of more $P_{m}$ frames could be feasible without significantly impacting the reconstruction quality of following frames. Stimulated by this notion, our research endeavors to tailor the GoP structure dynamically, enhancing codec performance for machine vision tasks. To approach this issue, we adopt FVC [15] as the foundational codec, select a GoP size of 10, and address the detection task utilizing the MOT Dataset. The optimization problem is thus defined:

where $R$ denotes the bitrate, $L_{det}$ denotes the detection loss, $\lambda_{p}$ is a weighting factor that balances the effects of bitrate and detection loss, and $\theta$ denotes the setting of GoP structure here.

Obtaining a GoP, each of the 9 predicted frames can be either $P$ frame or $P_{m}$ frame, which can be dynamically decided. To explore the optimal GoP structure for target function 2, and evaluate the performance gap between DivGoP and optimal GoP structure, we use a deep-first-searching (DFS) algorithm, where we use different $\lambda$ for different bitrate models. The DFS algorithm searches for the structure that makes the target function smallest for each GoP. Results are shown in the “FVC DFS” curve of Fig. 5. The application of the DFS algorithm led to a substantial increase in Bpp-mAP performance,, achieving about 32.3% bitrate saving in terms of the BD-RATE metric, while the “DivGoP” method achieves about 18.6% bitrate saving. Similar conclusions can also be observed on DCVC [19] and TCM [30], reinforcing our hypothesis that distinct videos necessitate unique GoP structures tailored for machine vision tasks. However, the DFS algorithm is extreamly time consuming, and the labels do not exist during real inference process, making the DFS not avaliable to be applied in real time encoding. How to further find an “adaptive GoP structure” in an available time complexity is worth exploring.

From this perspective, we introduce our GoP selection method to adaptively predict the GoP structure at encoding stage, which has a low time complexity and is available in application. Illustrated in Fig. 2 (c), our GoP selection network operates in two phases: the pre-analysis and the GoP prediction stages. Initially, input frames undergo pre-analysis, where a detector identifies objects and generates an overlapping mask for each frame, while the RAFT [35] optical flow estimation method calculates optical flow features for each predicted frame using its adjacent predecessor. These features are then refined by masking with the object bounding box mask to produce masked optical flow features, alongside motion feature priors (mean and variance, averaged by channel) derived from the codec’s motion encoder. Subsequently, these processed features are input into the GoP prediction network, which employs a feature extractor with convolution blocks for downsampling and aggregation, followed by adaptive average pooling and linear layers that output a normalized $s_{logit}$ for this GoP, which comprises two elements that sum to one. The averaged detection confidence information from the detector is also injected into the linear layers as meta data.

Upon acquiring the $s_{\text{logit}}$, we employ the Gumbel softmax sampling technique throughout the training phase to iteratively generate a binary value for each frame. This process culminates in the formation of a GoP structure vector, where the binary value—$0$ for a $P_{m}$ frame and $1$ for a $P$ frame—specifies the type of each frame within the GoP. During inference stage, we apply a softmax operation on $s_{\text{logit}}$ to derive probabilities, which are then used to evenly distribute $1$s and $0$s, thereby determining the actual encoded GoP structure. Similar to our mode prediction network, the loss function for optimizing GoP structure selection network is given by:

where the $\bar{R}$ and $\bar{\mathcal{L}_{t}}$ denote the GoP-wise average coding bitrate and average loss of downstream vision task.

In Fig. 6, we present a comparison between our method and various codecs, specifically DCVC, FVC, and TCM. It is evident that our approach yields a markedly improved trade-off in terms of Bpp-MOTA, Bpp-mAP, Bpp-MOTP and Bpp-FN metrics. When compared to the baseline codecs FVC, DCVC and TCM, our control method results in bitrate savings of approximately 27.54%, 41.82% and 12.64%, respectively, for equivalent MOTA metrics. With DCVC established as the anchor baseline, Table 1 displays the BD-RATE values, where our controlled TCM, controlled DCVC and controlled FVC codecs demonstrate the best, second-best and third-best performance, respectively. Additionally, we contrast our method with the task-oriented “One-to-one Codecs” of VCM, with detailed results provided in the supplementary materials.

For the evaluation of the video object detection task, we juxtapose our method with established codecs such as FVC, and TCM, with the comparative results illustrated in Fig. 6(e). Our method is noted to achieve a superior trade-off, particularly in the Bpp-mAP50 metric, when applied to the downstream model YOLOV. For instance, in comparison to the FVC and TCM baseline codecs, our method facilitates bitrate reductions of approximately 21.76% and 20.42%, respectively, while maintaining the same mAP50 metric. The uncompressed video represents the upper bound for the mAP50 metric, recorded at 76.4, and is delineated by the grey line within the figure.

We conducted an evaluation of our method on the UCF101 dataset, benchmarking against TCM and FVC codecs. The outcomes of this comparison are depicted in Fig. 6(f). For the task of video action recognition, rather than initiating a new set of network trainings, we employed pre-trained mode prediction networks previously utilized for the MOT task and use the “DivGoP” structure for validation. The original videos were compressed using the various codecs, and the resulting decoded videos served as inputs to the TSM model as cited in Lin et al. [23]. The experimental results indicate that our method maintains efficacy in this context, securing approximately 21.81% and 26.03% in bitrate savings for TCM and FVC, respectively.

When it comes to the performance of reconstructing the video for human viewing, the situation is straightforward for conventional video codecs, no matter learned or traditional ones: just encode the frames, transmit and decode the compressed bitstream, and measure their quality using appropriate metrics (PSNR, MS-SSIM, etc). In our method, there’re two options: Control the encoding procedure for human viewing, which is the ideal way for best frame reconstruction quality, or just continue the encoding procedure used for machine vision tasks.

Note that our method does not change the weights of the original encoder and decoder in pre-trained DVC, so using this option can directly recover the human viewing quality to original best performance of the codecs. On the other hand, we also evaluate the human viewing quality of the encoded bitstream for machine, using “DivGop” encoding structure on HEVC Class B and Class C test datasets. These are referred to as “FVC DivGoP” and “TCM DivGoP” in Fig. 7. Results show that our control for machine does decrease the video reconstruction quality of the original TCM codec by about 1dB in PSNR. Besides, it is also observed that the PSNR of residual codec FVC drops more sharply than that of conditional codecs like TCM. However, these are acceptable since we can always adjust the encoder to compress the video for human viewing when it’s needed.

Mode Prediction. In our method, we propose mode prediction network to decide the coding modes of motion and residual/contextual feature elements for machine vision, mainly targeting on reducing the bitrate and keep the crucial information for important objects in the frame. Moreover, mainstream DVC, including FVC, DCVC, TCM and others, use motion estimation and warping operation to generate a rough predicted frame, which we call $P_{r}$ frame here. The $P_{r}$ frame also meets our requirement of “consuming lower code streams but retaining object and motion information” to a certain extent. So comparing it with the $P_{m}$ frame used in our method is also a point worthy of attention. In “DivGoP” structure, we use the $P_{r}$ frame to replace the original $P_{m}$ frames, and the GoP structure will be $[I,P_{r},P,P_{r},...]$. As shown in the left figure in Fig. 8, based on DCVC and FVC, we compare the structure using $P_{r}$ frames with the original “DivGoP” structure using $P_{m}$ frames, corresponding to the “DCVC/FVC DivGoP w $P_{r}$” and “DCVC/FVC DivGoP” curves respectively. It is observed that the “DivGoP w $P_{r}$” curve also achieves about 28.68% and 16.24% bitrate savings for DCVC and FVC, respectively. But our “DivGoP” curve shows better trade-off, by achieving 32.67% and 19.43% bitrate savings.