LCCM-VC: Learned Conditional Coding Modes for Video Compression

Let $X$ and $Y$ be two random variables and $R=X-Y$, then

$$\begin{split}H(X|Y)=H(R+Y|Y)&\stackrel{{\scriptstyle\text{(a)}}}{{=}}H(R|Y)\\ &\stackrel{{\scriptstyle\text{(b)}}}{{\leq}}H(R)=H(X-Y),\end{split}$$

(1)

where $H(\cdot)$ is the entropy, $H(\cdot|\cdot)$ is the conditional entropy, (a) follows from the fact that given $Y$, the only uncertainty in $R+Y$ is due to $R$, and (b) follows from the fact that conditioning does not increase entropy [17].

Now consider the following Markov chain $X\to Y\to f(Y)$ where $f(\cdot)$ is an arbitrary function. By the data processing inequality [17], we have $I(X;f(Y))\leq I(X;Y)$, where $I(\cdot;\cdot)$ is the mutual information. Expanding the two mutual informations as follows: $I(X;f(Y))=H(X)-H(X|f(Y))$ and $I(X;Y)=H(X)-H(X|Y)$, and applying the data processing inequality, we conclude

$$H(X|Y)\leq H(X|f(Y)).$$

(2)

In video compression, coding modes are constructed via predictors, for example inter coding modes use other frames to predict the current frame, while intra coding modes use information from the same frame for prediction. Let $X=\mathbf{X}_{t}$ be the current frame, $Y=\{\mathbf{X}^{(1)},...,\mathbf{X}^{(n)}\}$ be a set of $n$ candidate predictors, and $\mathbf{X}_{p}=f(\mathbf{X}^{(1)},...,\mathbf{X}^{(n)})$ be a predictor for $\mathbf{X}_{t}$ from $\{\mathbf{X}^{(1)},...,\mathbf{X}^{(n)}\}$. Function $f(\cdot)$ could, for example, use different combinations of $\{\mathbf{X}^{(1)},...,\mathbf{X}^{(n)}\}$ in different regions of the frame. Further, let $f^{*}(\mathbf{X}^{(1)},...,\mathbf{X}^{(n)})$ be an optimal predictor for $\mathbf{X}_{t}$ that minimizes conditional entropy. Then, based on (1) and (2),

$$\begin{split}H(\mathbf{X}_{t}|\mathbf{X}^{(1)},...,\mathbf{X}^{(n)})&\stackrel{{\scriptstyle\text{(a)}}}{{\leq}}H(\mathbf{X}_{t}|f^{*}(\mathbf{X}^{(1)},...,\mathbf{X}^{(n)}))\\ &\stackrel{{\scriptstyle\text{(b)}}}{{\leq}}H(\mathbf{X}_{t}|f(\mathbf{X}^{(1)},...,\mathbf{X}^{(n)}))\\ &=H(\mathbf{X}_{t}|\mathbf{X}_{p})\stackrel{{\scriptstyle\text{(c)}}}{{\leq}}H(\mathbf{X}_{t}-\mathbf{X}_{p}),\end{split}$$

(3)

where (a) follows from (2), (b) follows from the fact that $f^{*}(\cdot)$ is the optimal predictor, and (c) follows from (1). Also note that if $m>n$, then

$$H(\mathbf{X}_{t}|f^{*}(\mathbf{X}^{(1)},...,\mathbf{X}^{(m)}))\leq H(\mathbf{X}_{t}|f^{*}(\mathbf{X}^{(1)},...,\mathbf{X}^{(n)})),$$

(4)

because an optimal predictor with more candidates ($m$) can, at the very least, choose to ignore $m-n$ of them and achieve the same performance as the predictor with $n$ candidates.

Our proposed codec is built on the idea shown in (4). By generating a large number of candidate predictors (modes), we want to minimize the bitrate needed for coding $\mathbf{X}_{t}$ conditioned on an optimal combination of these modes. Note that the theory promises even better performance via multi-conditional coding: inequality (a) in (3). However, this requires estimating conditional probabilities in very high-dimensional spaces, and is not pursued in this paper.

Fig. 1 depicts the structure of our proposed video compression system. It consists of three main components: 1) motion coder, 2) mode generator, and 3) inter-frame coder. The exact functionality of each of these components is described below.

Motion coder: Given the current frame $\mathbf{X}_{t}$ and its reconstructed reference frame $\widehat{\mathbf{X}}_{t-1}$, we first feed them to a learned optical flow estimation network like PWC-Net [13], to obtain a motion flow map $\mathbf{F}_{t}$. The obtained flow is then encoded by the encoder (AE) of the HyperPrior-based coder from [1], and the obtained motion bitstream is transmitted to the decoder. At the decoder side, the transmitted flow map is reconstructed by the decoder (AD) of the HyperPrior coder to obtain $\widehat{\mathbf{F}}_{t}$. Then, $\widehat{\mathbf{X}}_{t-1}$ is warped by $\widehat{\mathbf{F}}_{t}$ using bilinear sampling [16] to obtain a motion-compensated frame $\overline{\mathbf{X}}_{t}$.

The above-described motion coder is only used for the first P frame in each group of pictures (GOP). For the subsequent P frames, we use the CANF-based motion coder shown in Fig. 2. Here, the extrapolation network from CANF-VC [16] is used to extrapolate a flow map $\mathbf{F}_{c}$ from the three previously-decoded frames $\widehat{\mathbf{X}}_{t-3}$, $\widehat{\mathbf{X}}_{t-2}$, $\widehat{\mathbf{X}}_{t-1}$ and two decoded-flow maps $\widehat{\mathbf{F}}_{t-2}$, $\widehat{\mathbf{F}}_{t-1}$. $\mathbf{F}_{t}$ is then coded conditionally relative to $\mathbf{F}_{c}$. I-frames are coded using ANFIC [8].

Mode generator: The goal of the mode generator is to produce additional coding modes to operationalize (4). For this purpose, the previous reconstructed frame $\widehat{\mathbf{X}}_{t-1}$, the motion-compensated frame $\widehat{\mathbf{X}}_{t}$, and the decoded flow map $\widehat{\mathbf{F}}_{t}$ are concatenated and fed to the mode generator to produce two weight maps $\boldsymbol{\alpha}_{t}$ and $\boldsymbol{\beta}_{t}$, which are of the same size as the input image. The mode generator is implemented as a simple convolutional network with structure $[C_{1},R_{1},C_{2}]$, where $C_{1}$ and $C_{2}$ are convolutional layers with 32 kernels of size $3\times 3$ (stride=1, padding=1), and $R_{1}$ is a LeakyReLU layer whose negative slope is $0.1$. Since $\boldsymbol{\alpha}_{t}$ and $\boldsymbol{\beta}_{t}$ are produced from previously (de)coded data, they can be regenerated at the decoder without any additional bits. These two maps are then fed to a sigmoid layer to bound their values between 0 and 1. Then, a frame predictor $\widetilde{\mathbf{X}}_{t}$ is generated as:

$$\widetilde{\mathbf{X}}_{t}=\boldsymbol{\beta}_{t}\odot\overline{\mathbf{X}}_{t}+(\mathbf{1}-\boldsymbol{\beta}_{t})\odot\widehat{\mathbf{X}}_{t-1},$$

(5)

where $\odot$ denotes Hadamard (element-wise) product and $\mathbf{1}$ is the all-ones matrix. Moreover, $\boldsymbol{\alpha}_{t}$ is multiplied by both $\widetilde{\mathbf{X}}_{t}$ and $\mathbf{X}_{t}$, and the resultant two frames, $\boldsymbol{\alpha}_{t}\odot\widetilde{\mathbf{X}}_{t}$ and $\boldsymbol{\alpha}_{t}\odot\mathbf{X}_{t}$, are fed to the CANF-based conditional inter-frame coder for coding $\boldsymbol{\alpha}_{t}\odot\mathbf{X}_{t}$ conditioned on $\boldsymbol{\alpha}_{t}\odot\widetilde{\mathbf{X}}_{t}$. Note that $\boldsymbol{\alpha}_{t}$, $\boldsymbol{\beta}_{t}$, $\widehat{\mathbf{X}}_{t-1}$ and $\widetilde{\mathbf{X}}_{t}$ are available at the decoder.

Inter-frame coder: The inter-frame coder codes $\boldsymbol{\alpha}_{t}\odot\mathbf{X}_{t}$ conditioned on $\boldsymbol{\alpha}_{t}\odot\widetilde{\mathbf{X}}_{t}$ using the inter-frame coder of CANF-VC to obtain $\widecheck{\mathbf{X}}_{t}$ at the decoder. The final reconstruction of the current frame, i.e. $\widehat{\mathbf{X}}_{t}$, is then obtained by:

$$\widehat{\mathbf{X}}_{t}=\widecheck{\mathbf{X}}_{t}+(\mathbf{1}-\boldsymbol{\alpha}_{t})\odot\widetilde{\mathbf{X}}_{t}.$$

(6)

In the limiting case when $\boldsymbol{\beta}_{t}\to\mathbf{0}$, the predictor $\widetilde{\mathbf{X}}_{t}$ becomes equal to $\widehat{\mathbf{X}}_{t-1}$, and when $\boldsymbol{\beta}_{t}\to\mathbf{1}$, $\widetilde{\mathbf{X}}_{t}$ becomes equal to the motion-compensated frame $\overline{\mathbf{X}}_{t}$. For $\mathbf{0}<\boldsymbol{\beta}_{t}<\mathbf{1}$, the predictor $\widetilde{\mathbf{X}}_{t}$ is a pixel-wise mixture of $\widehat{\mathbf{X}}_{t-1}$ and $\overline{\mathbf{X}}_{t}$. Hence, $\boldsymbol{\beta}_{t}$ provides the system with more flexibility for choosing the predictor for each pixel within the current frame being coded. Also, for pixels where $\boldsymbol{\alpha}_{t}\to 0$, $\widehat{\mathbf{X}}_{t}$ becomes equal to $\widetilde{\mathbf{X}}_{t}$, so the inter-frame coder does not need to code anything. This resembles the SKIP mode in conventional coders, and depending on the value of $\boldsymbol{\beta}_{t}$, the system can directly copy from $\widehat{\mathbf{X}}_{t-1}$, $\overline{\mathbf{X}}_{t}$, or a mixture of these two, to obtain $\widehat{\mathbf{X}}_{t}$. When $\boldsymbol{\alpha}_{t}\to\mathbf{1}$, only the inter-frame coder is used to obtain $\widehat{\mathbf{X}}_{t}$. In the limiting case when $\boldsymbol{\alpha}_{t}\to\mathbf{1}$ and $\boldsymbol{\beta}_{t}\to\mathbf{1}$, the proposed method would reduce to CANF-VC [16].

Note that a somewhat similar approach was proposed in [14]. However, [14] used only one weight map, which is similar to our $\boldsymbol{\alpha}$ map, and this map was coded and transmitted to the decoder. In our proposed system, two maps, $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$, are used to create a larger number of modes. Moreover, these two maps can be constructed using previously (de)coded information, so they can be regenerated at the decoder without any additional bits to signal the coding modes.

We trained the proposed LCCM-VC on the VIMEO-90K Setuplet dataset [18], which consists of 91,701 7-frame sequences with fixed resolution $448\times 256$, extracted from 39K video clips. We randomly cropped these clips into $256\times 256$ patches, and used them for training LCCM-VC using a GOP of $N=5$ frames. We employed the Adam [19] optimizer with the batch size of 4. We adopted a two-stage training scheme. In the first stage, we froze the CANF-based conditional coders with their pre-trained weights, and optimized the remainder of the model for 5 epochs with the initial learning rate of $10^{-4}$. In the second stage, we trained the entire system end-to-end for 5 more epochs with the initial learning rate of $10^{-5}$. Four separate models were trained for four different bitrates using the following loss function:

$$\mathcal{L}=\sum_{i=1}^{N}\frac{\eta_{i}}{\sum_{j=1}^{N}\eta_{j}}\cdot\mathcal{L}_{i},$$

(7)

where $\eta_{i}=i$, and $\mathcal{L}_{i}$ is the RD loss of the $i$-th training frame defined in [16] with $\lambda_{1}\in\{256,512,1024,2048\}$ and $\lambda_{2}=0.01\cdot\lambda_{1}$. Note that [16] used $\mathcal{L}=\sum_{i}\mathcal{L}_{i}$ as the training loss, without weighting. In our experiments, we first trained the model with $\lambda_{1}=2048$ (highest rate), and all lower-rate models were then initialized from this model.

We evaluate the performance of LCCM-VC on three datasets commonly used in learning-based video coding: UVG [20] (7 sequences), MCL-JCV [21] (30 sequences), and HEVC Class B [22] (5 sequences). Following the common test protocol used in the recent literature [16], we encoded only the first 96 frames of the test videos, with the GOP size of 32.

We used the following video codecs as benchmarks: x265 (‘very slow’ mode) [23], HEVC Test Model (HM 16.22) with LDP profile [24] , M-LVC [25], DCVC [26], and CANF-VC [16]. As the quality of the I-frames has a significant role on the RD performance of video codecs, in order to have a fair comparison, we used ANFIC [8] as the I-frame coder for all learned codecs in the experiment. Note that ANFIC achieves state-of-the-art performance for static image coding [8].

Similar to the existing practice in the learned video coding literature [16, 25, 26], to evaluate the RD performance of various methods, the bitrates were measured in bits per pixel (BPP) and the reconstruction quality was measured by both RGB-PSNR and RGB-MS-SSIM. Then the RD performance is summarized into BD-Rate [27].

In Fig. 3, we plot RGB-PSNR vs. BPP (top) and RGB-MS-SSIM vs. BPP curves (bottom) of various codecs on the three datasets. It is notable that both LCCM-VC and CANF-VC achieve better performance than HM 16.22 at higher bitrates/qualities, whereas HM 16.22 has slight advantage at lower bitrates. All three codecs offer comparable performance at medium bitrates. Tables 1 and 2 show BD-rate (%) relative to the x265 anchor using RGB-PSNR and RGB-MS-SSIM, respectively, with negative values showing an average bit reduction (i.e., coding gain) relative to the anchor. The best result in each row is shown in blue, and the second best result is shown in red. As seen in the tables, LCCM-VC has the best RGB-PSNR results among learned codecs, and best RGB-MS-SSIM results overall.

It should be mentioned that choosing x265 as the anchor – in keeping with the established practice in learned video coding – has a consequence of excluding high-quality portions of the RD curves from BD-Rate computation. Yet, high quality is where LCCM-VC (and CANF-VC) have the biggest advantage over HM 16.22, as seen in Fig. 3. In fact, if we choose HM 16.22 as the anchor, then LCCM-VC has BD-Rates of –2.8%, 3.4%, and –1.1% (using RGB-PSNR), so in fact, it beats HM 16.22 on two out of three datasets even in terms of RGB-PSNR, and the gains are in the high-quality region.