Conditional and Residual Methods in Scalable Coding for Humans and Machines

Our theoretical analysis is performed in the lossless case to motivate the proposed baselines for our lossy approaches. In conditional coding, we model $H(Y_{b})+H(Y_{c}|Y_{t})$, having $H(Y_{c})$ as a lower bound:

	$\displaystyle H(Y_{c})$	$\displaystyle\leq H(Y_{c})+H(Y_{t}|Y_{c})=H(Y_{c},Y_{t})$
		$\displaystyle=H(Y_{t})+H(Y_{c}|Y_{t})\leq H(Y_{b})+H(Y_{c}|Y_{t}),$		(1)

where we used $H(Y_{t})\leq H(Y_{b})$ due to the data processing inequality. This bound is tight when $H(Y_{t}|Y_{c})=0$ and $H(Y_{b}|Y_{t})=0$, or equivalently, when $H(Y_{b})=I(Y_{c};Y_{t})$. This corresponds to a decrease of information in $H(Y_{c}|Y_{t})$ of $H(Y_{b})$. An upper bound is obtained by:

	$\displaystyle H(Y_{b})+H(Y_{c}|Y_{t})$	$\displaystyle=H(Y_{b})+H(Y_{c})-I(Y_{c};Y_{t})$
		$\displaystyle\leq H(Y_{b})+H(Y_{c}).$		(2)

This bound is tight when $I(Y_{c};Y_{t})=0$, which corresponds to $Y_{c}$ and $Y_{t}$ being independent.

We provide an upper baseline for the conditional approach by using a standalone enhancement representation $Y_{e}$ generated without relying on any side information, and measuring $\hat{H}(Y_{e})$, where $\hat{H}(\cdot)$ is an entropy estimate. As a lower baseline we use $\hat{H}(Y_{b})+\hat{H}(Y_{e})$. This is motivated by considering that $Y_{e}$ can be more efficient as a task representation than $Y_{c}$, and by the bounds in (3.1) and (3.1).

It has been shown that conditional coding is an upper bound of residual coding [16, 17]:

	$\displaystyle H(X|X_{p})$	$\displaystyle=H(X_{r}+X_{p}|X_{p})=H(X_{r}|X_{p})$		(3)
		$\displaystyle=H(X_{r})-I(X_{p};X_{r})\leq H(X_{r})$		(4)
		$\displaystyle=H(X|X_{p})+I(X_{p};X_{r}).$		(5)

Here (3) uses the fact that having observed $X_{p}$, the only uncertainty in $X_{r}+X_{p}$ is due to $X_{r}$. The inequality in (4) uses the non-negativity of mutual information. $H(X_{r})$ is rewritten in (5) using the definition of mutual information and once again the fact that given $X_{p}$, the only uncertainty in $X-X_{p}$ is due to $X$.

The term $I(X_{p};X_{r})$ in (4) acts as a penalty term on the residual formulation. To minimize it, the residual $X_{r}$ and the prediction $X_{p}$ must be as independent from each other as possible. This can be achieved when $X_{p}$ collapses values in $X$ so that $H(X_{r})$ decreases, or when $X_{p}$ produces a constant value for different values in $X$, reducing $H(X_{p})$. Reducing $H(X_{p})$ increases $H(X|X_{p})$, which in turn could have the adverse effect of increasing $H(X_{r})$, as shown in (5).

In our proposed method we train to minimize $D_{r}$ and $\hat{H}(Y_{r})$. By extension, we also minimize $\hat{H}(X_{r})$. As shown in (5), this reduces both $H(X|X_{p})$ and $I(X_{p};X_{r})$. Hence, this optimization procedure encourages the learnable function $h_{r}(\cdot)$ to create a representation $X_{p}$ that recovers the input $X$ as accurately as possible, while at the same time being as independent as possible from the resulting residual $X_{r}$.

Note that enforcing the similarity of $X_{p}$ and the original input $X$ may not be an optimal procedure, since even though such an optimization will decrease $H(X|X_{p})$, it may lead to an increase in $I(X_{p};X_{r})$. This explains why in preliminary experiments, we found that having a function $h_{r}(\cdot)$ that explicitly reconstructs $X$ does not perform as well as our proposed method.

Due to the previous considerations stemming from our proposed method, we find that $H(Y_{c}|Y_{t})=H(Y_{r})$ can be easier to achieve. This motivates us to compare $\hat{H}(Y_{b})+\hat{H}(Y_{r})$ against the same baselines used in the conditional approach.

To conditionally code a representation $Y_{c}$ that exploits as much information as possible from $Y_{t}$, we model the spatial and dimensional dependencies between and within representations using a CNN [18]. Our proposed entropy model strikes a balance between complexity and accuracy. We group channels with a fixed size $K$ [18]. Within each group, the same location across channels are processed in parallel, using as context all locations in the previous groups and all previous locations across all channels of the current group, within the receptive field of the convolutional layer. Similarly to [8], locations in the spatial domain are processed in a top-to-bottom, left-to-right fashion. The Markov property is enforced by a mask applied to the convolution kernels. Fig. 2(a) shows the kernel mask for a single output channel of a layer.

Unlike previous work, the CNN architecture of our entropy model has scalable residual connections and deeper layers with kernels sizes larger than one for its auto-regressive convolutions. This removes some of the modelling limitations imposed by similar entropy models. The CNN architecture has blocks of three layers in which the input channels are scaled up, transformed in a higher-dimensional space, and scaled down back to the original number of channels. The residual connections are introduced in-between these blocks, such that the inputs can be re-scaled differently across the channel dimension. To maintain the Markov property when the number of channels changes, the group sizes are re-scaled accordingly and the channels can only change in multiples of $M$. Fig. 2(b) shows the architecture overview of a single block in the CNN.

The conditional representation is available as another channel group and is transformed by the CNN in the same way as other groups, except that its context is restricted to the receptive field within that group. All elements of the conditioned representation have access to this group.

Similarly to [8] and more recent works, the predictions of the entropy model correspond to the means $W$ and scales $\Sigma$ of a univariate Gaussian distribution assigned to each element in the representation. The symbols are obtained by $Q=\lfloor Y-W\rceil$, and the corresponding probability is $P_{\mathcal{N}(\mathbf{0},\Sigma)}[|Q|\leq|Q|\pm\nicefrac{{1}}{{2}}]$. During training, the rounding operation is simulated by adding uniform noise $\mathcal{U}(-\nicefrac{{1}}{{2}},\nicefrac{{1}}{{2}})$.

As an architecture for learnable compression, we use a simplified version of the work in [11]. We drop the side information components from the coder, introduced as a hyper-prior in [8]. We also remove the attention layers introduced in [10]. To reduce the memory footprint and speed up the training procedure, we incrementally reduce the channels in the first layers of the analyzers and the last layers of the synthesizers.

The base and enhancement tasks have this same architecture. The base generates a representation with the same dimensionality and resolution as the input $X$ but reconstruction of the input is not enforced. This representation is the input for the computer vision model. The coder and the computer vision model are trained together end-to-end.

In the conditional approach, $h_{c}(\cdot)$ is a traditional CNN composed of blocks with residual connections between them. Each block has three convolutional layers that perform a transformation in a higher dimensionality and scales the output back to a lower dimensionality. The first half of the blocks maintain the same dimensionality as $Y_{b}$, while the second half transitions to the same dimensionality as $Y_{c}$, obtaining $Y_{t}$. The resolution is maintained across the network as both $Y_{b}$ and $Y_{c}$ have the same resolution. In the residual approach, $h_{r}(\cdot)$ uses the same architecture as our synthetizers, to transform $Y_{b}$ into $X_{p}$. The synthetizer upscales the representation to match the resolution and dimensionality of $X$.

A representation $Y_{b}$ that is optimized for $D_{b}$ can contain information that might not be beneficial for reconstruction on its own. Moreover, the information in $Y_{b}$ is represented in a way that is suitable for the computer vision task $T$ and bringing it all back to the image feature space through $h_{r}(\cdot)$ can be challenging. To overcome these obstacles, we add a small reconstruction penalty on a transformed $Y_{b}$ to the rate-distortion Lagrange minimization formulation:

$\displaystyle\mathcal{L}_{b}=D_{b}+\lambda_{b}\hat{H}(Y_{b})+\beta\,\mathbb{E}[d_{e}(\hat{h}_{r}(Y_{b}),X)],$

(6)

where $\hat{h}_{r}(\cdot)$ is an auxiliary network with the same architecture as the other syntherizers and $\lambda_{b}$ and $\beta$ are hyper-parameters. For the enhancement representations, we use the traditional rate-distortion loss function [6]:

$\displaystyle\mathcal{L}_{c}=D_{c}+\lambda_{e}\hat{H}(Y_{c}|Y_{t}),$

$\displaystyle\mathcal{L}_{r}=D_{r}+\lambda_{r}\hat{H}(Y_{r}),$

for the conditional and residual approaches, respectively. During training, either the base network remains frozen or the gradients from the reconstruction network do not flow into the base network.

Cityscapes is a set of images of urban scenes for semantic understanding [20]. We use DeepLabV3 [22] as the computer vision model for segementation with MobileNetV3 [23] as a back-end. Here, $d_{b}(\cdot,\cdot)$ is the per-pixel multi-class cross-entropy, although we report the mean intersection over union (mIoU) metric. We set $\beta=0.1$ and report the results on the validation dataset. For data augmentation, we use random crops of $768\times 768$ pixels, random horizontal flips and color jittering. The front-end of the model corresponding to the coder is trained with Adam using a learning rate of $10^{-4}$, while the classifier is trained with stochastic gradient descent using momentum and a learning rate of $10^{-2}$. A $\ell_{2}$ loss is added to the weights of the classifier to prevent over-fitting, with a scale factor of $10^{-4}$.

Fig. 3(a) shows the rate-distortion curves for the conditional and residual approach. We notice that these lines lie in between the baselines, producing rate-distortion points that respect them. Compared to the rate-distortion curve of the lower baseline, the conditional approach has a BD-Rate [24] of $-16.56\%$, whereas the residual approach achieves a $-14.6\%$ rate reduction. Thus, the conditional approach performs marginally better than the residual approach in terms of BD-Rate. Looking at the ratio between these BD-Rate scores and the BD-Rate score achieved by the upper baseline, we can compute the percentage of the base representation utilized. As such, the conditional approach uses $43.01\%$ of the side information rate, whereas the residual approach uses $37.91\%$. In the lowest-compression settings under both approaches, the utilization is higher.

Fig. 3(c) shows the rate-distortion performance of the base task. The chosen $\beta$ value places a penalty on both the rate and the task performance but allows the base representation to be exploited by the architecture. We attribute the small imperfections in this rate-distortion curve to the choice of $\beta$ and the limitations of the training algorithm.

COCO 2017 has 123,287 domain-agnostic images for object detection and segmentation [21]. We use Faster R-CNN [25] for object detection with ResNet-50 [26] as the back-end. For this task, $d_{b}(\cdot,\cdot)$ is the sum of the different loss functions employed by this architecture. We report the mean average precision (mAP) metric computed according to [21], and set $\beta=0.05$. As data preprocessing for training, we use random horizontal flips and generate batches with similar aspect ratios grouped in 3 clusters. Images inside a batch are resized to the minimum size and the bounding boxes are adjusted accordingly when training the computer vision task. When training for reconstruction, the images in a batch are center-cropped to their minimum size. All weights are trained with Adam using a learning rate of $10^{-4}$.

As shown in Fig. 3(b), the performance of both approaches is comparable, with a $-4.14\%$ and a $-2.47\%$ BD-Rate improvement over the lower baseline, for the conditional and residual methods, respectively. We achieve a utilization of the base in terms of BD-Rate of 49.24% and 29.32% for the conditional and residual approaches, respectively. Fig. 3(d) shows the base-distortion performance of the base task. Compared to semantic segmentation on Cityscapes, the task model is better at reaching the uncompressed task performance. Also, for a similar distortion penalty, this task uses more rate. The rate-distortion curves obtained by the reconstruction task on the COCO dataset are almost an order of magnitude larger than the ones from Cityscapes. This can be explained by the simplicity of the content in the images found in Cityscapes, and the higher amount of artifacts found in the COCO dataset due to compression.