Scalable Image Coding for Humans and Machines

The processing pipeline described by (1) and (2) forms a Markov chain $\mathbf{X}\to\widehat{\mathbfcal{Y}}\to\widehat{\mathbf{X}}\to\mathbfcal{F}\to T$. This chain is illustrated in Fig. 3 (top), where the inference task $T$ is object detection. Applying the data processing inequality (DPI) [39] to the Markov chain, we get

where $I(\cdot\,;\,\cdot)$ denotes mutual information [39]. We can draw two conclusions from this analysis. First, the latent representation $\widehat{\mathbfcal{Y}}$ contains enough information to compute $T$, because it contains enough information to reconstruct $\widehat{\mathbf{X}}$, from which $T$ can be computed. Second, $\widehat{\mathbfcal{Y}}$ carries less information about $T$ than it does about $\widehat{\mathbf{X}}$. This motivates our approach to latent-space scalability - we construct $\widehat{\mathbfcal{Y}}$ in such a way that only part of it is used to compute $T$, while all of it is used to reconstruct $\widehat{\mathbf{X}}$. Note that the latent space $\widehat{\mathbfcal{Y}}$ is not the same as $\mathbfcal{F}$ (the space from which $V^{(\textup{back-end})}$ computes $T$), so we will also need to construct the latent space transform from $\widehat{\mathbfcal{Y}}$ to $\mathbfcal{F}$ in order to bypass $\widehat{\mathbf{X}}$ when only $T$ is needed. Details of the latent space transforms will be discussed in Section III-C.

We will present two embodiments of the above idea: a 2-task (2-layer) system, and a 3-task (3-layer) system. The 2-layer system supports object detection from the base layer and input reconstruction from the full latent space, as explained above. The 3-layer system supports object detection ($T_{1}$), object segmentation ($T_{2}$), and input reconstruction ($\widehat{\mathbf{X}}$). The processing chain for such a system can be represented as $\mathbf{X}\to\widehat{\mathbfcal{Y}}\to\widehat{\mathbf{X}}\to\mathbfcal{F}\to T_{2}\to T_{1}$, and is illustrated in Fig. 3 (bottom). Note that it is possible to derive object detection results ($T_{1}$) from segmentation results ($T_{2}$) by simply placing bounding boxes around segmented objects and copying the object class. This implies $I(\widehat{\mathbfcal{Y}};T_{2})\geq I(\widehat{\mathbfcal{Y}};T_{1})$, and moreover, information required for object detection ($T_{1}$) is a subset of that required for segmentation ($T_{2}$). For this reason, in our 3-layer system, object detection is placed in base layer, segmentation is supported by the base and the first enhancement layer, while the whole latent space supports input reconstruction.

The processing chains in Fig. 3 are shown only to illustrate the DPI, and explain the resulting design of the latent space(s) in our systems. In practice, there is no need to reconstruct the input image $\widehat{\mathbf{X}}$ if one wants only to perform object detection or segmentation. This can be achieved directly from the corresponding portion of the latent space through the appropriate latent space transform, as discussed in Section III-C.

Fig. 4 shows an embodiment of our 2-task (2-layer) system. In particular, we adopt the baseline network [16] using mean and scale hyperprior to capture spatial dependencies in $\mathbfcal{Y}$, estimated by entropy parameter (EP) module and context model (CTX) for arithmetic encoder/decoder (AE/AD). Also, we adopt the configurations for Analysis Encoder, Synthesis Decoder, and Hyper Analysis/Synthesis without attention layers from [16]. The Analysis Encoder transforms the input image $\mathbf{X}$ into $\mathbfcal{Y}\in\mathbb{R}^{N\times M\times C}$, with $C=192$ as in [15, 16]. The latent representation $\mathbfcal{Y}$ is split into sub-latents $\{\mathbfcal{Y}_{1},\mathbfcal{Y}_{2}\}$, where $\mathbfcal{Y}_{1}=\{\mathbf{Y}_{1},\mathbf{Y}_{2},...\mathbf{Y}_{i}\}$ represents the base layer and $\mathbfcal{Y}_{2}=\{\mathbf{Y}_{i+1},\mathbf{Y}_{i+2},...,\mathbf{Y}_{C}\}$ represents the enhancement layer. The quantized sub-latent $\widehat{\mathbfcal{Y}}_{j}$ is coded using its own context model (CTX_m,j) and entropy parameter module (EP_j). The side bitstream, which contains coded hyper-parameters, is used by both layers.

Details of entropy coding related-blocks along with dimensions of relevant tensors are presented in Fig. 5. Let $\widehat{\mathbfcal{Y}}_{j}\in\mathbb{R}^{N\times M\times L_{j}}$ be the $j$-th quantized sub-latent, where $L_{j}$ is the number of channels in the $j$-th sub-latent. CTX_m,j is the context model for $\widehat{\mathbfcal{Y}}_{j}$ associated with a masked convolutional layer [15, 16] with $5\times 5$ kernels (Masked Conv (5) in the figure), to produce an output tensor with $2L_{j}$ channels. This output tensor is concatenated with $\mathbfcal{H}_{j}\in\mathbb{R}^{N\times M\times 2L_{j}}$, which is the part of Hyper Synthesis output $\mathbfcal{H}\in\mathbb{R}^{N\times M\times 2C}$ corresponding to $\widehat{\mathbfcal{Y}}_{j}$. The concatenated tensor is input to EP_j, which consists of three convolutional layers with $3\times 3$ kernels (Conv (3) in the figure), to estimate Gaussian means and variances for subsequent entropy coding. Quantized hyperpriors for all sub-latents (input to Hyper Synthesis) are encoded as side bitstream. We experimentally observed that the side bitstream accounts for 2–3% of total bits, on average.

At the decoder, hyperpriors are reconstructed from the side bitstream. The base-layer sub-latent $\widehat{\mathbfcal{Y}}_{1}$ is reconstructed from the the base bitstream using the hyperpriors. $\widehat{\mathbfcal{Y}}_{1}$ is sufficient for object detection. In case input reconstruction is also needed, $\widehat{\mathbfcal{Y}}_{2}$ is reconstructed from the enhancement bitstream using the hyperpriors, and the input image is reconstructed from $\widehat{\mathbfcal{Y}}_{1}\cup\widehat{\mathbfcal{Y}}_{2}$.

Although this example involves a 2-task (2-layer) system for simplicity of explanation, it is possible to construct scalable systems with more layers by following the same principles. In fact, in Section IV, we also demonstrate a 3-task (3-layer) system that supports object detection (base layer), object segmentation (first enhancement layer) and input reconstruction (second enhancement layer). In this system, the latent space $\widehat{\mathbfcal{Y}}$ is partitioned into three parts: $\{\widehat{\mathbfcal{Y}}_{1},\widehat{\mathbfcal{Y}}_{2},\widehat{\mathbfcal{Y}}_{3}\}$, where $\widehat{\mathbfcal{Y}}_{1}$ supports object detection, $\widehat{\mathbfcal{Y}}_{1}\cup\widehat{\mathbfcal{Y}}_{2}$ supports object segmentation, and the whole latent space $\widehat{\mathbfcal{Y}}_{1}\cup\widehat{\mathbfcal{Y}}_{2}\cup\widehat{\mathbfcal{Y}}_{3}$ supports input reconstruction.

The base layer $\widehat{\mathbfcal{Y}}_{1}$ will support the base computer vision task, say object detection, for example. However, $\widehat{\mathbfcal{Y}}_{1}$, in general, does not match any of intermediate latent spaces of any of the object detectors. Hence, we introduce the latent space transform (LST) to map $\widehat{\mathbfcal{Y}}_{1}$ to an intermediate latent space of an object detector we want to use at the decoder. Specifically, let $\mathbfcal{F}_{1}^{(l)}$ be the feature tensor at the output of the $l$-th layer of a chosen object detector $V_{1}$, that is, $\mathbfcal{F}_{1}^{(l)}=V_{1}^{(\textup{front-end})}(\widehat{\mathbf{X}},\psi)$, as in (2). Then the goal of the base LST is to map $\widehat{\mathbfcal{Y}}_{1}$ to $\mathbfcal{F}_{1}^{(l)}$. More generally, for the $j$-th task, we use $\{\widehat{\mathbfcal{Y}}_{1},...,\widehat{\mathbfcal{Y}}_{j}\}$ from the encoded latent space, and the goal of $j$-th LST is to map $\{\widehat{\mathbfcal{Y}}_{1},...,\widehat{\mathbfcal{Y}}_{j}\}$ to $\mathbfcal{F}_{j}^{(l)}=V_{j}^{(\textup{front-end})}(\widehat{\mathbf{X}},\psi)$, where $V_{j}$ is the DNN implementing the $j$-th task.

Fig. 6 shows the structure of our LST, which consists of residual blocks (RBs) similar to those in the Synthesis Decoder [16]. A RB consists of two convolutional layers with $3\times 3$ kernels (Conv (3) in the figure) followed by Leaky ReLU. A RB with upsampling factor $r_{k}$ (shown as $\uparrow r_{k}$) increases spatial dimension by $r_{k}$ and also applies the inverse GDN [40]. Upsampling factors $r_{k}$ are chosen based on the dimensions of $\{\widehat{\mathbfcal{Y}}_{1},...,\widehat{\mathbfcal{Y}}_{j}\}$ and $\mathbfcal{F}_{j}^{(l)}$. The last activation function is set to be the same as for the output of $V_{j}^{(\textup{front-end})}$, to produce a similar dynamic range for the output tensor. The output of the LST is $\widetilde{\mathbfcal{F}}^{(l)}_{j}$, an approximation to the desired $\mathbfcal{F}_{j}^{(l)}$, and the LST is trained to minimize their difference.

In this section we provide further insight into the operation of the proposed scalable framework from an information-theoretic perspective. Empirical information measurements are performed on a 2-layer system whose base task is object detection using a YOLOv3 [27] back-end, and the enhancement task is input reconstruction. The LST maps the base-layer features $\mathbfcal{Y}_{1}$ to $\widetilde{\mathbfcal{F}}_{1}^{(13)}$ of YOLOv3. The network was trained for 300 epochs with $\gamma_{1}=0.006$ in (6). Other training details are presented in Section IV-A.

Entropy estimates of the base features $\mathbfcal{Y}_{1}$ and the entire latent representation $\mathbfcal{Y}$ are obtained using rate estimates [15]

Fig. 7 shows the results. The right-hand side Y-axis shows $H(\mathbfcal{Y})$ in bits per pixel (bpp), computed as total bits divided by input image resolution. The left-hand side Y-axis shows the percentage of object detection information out of the total rate, that is

Fig. 7(a) shows the results for $\lambda=0.0483$ in (4), while Fig. 7(b) shows the results for $\lambda=0.0035$. In both cases, the number of base channels $i\in\{64,96,128\}$.

From Fig. 7, it can be seen that as $\lambda$ decreases from $0.0483$ in Fig. 7(a) to $0.0035$ in Fig. 7(b), the total rate reduces from the range $[1.45,1.55]$ bpp down to $[1.10,1.17]$ bpp. This is clear from (4) because, as $\lambda$ decreases, the importance of $R$ in the Lagrangian cost increases, so there is more pressure to reduce the rate. The same happens with the object detection information rate. With $i=128$ feature channels dedicated to object detection, in Fig. 7(a) the object detection information rate is approximately $0.8\cdot 1.45=1.16$ bpp, and this falls down to approximately $0.92\cdot 1.13=1.04$ bpp in Fig. 7(a), after 300 epochs. The same trend holds for other $i$’s.

We can also notice that the fraction of the total rate devoted to the base layer does not scale proportionally to the fraction of the latent space occupied by the base layer. For example, when the number of base layer channels halves from $i=128$ to $i=64$, the percentage of the total rate it occupies only reduces from $80\%$ to $55\%$ in Fig. 7(a), and from from $92\%$ to $82\%$ in Fig. 7(b). This is due to the fact that the base layer serves two purposes - it is responsible for object detection, and also plays a role in input reconstruction. Hence, through RD optimization, it receives more rate than would be expected from the fraction of the latent space it occupies.

It is well known that scalable extensions of conventional codecs, such as H.264/AVC [41] and H.265/HEVC [42], suffer a rate increase of about 15-25% per enhancement layer compared to single-layer coding [43] in terms of the rate-distortion performance. This is due to information redundancy between layers. It is natural to ask whether such redundancy exists in our scalable coding framework? Ideally, $I(\widehat{\mathbfcal{Y}}_{1};\widehat{\mathbfcal{Y}}_{2})=0$, meaning that base and enhancement layer are independent, and they can be coded separately without loss of coding efficiency. If this were the case, then because $\widehat{\mathbfcal{Y}}_{1}\to\widetilde{\mathbfcal{F}}_{1}^{(13)}$ through LST, we would have $I(\widehat{\mathbfcal{Y}}_{2};\widetilde{\mathbfcal{F}}_{1}^{(13)})=0$. To test this, we measure the mutual information between $\widehat{\mathbfcal{Y}}_{j},j\in\{1,2\}$ and $\widetilde{\mathbfcal{F}}_{1}^{(13)}$. Mutual information measurements are obtained using the Kernel Density Estimator (KDE) from [44], as follows.

We first divide sub-latent tensors $\widehat{\mathbfcal{Y}}_{j}$ into fibers $\widetilde{\mathbfcal{Y}}_{j}\in\mathbb{R}^{1\times 1\times L_{j}}$, where $L_{1}=128$ and $L_{2}=64$. Also $\widetilde{\mathbfcal{F}}_{1}^{(13)}$ is divided into spatially corresponding fibers with the size of $2\times 2\times 256$, as illustrated in Fig. 8. Then, we cluster the estimated output feature fibers into $K=16$ clusters (similarly to [45]) with cluster index denoted $\overline{F}_{1}$, and then compute

Due to clustering, $I(\widetilde{\mathbfcal{Y}}_{j};\overline{F}_{1})$ is an underestimate of $I(\widetilde{\mathbfcal{Y}}_{j};\widetilde{\mathbfcal{F}}_{1}^{(13)})$, but the measurements still provide useful insight into the behavior of the system. Fig. 9 shows the evolution of the estimated mutual information over 400 epochs for $\lambda=0.0483$ and $i=128$. Since $\widehat{\mathbfcal{Y}}_{1}\to\widetilde{\mathbfcal{F}}_{1}^{(13)}$, we see that $I(\widetilde{\mathbfcal{Y}}_{1};\overline{F}_{1})$ remains relatively high, around 3.76 bits per fiber, throughout the 400 epochs. Meanwhile, $I(\widetilde{\mathbfcal{Y}}_{2};\overline{F}_{1})$ reduces down to 1.20 bits per fibre, showing that the enhancement layer carries less and less information about object detection as the training goes on. While $I(\widetilde{\mathbfcal{Y}}_{2};\overline{F}_{1})$ does not reach the ideal value of zero, the decreasing trend is evident, which confirms that the loss function (4)-(6) provides appropriate information steering to various portions of the latent space. At the same time, since the base and enhancement latents are coded independently, a non-zero mutual information between them is an indication of inefficiency of scalable coding. Indeed, as will be seen in Section IV-C, in terms of input reconstruction, our 2-layer system is less efficient than [16], while our 3-layer system is less efficient than our 2-layer system, even though all three systems share the same encoder architecture. This loss of efficiency due to scalability has also been observed in earlier studies on scalable coding [43], so it is not a new phenomenon. We believe our approach of looking at the problem through the lens of mutual information will contribute to further advances in this area.

One may also ask what happens if a new task needs to be added to the system post-hoc, after the initial system has already been trained? This is a challenge for all coding-for-machine approaches, because the features trained for initial tasks may not be appropriate for the new task. In general, some modifications need to be made, but there are several possibilities, depending on how efficient the new system needs to be and whether or not it is possible to retrain the encoder. Four our proposed systems, the possibilities are as follows:

1. 1.

   If the new task happens to be a subset of an existing task,²²2For example, an existing task is detection of 10 different breads of dogs and 5 breeds of cats, while the new task is just detecting ‘dogs’ and ‘cats’. then one can simply train a new LST from the portion of the latent space dedicated to the previous task into the latent space of the new back-end, because all the information is already available there. In this case, encoder does not need to be retrained, only the LST.

2. 2.

   If the new task is not a subset of any of the previous tasks, one could retrain the encoder to include the new task and allocate a portion of the latent space to it.

3. 3.

   As shown in Section III-A, any task $T$ is a subset of the input reconstruction task. Therefore, one can simply train a LST from the full latent space into the latent space of the new back-end. This would avoid retraining the encoder, but the new task would likely operate at a higher bitrate compared to approach 2) above, because the full latent space needs to be decoded.

Our multi-task networks are trained in two stages. In the first stage, the networks are trained on CLIC [20] and JPEG-AI [47] datasets. Randomly cropped patches with size of $256\times 256$ from both datasets are used as input. We also set the mini-batch size to 16. Adam optimizer with a fixed learning rate of $10^{-4}$ is used for 400 epochs on a GeForce RTX 2080 GPU with 11 GB RAM. Then, we change the dataset to VIMEO-90K [48] to continue the training for another 300-400 epochs in the second stage. Likewise, randomly cropped patches are drawn from the dataset, but this time the learning rate decreases with polynomial decay for every 10 epochs. Six values of $\lambda$, shown in Table I, are used in (4) to produce six versions of the trained networks, as in [49]. The total number of latent-space channels is set to $C=192$ in each case, as in [14, 15, 16]. Specific details of the 2- and 3-layer networks are given below.

Our multi-task networks are evaluated on relevant datasets associated with targeted tasks, in terms of task accuracy vs. bitrate.

Benchmarks: We compare our proposed networks against several benchmarks, including the latest standard codecs such as HEVC [42] and VVC [51], as well as five recent DNN-based image codecs [15, 16, 17, 18]. Specifically, the reference software versions for HEVC and VVC are HM-16.20 [52] and VTM-12.3 [53], and are used in all intra configuration [54, 55] with quantization parameters QP $\in\{22,25,28,...,40\}$. To encode RGB images using the standard codecs, these images are first converted to YUV444 in accordance with ITU-R BT.709 [56], then the chrominance channels are sub-sampled to create the YUV420 version of the image, which can then be fed to an HEVC or VVC encoder. After decoding, chrominance channels of the YUV420 output are interpolated to YUV444 using bilinear interpolation, then converted to RGB in accordance with ITU-R BT.709 [56]. DNN-based codecs are trained to compress RGB images directly, so no conversion to YUV is needed. Finally, decoded RGB images are used as input to the pre-trained computer vision models³³3In each case, the same model whose back-end is used in our multi-task network. to examine task-specific accuracy.

Two-layer network: Our two-layer network supports object detection in the base layer using the YOLOv3 [27] back-end. We first evaluate the object detection performance on the COCO2014 validation set [57], which includes about 5,000 images. Since most vision networks resize the input to a specific resolution before processing, we also resize input images to $512\times 512$ using bilinear interpolation without letterboxing, in order to generate (via LST) a feature tensor $\widetilde{\mathbfcal{F}}^{(13)}_{1}\in\mathbb{R}^{64\times 64\times 256}$ that can be directly fed into the YOLOv3 back-end. The same resizing is done with benchmark image codecs.

Fig. 11 presents a comparison of our object detection performance against various benchmarks in terms of bitrate in bits per pixel (bpp) vs. mean Average Precision (mAP), where bpp is computed by dividing the total number of coded bits (base and side bitstreams) by the number of input pixels. The mAP uses the Intersection of Union (IoU) threshold of 0.5. In the figure, the black dashed line shows the default mAP performance of 55.85% when using the COCO2014 validation images as input to YOLOv3 [27] with pre-trained weights. The average bitrate of the images in the COCO2014 validation set, coded with JPEG, is 4.80 bpp, which is indicated in the legend in the figure. Our object detection achieves the best rate-accuracy performance, with mAP loss of about 1% at 0.74 bpp, where most benchmarks suffer a mAP loss of about 2%. Moreover, at 0.56 bpp, our method operates within a 2% mAP loss margin, whereas most benchmarks have lost about 4% mAP at this point. Cheng et al. [16] shows the best performance among the benchmarks, but there is still significant gap between it and our proposed method.

Table II (first three columns) summarizes object detection vs. bitrate results using extended versions of Bjøntegaard Delta (BD) metric [58, 59]. For the BD-mAP metric, positive numbers represent an average increase of mAP at the same bitrate. For BD-rate, negative numbers indicate average bit savings at the same accuracy. Our method shows a noticeable bit savings and increased accuracy compared to all benchmarks. For example, against HEVC, we achieve BD-rate savings of –47.9%, and BD-mAP gain of 4.55%. Against Cheng et al.[16], we achieve BD-rate savings of –37.4%, and BD-mAP gain of 2.89%. The greatest bit reduction of 41.3% is accomplished against Minnen et al.[15], also marginally less significant reductions are achieved against other DNN-based networks[17, 18].

Three-layer network: The three-layer network supports object detection in the base layer and object segmentation in the first enhancement layer. The corresponding back-ends use ResNet-50-based Faster [46] and Mask [28] R-CNN with pre-trained weights [50], respectively. Since the benchmarks’ performance of two tasks using those R-CNNs are evaluated on the COCO2017 validation set [57] and publicly reported via Detectron2 [50], we use the same dataset for fair evaluation and comparison with our proposed network. Faster R-CNN and Mask R-CNN have a constraint on the input resolution that the shorter edge must be less than or equal to 800 pixels according to given default configuration. Hence, we resize the test images to meet the constraint using bilinear interpolation prior to the experiment, then use the resized images as input to our three-layer network. As a result, generated feature tensors $\widetilde{\mathbfcal{F}}_{1}^{(4)}$ and $\widetilde{\mathbfcal{F}}_{2}^{(4)}$ from the base and the first enhancement layer can be fed into the Faster and Mask R-CNN back-ends, respectively. The resized images are also used as input to benchmark image codecs.

Fig. 12 shows the performance of our three layer network and the benchmarks on both tasks. In Fig. 12(a) and (b), dashed black lines show the default mAP performance of 40.2% and 37.2% at 4.80 bpp on average⁴⁴4Although the COCO2014 and COCO2017 validation sets are not identical, the average bitrate of images in each set, coded by JPEG, is 4.80 bpp. for object detection and segmentation, respectively. Here, mAP is obtained using the IoU threshold from 0.5 to 0.95 with a step size of 0.05. On both tasks, our 3-layer network shows excellent performance: less than 1% mAP drop down to 0.15 bpp, and less than about 1.5% mAP drop at 0.1 bpp. Meanwhile, all benchmarks lose 1% mAP already at 0.4 bpp, while at 0.1 bpp, they have lost 7-8% mAP.

Table II (last four columns) shows object detection and segmentation performance vs. bitrate in terms of extended versions of BD metrics [59]. Here, the gains of our network against the benchmarks are even higher. Against VVC, which was the best-performing benchmark, our network achieves BD-rate savings of –73.2% on object detection and –71.2% on object segmentation. At the same time, the BD-mAP gains against VVC are 2.33% on object detection and 2.34% on object segmentation.

The highest enhancement layer of our 2- and 3-layer networks supports input reconstruction for human viewing. The performance here is examined on the Kodak dataset [60], which includes 24 uncompressed RGB images.

Benchmarks: Our network performance is compared with the benchmarks, which include standard codecs [42, 51] and DNN-based codecs [15, 16, 17, 18], all optimized for MSE. The results reported here for HEVC and VVC are borrowed from CompressAI [49]. For HEVC [42] and VVC [51], HM-16.20 [52] with range extension profile and VTM-9.1 [61] were used, respectively. RGB images were first converted to YUV444 in accordance with BT.709 [56], then directly input to the codecs. Decoded images in the YUV444 format were converted back to RGB in accordance with BT.709 [56]. Since each input image is coded separately, all intra configurations following the common test conditions [54, 55] was used. Images were coded using various QPs from 12 to 47 with a step size of 5, and we show the RD points that cover the range of bitrates acheived by DNN-based codecs. We also borrowed JPEG results within this bitrate range from CompressAI [49].

Fig. 13(a) shows PSNR (RGB) vs. bitrate curves. VVC achieves the best performance among all methods in this figure. The method from which our backbone is derived, Cheng et al. [16], achieves the second-best performance. Our 2-layer network shows competitive performance compared to [16] at lower bitrates, but the reconstruction quality slightly degrades (by about 0.3dB) compared to [16] at higher bitrates. This is the price to pay for scalability and supporting object detection in the base layer. Compared to [17, 18] and other conventional codecs, our network still outperforms, while simultaneously supporting scalability. Table III shows BD-rate [58, 59] comparisons among various codecs. Overall, our 2-layer network has a loss of 10.17% against VVC and 4.49% against [16], but saves –3.58%, –5.58%, –7.01%, –14.27%, and –63.99% of bits compared to Minnen et al. [15], Hu et al. [17], Lee et al. [18], HEVC, and JPEG, respectively, while providing scalability.

Our 3-layer network is less efficient in terms of rate-distortion performance on input reconstruction. As shown in the last row of Table III, the 3-layer network suffers an increase of 18.84% BD-rate compared to the 2-layer network. Nonetheless, it is worthwhile to remark that earlier work on conventional scalable codecs suggests that adding one scalability layer costs about 15-25% in terms of BD-rate [43], so the performance of our 3-layer network seems reasonable in this context. It is still comparable with HEVC (1.38% increase in BD-rate) and much better than JPEG, while providing 3-layer scalability and superior efficiency on computer vision tasks. Overall, Fig. 13(a) shows that both our 2- and 3-layer networks are comparable with state of the art codecs in terms of PSNR at lower bitrates, but their relative PSNR performance degrades at higher bitrates.

In addition to PSNR results, we also provide performance comparisons in terms of MS-SSIM[62] vs. bitrate in Fig. 13(b). It is known that DNN-based codecs perform very well on MS-SSIM (even if they are optimized for MSE), and this is also evident in Fig. 13(b) and Table III. In particular, our 2-layer network achieves the best results in this comparison, showing coding gains against all benchmarks, with –1.9% savings compared to the best benchmark [16]. Even our 3-layer network now shows a gain of almost –18% relative to HEVC, and comparable performance (with a gap around 2%) relative to VVC and [15]. We believe this improved performance of our networks in terms of MS-SSIM is due to better latent representations related to objects features associated with machine vision tasks in the lower layers, which encourages higher structural quality of the reconstructed input.

Finally, we consider a comparison with [35]. While direct comparison is not possible since the tasks (and the corresponding image datasets) are different, an indirect comparison via BPG (HEVC Intra) is possible based on presented results. According to Fig. 9 in [35], when it comes to input reconstruction, the approach in [35] is less efficient than HEVC in terms of PSNR, and comparable to HEVC on MS-SSIM. On the other hand, from Table III, our 2-layer network is 14.27% more efficient than HEVC on PSNR, while our 3-layer network is slightly (about 1.3%) less efficient than HEVC in terms of PSNR. Meanwhile, both our networks are significantly more efficient than HEVC Intra on MS-SSIM. Hence, it appears that our networks are more efficient than [35] in terms of input reconstruction.

We compare the run time complexity of our methods with DNN-based benchmarks [15, 16, 17, 18] by measuring the average encoding and decoding time per image on a GeForce RTX 2080 GPU with 11 GiB RAM. We evaluate the DNN-based codecs on the Kodak dataset [60]. Our methods target the input reconstruction at the highest enhancement layer, but entail extra computation related to sub-latent coding in the lower layer(s) for machine vision task(s). Meanwhile, the benchmarks target only input reconstruction. Three benchmarks[15, 16, 17] and ours are implemented based on CompressAI [49], while [18] is developed using Tensorflow-Compression [63]. All models run on a single GPU, but some of the computation for some models [16, 18], including ours, has to run on a CPU (e.g., auto-regressive context modelling for entropy coding), which slows the system down.

Table IV presents run-time complexity of the models, and ours has the second-largest encoding and decoding time, after [18]. Compared to [16], which is the baseline network we used as the backbone for our models, run-time complexity increases in accordance (roughly proportionally) with the number of layers. This seems reasonable. Nonetheless, complexity is an issue for all DNN-based codecs, including ours.

In the proposed approach to latent space scalability, the LST plays a crucial role of transforming a portion of the encoder’s latent space into another latent space containing task-relevant features. In this section, we perform an ablation study on the LST in our 2-layer network, by gradually removing residual blocks (RBs, shown in Fig. 6) and examining the impact of such removal on both outputs of the network. The study is performed on the highest-rate 2-layer model (${\lambda}=0.0483$) for the first 400 epochs of training. The results are shown in Fig. 14. Fig. 14(a) shows the feature distortion in the target latent space (which is the latent space of YOLOv3) for four cases: LST with one set of RBs,⁵⁵5One “set of RBs” consists of one RB and one RB w/ Upsample, illustrated in Fig. 6. LST with two sets of RBs, LST with three sets of RBS, and the proposed LS, which has four sets of RBs. As seen in the figure, increasing the number of RBs improves the ability of the LST to match the target features of YOLOv3 and reduces the distortion in the target latent space. With four sets of RBs (the proposed LST), we seem to have reached diminishing returns in terms of performance vs. complexity, so four sets of RBs seems to be a reasonable choice for the LST.

Fig. 14(b) shows the impact of LST ablation on input reconstruction. Even though the LST is in the object detection processing pipeline, since the network is trained end-to-end, a change in the LST has an impact on input reconstruction as well. We see that the performance of input reconstruction improves as the number of sets of RBs increases to four, which is the proposed LST. In essence, the ability of LST to better extract object detection-relevant features from the base layer helps the system better distribute the information between the base and enhancement layers, thereby improving both tasks.

Fig. 15 shows the bitsteam composition of six versions of our networks, from those trained for the lowest bitrate (quality index 1), to those trained for the highest bitrate (quality index 6). For each index, left bar corresponds to the 2-layer network, and the right bar to the 3-layer network. There are relatively few bits in the side bitstream (blue), which accounts for less than 2% of the total bitstream. The base layer (green) accounts for most bits, and its fraction drops from over 93% at lowest bitrates to less than 85% at highest bitrates for the 2-layer network, and less than 70% for the 3-layer network. The first enhancement layer (orange, denoted “Enh.” in the figure) and the second enhancement layer in the 3-layer network (purple, denoted “Top”) take up progressively more bits at higher bitrates. Considering the fact that our networks show more competitive performance of input reconstruction at lower bitrates (Fig. 13), these results seem to indicate that the base layer conveys significant information for input reconstruction as well, in addition to enabling object detection.

One of the key contributions of the present paper is that our scalable DNN-based image coding approach is able to support birate-efficient high-quality machine vision without resorting to input reconstruction. Fig. 16 shows two examples of the outputs of our 3-layer network, along with the results obtained by the benchmarks. For each example, the first row shows the input image and the reconstructed images with the corresponding bitrate and RGB PSNR (bpp/dB). The next two rows show the results of object segmentation and object detection. For the benchmarks, reconstructed image is fed to the corresponding model (Faster R-CNN [46] for detection, Mask R-CNN [28] for segmentation) with pre-trained weights to obtain the results. For our 3-layer network, only the corresponding part of the bitstream is decoded. Hence, since the input is not reconstructed in these cases, the results are shown on the empty background, and the corresponding rate is indicated below the image.

In the first example, our network successfully detects and segments all three objects, with bitrates of 0.195 bpp for detection and 0.205 bpp for segmentation. In contrast, all benchmarks lead to mislabelling of a horse on the right as a person, even in the case of object detection, despite the fact they use more bits than our base layer. In the second example, benchmark-coded images lead to some missing objects, while our network correctly detect them all. For example, the image coded by [15] leads to missing the second person from the right in the background, as well as the baseball. Also, image coded by [16] leads to missing the person in the background. With conventional codecs (HEVC and VVC), either the person or the baseball are missed. These examples illustrate why our 3-layer network provides superior performance in terms of object detection and segmentation in Fig. 12.