CPIPS: Learning to Preserve Perceptual Distances in End-to-End Image Compression

The field of learned image compression has witnessed significant advancements with the introduction of convolutional neural networks. Several approaches have been proposed in the literature, starting with Ballé et al. [12] that surpassed traditional codecs like JPEG [18] and JPEG 2000 [28] in terms of PSNR and SSIM metrics. Minnen et al. [13] further improved coding efficiency by employing a joint autoregressive and hierarchical prior model, surpassing the performance of the HEVC [29] codec. More recently, Cheng et al. [15] developed techniques that achieved comparable performance to the latest coding standard VVC [30]. Several comprehensive survey and introduction papers [31, 32, 10] have summarized these advancements in end-to-end learned compression.

Currently, there are two remaining challenges [33] in this field: computational complexity and subjective image quality. The neural compressor employs high-capacity networks to end-to-end model data dependency in exchange for better bitrate-distortion (BD) efficiency. The channel-conditional method proposed by Minnen et al. [14] achieves performance close to VVC but at the cost of high computational complexity (600K FLOPS/pixel). Regarding image quality, Valenzise et al. [34] conducted subjective tests on DNN-based methods and observed that these methods produce artifacts that are difficult to evaluate using traditional metrics like PSNR. They concluded that PSNR is inadequate for evaluating DNN-based methods. Upenik et al. [35] benchmarked a set of DNN-based image codecs using a crowdsourcing-based subjective quality evaluation procedure with Differential Mean Opinion Scores (DMOS). Their results demonstrate that learning-based approaches can achieve promising bitrate-DMOS performance compared to HEVC. However, despite their superior subjective scores, these DNN-based image codecs are optimized with pixel difference-based distortion functions.

The evaluation of image codec quality traditionally relies on full-reference image quality assessment (FR-IQA) metrics, which measure the similarity between the reconstructed image and the original image as perceived by human observers. In addition, to mean square error (MSE) or PSNR, various FR-IQA metrics, such as SSIM variants [36, 37], PIM [25], and DISTS [27], have been proposed to predict subjective image quality judgments. Johnson et al. [38] proposed using the feature vector distance from the VGG network [39] as a perceptual loss for image transformation tasks based on the hypothesis that the same image features used for image classification are also helpful for other tasks.

Zhang et al. [26] introduced the BAPPS dataset, which includes a large-scale collection of human judgments on image pairs, and trained the Learned Perceptual Image Patch Similarity (LPIPS) metric. LPIPS was found to be more aligned with human judgments than traditional quality metrics such as L2, PSNR, and SSIM. Ding et al. [40] conducted an interesting study to evaluate whether DNN-based quality metrics can be used as objectives for optimizing image processing algorithms. Developing effective perceptual quality metrics for image tasks remains a challenging problem.

Lossy image coding standards such as JPEG and MPEG have primarily focused on achieving high compression rates for human consumption of multimedia data. However, with the rise of IoT devices, drones, and self-driving cars, there is a growing need for efficient compressed representations that cater not only to human vision but also to image processing and machine vision tasks. Techniques such as image data hiding [19], image denoising [20], and image super-resolution [41] have been developed to operate directly on neural compressed latent spaces.

Le et al. [21] proposed an inference-time content-adaptive fine-tuning scheme that optimizes the latent representation to improve compression efficiency for machine consumption. Duan et al. [22] employed transfer learning to perform semantic inference directly from quantized latent features in the deep compressed domain without pixel reconstruction. Choi et al. [42] introduced scalable image coding frameworks based on well-developed neural compressors, achieving up to 80% bitrate savings for machine vision tasks.

We illustrate the Left-UNet architecture in Fig. 2 and provide details in Table 1. The feature extraction network $\mathcal{F}$ for the image classification task uses the parameterized ReLU as the activation function for all layers, while the encoder network $\mathcal{G}_{e}$ employs Generalized Divisive Normalization (GDN) at the end of each downsampling layer. GDN, proposed by Ballé et al. [43], is inspired by modeling neurons in biological visual systems and has been proven effective in Gaussianizing image densities for a superior rate-distortion trade-off.

The extracted image features are then average pooled and connected to a linear layer with 1,000 neurons to optimize the classification loss $\mathcal{L}_{C}$ using cross-entropy:

It is known that a high-capacity neural network trained for a high-level vision task implicitly learns to reason about relevant semantics [38]. Our goal is not to solve the classification problem directly. Instead, we aim to design a moderately-sized network that can learn semantic features without significantly increasing the encoder-decoder complexity.

A typical learned neural codec consists of an encoder-decoder pair, a quantization module, and an entropy coder. Given an input image $x\in\mathcal{X}$, the neural encoder $\mathcal{G}_{e}$ transforms $x$ into a latent representation $y=\mathcal{G}_{e}(x)$, which is later quantized to a discrete-valued vector $\hat{y}$. The discrete probability distribution $P_{\hat{y}}$ is estimated using a neural network and then encoded into a bitstream using an entropy coder. The rate of this discrete code, $\mathsf{R}$, is lower-bounded by the entropy of the discrete probability distribution $H(P_{\hat{y}})$. On the decoder side, we decode $\hat{y}$ from the bitstream and reconstruct the image $\hat{x}=\mathcal{G}_{d}(\hat{y})$ using the neural decoder. The distortion, $\mathsf{D}$, is measured by a perceptual metric $d(x,\hat{x})$. Overall, we optimize the network parameters for a weighted sum of the rate and distortion, $\mathsf{R}+\lambda\mathsf{D}$, over a set of images.

Table 2 illustrates the decoder network $\mathcal{G}_{d}$, which is designed to complement the encoder. In the generic neural codec concept, the innermost latent vector $\hat{y}^{5}$ is equivalent to the discrete-valued vector $\hat{y}$. During image reconstruction, the intermediate output vectors $e^{l}$ from each upsampling layer conv_l_2 play a crucial role because they represent learned multi-scale semantic layers, which are equivalent to the feature layers of a VGG-16 network.

Like [12], we employ kernel density estimation with a neural network to obtain the probability distribution $P_{\hat{y}}$. The rate loss $\mathsf{R}$ is computed as follows:

In our experiments, we utilize the MSE as the distortion function. However, alternative quality metrics such as SSIM variants [36, 37] can be employed to fit perceptual quality better. The distortion loss $\mathsf{D}$ is defined as:

Although the intermediate convolution output features are seldom used in most machine learning tasks, these features, which are tuned to be predictive of essential structures, exhibit a high correlation with human perceptual similarity [26]. However, storing intermediate latent features in the context of data compression becomes impractical if the final bottleneck layer contains sufficient information for the decoder to reconstruct the image. Another approach to mitigate storage waste is to reduce the number of downsampling layers. However, modeling the sensory cortex in the visual system [3] requires at least five layers of feature extraction to generate neural responses, a finding that our experiments validate as well. Consequently, we utilize the intermediate output $e^{l}$ from the decoder as a proxy for multi-scale semantic features and apply a regularizer to constrain the decoder. Specifically, we employ the $l_{1}$ distance to define our regularization loss:

To initialize the Left-UNet encoder $\mathcal{G}_{e}$ and an auxiliary classifier, we utilize pre-trained semantic features from the image classification task mentioned in Section 3.1. Subsequently, we train an end-to-end image compression network using the overall loss function:

The hyper-parameter $\lambda$ represents the rate-distortion trade-off, which can be adjusted according to the desired image quality factor $Q$. We set $\alpha=0.3$ and $\beta=1.0$ for our experiments.

Through joint compression-classification training, the weights of the Left-UNet encoder are initially initialized with pre-trained semantic features. Subsequently, the gradient descent optimizer updates the encoder-decoder weights to analyze and synthesize the image while improving classification accuracy.

To obtain the distance between two images, denoted as $x$ and $x_{0}$, we follow the same procedure as LPIPS [26] by learning a linear layer $w$ on the BAPPS dataset. This linear layer assigns weights to the compressed latents and intermediate decoder outputs. Fig. 3 illustrates the process of obtaining the distance using entropy-decoded $\hat{y}^{5}$ and feature outputs $e^{l}$ from our decoder network $\mathcal{G}_{d}$. We extract feature maps $\hat{y}^{5},e^{l}\in\mathbb{R}^{C_{l}\times H_{l}\times W_{l}}$ for all layers $l$ and normalize them in the channel dimension. The activations are then scaled channel-wise using the vector $w^{l}\in\mathbb{R}^{C_{l}}$, and the $l_{2}$ distance is computed. Finally, we average across the spatial dimensions and all layers to obtain the following:

Eq. (7) calculates the final distance between image $x$ and $x_{0}$, that is:

Furthermore, we train another smaller network, denoted as $\mathcal{D}$, to predict perceptual judgments $h$ from the distance pair $(d_{0},d_{1})$ on the BAPPS 151k patches 2AFC (two alternative forced choice) dataset.

To implement our CPIPS, we utilize the CompressAI¹¹1https://github.com/InterDigitalInc/CompressAI [44] implementation of the hyperprior neural compressor [12] and the official release²²2https://github.com/richzhang/PerceptualSimilarity of LPIPS. We pre-train the image classification task on the ImageNet dataset, which consists of 1.2 million images. The training is performed using the PyTorch Adam optimizer with a learning rate 0.0001 for 120 epochs. Following that, we jointly train the compression-classification task with the pre-trained weights for 150 epochs, employing the Adam optimizer with a learning rate of 0.0001.

Regarding CPIPS weights $w$ and the judgment network $\mathcal{D}$, we train them for ten epochs using the BAPPS 2AFC dataset, as mentioned in the original LPIPS paper.

Table 3 displays our top-1 and top-5 accuracy compared to high-capacity deep networks such as VGG-16. The achieved top-1 accuracy of 60.11% is considered favorable, indicating that the pre-trained weights can serve as a suitable initialization for the Left-UNet encoder $\mathcal{G}_{e}$.

We compare our method with LPIPS and traditional L2 and SSIM metrics, in terms of the accuracy of image judgments against human ratings on the BAPPS dataset. Table 4 and Fig. 4 present the results.

Evidently, the metrics incorporating learned semantic features, such as LPIPS, Left-UNet, and CPIPS, exhibit a higher correlation with human judgments compared to L2 and SSIM. While Left-UNet does not achieve the same level of accuracy as LPIPS, it serves as an upper bound for our proposed CPIPS since they share the same feature extraction convolution layers. Our CPIPS achieves similar accuracy to Left-UNet in the CNN, DeBlur, and Color subsets but experiences a more considerable drop in accuracy in the Traditional, Super-Res, and Frame-Interp subsets. We attribute this drop to two factors: 1) the rate-distortion optimization process influencing the semantic properties of the latent vectors, thereby affecting the perceptual representation, and 2) the multi-scale feature maps $e^{l}$ serving as proxies for the feature extraction vectors $y^{l}$ reconstructed in the decoding stages through the regularization loss. Investigating and improving upon these factors are left as future work.

Qualitatively, we select some sample image patches from the BAPPS dataset and present their different judgments in Fig. 5. We can see that the L2 and SSIM cannot reflect human perceptual preferences. At the same time, the CPIPS and LPIPS align with the ground truth better. The second image pair in Fig. 5 demonstrates that the SSIM has a strong bias with structures and tends to be impacted by additive noises.

We assessed the computation time of the metrics on an Intel i7-9700K workstation with an Nvidia GTX 3090 GPU. To compare our CPIPS metric with LPIPS and DISTS³³3https://github.com/dingkeyan93/DISTS, we used the Kodak dataset [45] and calculated the average time cost, as shown in Table 5. Due to utilizing of a less complex neural network that only requires decoding the bitstream and intermediate features, our CPIPS method is approximately 50 times faster.