Exploring Compressed Image Representation as a Perceptual Proxy: A Study

We consider a natural image $x$ as a point in the signal space that triggers stimuli in the brain’s sensory cortex through the visual system. The effectiveness of a deep neural network (DNN) lies in its ability to learn a complex transformation $\rho_{s}$ that maps stimuli $x_{i}$ to points $s_{i}$ in the semantic space for object classification, as illustrated in Figure 1 (a). Equivalently, research in image quality assessment aims to learn a perceptual mapping $\rho_{p}$ such that the Euclidean distance between points $z_{i}$ in the perceptual space highly correlates with human perception.

In the field of neural science [1], large capacity goal-driven hierarchical convolutional neural networks (HCNNs) like the VGG network [2], which are trained for image classification, have been demonstrated to effectively model neural stimuli and population responses in higher visual cortical areas. Consequently, the VGG network can map images into a feature vector space where distances between vectors closely align with the image’s structures and semantics. Rather than employing per-pixel difference loss, studies have shown that utilizing perceptual loss from VGG leads to superior image quality in various image transformation tasks, such as style transfer [3, 4] and super-resolution [5].

Zhang et al. [6] introduced the Learned Perceptual Image Patch Similarity (LPIPS) metric, which involves learning a set of linear weights $w$ applied to HCNN-extracted vectors to align with human perceptual judgments. LPIPS represents the pioneering attempt to employ a semantic transformation as a proxy for perceptual transformation. Essentially, they repurpose a semantic model as a perceptual model, where $\rho_{p}=\omega(\rho_{s}(x);w)$, with $\omega$ representing the learned linear mapping. Huang et al. [7] proposed the Compressed Perceptual Image Patch Similarity (CPIPS) metric, which incorporates both an analysis transform $g_{a}$ and a synthesis transform $g_{s}$ within an end-to-end learned image compression codec [8] as a goal-driven HCNN for image reconstruction. In CPIPS, the analysis transform is pre-trained on ImageNet object classification to achieve a certain level of accuracy. Subsequently, it is jointly optimized for rate-distortion loss and classification loss. The CPIPS affirms that with a mapping function $\omega$ applied to the compressed latent representation, we can conduct human perceptual distance judgments with an accuracy comparable to LPIPS. This work extends CPIPS further and can be conceptually illustrated in Figure 1 (b). We study the following two key questions:

1) To what extent can the image encoder network be effectively utilized from both perceptual and semantic perspectives?

2) What outstanding tasks are for a neural codec to serve as a perceptual space transform?

Learned Image Compression

The end-to-end learned image compression approaches have been proposed in the literature, starting with Ballé et al. [8] that surpassed traditional codecs like JPEG and JPEG 2000 regarding PSNR and SSIM metrics. Minnen et al. [9] further improved coding efficiency by employing a joint autoregressive and hierarchical prior model, surpassing the performance of the HEVC [10] codec. Cheng et al. [11] developed techniques that achieved comparable performance to the latest coding standard VVC [12]. Several comprehensive surveys and introduction papers [13, 14] have summarized these advancements in end-to-end learned compression.

Image Quality Assessment

The evaluation of image coding quality relies on full-reference image quality assessment metrics, which gauge the similarity between the reconstructed and the original images as human observers perceive. In addition to the traditional pixel-based PSNR, metrics such as SSIM [15], GMSD [16], and NLPD [17] have been widely employed. Recently, new DNN-based methods like LPIPS [6], PIM [18], and DISTS [19] have been proposed. These approaches have demonstrated superior predictive performance for subjective image quality. Their efficacy has been confirmed through validation against human judgments on benchmark datasets like BAPPS [6] and CLIC2021 [20], which comprise a comprehensive collection of two-alternative force choice (2AFC) human judgments.

Semantic Features as Perceptual Loss

The feature space of the VGG network has been leveraged by Gatys et al. [3] to generate high-quality images for style transfer. Johnson et al. [4] further introduced the feature reconstruction loss and style reconstruction loss computed up to the VGG network layer relu3_3 and relu4_3, respectively. Features extracted from higher VGG layers encapsulate high-level semantics, including color, texture, and shape, collectively representing an image’s style. Johnson’s work encompasses the joint optimization of content and style loss, expanding beyond style transfer to encompass generic image transformation tasks.

In the domain of image super-resolution, Ledig et al. [5] introduced SRGAN, which incorporates adversarial training and a perceptual loss function comprising an adversarial loss and a content loss. To capture low-level image details, they define the content loss based on the VGG network up to the relu2_2 layer. SRGAN has demonstrated remarkable results in achieving visually appealing 4X super-resolution and has garnered superior mean opinion scores (MOS) compared to prior methods. In addition to these two image tasks, other image reconstruction tasks [21] have also used perceptual loss to enhance the visual quality.

Transform coding plays a crucial role in conventional image compression as it separates the task process of decorrelating a source from its coding process [22]. Popular transforms such as the Discrete Cosine Transform (DCT) and the Wavelet transform are linear and unitary. Consider $U$ as a unitary transform in $Z=UX$, then the energy in the transform domain equals the energy in the original domain: $\lVert Z\rVert^{2}=\lVert X\rVert^{2}$. If $z_{1}=Ux_{1}$ and $z_{2}=Ux_{2}$, we have

A unitary transform preserves Euclidean distances from the signal space to the latent space. As a result, leveraging the latent representation in traditional image compression for high-level semantic computation is equivalent to processing in the signal domain. However, the development of complex non-linear transforms through deep neural networks in modern learned image compression [8, 9] has made it possible to perform processing and analysis in the compressed domain, giving rise to the advocacy of Video Coding for Machines (VCM) [23].

CPIPS: An Image Encoder Tuned with Semantics

In CPIPS [7], the weights of the analysis transform are initialized with the semantic features pre-trained on ImageNet. Through joint compression classification training, the gradient descent optimizer updates the encoder-decoder weights to analyze and synthesize the image while improving classification accuracy. We enhance CPIPS by removing the regularization loss and leveraging the encoder instead of the decoder. Figure 2 shows the detailed network architecture.

To obtain the distance between two images, denoted as $x$ and $x_{0}$, we follow the same procedure as LPIPS [6] by learning a linear layer $w$ on the BAPPS dataset. We extract feature maps $f^{l}\in\mathbb{R}^{C_{l}\times H_{l}\times W_{l}}$ for all layers $l$ and normalize them in the channel dimension. The activations are then scaled channel-wise using the vector $w_{l}\in\mathbb{R}^{C_{l}}$, and the $l_{2}$ distance is computed. Finally, we average across the spatial dimensions and sum up all layers for the final distance:

where $\odot$ denotes the convolution operator. We jointly train the compression classification loss to achieve 51.58% and 76.27% for top-1 and top-5 accuracy, respectively. The enhanced CPIPS performs superior to LPIPS on the BAPPS human perceptual judgment dataset, as shown in Table 3. However, its coding efficiency is somewhat compromised, surpassing only JPEG’s.

Hyperprior encoder and Semantic-tuned Hyperprior

A perceptual distance-preserving image encoder only makes sense when its coding efficiency remains superior to traditional hand-crafted codecs. To address this, we turn to the well-known hyperprior codec [8], which has been reported to outperform HEVC in terms of PSNR metric. Given that image reconstruction is a well-defined goal-driven HCNN, we assess the feasibility of utilizing the hyperprior’s analysis transform as the perceptual transform. Additionally, we apply the same joint compression-classification optimization strategy to fine-tune the hyperprior encoder/decoder as we wonder about the semantic-aware benefits derived from the auxiliary image classification task. Table 1 reports the Bjøntegaard-Delta (BD) metric compared to JPEG on the Kodak dataset. The fine-tuning of the hyperprior encoder significantly reduces its coding efficiency but remains superior to traditional codecs.

Style Transfer and Super-resolution

In addition to perceptual similarity judgments, we assess the efficacy of the analysis transform as a perceptual loss for image transformation tasks. We employ Gatys’ approach [3] and SRGAN’s technique [5] for style transfer and super-resolution, utilizing CPIPS, hyperprior codec, and the hyperprior-tuned encoder as the perceptual loss networks. The feature layers we extract are detailed in Table 2. We focus on early layers from the analysis transform to capture low-level structures as content loss for style transfer. Conversely, we extract full layers to represent semantic features in super-resolution, as the original paper suggests. Each encoder network includes a final Gaussian-conditioned hyperprior bottleneck layer, reflecting the rate constraint imposed on the latent vectors.

For our proposed CPIPS, we begin by pre-training the analysis transform on the ImageNet dataset for 90 epochs, focusing on an image classification task, resulting in a top-1 accuracy of 59.68%. Subsequently, we jointly train the compression classification task, utilizing the pre-trained weights for an additional 120 epochs. This phase involves employing the Adam optimizer with a learning rate set to 0.0001. In the case of the hyperprior method, we use the CompressAI implementation¹¹1https://github.com/InterDigitalInc/CompressAI along with a pre-trained model configured at quality setting 8. We apply the same joint compression-classification optimization strategy to fine-tune the hyperprior codec, conducting this process for 120 epochs, resulting in a semantic-tuned hyperprior, denoted as Hyperprior-tune.

To evaluate our approach, we compare the perceptual judgment results with various quality metrics, namely GMSD, NLPD, LPIPS²²2https://github.com/richzhang/PerceptualSimilarity, PIM³³3https://github.com/google-research/perceptual-quality, and DISTS⁴⁴4https://github.com/dingkeyan93/DISTS, on two datasets: BAPPS [6] and CLIC2021 [20]. These datasets consist of 151k and 122k patches, respectively, and employ a 2AFC judgment methodology.

For style transfer, we adopt a separate model for each style, fine-tuning the weights for content and style loss based on the activation magnitude of different networks to achieve visually appealing results. In the case of super-resolution, we conduct experiments on three widely recognized benchmark datasets: Set5, Set14, and BSD100. All experiments are performed with a scale factor of 4X between low and high-resolution images.

Perceptual Distance Judgments

The quality metrics are compared on the benchmark datasets, as shown in Table 3. It is evident that there exists a significant gap between modern DNN-based quality metrics and traditional pixel-based metrics. The PIM exhibits the best performance among specially designed perceptual metrics, and the DISTS metric also outperforms the LPIPS. Notably, the DISTS metric also leverages the VGG network as a fundamental building block.

The off-the-shelf perceptual model, CPIPS, closely trails behind a custom-tailored quality metric like DISTS. Our CPIPS, built on a neural encoder, performs slightly better than DISTS in the BAPPS dataset but is less competitive in the CLIC dataset. Surprisingly, the CPIPS built on the original hyperprior codec, referred to as CPIPS-hyper, delivers a competitive performance in the BAPPS dataset and is superior in the CLIC dataset, closely rivaling a well-designed quality metric like DISTS. We argue that this phenomenon echoes the conclusion from [1], where a well-trained neural network on an image reconstruction task is equivalent to a goal-driven HCNN. Another finding is that fine-tuning the joint compression classification optimization on the hyperprior encoder does not have a meaningful impact on perceptual prediction performance. We attribute this to the network architecture design problem: a pre-trained network good at classification like CPIPS is challenging to achieve satisfying coding efficiency while fine-tuning a highly efficient hyperprior codec cannot improve perceptual prediction. Investigating and improving upon the above findings are left as future work. Despite this, we confirmed that a vanilla hyperprior encoder performs closely, rivaling a well-designed quality metric like DISTS.

Style Transfer

We assess the analysis transform’s ability to extract high-level features for style and transfer them to another content image. Figure 3 showcases the qualitative results obtained using different networks as a perceptual loss. The CPIPS encoder effectively disentangles content and style features through semantic fine-tuning, allowing for style transfer similar to the VGG network. However, the stylish effect, while present, may not be as pronounced as with VGG, according to subjective evaluation. On the other hand, the raw hyperprior encoder struggles to differentiate between style and content features, resulting in less meaningful outputs. By applying semantic fine-tuning similar to CPIPS, the hyperprior’s performance is greatly enhanced, producing stylish results comparable to those of CPIPS. Therefore, the vanilla neural encoder needs to be fine-tuned to accomplish the style transfer task.

Super-resolution

While it is acknowledged that SRGAN, with a perceptual loss, yields higher mean square errors, it often achieves superior mean opinion scores. We begin by quantifying and presenting the objective measurements in Table 4. When utilizing the CPIPS encoder as the perceptual loss, we obtain quantitative results comparable to those achieved with the VGG network. Consequently, the raw and tuned hyperprior encoder networks outperform the VGG network regarding PSNR and SSIM metrics. This result could be attributed to the fact that neural codecs are designed to excel in image reconstruction tasks.

In terms of visual quality, as illustrated in Figure 4, all the analysis transforms yield sharpened and visually enhanced outputs. Remarkably, using the VGG network introduces finer details than other methods. This observation suggests that the analysis transform derived from a neural codec could be a reliable proxy for perceptual loss in specific image transformation tasks.

Rate-Distortion Trade-offs

In lossy compression, defining distinct quality factors for low and high-bit-rate applications through rate-distortion optimization is a common practice. We evaluated the influence of bit-rate trade-offs on the efficacy of perceptual representation. Our experiments revealed that even with the lowest bit-rate setting, such as quality 1, the accuracy of perceptual judgment, style transfer effectiveness, and super-resolution quality remained remarkably close to that achieved with a high-quality setting.

This study demonstrated that the analysis transform within a neural codec is highly proficient in extracting feature representations suitable for perceptual distance judgment and various image transformation tasks. Specifically, we’ve shown its effectiveness as a perceptual loss in super-resolution tasks, generating visually pleasing outputs with or without classification fine-tuning. In the case of style transfer, fine-tuning the analysis transform is crucial for producing stylish content. Suppose one seeks to repurpose the neural encoder as a perceptual transform. In that case, it’s imperative to carefully re-examine the network architecture design, pre-training techniques, and rate-distortion-semantic optimization. Further exploration into auxiliary tasks beyond image classification may also serve as avenues for semantically fine-tuning the neural encoder.