MarsQE: Semantic-Informed Quality Enhancement for Compressed Martian Image

Recently, extensive works  [8, 15, 16, 9, 17, 10, 18, 11] have reached remarkable performance for enhancing the quality of Earth images, due to the booming development of Convolutional Neural Networks (CNNs). Concretely, as the pioneer of CNN-based approach, a shallow four-layer Artifacts Reduction Convolutional Neural Network (AR-CNN) [8] was proposed to enhance the quality of JPEG-compressed images. AR-CNN processively performs feature extraction, feature enhancement, feature mapping, and image reconstruction. Motivated by dual-domain sparse coding and quantization prior of JPEG compression, the Deep Dual-Domain (D3) [15] approach and Deep Dual-domain Convolutional neural Network (DDCN) [16] were proposed to mitigate JPEG compression artifacts. To be more specific, D3 and DDCN improve the quality of compressed images by incorporating Discrete Cosine Transform (DCT) priors into network branches. Notably, as a milestone of CNN-based approaches, Denoising Convolutional Neural Network (DnCNN) [9] was proposed, which utilizes some effective techniques including the residual learning [19] and batch normalization [20]. With a 20-layer deep network, DnCNN can remove both Additive White Gaussian Noise (AWGN) and JPEG artifacts, surpassing most traditional approaches such as Block Matching and 3-D filtering (BM3D) [21]. Afterwards, Liu et al. [17] presented a Multi-level Wavelet CNN (MWCNN) by utilizing Discrete Wavelet Transform (DWT) and Inverse Wavelet Transform (IWT). Specifically, DWT and IWT are introduced to replace pooling and upsampling operations to enlarge receptive field.

More recently, Xing et al. [10] proposed a RBQE approach for the purpose of enhance quality in a resource-efficient manner. By utilizing dynamic inference structure, RBQE approach can effectively and efficiently remove both compressed artifacts. Guo et al. [18] proposed Convolutional Blind Denoising Network (CBDNet) to improve the generalization capability for real-world degradations. The CBDNet was designed with an estimation subnetwork and a non-blind denoising subnetwork, such that effective blind quality enhancement can be achieved. Later, the Defocus-Aware Quality Enhancement (DAQE) approach [11] was proposed to improve the quality of compressed images with a region-wise divide-and-conquer strategy, which utilizes image defocus to discern region-wise quality differences.

The aforementioned quality enhancement approaches for Earth images have employed datasets such as Berkeley Segmentation Data Set and Benchmarks 500 (BSDS500) [22] and DIV2K [13]. These datasets comprise a large variety of contents with natural and artificial scenes, e.g., plants, man-made objects, cityscapes, etc. With these datasets, the aforementioned approaches can take advantage of the rich semantics in Earth images. In fact, different from Earth images, Martian images have limited semantics due to the simpler scene. Therefore, we propose utilizing the limited-semantic characteristics to enhance the quality of Martian images.

Some vision works have been conducted on Martian images to assist Mars exploration. These vision works mainly focus on high-level vision tasks [23, 24, 25, 26], including terrain classification and semantic segmentation. Specifically, Michael et al. [23] employed a deep CNN architecture, called DeepLabV3+, for semantic segmentation on Martian terrain images. The classification accuracy achieves over 96%, showing the potential of auto-driving Mars rovers using deep-learning. To make a step forward, Hu et al. [24] presented a hybrid network (DeepLabV3+/EfficientNet) to further improve the classification accuracy, such that the different areas of Mars can be judged. From the perspective of geological taxonomy, Tejas et al. [25] developed a self-supervised deep clustering algorithm to support rapid and robust terrain categorization. By filtering data with high confidence of labels, Gagan et al. [26] designed SegFormer, a semantic segmentation framework utilizing vision transformer, for segmenting and identifying the terrain images of Mars. In this way, the SegFormer can effectively segment terrain into three categories, i.e., sand, soil, and rock. In addition to the high-level vision works, a few works focus on low-level vision tasks, such as image super-resolution and image compression [27, 14]. Wang et al. [27] addressed the challenge of reconstructing high-resolution Martian images from low-resolution ones by estimating the blur kernels. Specifically, they proposed a novel degradation network specifically designed for Martian images. Ding et al. [14] proposed a deep Martian image compression network with a non-local block to explore both local and non-local dependencies among Martian image patches.

To the best of our knowledge, no prior work has been dedicated to the vital task of enhancing the quality of Martian images. This becomes particularly significant for geologic analysis and science popularization, since Martian images invariably undergo compression for transmission, leading to the presence of noticeable artifacts [4]. Therefore, this paper makes the pioneering effort of enhancing the quality of Martian images. Moreover, by leveraging the distinct and limited semantics inherent to Martian images, this paper effectively enhances the quality of Martian images, which may inspire further advancement in the domain of Martian vision processing.

Given the significant inter-image similarity in Martian images (as noted in Finding 1), MarsQE take advantages of texture-similar reference images for quality enhancement. As shown in the dot-dash box of Figure 6, the Semantic-based Matching Module (SMM) is designed to efficiently obtain these reference images.

Reference dictionary construction: Findings 2 and 3 reveal that Martian images can be efficiently categorized into a few texture-distinct semantic classes. Inspired by this characteristic, SMM constructs a compact but comprehensive semantic-based dictionary. Specifically, SMM first divides the raw images into non-overlapping $S\times S$ patches. Then, these patches are categorized into $K$ semantic classes, utilizing Martian terrain segmentation [26]. Finally, for the $k$-th semantic class where $k$ ranges from $1$ to $K$, SMM selects $N_{k}$ patches to construct a reference dictionary $\{\mathbf{R}^{k}_{i}\}_{i=1}^{N_{k}}$. The corresponding compressed patches $\{\hat{\mathbf{R}}^{k}_{i}\}_{i=1}^{N_{k}}$ are also included for reference.

Reference patch matching: SMM conducts patch-wise reference matching using the obtained reference dictionary. Specifically, the input compressed image is divided into non-overlapping patches. Each compressed patch $\mathbf{P}_{\text{comp}}$ is semantically classified and matched with the appropriate patches in the same-semantic-class dictionary. The matching process is formulated as:

In the above formulation, $D_{\text{KL}}(x\|y)$ denotes the Kullback-Leibler (KL) divergence [30] between vectors $x$ and $y$; $\operatorname{\text{LBP}}(\cdot)$ denotes the LBP [28] texture descriptor. As a result, the semantically consistent reference patches $\mathbf{R}^{k}_{n}$ and $\hat{\mathbf{R}}^{k}_{n}$ are selected for each compressed patch $\mathbf{P}_{\text{comp}}$ as $\mathbf{P}_{\text{ref}}$ and $\hat{\mathbf{P}}_{\text{ref}}$, respectively.

Given the obtained raw and compressed reference patches, the MarsQE approach performs semantic-informed quality enhancement in a patch-wise manner. According to Findings 2 and 3, patches from the same semantic classes exhibit similar texture patterns. Thus, MarsQE uses semantic-related textures from reference patches to guide the enhancement of input patches. Specifically, the overall network is designed with a $L$-level encoder-decoder architecture. The network initially extracts transferable features with semantic-related textures at various scales, and then fuses these features with decoded features to restore texture details. Finally, the fused features are enhanced through the Supervised Attention Module (SAM) [31], allowing for efficient feature propagation to the post-enhancement network.

Semantic-related texture extraction: In fact, reference patches are rich in semantic-related texture details, crucial for enhancing the quality of input compressed patches. As depicted in the yellow box of Figure 6, the network employs an encoder to extract multi-scale semantic-related textures from the compressed input, raw reference, and compressed reference patches. The extraction flows for these patches are represented by blue, yellow, and red lines, respectively. Texture Extracting Modules (TEMs) are integral to this process, designed to extract transferable features at each scale. Consider one input patch for example as shown in Figure 7(a). The encoded features of $\mathbf{P}_{\text{comp}}$, $\hat{\mathbf{P}}_{\text{ref}}$, and $\mathbf{P}_{\text{ref}}$, denoted as $\mathbf{F}_{\text{comp}}$, $\hat{\mathbf{F}}_{\text{ref}}$, and $\mathbf{F}_{\text{ref}}$, are fed into TEM as query, key and value. TEM unfolds these features into non-overlapping $d\times d$ feature blocks, resulting in $\{\mathbf{B}_{\text{comp}}\}$, $\{\hat{\mathbf{B}}_{\text{ref}}\}$, and $\{\mathbf{B}_{\text{ref}}\}$. For each block $\mathbf{B}_{\text{comp}}^{i}$, TEM identifies the most relevant block in $\{\hat{\mathbf{B}}_{\text{ref}}\}$ as follows,

where ${D}_{i}$ indicates the index and ${S}_{i}$ represents the similarity score. The transferable feature blocks $\{\mathbf{B}_{\text{trans}}\}$ are then generated from $\{\mathbf{B}_{\text{ref}}\}$:

Finally, we aggregate $\{\mathbf{B}_{\text{trans}}^{n}\}$ to form the transferable features $\mathbf{F}_{\text{trans}}$, thereby extracting the semantic-related textures.

Semantic-related texture fusion: After extracting the semantic-related textures, the learnable decoder employs these textures to enrich the decoded feature. This process is visually detailed in the red box of Figure 6. At each decoder level, the decoded features are up-sampled and combined with corresponding encoded features via skip connections. These features are then fused with the extracted transferable features using the Texture Fusing Module (TFM), as depicted in Figure 7(b). TFM initially calculates the residual $\mathbf{F}_{\text{res}}$, which represents the difference between transferable features $\mathbf{F}_{\text{trans}}$ and decoded features $\widetilde{\mathbf{F}}_{\text{comp}}$. Subsequently, TFM generates attention maps to filter informative features from $\mathbf{F}_{\text{res}}$ for subsequent fusion. The process culminates with the fusion of $\widetilde{\mathbf{F}}_{\text{comp}}$ and the filtered features, output through a convolution layer, resulting in the fused features $\mathbf{F}_{\text{fused}}$. Mathematically, the above process is represented by

In the above, $C_{3\times 3}(\cdot)$ represents a convolution layer with a kernel size of $3\times 3$, and $\operatorname{\text{CA}}(\cdot)$ signifies a channel attention layer [32]. At the output side of the network, the final decoded features $\widetilde{\mathbf{F}}_{1,\text{comp}}$ are spatially recombined to the input image size and fed into SAM [31] to generate an enhanced image $\mathbf{I}_{\text{enh}}$. This way, the semantic-related textures from reference patches are effectively utilized to produce a semantically enriched enhanced image.

The main network of MarsQE has produced a high-quality enhanced image. In addition, we develop a post-enhancement network to mitigate blocking effects caused by patch-wise enhancement, such that semantic details can be enriched. It is also designed with an encoder-decoder architecture. In the encoder stage, the features from the main network are combined with the encoded features of the input compressed image $\mathbf{I}_{\text{comp}}$. These features are further refined at each level by the Quality Enhancement Module (QEM), as illustrated in Figure 7(c). QEM is mainly composed of Layer Normalization (LN) [33], Gaussian Error Linear Unit (GELU) [34], and Channel Attention (CA) [32].

To exploit the intra-image similarity in Martian images (highlighted in Finding 1), the network incorporates the Non-Local Module (NLM) for each compressed image. Given the texture-distinct semantic classes obtained in Section IV-A, we apply NLMs within each semantic class, significantly accelerating the post-enhancement process. The functionality of NLM is mathematically represented as

Here, $\mathbf{F}_{i}$ and $\mathbf{F}_{i+1}$ represent the input and output features of NLM, respectively. The notations $C_{1\times 1}(\cdot)$, $L_{\text{NLA}}(\cdot)$, $L_{\text{LN}}(\cdot)$, and $L_{\text{GELU}}(\cdot)$ correspond to a convolution layer with a kernel size of $1\times 1$, a non-local attention layer, a LN layer, and a GELU activation layer, respectively. The features produced by NLM are then fed into the decoder, resulting in a final enhanced image $\mathbf{I}_{\text{enh}}^{\text{+}}$, exhibiting higher quality than $\mathbf{I}_{\text{enh}}$. The overall framework of MarsQE is trained in an end-to-end manner with the loss function of Mean Squared Error (MSE) between the ground truth image $\mathbf{I}_{\text{raw}}$ and two enhanced images:

In summary, our MarsQE approach effectively enhances the quality of compressed Martian images by leveraging semantic-related intra- and inter-image similarities.

In this section, we present details about the datasets, hyper-parameters, and training strategy of our experiments for evaluating the effectiveness of MarsQE.

Martian image datasets. Several Martian image datasets [23, 35, 14] have been established through the cameras mounted on Mars rovers. In particular, Mars32K [35] consists of 32,368 color Martian images collected by the Curiosity rover. Moreover, a notable large-scale labeled dataset, AI4MARS [23], collects over 35K images from the Curiosity, Opportunity, and Spirit rovers of NASA. With crowd-sourced labels annotated by the rover planners and scientists, AI4MARS was proposed as a high-quality dataset for training and evaluating models for terrain classification on Mars. Recently, Ding et al. [14] have presented a Martian image dataset, i.e., MIC, which consists of high-resolution and large-scale Martian color images captured by the latest Perseverance rover [36]. Specifically, the MIC dataset comprises 3,860 raw colorful Martian images captusred by the left Mastcam-Z camera on the Perseverance rover, all at a resolution of $1152\times 1600$. These images are randomly divided into 3,088 images for training, 386 images for validation, and 386 images for testing. As detailed in Table II, we adopt the training, validation, and test sets of MIC as the same usage in our experiments. Moreover, the AI4MARS and Mars32K datasets are used as our test sets, since they have no ground-truth images and can only be used for qualitative evaluation.

Martian image compression. The image compression codec of JPEG [3] has been widely used for Martian images transmission in recent missions, e.g., MSL mission [4]. Therefore, our experiments mainly focus on enhancing the quality of JPEG-compressed Martian images. In addition, we include the High-Efficiency Video Coding with Main Still image Profile (HEVC-MSP)/Better Portable Graphics (BPG) [37, 38] codec for Martian image compression, considering its prevalence in quality enhancement approaches for natural images [39, 40, 10, 11, 41].

We adopt four compression settings for each codec, with the quality factor (QF) being 20, 30, 40, and 50 for JPEG, and the quantization parameter (QP) being 27, 32, 37, and 42 for BPG. Note that these settings have also been adopted by prevalent quality enhancement approaches for natural images.

Hyper-parameters and training. In the MarsQE approach, parameters $S$, $d$, and $L$ are set to 128, 4, and 3, respectively. All convolution operators are with a stride of 1 and a padding of 1. To classify the input patches, we adopt an off-the-shelf Transformer-based method for Martian terrain segmentation [26]. This method classifies image patches into 4 classes, and the parameter $K$ is thus set to 4. When constructing the dictionary for reference matching, we select 6 patches for each semantic class; therefore, the parameters $\{{N_{i}}\}_{i=1}^{K}$ are all set to 6. As for the loss function, we set $\lambda$ to 1. During the training process, the Adam [42] optimizer is applied with an initial learning rate of $10^{-4}$. The cosine annealing schedule [43] is also applied to automatically decrease the learning rate. We set the training batch size to 16. A workstation with one CPU (12th Gen Intel Core i9-12900KF) and four GPUs (NVIDIA GeForce RTX 4090) is adopted in our experiments.

We evaluate the performance of our MarsQE approach on the quality enhancement of compressed Martian images. Since there exist no quality enhancement approaches for Martian images, we compare MarsQE with widely-used approaches for natural images, including AR-CNN [8], DCAD [39], DnCNN [9], CBDNet [18], RBQE [10], and DAQE [11]. For fair comparison, all approaches are re-trained on our training set in a non-blind manner, i.e., one model is trained for each compression setting.

Quantitative performance. To quantify the efficacy of our and compared approaches, we measure the performance of quality enhancement in terms of several metrics, including Peak Signal-to-Noise Ratio (PSNR), Peak Signal-to-Noise Ratio including Blocking effects (PSNR-B) [44], Structural Similarity Index Measure (SSIM) [45], Multi-Scale Structural Similarity (MS-SSIM) index [46], and Learned Perceptual Image Patch Similarity (LPIPS) [47]. As shown in Table III, our MarsQE achieves the best performance over the MIC test set in terms of all metrics. Specifically, for the JPEG-compressed Martian images with QF $=30$, the average PSNR of MarsQE is 34.61 dB, which is 2.17 dB higher than the JPEG baseline, and 0.25 dB higher than that of the second-best approach. In addition, the average PSNR-B is 34.03 dB, which is 3.49 dB higher than the JPEG baseline, and 0.21 dB higher than that of the second-best approach. Similar results can be found for other QF settings and metrics. For the BPG-compressed Martian images with QP $=37$, the average PSNR of MarsQE is 33.46 dB, which is 1.25 dB higher than the BPG baseline, and 0.20 dB higher than that of the second-best approach. In addition, the average PSNR-B is 33.25 dB, which is 1.29 dB higher than the BPG baseline, and 0.19 dB higher than that of the second-best approach. Similar results can be found for other QP settings and metrics. In summary, our MarsQE approach achieves state-of-the-art quality enhancement performance on both JPEG and BPG-compressed Martian images.

Rate-distortion performance. We further evaluate the rate-distortion performance of our MarsQE approach in Figure 8 and Table IV. Figure 8 shows the rate-distortion curves of different approaches over the MIC test set. As can be seen from this figure, the rate-distortion curves of MarsQE are higher than those of other approaches, indicating a better rate-distortion performance of MarsQE. Then, we quantify the rate-distortion performance by evaluating the reduction of Bjontegaard-rate (BD-rate) [48]. The results are presented in Table IV. As can be seen, for JPEG-compressed Martian images, the BD-rate reductions of our MarsQE approach on the MIC test set are averagely 38.94% and 27.18% with the distortion measured by PSNR and SSIM, respectively, considerably better than other approaches. Similar results can be seen for BPG-compressed Martian images. In summary, our MarsQE approach significantly advances the state-of-the-art rate-distortion performance for quality enhancement on Mars images.

Qualitative performance. Figure 9 compares the visual results of our MarsQE and compared approaches over the MIC test set with ground-truth images. It can be seen in Figure 9, our MarsQE approach successfully restores the ground-truth color and edge texture of Martian surface. To be specific, the color of stones (a) and the edge of sands (b) recovered by our MarsQE are significantly better than those recovered by other enhancement approaches. Besides, our MarsQE approach suppresses the compression artifacts around these edges, while these artifacts can hardly be reduced by other compared approaches. We also present the visual results of residual-to-raw images in Figure 10. It is evident that our MarsQE approach effectively restores the clear sky with the smallest residual values among all approaches. To summarize, our MarsQE approaches outperforms the compared approaches in subjective quality, especially in restoring details and suppressing compression artifacts.

Efficiency. We measure the efficiency of our MarsQE approach in terms of the number of parameters and FLoating point OPerations (FLOPs). Notably, our MarsQE approach outputs an enhanced image without the need for post-processing, as detailed in Section IV-B. This version is referred to as MarsQE-lite. As depicted in Figure 11, MarsQE-lite surpasses the second-best, CBDNet, by 0.08 dB in PSNR while having 67.0% fewer parameters. Even when compared with RBQE and DAQE, which have fewer parameters than CBDNet, MarsQE-lite stands out both in terms of PSNR performance and parameter efficiency. To achieve a higher PSNR improvement, MarsQE further enhances the PSNR score by an additional 0.25 dB using only 26.6% more parameters compared to CBDNet. Although some approaches, like AR-CNN and DCAD, have fewer parameters than both our MarsQE approach and CBDNet, their PSNR performance lags by at least 0.40 dB behind MarsQE-lite and 0.59 dB behind MarsQE, respectively. We also assess computational complexity in terms of FLOPs. Compared to the second-best, CBDNet, our MarsQE-lite achieves a higher PSNR with 10.3% fewer FLOPs, whereas MarsQE achieves a 0.25 dB higher PSNR score with just 30.7% more FLOPs. Specifically, for an input image at a resolution of $1152\times 1600$, CBDNet requires 1.10 TMACs, while MarsQE-lite requires only 0.99 TMACs and MarsQE requires 1.44 TMACs. In summary, our MarsQE-lite approach outperforms other compared approaches in both efficiency and enhancement performance, while our MarsQE secures higher-performance results with a modest increase in computational resources.

Network modules. In our MarsQE approach, we have introduced several crucial components. First, TEM is designed to extract semantic-related texture pattern from reference patches. In addition, TFM is developed to fuse semantic-related texture pattern for quality enhancement. To assess the significance of these reference-based modules, we conduct systematic ablation studies by removing TEM and TFM from the MarsQE framework, resulting in two distinct networks labeled as (A) and (B), as illustrated in Table V. Similarly, we apply a gradual module ablation strategy to the MarsQE-lite approach, generating two additional networks, denoted by (C) and (D). These ablations involved the removal of TEM and the substitution of TFM with a simple addition operator. Following these modifications, we re-train and evaluate all these networks using our dataset compressed by JPEG at QF $=30$.

The results, as shown in Table V, reveal that (1) the removal of TFM degrades PSNR by 0.04 dB for MarsQE, and (2) the further removal of TEM leads to an additional PSNR degradation of 0.10 dB. Similar degradation can also be observed for MarsQE-lite. These results indicate that the reference-based modules, i.e., TEM and TFM, positively contribute to the overall performance of image quality enhancement, underscoring their importance in our approach.

Semantics-based reference. During the construction of the reference dictionaries, we introduce four semantically distinct reference dictionaries for MarsQE, encompassing sky, sand, soil, and rock reference patches. To validate the effectiveness of these dictionaries, we adopt each dictionary independently for reference, rather than using all dictionaries collectively. For fair comparison, all validations are performed on our test set compressed by JPEG at QF $=30$. The exclusive use of sky, sand, soil, and rock reference dictionaries results in a degradation of PSNR values over our test set from 34.61 to 34.53, 34.57, 34.59, and 34.56 dB, respectively. Moreover, we evaluate the PSNR results for each semantic class of patches. As presented in Figure 12, when the semantic class of patches is consistent with that of dictionary, the PSNR values achieve their highest. The PSNR degradation of semantically mismatched references reaches up to 0.15 dB when sky reference patches are provided to soil test patches (i.e., from 1.98 to 1.83 dB). We also provide visual results with different reference in Figure 13. It can be seen that using reference patches with differing semantics can significantly impair the restoration of color and edges in rocks. In summary, the semantics-based reference pose a positive effect on our semantic-informed quality enhancement in terms of both quantitative and qualitative performance.

We further validate our MarsQE approach to the datasets beyond the MIC dataset [14], i.e., AI4MARS [23] and Mars32K [35], without fine-tuning the MarsQE network. Recall that MarsQE is trained on the MIC dataset with ground-truth RAW images captured by the Perseverance rover. In comparison, the AI4MARS and Mars32K datasets contain only compressed images captured by Curiosity, Opportunity, and Spirit rovers, without ground-truth counterparts. As a result, this extended validation poses a significant challenge to MarsQE with unseen data that lack ground-truth images and with unknown compression settings. Notably, in AI4MARS, 25.7% of the images, captured by Curiosity’s Navigation camera (Navcam), are grey-scale, posing a further challenge due to the absence of color information.

To be specific, we test our MarsQE model, which is pre-trained on MIC compressed by JPEG with QF $=50$, on images from Mars32K and AI4MARS. Given the absence of ground-truth data, we present the qualitative results and no-reference quality assessment results of MarsQE. The qualitative results, as illustrated in Figure 14, underscore MarsQE’s proficiency in suppressing compression artifacts, particularly the ringing effect observed at the boundaries of stones, sand, and rovers. In addition, we evaluate the quality of these images in terms of two widely-used no-reference quality metrics, i.e., Perceptual Index (PI) [49] and Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [50]. Note that smaller PI and BRISQUE signify a higher degree of perceptual quality. As shown in Figure 14, both metrics demonstrates the effectiveness of MarsQE on enhancing unseen compressed images with improved quality. To summarize, the MarsQE approach not only showcases its adaptability and robustness through these assessments but also confirms the potential applicability across a spectrum of Martian exploration missions, evidencing a broad generalization capability of our MarsQE approach.