ASANet: Asymmetric Semantic Aligning Network for RGB and SAR image land cover classification

The significance of LCC in remote sensing images cannot be overstated, given its pivotal role in addressing a wide range of practical application requirements (Li et al., 2022a). Within this context, semantic segmentation is extensively employed to guarantee accurate classification of various land cover categories. Initially, researchers relied on machine learning techniques, such as random forest (Rodriguez-Galiano et al., 2012) and K-means clustering (He et al., 2014), to segment high-resolution remote sensing images, which aided in the efficient extraction of surface features.

With the rapid advancement of deep learning techniques, models based on Convolutional Neural Networks (Yang et al., 2018) and the Transformer (Zhou et al., 2023) series have become widely utilized in remote sensing image analysis. For instance, SLCNet (Yu and Ji, 2023), which integrates global and local feature correlations, demonstrates superior performance across three remote sensing datasets. Nevertheless, fully supervised semantic segmentation methods (Costa et al., 2018; Zhao et al., 2023) are increasingly challenged in meeting the demands of large-scale land mapping tasks. (Ma et al., 2023) introduced an alternative approach in which an unsupervised domain adaptation method for LCC is developed, leveraging local consistency and global diversity. This method utilizes a subset of annotated source domain data along with unlabeled target domain data to improve the generalization capability of the semantic segmentation model. This adaptation strategy enhances the model’s adaptability for LCC in diverse geographical areas. Additionally, a prominent remote sensing big-model (Osco et al., 2023) aims to promote the extensive application of semantic segmentation models in real-world scenarios. While these strategies contribute to making the segmentation process more intelligent, challenges persist when dealing with RGB images captured under challenging conditions and extreme weather. Such conditions can pose difficulties for models to learn effectively and segment feature classes accurately.

SAR has garnered considerable attention from researchers due to its microwave-active imaging mechanism, which enables all-weather imaging capabilities. Early studies (Orlíková and Horák, 2019) exploited the sensitivity of different polarization SAR images to feature types for LCC using machine learning. Additionally, the segmentation of water and soil areas (Guo et al., 2022) could be relatively easily achieved by utilizing the water path scattering property of SAR. However, the coherent imaging mechanism of SAR introduces a high degree of variability at the individual pixel level, known as coherent scattering (Lu, 2021), which poses significant challenges for LCC. The GLNS (Liu et al., 2022) addresses this challenge by integrating both local and global features to extract more efficient and discriminative features and addresses the issue of excessive inter-class distances in SAR by incorporating a twofold loss function. (Liu et al., 2019) proposed combining representation learning with statistical analysis to capture the statistical properties of SAR in feature space. While these methods offer various perspectives for mitigating the interference caused by coherent scattering in SAR, the LCC task based on SAR remains challenging. This challenge is primarily due to the lack of rich texture and the absence of color information in SAR images.

In challenging real-world application scenarios, single modal algorithms often struggle to fully meet the task requirements. For example, in tasks like semantic segmentation for autonomous driving in foggy conditions (Rizzoli et al., 2022), intelligent interpretation of remote sensing images with cloud cover (Li et al., 2023a), or medical image diagnosis relying on multimodal data (Hermessi et al., 2021), the complexities of these real-world scenarios have led to the development of multimodal segmentation algorithms.

As shown in Fig. 1a), the model trains on the fused images generated from the data of the two modalities to obtain segmentation results, or it fuses the segmentation results of the two modalities and calculates the loss. (Xu et al., 2023b) proposed generating the RGB cloud-removed image from the RGB-SAR dataset, which is subsequently used for LCC. Some scholars have also employed methods such as Principal Component Analysis, HSV Color Space, and Generative Adversarial Networks (Bermudez et al., 2019) to create fused images, which are then input into the network for training. The quality of the generated images significantly affects the segmentation results (He and Yokoya, 2018). (Hang et al., 2020) introduced a network where the post-processing results are obtained through the weighted summation of the segmentation results from the two modalities. The loss is calculated by comparing these results with the ground truth, providing feedback for network learning. However, this method encounters challenges in learning the complementary relationship between the two modalities. To address these issues, MMFNet (Li et al., 2023b) computes phase features of RGB and SAR images as intermediate features for multi-level training, thereby avoiding direct feature interaction between the modalities. Nonetheless, the multi-level network model presents a complex structure requiring longer training time.

Typically, symmetric Siamese networks are employed to construct the multimodal framework, which facilitates the adaptive learning of feature correlations between the two modalities. As depicted in Fig. 1b), feature fusion occurs within the encoder or decoder section of the framework, and one or more segmentation heads are deployed to generate the final segmentation outcome. Methods such as FuseNet (Hazirbas et al., 2017) and the network proposed by (Gao et al., 2023) fuse feature maps from both modalities at the decoder stage and then feed them into the segmentation head for the segmentation process. CMFNet (Ma et al., 2022b) and FTransUNet (Ma et al., 2024) propose the use of cross-scale fusion modules, which are designed to integrate features across different modalities and at multiple levels. This fusion strategy integrates the feature extraction component, facilitating the learning of feature correlations. However, it does not provide a feedback mechanism to the branches for feature interaction. The attention weights for feature interactions within SwinTFNet are adjusted exclusively for the RGB branches. CMX (Zhang et al., 2023a) introduces an attention module named CM-FRM, which is specifically designed to learn the weights of two modal channel and spatial features through feature interaction. These interacted feature maps are then passed to the subsequent layer for further learning. The CM-FRM module enhances the network’s capability to integrate feature information from another modality within the encoder. Other networks, including ACNet (Hu et al., 2019), SA-Gate (Chen et al., 2020), and CroFuseNet (Wu et al., 2023), also utilize distinct attention modules in their encoder sections to calculate feature weights between the two modalities. These weights are employed to selectively fuse features across the two modalities. While these interactive modules calculate feature weights that highlight the correlation between features of the two modalities and underscore the significance of feature fusion, they often fail to consider the feature independence within each modality. These interaction modules highlight the correlation between features of two modalities and emphasize the importance of feature fusion. However, they often ignore the feature independence within the RGB and SAR modalities, and only output a feature weight that leads to the same feature representation for both modalities to learn.

The overall network structure of ASANet is provided in Fig. 2, which is structured on a dual-branch encoder and decoder architecture. In the encoder, we employ two branches where weights are not shared to concurrently extract features from RGB and SAR images, thereby capturing the unique characteristics of each modality in a parallel and interactive manner. The model’s residual structure enables it to discern and leverage the substantial differences among various data modalities, resulting in the generation of more effective and complementary features for subsequent stages of feature extraction and interaction. To optimally fuse the two modal features after focusing on the SFM, we propose the CFM. The decoder can be based on nearly any design from SOTA segmentation networks. For broad applicability, we select UPerNet (Xiao et al., 2018) as our preferred decoder design.

Based on the analysis above, it is clear that the feature information required for RGB and SAR images is usually different. Consequently, this study introduces an innovative SFM that obtains different information to adjust the features of parallel branches. Drawing from the channel attention mechanism (Hu et al., 2018), the SFM handles multimodal features along the channel axis. Unlike previous methods which use a single attention matrix to weigh different branches, our approach capitalizes on both consistent and complementary features. The architecture of the SFM is shown in Fig. 3. Its design allows the SFM to compute attention matrices for individual branches, focusing on distinctive and complementary features between them. This provides a robust foundation for subsequent network layers to accurately calibrate and align features from the two modalities.

Specifically, the process comprises three main steps. We denote the feature maps of the two modalities as $\mathrm{F_{RGB}\in\mathbb{R}^{C\times H\times W}}$ and $\mathrm{F_{SAR}\in\mathbb{R}^{C\times H\times W}}$, respectively, where C, H, and W correspond to the number of channels, height, and width of the feature maps. Superscripts r (RGB) and s (SAR) are utilized to distinguish between the feature maps of the two modalities.

Obtain differential features: As previously mentioned, to capture the complementarity between different modalities, we extract complementary features from each modality separately. Following the principle of differential amplification (Qingyun and Zhaokui, 2022), the SFM performs a pixel-wise subtraction between the $\mathrm{F_{RGB}}$ and $\mathrm{F_{SAR}}$ feature maps to obtain the differential feature map $\mathrm{F_{\bigtriangleup}^{r}\in\mathbb{R}^{C\times H\times W}}$ for the RGB modality. Conversely, for the SAR modality, a subtraction is performed between $\mathrm{F_{RGB}}$ and $\mathrm{F_{SAR}}$ to obtain the differential feature map $\mathrm{F_{\bigtriangleup}^{s}\in\mathbb{R}^{C\times H\times W}}$. The first step can be described as Eq. 1:

where - denotes the pixel-wise subtraction operation.

Adaptive refinement learning: Furthermore, we apply global max-pooling to the differential feature maps $\mathrm{F_{\bigtriangleup}^{r}}$ and $\mathrm{F_{\bigtriangleup}^{s}}$ to obtain channel weights $\mathrm{s_{1}^{r}\in\mathbb{R}^{C\times 1\times 1}}$ and $\mathrm{s_{1}^{s}\in\mathbb{R}^{C\times 1\times 1}}$. Subsequently, we introduce multiple convolutional modules resembling channel-wise perceptrons to perform fine-grained learning on $\mathrm{s_{1}^{r}}$ and $\mathrm{s_{1}^{s}}$, resulting in global differential weights $\mathrm{s_{2}^{r}\in\mathbb{R}^{C\times 1\times 1}}$ and $\mathrm{s_{2}^{s}\in\mathbb{R}^{C\times 1\times 1}}$. Eq. 2 describes the second step.

where $\mathrm{\mathcal{F}_{GMP}}$ denotes global max-pooling, $\mathrm{\mathcal{F}_{1}}$ represents a convolutional layer with an output dimension of $\mathrm{max(32,\frac{C}{16})}$, $\mathrm{\mathcal{F}_{2}}$ is a convolutional layer with an output dimension of C, and $\mathrm{\mathcal{F}_{GELU}}$ signifies the GELU activation function.

Correction of parallel branches: The formula for correcting the original feature maps is as follows:

where $\mathrm{F_{SFM}^{r}\in\mathbb{R}^{C\times H\times W}}$ and $\mathrm{F_{SFM}^{s}\in\mathbb{R}^{C\times H\times W}}$ represent the output feature maps of the dual branches, respectively. $\mathrm{\sigma}$ is the sigmoid function and $\mathrm{\otimes}$ is the pixel-wise multiplication.

The feature information in RGB and SAR images is often complementary. The challenge lies in effectively calibrating and aligning the complementary information from both modalities. Traditional feature fusion strategies, such as simple concatenation or summation of feature maps, may inadvertently amplify the noise present in both modalities, thereby negatively impacting the final classification performance. To overcome this issue, inspired by CBAM (Woo et al., 2018), we propose a CFM for feature fusion. As illustrated in Fig. 4, the CFM adopts a concatenated attention mechanism to learn deep representations of RGB and SAR features in both channel and spatial dimensions. This fusion approach can adaptively calibrate and align the complementary information from the two modalities, reducing the interference of noise information and improving the model’s performance.

Suppose the input feature maps to the CFM are $\mathrm{F_{RGB}\in\mathbb{R}^{C\times H\times W}}$ and $\mathrm{F_{SAR}\in\mathbb{R}^{C\times H\times W}}$, where C, H, and W represent the number of channels, height, and width of the feature maps, respectively. The superscripts r (RGB) and s (SAR) are used to distinguish between the feature maps of the two modalities.

In the first attention module, we align and calibrate the features from the two branches along the channel dimension. Initially, the feature maps $\mathrm{F_{RGB}}$ and $\mathrm{F_{SAR}}$ are concatenated. Subsequently, the Concatenated feature map is fused through convolutional layers, followed by global average-pooling to obtain the feature vector $\mathrm{s\in\mathbb{R}^{C\times 1\times 1}}$. GAP generate channel-wise statistics by shrinking concatenated feature through its spatial dimensions. The feature vector s is then input into two separate Multi-Layer Perceptrons (MLPs) with non-shared weights to capture channel-wise dependencies. After passing through the sigmoid function, different fusion channel weights are obtained for each modality. These weights are then multiplied with the original feature maps to obtain the feature maps $\mathrm{F_{s}^{r}\in\mathbb{R}^{C\times H\times W}}$ and $\mathrm{F_{s}^{s}\in\mathbb{R}^{C\times H\times W}}$. The process is described in formulaic terms as follows:

where $\mathit{cat}$ refers to concatenating feature maps along the channel dimension, $\mathrm{\mathcal{F}_{GAP}}$ denotes global average-pooling, $\mathrm{\mathcal{F}_{1}^{c}}$ represents a convolutional layer with an input dimension of 2C and an output dimension of c. $\mathrm{\mathcal{F}_{MLP_{1}}}$ and $\mathrm{\mathcal{F}_{MLP_{2}}}$ signify MLP. $\mathrm{\sigma}$ represents the sigmoid activation function, $\mathrm{\otimes}$ denotes the pixel-wise multiplication operation. CFM learn different channel attention weights, enabling the adaptive selection and calibration of the original features from the two branches.

In the second attention module, we align and calibrate the features from the two branches along the spatial dimensions. Initially, the feature maps $\mathrm{F_{s}^{r}}$ and $\mathrm{F_{s}^{s}}$ are concatenated, and the concatenated feature map undergoes processing through a convolutional layer to fuse the information, yielding feature map z. This feature map $\mathrm{z\in\mathbb{R}^{1\times H\times W}}$ is then convolved with two sets of distinct weights to enhance the feature representation. The softmax function is applied to derive the weight scores for the respective modal space dimensions, which represent the importance of each pixel. These scores are used to weight the feature maps $\mathrm{F_{s}^{r}}$ and $\mathrm{F_{s}^{s}}$, resulting in the fusion spatial attention weighted feature maps $\mathrm{F_{z}^{r}}$ and $\mathrm{F_{z}^{s}}$.

where $\mathit{cat}$ refers to concatenating feature maps along the channel dimension, $\mathrm{\mathcal{F}_{2}^{c}}$ represents a convolutional layer with an input dimension of 2C and an output dimension of c, $\mathrm{\mathcal{F}_{3}}$ and $\mathrm{\mathcal{F}_{4}}$ represent convolutional layers with both input and output dimensions C. $\mathrm{\otimes}$ represents pixel-by-pixel multiplication, and $\mathrm{\mathcal{F}_{softmax}}$ is a softmax function. CFM calculate the spatial attention weights for different modalities, adaptively strengthening the effective features and suppressing the invalid ones.

Finally, According to Eq. 6, $\mathrm{F_{z}^{r}}$ and $\mathrm{F_{z}^{s}}$ are pixel-wise added to produce the composite output feature map $\mathrm{F_{CFM}\in\mathbb{R}^{C\times H\times W}}$.

where + denotes the pixel-wise addition (PWA) operation.

In this paper, all model training and testing are conducted on two Quadro RTX 8000 GPUs, which are running on the Ubuntu 18.04 Linux platform. The batch size per GPU is set to 4, and a total of 80,000 iterations are performed. The model is validated every 8000 iterations. The metrics reported in this paper are indicators of the number of iterations corresponding to the best mIoU. The results for all models are based on the accuracy of our own reproduction. The hyperparameter settings for each model are listed in the table. Specifically, pre-trained weights from ImageNet22k are adopted to initialize two ConvNeXtV2-tiny (Woo et al., 2023) backbone networks within ASANet. The primary data augmentation techniques include random scaling, random cropping, and random flipping. To ensure the fairness of the experiment, all experimental configurations outlined in this paper were strictly adhered to throughout the study. The evaluation metrics include mIoU, Kappa, and Overall Accuracy (OA), which are commonly employed in semantic segmentation assessments. We also used floating point operation count (FLOPs), the number of model parameters, and FPS to evaluate the complexity of the model (see Table 1).

To evaluate the models’ performance, we conducted a detailed comparison of six multimodal semantic segmentation models: CMX (Zhang et al., 2023a), SA-Gate (Chen et al., 2020), FTransUNet (Ma et al., 2024), CMFNet (Ma et al., 2022b), AFNet (Xu et al., 2021), and FuseNet (Hazirbas et al., 2017). The mIoU performance of various methods on three datasets is presented in Table 2, where our proposed ASANet demonstrates SOTA results. ASANet inference is 15.2 FPS faster than the suboptimal model CMX. Furthermore, we tested the UperNet model, which employs the ConvNeXtV2-tiny backbone, for single-modal experiments. All models were configured to ensure comparability, facilitating a fair evaluation of both multimodal and single-modal segmentation capabilities.

In this section, we conduct ablation experiments based on the PIE-RGB-SAR dataset. The results of these ablation experiments between modules are presented in Table 6. Specifically, the third row of results shows the twin network framework with the SAR mode, which uses the PWA (as described in Eq. 6) to fuse feature maps of the two modes before they are input into the decoder. To evaluate the effectiveness of the CFM, we replace PWA with the CFM. In assessing the effectiveness of the SFM, we integrate the SFM for feature interaction within the network framework, as shown in the third and fourth rows.

The backbone network consists of four stages that output features at different scales: stage-1, stage-2, stage-3, and stage-4. In this section, we choose a baseline model that utilizes PWA module for fusion in all stages. Furthermore, SFM and CFM modules are introduced at various stages for ablation experiments. The fusion results for each stage are presented in Table 7. Our experimental findings indicate a decrease in accuracy when the SFM and CFM modules are introduced at stage-1 and stage-4. This reduction in accuracy can be attributed to the insufficient semantic information in the feature maps of the lowest stage and the limited local information in the feature maps of the highest stage. The complex feature interactions and fusions that are unique to stage-1 or stage-4 tend to amplify noise from the two modalities, thereby resulting in adverse effects.

The outcomes of incorporating SFM and CFM across different stages indicate that water bodies, forests, and croplands exhibit superior performance at stage-2, with roads and cities performing optimally at stage-3. The local details and texture features in stage-2 benefit significantly from integration. For categories that require expansive receptive fields and rich semantic information, stage-3 offers the most effective fusion. However, at stage-4, the drop in accuracy is more significant, due to the reduced availability of location-specific information.

The network that integrated SFM and CFM modules across all stages achieved the highest overall accuracy. However, the $\mathrm{IoU_{road}}$ and $\mathrm{IoU_{other}}$ decreased compared to the network where only the stage-3 was enhanced. This suggests that features fused across different stages may negatively impact certain categories. For detection tasks requiring high accuracy for specific categories, combining fusion strategies across different stages can yield superior results.

To assess the model’s effectiveness and superiority in capturing large-scale cloudy scenes, we generated RGB images under simulated cloudy conditions using GIMP. (The foggify filter was used to perform the simulation, with fog color set to white, turbulence set to 1.5, and opacity set to 100.) The simulated image is depicted in Fig. 13. Adhering to the methodology employed in PIE-RGB-SAR, we cropped the images and segmented the dataset. The experimental outcomes for diverse models are summarized in Table 8.

When compared to the original RGB image, the kappa, OA, and mIoU of the simulated S-RGB single-mode image decreased by 2.18%, 1.53%, and 3.72%, respectively. This suggests that foggy conditions can adversely affect detection outcomes. Nevertheless, in the simulated environment, our ASANet achieved the best performance, with the mIoU index 1.07% to 21.06% higher than the corresponding indices of other models. The advantage of ASANet is more obvious in the road and water categories.