MANet: Fine-Tuning Segment Anything Model for Multimodal Remote Sensing Semantic Segmentation

Semantic segmentation is a critical preprocessing step in remote sensing image processing, and leveraging multimodal information often yields better results than relying on a single modality. In recent years, the advent of deep learning has revolutionized the entire field of remote sensing, including semantic segmentation. Based on the classical encoder-decoder framework [8], numerous multimodal fusion approaches based on CNNs and Transformers have driven significant advancements in the field [30, 14, 15, 16]. ResUNet-a [31], an early architecture that based on CNNs, simply stacked multimodal data into four channels. vFuseNet [30] introduced a two-branch encoder to separately extract multimodal features, enabling deeper multimodal fusion through element-wise operations at the feature level. The introduction of Transformers [11, 12] has further enriched multimodal networks. For instance, CMFNet [14] employed CNNs for feature extraction and uses a Transformer structure to connect multimodal features across scales, emphasizing the importance of scale in multimodal fusion. Similarly, MFTransNet [16] used CNNs for feature extraction, while enhanced the self-attention module with spatial and channel attention for better feature fusion. FTransUNet [16] presented a multilevel fusion approach to refine the fusion of shallow- and deep-level remote sensing semantic features. Despite the great performance they achieved, we argue that existing models lack sufficient general knowledge, which poses a fundamental limitation to the advancement of multimodal fusion methods.

The Segment Anything Model (SAM) [17] holds a unique position as a general-purpose image segmentation model. This vision foundation model was trained on a very large visual corpus. It endows SAM with a remarkable ability to generalize to unseen objects, making it well-suited for applications in diverse scenarios. Nowadays, SAM has already been applied across various fields, such as autonomous driving [32], medical image processing [33] and remote sensing [34, 35, 36, 37]. In remote sensing, SAMRS [34] leveraged SAM to integrate numerous existing remote sensing datasets, using a new prompt named rotated bounding box. Furthermore, some recent works have considered fine-tuning SAM for remote sensing tasks such as semantic segmentation [27, 28, 38] and change detection [39, 40].

For single modality tasks, CWSAM [27] adapted SAM to synthetic aperture radar (SAR) images by introducing a task-specific input module and a class-wise mask decoder. MeSAM [28] incorporated an inception mixer into the SAM’s image encoder to retain high-frequency features and introduced a multiscale-connected mask decoder for optical images. For multimodal tasks, RingMo-SAM [38] introduced a prompt encoder tailored to the characteristics of multimodal remote sensing data, along with a category-decoupling mask decoder. It is observed that these methods focus on designing task-specific prompts and mask decoders. However, as discussed in Sec. I, the general knowledge in SAM is primarily centered in the image encoder. While these approaches successfully utilize SAM’s general knowledge to remote sensing, their complex architectures greatly hinder their adaptation to existing general semantic segmentation networks. Furthermore, there is currently no SAM-based multimodal approach designed for DSM data.

As discussed in Sec.I, SAM general knowledge is confined to its image encoder, specifically the ViT block, whose structure is shown in Fig.2(a). In [41], the Adapter was proposed to enhance the capabilities of ViT blocks for medical tasks through fine-tuning, as illustrated in Fig. 2(b). Instead of adjusting all parameters, the pre-trained SAM parameters remain frozen, while two Adapter modules are introduced to learn task-specific knowledge. Each Adapter consists of a down projection, a ReLU activation, and an up projection. The down projection compresses the input embedding to a lower dimension using a simple MLP layer, and the up projection restores the compressed embedding to its original dimension using another MLP layer. For a specific input feature $\bm{x}_{i}\in\mathbb{R}^{h\times w\times c}$, where $h$, $w$, and $c$ represent the height, width, and channels of the input feature, the Adapter’s process for generating the adapted feature can be expressed as:

where $\bm{W}_{d}\in\mathbb{R}^{c\times\hat{c}}$ and $\bm{W}_{u}\in\mathbb{R}^{\hat{c}\times c}$ are the down projection and up projection, respectively. $\hat{c}\ll c$ is the compressed middle dimension of the Adapter. After that, both the adapted feature $\bm{x}_{a}$ and the output of the original MLP branch are fused with $\bm{x}_{i}$ by residual connection to generate the output feature $\bm{x}_{o}$:

where $s$ is a scaling factor to weight the task-specific and task-agnostic knowledge.

Since the Adapter proposed in [41] was designed for single-modal data, it is referred to as the standard Adapter in the sequel.

Next, we extend the standard Adapter to multimodal tasks. The proposed MMAdapter is the core multimodal fine-tuning technology of this work. As illustrated in Fig. 3(a), we employ dual branches with shared weights to process multimodal information. The Adapter after the Multi-head Attention is retained to extract the features of each modality independently, while the Adapter during the MLP stage is replaced with the proposed MMAdapter. The detailed structure of MMAdapter is shown in Fig. 3(b). While preserving the core structure of the Adapter, the MMAdapter enables modality interaction through the introduction of two weighting coefficients, $\lambda_{1}$ and $\lambda_{2}$. For two specific multimodal input features, $\bm{x}_{i}\in\mathbb{R}^{h\times w\times c}$ and $\bm{y}_{i}\in\mathbb{R}^{h\times w\times c}$, the process of generating the adapted features with MMAdapter can be described as:

where $\bm{W}_{dx},\bm{W}_{dy}\in\mathbb{R}^{c\times\hat{c}}$ and $\bm{W}_{ux},\bm{W}_{uy}\in\mathbb{R}^{\hat{c}\times c}$ are the down projections and up projections, respectively. After that, the multimodal output features $\bm{x}_{o}$ and $\bm{y}_{o}$ are generated using $\lambda_{1}$ and $\lambda_{2}$ as follows:

During the fine-tuning stage, only the newly added parameters are optimized, while the other parameters remain fixed. Detailed annotations are provided in Fig. 2(b), while other subfigures omit these annotations for clarity and conciseness. Finally, it is worth mentioning that the design shown in Fig. 3 can be generalized to more than two modalities in a straightforward manner.

Fig. 4 shows an overview of the proposed MANet. The input to MANet is first processed by SAM’s image encoder endowed with MMAdapter, responsible for extracting and fusing multimodal remote sensing features using the multimodal fine-tuning mechanism. The output is then fed into DFM, which receives two single-scale multimodal outputs from the encoder and expands them into two sets of multiscale multimodal features using pyramid modules. These high-level abstract features are then fused by four squeeze-and-excitation (SE) Fusion modules to generate a group of multiscale features. Finally, the outputs of DFM are passed to the decoder to produce segmentation prediction maps. In this section, we will elaborate on the key components of the proposed MANet.

• •

  SAM’s image encoder: We denote the optical images and their corresponding DSM data as $\bm{X}\in\mathbb{R}^{H\times W\times 3}$ and $\bm{Y}\in\mathbb{R}^{H\times W\times 1}$, respectively, where $H$ and $W$ represent the height and width of the inputs. The SAM’s image encoder, employing a non-hierarchical ViT backbone, first embeds the input into a size of $\mathbb{R}^{h\times w\times c}$, where $h=\frac{H}{16}$, $w=\frac{W}{16}$, and $c$ is the embedding dimension. Then stacked ViT Blocks is used to extract features. This feature size is maintained throughout the encoding process [42]. As illustrated in Fig. 4, both $\bm{X}$ and $\bm{Y}$ are input into the SAM’s image encoder. It is important to note that the same SAM encoder is used for DSM data, which demonstrates that SAM can be utilized to extract features from non-image data. The SAM’s image encoder extracts and fuses multimodal features, generating high-level abstract features $\bm{F}_{x}\in\mathbb{R}^{h\times w\times c}$ and $\bm{F}_{y}\in\mathbb{R}^{h\times w\times c}$ through the multimodal fine-tuning modules.

  

• •

  DFM: Multiscale features play a critical role in semantic segmentation tasks since dense predicting requires both global and local information. As shown in Fig. 5(a), two pyramid modules, each consisting of a set of parallel convolutions or deconvolutions, are used to generate multiscale feature maps. Starting with the plain ViT feature map at a scale of $\frac{1}{16}$, we produce feature maps at scales of $\{\frac{1}{4},\frac{1}{8},\frac{1}{16},\frac{1}{32}\}$ using convolutions with strides of $\{\frac{1}{4},\frac{1}{2},1,2\}$, where fractional strides indicate deconvolutions [42]. These simple pyramid modules generate two sets of multimodal multiscale features, denoted as $\bm{F}_{x}^{i}$ and $\bm{F}_{y}^{i}$, where $i=\{1,2,3,4\}$ represents the scale index. Subsequently, four SE Fusion modules [16] are employed to further integrate the multimodal features. It is worth mentioning that more advanced fusion modules can yield further improved segmentation performance.

  As illustrated in Fig. 5(b), the SE Fusion module begins by aggregating the global information from the multimodal features. For the $i$-th fusion module, with an input channel size of $C_{i}$, the squeeze and excitation process is performed through two convolutional operations with a kernel size of $1\times 1$, followed by the ReLU and Sigmoid activations. The multimodal outputs are then weighted and combined element-wise, producing the enhanced fused features denoted by $\bm{F}_{f}^{i}$. The outputs from the four SE Fusion modules form the multiscale fusion features, denoted as $\bm{F}_{f}^{\emph{1-4}}$, which are fed into the decoder for processing. The decoder introduced in UNetformer [29] is employed in this work that reconstructs abstract semantic information into a segmented map by focusing on both global and local information.

Vaihingen: It contains $16$ fine-resolution Orthophotos, each averaging $2500\times 2000$ pixels. These Orthophotos consist of three channels: Near-Infrared, Red, and Green (NIRRG), and come with a normalized DSM at a $9$ cm ground sampling distance. The $16$ Orthophotos are split into a training set of $12$ patches and a test set of $4$ patches. To improve the storage and reading efficiency of large patches, a sliding window of size $256\times 256$ is utilized rather than cropping the patches into smaller images in both training and test stages, which results in about $960$ training images and $320$ test images.

Potsdam: This dataset is much larger than the Vaihingen dataset, which consists of $24$ high-resolution Orthophotos, each with the size of $6000\times 6000$ pixels. It includes four multispectral bands: Infrared, Red, Green, and Blue (IRRGB), along with a normalized DSM of $5$ cm. The last three bands in this dataset are utilized to diversify our experiments. The $24$ orthophotos are split into $18$ patches for training and $6$ for testing. Using the same sliding window approach, this dataset contains $10368$ training samples and $3456$ test samples.

The Vaihingen and Potsdam datasets classify five main foreground categories: Building (Bui.), Tree (Tre.), Low Vegetation (Low.), Car, and Impervious Surface (Imp.). Additionally, a background class labeled as Clutter contains indistinguishable debris and water surfaces. Visual examples from both datasets are presented in Fig. 6. Notably, the sliding window moves with a smaller step size, and the overlapping areas are averaged to the reduced boundary effect during the test stage.

All experiments were conducted using PyTorch on a single NVIDIA GeForce RTX 3090 GPU with 24GB of RAM. The stochastic gradient descent algorithm was used to train all the models under consideration with a learning rate of $0.01$, momentum of $0.9$, weight decay of $0.0005$, and a batch size of $10$. The batch size is reduced to $6$ when ViT-H is applied to meet the memory limit. Basic data augmentation techniques, including random rotation and flipping, are applied after sample collection with the sliding window.

To assess the semantic segmentation performance on multimodal remote sensing data, we use overall accuracy (OA), mean F1 score (mF1), and mean intersection over union (mIoU). These standard metrics enable a fair comparison between the proposed MANet and other state-of-the-art methods. Specifically, OA evaluates both foreground classes and the background class, while mF1 and mIoU are calculated for the five foreground classes.

We benchmarked the proposed MANet against fourteen state-of-the-art methods, including ABCNet [43], PSPNet [47], MAResU-Net [44], vFuseNet [30], FuseNet [9], ESANet [45], SA-GATE [48], CMGFNet [46], TransUNet [49], CMFNet [14], UNetFormer [29], MFTransNet [15], FTransUNet [16], and RS³Mamba [50], most of which were specifically designed for remote sensing tasks. In our experiments, ABCNet, PSPNet, MAResU-Net, UNetFormer, and RS³Mamba utilized only the optical images, to highlight the impact of DSM data and demonstrate the advantages of multimodal methods over single-modal ones. The other methods are state-of-the-art multimodal models based on different network architectures, including CNN and Transformer. The comparative quantitative results are presented in Table II and Table III.

To illustrate the necessity of multimodal fine-tuning, we conducted a modality analysis, with the results presented in Table IV. In the first experiment, we only used single-modality data and applied the standard Adapter mechanism to fine-tune the SAM’s image encoder. This experiment highlights the importance and need for multimodal information. In the second experiment, the SAM’s image encoder retained the standard Adapters but excluded the proposed MMAdapter. Therefore, this experiment could still extract remote sensing multimodal features independently, but it lacked crucial information fusion during the encoding process.

Inspection of Table IV reveals the significant effectiveness of incorporating multimodal information. The improvement is particularly pronounced in the categories of Building and Impervious surface, which tend to have significant and stable surface elevation information. Additionally, this enhancement strengthens the model’s ability to distinguish other categories. Overall, the integration of multimodal information provides comprehensive benefits across the board. The introduction of the MMAdapter enables more effective utilization of DSM information, significantly enhancing the model’s ability to extract and fuse multimodal information. As a result, the semantic segmentation performance is further improved.

The proposed MANet consists of two core components: the SAM’s image encoder with MMAdapter and the DFM. To validate their effectiveness, ablation experiments were conducted by systematically removing specific components. As shown in Table V, two ablation experiments were designed. In the first experiment, the DFM was removed from MANet, leading to a lack of deep analysis and integration of high-level abstract remote sensing semantic features. The second experiment follows the setup of the second experiments in Table IV.

It is important to note that removing all Adapters from the SAM’s image encoder severely degrades the model’s performance, as confirmed in Fig. 9, which also illustrates the effectiveness of the multimodal fine-tuning mechanism. Inspection of Fig. 9(c) and (e) reveal that SAM, without fine-tuning, cannot extract meaningful features from remote sensing data, rendering it unsuitable for semantic segmentation tasks. However, inspection of Fig. 9(d) and (f) shows that, after fine-tuning with the MMAdapter, the heatmaps change dramatically. Furthermore, Fig. 9(e) clearly demonstrates that SAM, despite being trained on RGB optical images, is also effective when applied to non-optical DSM data. It is observed that DSM can effectively provide supplementary information. Therefore, the fine-tuned SAM’s image encoder is capable of recognizing and segmenting remote sensing objects effectively in multimodal tasks.

Inspection of Table V indicates that both the MMAdapter and DFM are essential for enhancing the performance of the proposed MANet. Specifically, the MMAdapter facilitates continuous information fusion, allowing for extraction and fusion of multimodal information as the encoding depth increases. The DFM verifies the importance of high-level features in the semantic segmentation of remote sensing data. In this work, we primarily introduce a new framework for leveraging SAM, rather than emphasize the high-level feature fusion techniques. Replacing DFM with a more advanced fusion model is expected to result in further performance improvement.

The improved performance of MANet is largely attributable to the general knowledge provided by the vision foundation model, SAM, which is also a large model. However, the large model does not have advantages in terms of computational complexity or inference speed compared to existing general methods. Consequently, we focus on reporting the model’s trainable parameter number and memory footprint to measure its hardware requirements.

Table VI presents the results of the model scale for all methods compared in this work. As indicated in Table VI, the proposed parameter-efficient fine-tuning technique allows the large foundation models to be used on a single GPU while maintaining a manageable number of trainable parameters and memory costs. In our experiments, we successfully fine-tuned the ViT-L backbone on the same hardware with the same hyperparameters, achieving results that surpassed all existing methods. For the ViT-H backbone, we adjusted the batch size from $10$ to $6$ due to the GPU memory limitation. This reduction in batch size did not degrade performance, but further improved performance. These results prove the powerful feature extraction and fusion capabilities of the large vision foundation model. This work also offers valuable insights for exploring multimodal tasks with large models under constrained hardware conditions.