MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt

Traditional object ReID aims to extract features with single-modal input, such as RGB images or depth maps. In the past few years, single-modal object ReID has achieved remarkable progress. However, relying solely on single-modal input, these methods may extract limited information in complex scenarios. Meanwhile, the generalization capability of multi-modal data has sparked increasing attention in multi-modal object ReID (Zheng et al. 2021; Li et al. 2020; Yin et al. 2023; He et al. 2023; Wang et al. 2023). However, multi-modal input introduces additional challenges, such as distribution gaps among different modalities (Liang et al. 2022) and modality laziness (Crawford et al. 2023). To address above challenges, Zheng et al. (Zheng et al. 2021) propose to learn robust features through a progressive fusion for multi-modal person ReID. For multi-modal vehicle ReID, Li et al. (Li et al. 2020) propose to learn balanced multi-modal features. Based on Transformers (Vaswani et al. 2017), Pan et al. (Pan et al. 2023) employ a feature hybrid mechanism to control modal-specific information. Recently, Zhang et al. (Zhang et al. 2024) propose to select diverse tokens with the cooperation of multi-modal information. Despite promising results, these methods still face challenges in generalization capability and model complexity. Especially for previous Transformer-based methods, full fine-tuning introduces high computational complexity. Thus, with the strong transfer capabilities of large-scale pre-trained models, we introduce the powerful CLIP into multi-modal object ReID with fewer trainable parameters and FLOPs.

With the development of foundational models (Yang et al. 2024), the transfer of pre-trained knowledge is attracting increasing attention (Diao et al. 2024a, b). Currently, mainstream PEFT methods can be divided into three categories: prompt tuning, adapter tuning and LoRA tuning. Prompt tuning (Lester, Al-Rfou, and Constant 2021) usually inserts layer-wise learnable tokens into the frozen backbone and guides the model to learn task-specific knowledge. Adapter tuning (Houlsby et al. 2019) introduces a plug-and-play module into the backbone, typically consisting of an MLP or attention module. LoRA tuning(Hu et al. 2021) uses a low-rank approximation as a side branch to reshape the backbone. With the above methods, PEFT demonstrates promising performance with limited resources. However, knowledge transfer for multi-modal fusion has been insufficiently explored, particularly in multi-modal object ReID. Therefore, we propose the PFA and SRP to leverage adapter tuning and prompt tuning for multi-modal object ReID. With the parallel adapters and synergistic transformations across multi-modal prompts, we can fully integrate the complementary information with lower complexity.

Recently, Mamba (Gu and Dao 2023) draws considerable attention (Wan et al. 2024; Xu et al. 2024) with the superior scalability of State Space Models (SSM) (Gu et al. 2021). By making the Structured State Space Sequence Model (S4) (Gu, Goel, and Ré 2021) data-dependent with a selection mechanism, Mamba outperforms Transformers in handling long sequences with linear complexity. In computer vision, ViS4mer (Islam and Bertasius 2022) and TranS4mer (Islam et al. 2023) demonstrate the effectiveness of SSM. To fully model the interactions between image patches, Vision Mamba (Zhu et al. 2024) and VMamba (Liu et al. 2024b) employ different scanning strategies. However, most of these methods focus on single-modal tasks. Additionally, attention-based aggregation methods often face quadratic complexity. To address these issues, we design the intra-modality and inter-modality aggregation with Mamba, which together form the complete MA block. This approach integrates complementary information with linear complexity and complements the progressive fusion from PFA and SRP, enhancing feature robustness for ReID.

To transfer CLIP’s knowledge to ReID tasks, we introduce the Parallel Feed-Forward Adapter (PFA). Previous methods (Diao et al. 2024b) typically insert adapters sequentially into networks. However, this can disrupt the original information flow, leading to suboptimal feature transformations. Especially in multi-modal settings, it is crucial to maintain the integrity of input features. To address these issues, our PFA is designed as a parallel branch to the Feed-Forward Network (FFN) rather than as a sequential operation. Taking the NIR modality as an example, as shown in Fig. 2 (a), the tokenized features $F_{1}^{n}$ are first processed through the initial Transformer layer with the following equation:

$$\bar{F}^{n}_{1}=\mathcal{M}(\mathcal{L}({F}^{n}_{1}))+{F}^{n}_{1},$$

(1)

where $\mathcal{L}$ represents the Layer Normalization (LN) (Ba, Kiros, and Hinton 2016) and $\mathcal{M}$ denotes the multi-head self-attention (Dosovitskiy et al. 2020). The output $\bar{F}^{n}_{1}$ is then fed into the PFA, undergoing the following transformation:

$$\bar{F}^{n_{PFA}}_{1}=\xi(\delta(\xi(\bar{F}^{n}_{1}))).$$

(2)

Here, $\xi$ is a linear layer and $\delta$ is the GELU activation function (Hendrycks and Gimpel 2016). Finally, we integrate the output $\bar{F}^{n_{PFA}}_{1}$ with the FFN, resulting in the feature $\hat{F}^{n}_{1}$:

$$\hat{F}^{n}_{1}=\mathcal{F}(\mathcal{L}(\bar{F}^{n}_{1}))+\bar{F}^{n_{PFA}}_{1}+\bar{F}^{n}_{1},$$

(3)

where $\mathcal{F}$ represents the FFN. This parallel structure preserves the original feature representations while enabling flexible and efficient knowledge transfer across different modalities. Additionally, PFA employs an ascending-then-descending pattern in linear layers. This design enhances its ability to adapt to complex multi-modal feature distributions, supporting more stable knowledge transfer.

To promote the joint learning of multi-modal features, we introduce the Synergistic Residual Prompt (SRP). Different from previous single modal prompt-based methods, our SRP is designed to integrate the modality-specific knowledge with synergistic transformations. Besides, current multi-modal prompt-based methods (Li et al. 2023; Khattak et al. 2023) often discard prompts from earlier layers, potentially resulting in the loss of discriminative information. To address these issues, our SRP introduces two key components: Synergistic Prompt (SP) and Residual Prompt (RP). The SP transfers modality-specific knowledge across different modalities, while the RP aggregates multi-modal information across layers with residual refinements.
Synergistic Prompt. Without loss of generality, we take the NIR modality as an example. In Fig. 2 (b), the NIR image is first tokenized by the patch embedding (Vaswani et al. 2017). The patch tokens $f_{\mathrm{pa}}^{n}\in\mathbb{R}^{D\times N_{pa}}$ are concatenated with the class token $f_{\mathrm{cls}}^{n}\in\mathbb{R}^{D}$ to form $F_{1}^{n*}\in\mathbb{R}^{D\times(1+N_{pa})}$:

$$F_{1}^{n*}=\left[f_{\mathrm{cls}}^{n},f_{\mathrm{pa}}^{n}\right],$$

(4)

where $[\cdot]$ is the concatenation operation. For the prompt tokens in NIR modality, they are composed of three parts: $p_{\mathrm{1}}^{n},p_{\mathrm{1}}^{r\to n},p_{\mathrm{1}}^{t\to n}\in\mathbb{R}^{D\times N_{pr}}$. Here, $p_{\mathrm{1}}^{n}$ is the randomly initialized NIR-specific prompt, while $p_{\mathrm{1}}^{r\to n}$ and $p_{\mathrm{1}}^{t\to n}$ are transferred from RGB and TIR modalities, respectively. Finally, all of them will be concatenated to form the input $F_{1}^{n}\in\mathbb{R}^{D\times(1+N_{pa}+3N_{pr})}$ of the first layer as follows:

$$F_{1}^{n}=\left[F_{1}^{n*},p_{\mathrm{1}}^{n},p_{\mathrm{1}}^{r\to n},p_{\mathrm{1}}^{t\to n}\right].$$

(5)

Here, $D$ is the embedding dimension and $N_{pa}$ is the number of patches. $N_{pr}$ is the number of prompts in each modality.

$$p_{\mathrm{1}}^{x\to n}=\mathcal{T}(p_{1}^{x})=\xi(\delta(\xi(p_{\mathrm{1}}^{x}))),$$

(6)

where $x$ can be $r$ or $t$. $\mathcal{T}$ is the transfer block composed of linear layers and a GELU activation function. Then, the output $\hat{F}_{1}^{n}$ of the first Transformer layer $\Phi_{1}$ is calculated with the input $F_{1}^{n}$ with the following equation:

$$\hat{F}_{1}^{n}=\mathrm{\Phi_{1}}(F_{1}^{n}).$$

(7)

Other modalities undergo similar operations. As a result, we can leverage SP to facilitate the transfer of discriminative modality-specific information across different modalities.
Residual Prompt. As shown in Fig. 2 (c), we first extract prompt tokens $\hat{p}_{1}^{n},\hat{p}_{1}^{r\to n},\hat{p}_{1}^{t\to n}$ from $\hat{F}_{1}^{n}$, with NIR modality as an example. Then, we average these prompts to form the fused prompts $\hat{p}_{1}^{f}\in\mathbb{R}^{D\times N_{pr}}$. Afterwords, we add the fused prompts $\hat{p}_{1}^{f}$ with linear layers to the new NIR-specific prompt $p_{2}^{n}$ to form the final prompts for the second layer:

$$p_{2}^{n^{{}^{\prime}}}=p_{2}^{n}+\xi(\delta(\xi(\hat{p}_{1}^{f}))),\hat{p}_{1}^{f}=\frac{\hat{p}_{1}^{n}+\hat{p}_{1}^{r\to n}+\hat{p}_{1}^{t\to n}}{3}.$$

(8)

Finally, the input $F_{2}^{n}$ of the second layer $\Phi_{2}$ is calculated as:

$$F_{2}^{n}=\left[F_{2}^{n*},p_{2}^{n^{{}^{\prime}}},\mathcal{T}(p_{2}^{r}),\mathcal{T}(p_{2}^{t})\right].$$

(9)

Other modalities and layers are processed in the same way:

$$\hat{F}_{l}^{n}=\mathrm{\Phi_{\mathnormal{l}}}(F_{l}^{n}),F_{l}^{n}=\left[F_{l}^{n*},p_{l}^{n^{{}^{\prime}}},\mathcal{T}(p_{l}^{r}),\mathcal{T}(p_{l}^{t})\right],$$

(10)

$$\hat{F}_{l}^{r}=\mathrm{\Phi_{\mathnormal{l}}}(F_{l}^{r}),F_{l}^{r}=\left[F_{l}^{r*},\mathcal{T}(p_{l}^{n}),p_{l}^{r^{{}^{\prime}}},\mathcal{T}(p_{l}^{t})\right],$$

(11)

$$\hat{F}_{l}^{t}=\mathrm{\Phi_{\mathnormal{l}}}(F_{l}^{t}),F_{l}^{t}=\left[F_{l}^{t*},\mathcal{T}(p_{l}^{n}),\mathcal{T}(p_{l}^{r}),p_{l}^{t^{{}^{\prime}}}\right],$$

(12)

where $l\in\{2,3,...,L\}$. With the synergistic transfer of modality-specific knowledge across different modalities and the residual refinements of multi-modal information, our SRP fully enhances the information synergism, promoting the feature discrimination. After the feature extraction from the backbone, we obtain $F^{n},F^{r},F^{t}\in\mathbb{R}^{D\times(1+N_{pa})}$ for each modality. Then, we extract class tokens $f^{n}_{cls},f^{r}_{cls},f^{t}_{cls}$ and concatenate them to form $f_{cls}\in\mathbb{R}^{3D}$ for supervision.

To efficiently model interactions between tokenized sequences from different modalities, we introduce Mamba Aggregation (MA) as a fusion strategy. As depicted in Fig. 3, MA is essentially composed of two parts: intra-modality MA and inter-modality MA. The intra-modality MA is designed to capture the discriminative information within each modality. While the inter-modality MA is introduced to fully integrate the complementary features across different modalities. In the intra-modality MA, we first extract patch tokens $f_{pa}^{n},f_{pa}^{r},f_{pa}^{t}\in\mathbb{R}^{D\times N_{pa}}$ from $F^{n}$, $F^{r}$ and $F^{t}$, respectively. In the left part of Fig. 3, each of them will undergo the Mamba block composed of Depth-Wise Convolution (DWConv) (Chollet 2017) $\mathcal{D}$, Batch Normalization (BN) (Ioffe and Szegedy 2015) $\mathcal{B}$, SSM $\Omega$ and SiLU activation function (Elfwing, Uchibe, and Doya 2018) $\beta$ with the following transformations:

$$\mathrm{\Theta}(\mathrm{\mathcal{X}})=\beta(\mathcal{B}(\mathcal{D}(\xi(\mathcal{X})))),\mathrm{\Psi}(\mathcal{X})=\mathrm{\beta}(\xi(\mathcal{X})),$$

(13)

$$f^{M}_{ssm}=\Omega(\mathrm{\Theta}(f_{pa}^{M}))\times\mathrm{\Psi}(f_{pa}^{M}),M\in\{n,r,t\}.$$

(14)

Here, $M$ denotes the modality. Then, features from different Mamba blocks are concatenated along the token dimension. After passing through a linear layer, the aggregated features are split by modality for the intra-modality MA:

$f_{ia}^{n},f_{ia}^{r},f_{ia}^{t}=\mathcal{S}\left(\xi\left(\left[f^{n}_{ssm},f^{r}_{ssm},f^{t}_{ssm}\right]\right)+\left[f_{pa}^{n},f_{pa}^{r},f_{pa}^{t}\right]\right),$

(15)

where $f_{ia}^{n},f_{ia}^{r},f_{ia}^{t}\in\mathbb{R}^{D\times N_{pa}}$. Here, $\mathcal{S}$ denotes the feature splitting along the token dimension. Thus, through a separable SSM for each modality, the intra-modality MA effectively captures interactions within each modality.

Furthermore, as shown in the right part of Fig. 3, $f_{ia}^{n}$, $f_{ia}^{r}$ and $f_{ia}^{t}$ will be sent to the inter-modality MA as follows:

$f_{ssm}^{a}=\Omega([\mathrm{\Theta}(f_{ia}^{n}),\mathrm{\Theta}(f_{ia}^{r}),\mathrm{\Theta}(f_{ia}^{t})])\times[\mathrm{\Psi}(f_{ia}^{n}),\mathrm{\Psi}(f_{ia}^{r}),\mathrm{\Psi}(f_{ia}^{t})].$

(16)

Here, $f_{ssm}^{a}\in\mathbb{R}^{D\times 3N_{pa}}$ represents the aggregated features obtained by modeling all modality tokens as the long sequences. Finally, we split the aggregated features into different modalities for the next MA block:

$$f_{ir}^{n},f_{ir}^{r},f_{ir}^{t}=\mathcal{S}(\xi(f_{ssm}^{a})+[f_{ia}^{n},f_{ia}^{r},f_{ia}^{t}]).$$

(17)

Through the cooperation of the intra-modality MA and inter-modality MA, MA fully models the interactions of patch tokens within and across modalities. After the stacked MA blocks, we concatenate the class tokens and the average of patch tokens of each modality. Then, we use a LN to stabilize the feature learning. With a linear reduction, we obtain the final features $f_{final}^{M},M\in\{n,r,t\}$ for each modality:

$$f_{final}^{M}=\xi(\mathcal{L}(\left[f_{cls}^{M},\mathcal{A}(f_{ir}^{M})\right])).$$

(18)

Here, $\mathcal{A}$ represents the average operation. Finally, we concatenate the features from different modalities to form $f_{ma}\in\mathbb{R}^{3D}$ for the loss supervision:

$$f_{ma}=[f_{final}^{n},f_{final}^{r},f_{final}^{t}].$$

(19)

With the above aggregation strategy, we can fully model the interactions across modalities with linear complexity.

As illustrated in Fig. 2, our objective function comprises two parts: losses for the image encoder of CLIP and the MA. For the backbone and MA, they are both supervised by the label smoothing cross-entropy loss (Szegedy et al. 2016) and triplet loss (Hermans, Beyer, and Leibe 2017):

$$\mathcal{L}_{g}=\mathrm{\lambda_{1}}\mathcal{L}_{ce}+\mathrm{\lambda_{2}}\mathcal{L}_{tri}.$$

(20)

Finally, the total loss for our framework can be defined as:

$$\begin{split}\mathcal{L}_{total}=\mathcal{L}_{g}^{CLIP}+\mathcal{L}_{g}^{MA}.\end{split}$$

(21)

Datasets and Evaluation Protocols. To fully evaluate the performance of our method, we conduct experiments on three multi-modal object ReID benchmarks. Specifically, RGBNT201 (Zheng et al. 2021) is a multi-modal person ReID dataset comprising RGB, NIR and TIR images. RGBNT100 (Li et al. 2020) is a large-scale multi-modal vehicle ReID dataset with diverse visual challenges, such as abnormal lighting, glaring and occlusion. MSVR310 (Zheng et al. 2022) is a small-scale multi-modal vehicle ReID dataset with more complex visual challenges. For evaluation metrics, we utilize the mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC) at Rank-K ($K=1,5,10$). Meanwhile, we report the GPU memory, trainable parameters and FLOPs for complexity analysis.
Implementation Details. Our model is implemented by using the PyTorch toolbox with one NVIDIA A100 GPU. We employ the pre-trained image encoder of CLIP (Radford et al. 2021) as the backbone. For the input resolution, images are resized to 256$\times$128 for RGBNT201 and 128$\times$256 for RGBNT100/MSVR310. For data augmentation, we employ random horizontal flipping, cropping and erasing (Zhong et al. 2020). For small-scale datasets (i.e., RGBNT201 and MSVR310), the mini-batch size is set to 64, with 4 images sampled for each identity and 16 identities sampled in a batch. For the large-scale dataset, i.e., RGBNT100, the mini-batch size is set to 128, with 16 images sampled for each identity. We set $\mathrm{\lambda_{1}}$ to 0.25 and $\mathrm{\lambda_{2}}$ to 1.0. We use the Adam optimizer to fine-tune the model with a learning rate of 3.5$\mathrm{e}^{-4}$. The warmup strategy with a cosine decay is used for learning rate scheduling. We set the total number of training epochs to 60 for RGBNT201/MVSR310 and 30 for RGBNT100, respectively.

Multi-modal Person ReID. As shown in Tab. 1, we compare our proposed MambaPro with state-of-the-art methods on RGBNT201. Within single-modal methods, PCB exhibits a remarkable mAP of 32.8%. For multi-modal methods, CNN-based methods like IEEE (Wang et al. 2022) and LRMM (Wu et al. 2025) achieve competitive performance with mAPs of 49.5% and 52.3%, respectively. However, Transformer-based methods outperform CNN-based methods. Especially, TOP-ReID (Wang et al. 2023) achieves a mAP of 72.3%. Moreover, our MambaPro achieves a 78.9% mAP and 83.4% Rank-1, surpassing TOP-ReID by 6.6% and 6.8%, respectively. These performance improvements verify the effectiveness of our proposed method.
Multi-modal Vehicle ReID. As depicted in Tab. 2, single-modal methods generally under-perform multi-modal methods. In single-modal methods, CNN-based methods like BoT (Luo et al. 2019) achieve better results than some Transformer-based methods. For multi-modal methods, LRMM (Wu et al. 2025) achieves a mAP of 78.6% on RGBNT100, showing the effectiveness of its low rank fusion strategy. Meanwhile, Transformer-based methods exhibit their superiority on integrating multi-modal information. Specifically, EDITOR (Zhang et al. 2024) achieves a mAP of 82.1%, surpassing LRMM by 3.5% on RGBNT100. Our MambaPro achieves a mAP of 83.9% on RGBNT100, surpassing EDITOR by 1.8%. Especially on the small-scale dataset MSVR310, our MambaPro achieves a mAP of 47.0%, surpassing EDITOR by 8.0%. These results clearly verify the effectiveness of our proposed methods in multi-modal object ReID tasks.

We conduct ablation studies on RGBNT201 to validate the effectiveness of different components. The locked image encoder of CLIP is denoted as CLIP(L) with frozen parameters. Meanwhile, we refer to the fully fine-tuned CLIP as CLIP(F). When MA is not applied, we use $f_{cls}$ for retrieval. Otherwise, we use $f_{ma}$ for retrieval.
Effect of Key Components. Tab. 3 shows the performance comparison with different components. Comparing Model A with Model B, we observe an impressive mAP improvement by fully fine-tuning CLIP, demonstrating the necessity of fine-tuning. With 32.9% of the trainable parameters compared with Model B, Model C outperforms the fully fine-tuned CLIP by 3.8% mAP, showing the effectiveness of PFA. Meanwhile, Model D also surpasses Model B with minimal additional parameters and FLOPs. Furthermore, through the combination of SRP and PFA, Model E achieves higher performance. Besides, with the efficient modeling of multi-modal long sequences, Model F achieves the best mAP of 78.9%. Model G, with the full fine-tuning of CLIP, achieves a comparable mAP of 78.7%. Comparing the last two rows, our method achieves robust performance no matter whether the backbone is fully fine-tuned or not. With our efficient learning framework, MambaPro transfers the knowledge from CLIP to multi-modal object ReID with superior performance and low complexity.

Effect of Different Prompt Mechanisms. Fig. 4 illustrates the details of different prompt mechanisms. Tab. 4 shows the corresponding performances. The first row represents that each modality has its own prompts without synergistic transfer. SP achieves better performance than Independent Prompt (IP), demonstrating the effectiveness of synergistic transformation between modalities. Both SRP (Separation) and SRP (Fusion) outperform SP, underscoring the effectiveness of residual connections in multi-modal prompts. The difference between SRP (Separation) and SRP (Fusion) lies in the $\hat{p}_{l}^{f}$ comes from the prompts of separate modalities or the fusion of them. SRP (Fusion) performs best by averaging tokens with the residual connections.
Effect of Different Adapters. Tab. 5 shows the performance with different adapters. Due to the weaker learning capability without non-linear transformations, LoRA generally achieves inferior performance compared with the Bottleneck Adapter (BNA). Meanwhile, with acceptable complexity, our PFA outperforms BNA by 4.7% mAP, verifying the effectiveness in multi-modal object ReID.
Effect of Different Aggregation Methods. In Tab. 6, we compare the performance with different aggregation methods. The Sum and Concat are simple aggregation methods that yield inferior results. The Transformer achieves better performance, demonstrating the effectiveness of attention-based aggregation. Our Intra-MA and Inter-MA outperform Transformers, showcasing the effectiveness of modeling multi-modal sequences with Mamba. Finally, our MA achieves the best performance, demonstrating the effectiveness of our proposed method.

Feature Distributions. In Fig. 5, we show the feature distributions with different modules. Comparing Fig. 5 (a) and Fig. 5 (b), CLIP(L) with PFA leads to more compact distributions across different IDs. Besides, with SRP, the gaps between different IDs in Fig. 5 (c) are further expanded. Finally, with MA, the feature distributions become more separable as shown in Fig. 5 (d). These visualizations strongly support the effectiveness of our proposed modules.
Feature Alignment of MA. In Fig. 6, we visualize the similarity distributions of different modalities before and after MA. The results clearly show that MA can fully align the distributions of different modalities, enhancing the feature aggregation. More details are in the supplementary material.

Mamba (Gu and Dao 2023) is a sequential modeling framework designed to effectively capture complex interactions within ultra-long sequences by leveraging State Space Models (SSMs). SSMs (Gu, Goel, and Ré 2021; Gu et al. 2021; Smith, Warrington, and Linderman 2022) are a type of sequence-to-sequence models inspired by continuous linear time-invariant (LTI) systems. With linear complexity, they can effectively capture the dynamics inherent in the system’s state variables, specifically defined as:

$$y(t)=Ch(t)+Dx(t),\dot{h}(t)=Ah(t)+Bx(t).$$

(22)

Here, $x(t)\in\mathbb{R}$ means the input, $h(t)\in\mathbb{R}^{N}$ denotes the hidden state and $y(t)\in\mathbb{R}$ represents the output. $\dot{h}(t)$ stands for the time derivative of $h(t)$. Besides, ${A}\in\mathbb{R}^{N\times N}$, ${B}\in\mathbb{R}^{N\times 1}$, ${C}\in\mathbb{R}^{1\times N}$ and ${D}\in\mathbb{R}$ are the system matrices, where $N$ is the hidden state dimension. Meanwhile, we often need to process discrete sequences like image and text. Thus, SSMs utilize zero-order hold discretization to discretize the system matrices. Then, the input sequence $x=\{x_{1},x_{2},...,x_{K}\}$ can be mapped to the output sequence $y=\{y_{1},y_{2},...,y_{K}\}$ with the following equations:

$$y_{k}=\overline{C}{h_{k}}+\overline{D}{x_{k}},h_{k}=\overline{A}{h_{k-1}}+\overline{B}{x_{k}}.$$

(23)

Although effective for sequence modeling, SSMs still faces limitations for the LTI property. To address this issue, the Selective State Space Model (S6) (Gu and Dao 2023) is introduced to make SSMs data-dependent. With the selection mechanism, S6 can effectively model the complex interactions in ultra-long sequences with linear complexity. In this work, we specifically design the intra-modality MA and inter-modality MA with S6, which can fully integrate complementary information from multiple modalities.

As shown in Fig. 7 (a), we take the NIR modality as an example to clarify it. In fact, SRP’s input consists of two parts: (1) prompt tokens $\hat{p}_{l-1}^{n},\hat{p}_{l-1}^{r\to n},\hat{p}_{l-1}^{t\to n}$ from the output of the last Transformer layer; (2) prompt tokens $p_{l}^{n},p_{l}^{r},p_{l}^{t}$ to be added in the current layer. The output includes $p_{l}^{n^{{}^{\prime}}}$ from RP and ${p}_{l}^{r\to n}$, ${p}_{l}^{t\to n}$ from SP. As shown in Fig. 7 (b), SP utilizes the Transfer Block (TB) to transfer complementary information from prompts of other modalities. Each modality utilizes its specific TB. RP aggregates information from $\hat{p}_{l-1}^{n},\hat{p}_{l-1}^{r\to n},\hat{p}_{l-1}^{t\to n}$ and injects the aggregated information into $p_{l}^{n}$, forming $p_{l}^{n^{{}^{\prime}}}$ with different levels of information. Finally, we concatenate the output of SRP with the class token and patch tokens, as the input of the current Transformer layer for further processing.

For the input resolution, images are resized to 256$\times$128 for RGBNT201 and 128$\times$256 for RGBNT100/MSVR310. For data augmentation, we employ random horizontal flipping, cropping and erasing (Zhong et al. 2020). A warmup strategy with a cosine decay is used for the learning rate scheduling. On the three datasets, we use a learning rate of 3.5$\mathrm{e}^{-4}$ for PEFT. To compare with the fully fine-tuned CLIP, we use a learning rate of 5$\mathrm{e}^{-6}$ on the CLIP backbone, while other parts are trained with 3.5$\mathrm{e}^{-4}$. In the small-scale datasets RGBNT201 and MSVR310, the mini-batch size is set to 64, 4 images sampled for each identity and 16 identities sampled for each batch. Both of the two datasets are trained with 60 epochs and the warm up epoch is set to 10. For the large-scale RGBNT100 dataset, to better utilize the data, we set the mini-batch size to 128, with 16 images sampled for each identity. This leads to a more stable training. Besides, we train the model for 30 epochs with a 5-epoch warm-up.

In Tab. 7, we compare the trainable parameters of our model with other methods, including both CNN-based methods (Zhou et al. 2019; Sun et al. 2018; Rao et al. 2021; Chang, Hospedales, and Xiang 2018; Li, Zhu, and Gong 2018; Qian et al. 2017; Li et al. 2020; Wang et al. 2022; Zheng et al. 2022; Guo et al. 2022) and Transformer-based methods (He et al. 2021; Wang et al. 2023; Crawford et al. 2023; Yin et al. 2023). In general, CNN-based methods have fewer parameters than Transformer-based methods. However, in Transformer-based methods, our MambaPro achieves the best performance with the fewest trainable parameters. In contrast, the parameter count of the other Transformer-based methods is at least 100M, with the highest reaching 324.53M. With comparable parameters as CNN-based methods, our MambaPro outperforms them by a large margin, verifying the effectiveness of our method.

Effect of $\mathrm{d_{state}}$ and $\mathrm{dt_{rank}}$ in MA. In Fig. 8, we compare the performance with different $\mathrm{d_{state}}$ and $\mathrm{dt_{rank}}$ in MA. In fact, $\mathrm{d_{state}}$ is the dimension of the hidden state $h_{k}$, while $\mathrm{dt_{rank}}$ is the rank when generating the discrete time steps $\mathrm{\Delta}$ in SSM. As shown in Fig. 8 (a), the performance first rises and then falls with the increase of $\mathrm{d_{state}}$. The best performance is achieved when $\mathrm{d_{state}}$ is set to 16. The reason is that a small $\mathrm{d_{state}}$ is not able to capture the complex multi-modal correlations, while a large $\mathrm{d_{state}}$ may lead to overfitting. As for $\mathrm{dt_{rank}}$, the performance is relatively stable. However, the performance exhibits an evident increase when $\mathrm{dt_{rank}}$ is bigger than 32. The bigger $\mathrm{dt_{rank}}$ can avoid information loss, which is beneficial for the performance improvement.
Effect of Varying Depths of Proposed Modules. In Fig. 9, we show the performance with different depths of PFA, SRP and MA. Meanwhile, we also perform the analysis on the length of prompts. Fig. 9 (a) shows that with more learnable prompts in each modality, the performance first increases and then decreases, achieving the best performance with $N_{pr}=4$. In Fig. 9 (b), the performance shows a large improvement with the depth increasing from 9-12 to 6-12. Furthermore, the best performance is achieved with SRP in every layer. Similar trends can be observed in Fig. 9 (c), with the best performance achieved with all layers. Meanwhile, the overall performance is higher than that of Fig. 9 (b), indicating the strong transfer capability of PFA. For Fig. 9 (d), stacked MA blocks improve the performance, with a 79.5% mAP achieved with 4 MA blocks. Considering the complexity, we choose 2 MA blocks as the best trade-off. These results verify the effectiveness of our proposed modules.
Effect of the Inserted Position of PFA. In Tab. 8, we compare the performance with different inserted positions of PFA. Within a Transformer layer, the main components include the Multi-Head Self-Attention (MHSA) and the Feed-Forward Network (FFN). Thus, we insert the PFA behind the FFN and the MHSA or on the side of the FFN and the MHSA, respectively. We observe that the PFA inserted on the side of original components achieves better performance. The main reason is that the stable structure of original components is not disturbed. By introducing additional knowledge from special domains with our proposed PFA, the pre-trained knowledge can be better utilized.

Effect of the Pre-trained Backbone. In fact, our framework can also change the backbone to ViT, which is pre-trained on the ImageNet. Thus, we show the performance with different pre-trained backbones in Tab. 9. Comparing the first line and the third line, we observe that CLIP (L) achieves better performance than ViT (L), which indicates the effectiveness of the CLIP pre-trained knowledge in multi-modal object ReID. Meanwhile, ViT (F) achieves competitive performance, showcasing the effectiveness of our modules.
Comparison with CLIP-based TOP-ReID. In Tab. 10, we compare our MambaPro with the CLIP-based TOP-ReID, with both models fully fine-tuned. The results show that TOP-ReID with CLIP underperforms compared to our MambaPro. Despite having nearly half the parameters, MambaPro achieves a 5.4% improvement in mAP and a 6.9% improvement in Rank-1. These findings suggest that our MambaPro can better leverage CLIP’s knowledge.

Effect of Mamba Block in MA. To clarify, the complete MA module includes both intra-modal and inter-modal interactions. A fair comparison is shown in Tab. 6 between Row 3 (standard Transformer applied across all three modalities) and Row 5 (using Mamba blocks). Mamba achieves a 1% mAP increase with nearly half the FLOPs of the Transformer. To further enhance modeling capability, we add Intra-MA, and the final MA achieves a 3.4% mAP improvement with FLOPs and parameters comparable to the Transformer block. Additionally, in Tab. 11, we validate MA’s effectiveness by replacing Mamba blocks in both MA-inter and MA-intra with Transformers, denoted as MA(Trans), verifying the superiority of our proposed Mamba-based MA.

In Tab. 12, we validate the effectiveness of key components on the vehicle dataset RGBNT100. Here, the memory consumption is measured with a batch size of 128. Comparing Model A with Model B, we observe a remarkable improvement in mAP, showcasing the necessity of knowledge transfer. In Model C, PFA outperforms the fully fine-tuned CLIP, showcasing the effectiveness of parallel knowledge transfer. Meanwhile, Model D shows competitive performance, showing the effectiveness of SRP. Furthermore, through the combination of SRP and PFA, Model E achieves higher performance. Finally, with the MA module, Model F achieves the best performance. Similarly, we also compare our method with the fully fine-tuned CLIP, which has a much higher complexity. In contrast, our method achieves better performance with lower complexity, validating the effectiveness of our proposed method on vehicle datasets.