Learning Progressive Modality-shared Transformers for Effective Visible-Infrared Person Re-identification

Visible-infrared person ReID aims to retrieve the same person under different image modalities. In fact, Wu et al. (Wu et al. 2017) first explicitly defined the VI-ReID task and built a large-scale and challenging dataset. Existing VI-ReID methods usually adopt dual-stream networks and mine modality-shared features. For example, Ye et al. (Ye et al. 2018) propose an effective dual-stream network to explore modality-specific and modality-shared features simultaneously. Lu et al. (Lu et al. 2020) propose a mechanism of feature information complementarity to exploit the potential of modality-specific features. Gao et al. (Gao et al. 2021) propose a multi-feature space joint optimization network to enhance modality-shared features. Zhang et al. (Zhang et al. 2021b) introduce a dual-stream network to achieve the global-local multiple granularity learning. Based on dual-stream networks, Zhu et al. (Zhu et al. 2020) propose a Heterogeneous-Center (HC) loss to reduce the modality gaps. Liu et al. (Liu, Tan, and Zhou 2020) further design a heterogeneous-center triplet loss and explore the parameter sharing methods to improve the feature representation ability. Ye et al. (Ye, Shen, and Shao 2020) introduce gray-scale images as auxiliary modalities and realize the homogeneous augmented tri-modal learning. Fu et al. (Fu et al. 2021) propose the cross-modality neural architecture search and improve the structural effectiveness for VI-ReID. Hao et al. (Hao et al. 2021) introduce a modality confusion mechanism and a center aggregation method to reduce the differences between modalities. Meanwhile, many image generation-based methods are developed to mitigate the large modality gap. For example, Dai et al. (Dai et al. 2018) introduce the GAN framework for cross-modality image generation, and propose the so-called cmGAN for feature learning. Furthermore, Wang et al. (Wang et al. 2019) propose the AlignGAN and convert visible images to infrared images with joint pixel and feature alignment. Choi et al. (Choi et al. 2020) attempt to disentangle cross-modality representations with hierarchical structures. Li et al. (Li et al. 2020) introduce an auxiliary X-modality to generate robust features and bridge the different modalities. Although the above methods are effective, they usually introduce additional image noises and huge computational costs. Thus, they are difficult to deploy in practical scenarios.

Transformers (Vaswani et al. 2017) are initially proposed in Natural Language Processing (NLP). Recently, they have been utilized for some computer vision tasks, including person Re-ID. For visible-visible person ReID, He et al. (He et al. 2021) improve the Vision Transformer (ViT) (Dosovitskiy et al. 2020) with a side information embedding and a jigsaw patch module to learn discriminative features. Zhu et al. (Zhu et al. 2021) add the learnable vectors of part tokens to learn part features and integrate the part alignment into the self-attention. Lai et al. (Lai, Chai, and Wei 2021) utilize Transformers to generate adaptive part divisions. Zhang et al. (Zhang et al. 2021a) propose a hierarchical aggregation Transformer for integrating multi-level features. Chen et al. (Chen et al. 2021a) propose an omni-relational high-order Transformer for person Re-ID. Ma et al. (Ma, Zhao, and Li 2021) propose a pose-guided inter-and intra-part relational Transformer for occluded person Re-ID. As for VI-ReID, Liang et al. (Liang et al. 2021) improve the single-modality Transformer and try to remove modality-specific information. Chen et al. (Chen et al. 2022) utilize the human key-point information and introduce a structure-aware positional Transformer to learn semantic-aware modality-shared features. Although all the above Transformer-based methods have achieved superior performances, they generally lack of desirable modality-invariant properties. In this work, we explore the progressive modality-shared Transformers and learn reliable features for the VI-ReID task.

Although previous weight-shared structures can capture more modality-shared features, they are also susceptible to modality-specific noises. Besides, the pre-trained weights on ImageNet (Deng et al. 2009) generally have a stronger reliance on low-level features, such as color or texture. Thus, directly using these pre-trained models may miss some modality-specific information. Considering above facts, we design a progressive learning strategy. The key idea is to remove color information of visible images through gray-scale images. It also helps to learn the modality-independent discriminative patterns. In this way, the negative effects from large modality gaps are effectively mitigated.

Formally, we denote the visible image and the infrared image as $x^{vis}$ and $x^{ir}$, respectively. Then, the gray-scale image corresponding to $x^{vis}$ can be denoted as $x^{gray}$. By feeding {$x^{vis}$, $x^{gray}$, $x^{ir}$} into a weight-shared Transformer $\mathcal{F(\cdot)}$, we can obtain their corresponding embedding vectors:

$$f^{v}=\mathcal{F}\left(x^{vis}\right),f^{g}=\mathcal{F}\left(x^{gray}\right),f^{ir}=\mathcal{F}\left(x^{ir}\right).$$

(1)

To reduce the impact of modality-specific information, we further propose a Progressive Hard Triplet Loss (PHT). Similar to most VI-ReID methods, in each mini-batch, we randomly select $P$ identities and then select each identity’s $K$ visible and $K$ infrared images. Then, the proposed progressive hard triplet loss can be denoted as:

$$L_{PHT}=\left\{\begin{array}[]{ll}L_{Intra},&X=\left\{x^{gray},x^{ir}\right\}\\ L_{Global},&X=\left\{x^{vis},x^{ir}\right\}\end{array}\right.$$

(2)

	$\displaystyle{L_{Intra}}=\sum_{i=1}^{PK}\left[\max_{\forall y_{i}=y_{j}}D\left(f_{i}^{g},f_{j}^{g}\right)-\min_{\forall y_{i}\neq y_{k}}D\left(f_{i}^{g},f_{k}^{g}\right)+m\right]_{+}$		(3)
	$\displaystyle+\sum_{i=1}^{PK}\left[\max_{\forall y_{i}=y_{j}}D\left(f_{i}^{ir},f_{j}^{ir}\right)-\min_{\forall y_{i}\neq y_{k}}D\left(f_{i}^{ir},f_{k}^{ir}\right)+m\right]_{+}.$

$${L}_{Global}=\sum_{i=1}^{2PK}\left[\max_{\forall y_{i}=y_{j}}D\left(f_{i},f_{j}\right)-\min_{\forall y_{i}\neq y_{k}}D\left(f_{i},f_{k}\right)+m\right]_{+}.$$

(4)

where $D(\cdot,\cdot)$ represents a distance metric. $y_{i}$ is the identity label of the $i$-th image. $[z]_{+}=\max(z,0)$. $m$ is a margin.

As shown in Fig. 3, we divide the entire training process into two stages. At the first stage, the framework takes {$x^{gray}$, $x^{ir}$} as input, independently sampling positive and negative samples within each modality. With $L_{intra}$, the framework mainly focuses on learning the modality-independent discriminative patterns, thus effectively alleviating the negative effects caused by the large gap between visible and infrared modalities. At the second stage, we replace the inputs with {$x^{vis}$, $x^{ir}$} to take full advantages of modality-specific information for more fine-grained learning. With $L_{global}$, the framework will no longer distinguish different modalities and select positive and negative samples only based on feature distances. This can keep the raw image information, and allow the model to be benefited from modality-specific information.

In real scenes, there are large modality differences between visible and infrared images. Thus, it is essential to extract modality-invariant features. As shown in Fig. 4 (a), the red backpack appears only in the visible modality, so over-reliance on such features will lead to failure in cross-modality retrieval. Therefore, we introduce the MSEL to appropriately suppress the unreliable features that only appear in one modality and enhance the utilization of reliable modality-invariant features.

To the above goal, we explore potential information from all samples in a mini batch. Formally, we denote the anchor features of the infrared and visible modality as $f^{ir}_{a}$ and $f^{vis}_{a}$, respectively. Without loss of generality, we take $f^{ir}_{a}$ as an example. Firstly, we calculate its average distance to other positive samples under the intra modality and cross modality, denoted as:

$$\mathrm{D}^{intra}=\frac{1}{K-1}\sum_{\begin{subarray}{c}i=1\\ i\neq a\end{subarray}}^{K}D\left(f_{a}^{ir},f_{i}^{ir}\right),$$

(5)

$$\mathrm{D}^{cross}=\frac{1}{K}\sum_{i=1}^{K}D\left(f_{a}^{ir},f_{i}^{vis}\right).$$

(6)

Then, the $L_{MSEL}$ is defined as:

$$L_{MSEL}=\frac{1}{2PK}\sum_{p=1}^{P}\left[\sum_{a=1}^{2K}\left(\mathrm{D}^{intra}_{a}-\mathrm{D}^{cross}_{a}\right)^{2}\right].$$

(7)

In Eq. 7, $L_{MSEL}$ penalizes the difference between $\mathrm{D}^{intra}$ and $\mathrm{D}^{cross}$. When discriminative features that appear only within one modality, then the difference between $\mathrm{D}^{intra}$ and $\mathrm{D}^{cross}$ will increase, and such anomalies will be captured by $L_{MSEL}$. During the bi-direction optimization process of $\mathrm{D}^{intra}$ and $\mathrm{D}^{cross}$, the unreliable features that appear in only one modality will be suppressed, while the more reliable features that appear in both modalities will be enhanced as shown in Fig. 4 (b). Fig. 5 (a) shows the geometric illustrations of the MSEL. It encourages feature embeddings subject to a spherical distribution.

Similar to visible-visible person ReID, the same person may suffer large intra-class differences due to typical variations in pose, point of view, illumination, etc. They greatly increase the difficulty of feature alignment between different modalities. To address this problem, we propose a Discriminative Center Loss (DCL) to exploit the example relationships between center instances and enhance the discriminative power of reliable modality-shared features.

Firstly, to obtain the robust representation of each identity, we compute the feature center under the two modalities by:

$$c_{y_{i}}=\frac{1}{2K}\left(\sum_{j=1}^{\mathrm{K}}f_{j}^{vis}+\sum_{k=1}^{\mathrm{K}}f_{k}^{ir}\right).$$

(8)

Here, $c_{y_{i}}$ denotes the feature center of the $y_{i}^{th}$ identity. Then we calculate the average distance of $c_{y_{i}}$ to all other negative samples as a dynamic margin, which can be denoted as:

$$d_{y_{i}}^{neg}=\frac{1}{2K(P-1)}\sum_{\forall y_{j}\neq y_{i}}\left\|f_{j}-c_{y_{i}}\right\|_{2}.$$

(9)

Finally, the $L_{DCL}$ is defined as:

$$L_{DCL}=\frac{\sum_{i=1}\limits^{P}\mathop{mean}\limits_{y_{j}=y_{i}}\left\|f_{j}-c_{y_{i}}\right\|_{2}}{\sum_{i=1}\limits^{P}\mathop{mean}\limits_{\left\|f_{k}-c_{y_{i}}\right\|_{2}<d_{y_{i}}^{neg}\atop y_{k}\neq y_{i}}\left\|f_{k}-c_{y_{i}}\right\|_{2}}.$$

(10)

By minimizing Eq. 10, the intra-class compactness and inter-class separability will be improved. Fig. 5 (b) shows the geometric illustrations of the DCL. The utilization of $L_{DCL}$ has two main advantages: 1) It can utilize modality-specific features and capture more potential relationships than the center-center solution. 2) The dynamic sampling through $d_{y_{i}}^{neg}$ can effectively focus on relatively difficult examples. The effectiveness will be verified by experiments.

For model training, we adopt a hybrid loss function for our progressive learning framework. At the first stage, we utilize the identity loss ${L}_{ID}$ (Zheng, Zheng, and Yang 2017) and $L_{Intra}$ to learn modality-independent features:

$$L_{1}=L_{Intra}+L_{ID}.$$

(11)

At the second stage, we further extract the reliable modality-shared features with $L_{MSEL}$ and enhance the discrimination with $L_{DCL}$. The loss function can be defined as:

$$L_{2}=L_{Global}+\lambda_{1}L_{MSEL}+\lambda_{2}L_{DCL}.$$

(12)

Here, the parameters $\lambda_{1}$ and $\lambda_{2}$ are used to balance the terms of $L_{MSEL}$ and $L_{DCL}$, respectively.

In this subsection, we conduct experiments to verify the effects of different modules on the SYSU-MM01 dataset under the all-search mode.

In this subsection, we compare our proposed PMT with other state-of-the-art methods on SYSU-MM01 and RegDB.