Parameter Sharing Exploration and Hetero-center Triplet Loss for Visible-Thermal Person Re-Identification

Triplet loss was first proposed in FaceNet [17], and then improved by mining the hard triplets [8]. The core idea is to form batches by randomly sampling $P$ identities, and then randomly sampling $K$ images of each identity, resulting in a mini-batch with $PK$ images. For each sample $x_{a}$ in the mini-batch, we can select the hardest positive and hardest negative samples within the mini-batch to form the triplets for computing the batch hard triplet loss,

which is defined for a mini-batch $X$, where a data point $x_{a}^{i}$ denotes the $a^{th}$ image of the $i^{th}$ person in the batch, $[x]_{+}=\max(x,0)$ denotes the standard hinge loss, $\|x_{a}-x_{p}\|_{2}$ denotes the Euclidean distance of data point $x_{a}$ and $x_{p}$, $\rho$ is the margin.

Due to our two-stream structure respectively extracting features for visible and thermal images, we introduce the following online batch sampling strategy. Specifically, $P$ person identities are first randomly selected at each iteration, and then we randomly select $K$ visible images and $K$ thermal images of the selected identity to form the mini-batch, in which a total of $2*PK$ images are obtained. This sampling strategy can fully utilize the relationship of all the samples within a mini-batch. In this manner, the sample size of each class is the same, which is important to avoid the perturbations caused by class imbalance. Moreover, due to the random sampling mechanism, the local constraint in the mini-batch can achieve the same effect as the global constraint in the entire set.

Eq. (5) shows that triplet loss computes the loss by comparison of the anchor to all the other samples. It is a strong constraint, perhaps too strict to constrain the pairwise distance if there exist some outliers (bad examples), which would form the adverse triplet to destroy other pairwise distances. Therefore, we consider adopting the center of each person as the identity agent. In this manner, we can relax the strict constraint by replacing the comparison of the anchor to all the other samples by the anchor center to all the other centers.

First, in a mini-batch, the center for the features of every identity from each modality is computed,

which is defined for a mini-batch, where $v_{j}^{i}$ denotes the $j^{th}$ visible image feature of the $i^{th}$ person in the mini-batch, while $t_{j}^{i}$ corresponds to the thermal image feature.

Therefore, based on our $PK$ sampling method, in each mini-batch, there are $P$ visible image centers {$c_{v}^{i}|i=1,\cdots,P$} and $P$ thermal centers {$c_{t}^{i}|i=1,\cdots,P$}, as shown in Fig. 3. In the following, all the computations are only performed on the centers.

The goal of metric learning is to make those features from the same class close to each other (intra-class compactness), while those features from different classes are far away from each other (inter-class separation). Therefore, in our VT cross-domain Re-ID, based on the $PK$ sampling strategy and calculated centers, we can define the hetero-center triplet loss as,

which is defined on mini-batch centers $C$ including both visible centers {$c_{v}^{i}|i=1,\cdots,P$} and thermal centers {$c_{t}^{i}|i=1,\cdots,P$}. For each identity, $L_{hc\_tri}$ concentrates on only one cross-modality positive pair and the mined hardest negative pair in both the intra- and inter-modality.

Comparing the general triplet loss $L_{bh\_tri}$ (Eq. (5)) to our proposed center-based triplet loss $L_{hc\_tri}$ (Eq. (7)), we replace the comparison of the anchor to all the other samples by the anchor center to all the other centers. This modification has two major advantages:

1. a)

   It reduces the computational cost, as shown in Table II. For a mini-batch with $2PK$ images, $L_{bh\_tri}$ requires computing pairwise distance $2PK\times(2K-1)$ for hardest positive sample mining and $2PK\times 2(P-1)K$ for hardest negative sample mining. In comparison, $L_{hc\_tri}$ only needs to compute the pairwise distance $2P$ for positive sample pairs (there are only $P$ cross-modality positive center pairs), and $2P\times 2(P-1)$ for hardest negative center sample mining. The computational cost is largely reduced.

2. b)

   It relaxes the sample-based triplet constraint to the center-based triplet constraint, which also preserves the property of handling both the intra-class and inter-class variations simultaneously on visible and thermal modalities in the common feature space. For each identity, minimizing the only cross-modality positive center pairwise distance can ensure intra-class feature compactness. The hardest negative center mining can ensure the inter-class feature distinguishable property both in visible and thermal modalities.

There are two kinds of center-based losses: the learned centers [25, 32] and the computed centers [39]. The main difference lies in the method of obtaining the centers. One learns them by pre-setting a center parameter for each class, while the other computes the centers directly based on the learned deep features.

The learned centers. The learned center loss [25] is first introduced in face verification to learn a center for the features of each class and penalizes the distances between the deep features and their corresponding centers. For our cross-modality VT Re-ID task with the $PK$ sampling strategy, the learned center loss can be extended in a bi-directional manner [12, 32, 35] as follows,

where $v_{j}^{i}$ denotes the $j^{th}$ visible image feature of the $i^{th}$ person in the mini-batch, while $t_{j}^{i}$ corresponds to the thermal image feature. $c^{i}$ is the $i^{th}$ class center for both visible and thermal modalities.

Comparing the learned center loss $L_{lc}$ (Eq. (8)) to our proposed hetero-center triplet loss $L_{hc\_tri}$ (Eq. (7)), there are the following differences. 1) $L_{hc\_tri}$ is in a comparison of the anchor center to centers rather than $L_{lc}$’s anchor sample to centers. 2) $L_{hc\_tri}$ computes the centers for visible and thermal modalities, while $L_{lc}$ unifies the $i^{th}$ class center for both visible and thermal modalities into one learned vector. 3) $L_{hc\_tri}$ is formulated by triplet mining the properties of both the inter-class separability and intra-class compactness, while $L_{lc}$ only focuses on the intra-class compactness, ignoring the inter-class separability.

As shown in Fig. 4, our proposed hetero-center triplet loss $L_{hc\_tri}$ concentrates on both of the inter-class separability and intra-class compactness, while the learned center loss $L_{lc}$ ignores the inter-class separability for both the intra- and inter modality. $L_{lc}$ only performs well on intra-modality intra-class compactness, but poorly on cross-modality intra-class compactness. This may be due to the hard training of learned center loss combined with identification loss, which leads to the unsatisfactory performance.

The computed centers. The other method for obtaining the center of each class is to calculate it directly based on the learned deep features [39]. We also adopt this approach. Instead of pre-setting a center parameter to be learned as Eq. (8), the centers are directly calculated as Eq. (6). In [39], the hetero-center loss $L_{hc}$ was proposed to improve the intra-class cross-modality similarity, penalizing the center distance between two modality distributions, which can be formulated as follows,

Comparing the hetero-center loss $L_{hc}$ (Eq. (9)) to our proposed hetero-center triplet loss $L_{hc\_tri}$ (Eq. (7)), the main difference is that $L_{hc}$ only focuses on the intra-class cross-modality compactness (the red arrows in Fig. 3), while our $L_{hc\_tri}$ additionally focuses on the inter-class separability for both the intra- and inter-modality (the grey arrows in Fig. 3) with triplet mining. In summary, $L_{hc}$ is only a part of our proposed $L_{hc\_tri}$.

Moreover, similar to some state-of-the-art VT Re-ID methods [6, 24, 35, 32, 39], for the sake of feasibility and effectiveness for classification, identification loss is also utilized to integrate the identity-specific information by treating each person as a class. The identification loss with label smooth operation is adopted to prevent overfitting the Re-ID model training. Given an image, we denote $y$ as the truth ID label and $p_{i}$ as the ID prediction logits of the $i^{th}$ class. The identification loss is calculated as follows,

where $N$ is the number of identities in the total training set, and $\xi$ is a constant to encourage the model to be less confident on the training set. In this work, $\xi$ is set to 0.1.

We adopt both the identification loss and hetero-center triplet loss for each part-level feature, while only the hetero-center triplet loss $L_{hc\_tri}^{g}$ is for the final concatenated global features. Therefore, the final loss is,

where $\lambda$ is a predefined tradeoff parameters.

SYSU-MM01 [27] is a large-scale dataset collected by 6 cameras, including 4 visible and 2 infrared cameras, captured in the SYSU campus. Some cameras are deployed in indoor environments and others are deployed in outdoor environments. The training set contains 395 persons, including 22,258 visible images and 11,909 infrared images. The testing set contains another 96 persons, including 3,803 infrared images for the query and 301 randomly selected visible images as the gallery set. In the all-search mode, the gallery set contains all the visible images captured from all four visible cameras. In indoor-search mode, the gallery set only contains the visible images captured by two indoor visible cameras. Generally, the all-search mode is more challenging than the indoor-search mode. We follow existing methods to perform 10 trials of gallery set selection in the single-shot setting [35, 32], and then report the average retrieval performance. A detailed description of the evaluation protocol can be found in [27].

RegDB [15] was constructed by dual-camera (one visible and one thermal camera) systems, and includes 412 persons. For each person, 10 visible images were captured by a visible camera, and 10 thermal images are obtained by a thermal camera. We follow the evaluation protocol in [31] and [35], where the dataset is randomly split into two halves, one for training and the other for testing. For testing, the images from one modality (default is thermal) were used as the gallery set while those from the other modality (default is visible) were used as the probe set. The procedure is repeated for 10 trials to achieve statistically stable results, recording the mean values.

Following existing works, cumulative matching characteristics (CMC), mean average precision (mAP) and the mean inverse negative penalty (mINP) are adopted as the evaluation metrics. CMC (rank-r accuracy) measures the probability of a correct cross-modality person image occurring in the top-r retrieved results. mAP measures the retrieval performance when multiple matching images occur in the gallery set. Moreover, mINP considers the hardest correct match that determines the workload of inspectors [33]. Note that all the person features are first $L2$ normalized for testing.

The implementation¹¹1https://github.com/hijune6/Hetero-center-triplet-loss-for-VT-Re-ID of our method is with the Pytorch framework. Following the existing person Re-ID works, the ResNet50 model is adopted as the backbone network for a fair comparison, and the pre-trained ImageNet parameters are adopted for the network initialization. Specifically, the stride of the last convolutional block is changed from 2 to 1 to obtain fine-grained feature maps with large body size. In the training phase, the input images are resized to $288\times 144$ and padded with 10, then randomly left-right flipped and cropped to $288\times 144$ for data augmentation. We adopt the stochastic gradient descent (SGD) optimizer for optimization, and the momentum parameter is set to 0.9. We set the initial learning rate as 0.1 for both datasets. The warmup learning rate strategy is applied to bootstrap the network to enhance performance. The learning rate ($lr$) at epoch $t$ is computed as follows,

We set the predefined margin $\rho=0.3$ for all triplet losses. For the $PK$ sampling strategy, we set $P=8$, $K=4$ for the RegDB dataset, and $P=6$, $K=8$ for the SYSU-MM01 dataset. For the tradeoff parameter, we set $\lambda=2.0$ for the RegDB dataset, and $\lambda=1.0$ for the SYSU-MM01 dataset. The dimension of part-level feature $d$ is set to $256$, and the number of part-level stripes $p$ is set to $6$.

As analyzed in Sec. III-A, the key point of the two-stream backbone network setting is how to split the well-designed CNN model to construct a modality-specific feature extractor with independent parameters and modality-shared feature embedding with shared parameters. Based on the AGW baseline [33] which is designed on top of BagTricks [14], we optionally build the following baseline network with the ResNet50 model. As shown in Fig. 6, the 3D feature maps outputted from the two-stream backbone network are pooled by the generalized-mean pooling (GeM) layer to obtain the 2D feature vector. Then the batch normalization (BN) neck is adopted to train the network, where triplet loss (Eq. (5)) is first utilized on the 2D feature vector, and then the identification loss is sequentially utilized on the batch normalized feature vector.

The results of different backbone splits on RegDB and SYSU-MM01 datasets are listed in Table III, from which we can observe that.

1. a)

   $s5$, without sharing any res-convolutional layers, obtains the worst performance on both the RegDB and SYSU-MM01 datasets, with large margins compared to other splits. $s5$ only shares the last fully connected layer to process the 1D feature vector without any person spatial structure information. It demonstrates the effectiveness of the 3D feature maps with person spatial structure information to describe a person.

2. b)

   $s0$, sharing all the backbone networks without the modality-specific feature extractor, obtains good performances on both the RegDB and SYSU-MM01 datasets. $s0$ equally treats both visible and thermal person images, without focusing additionally on the color information of visible images, focusing on the spatial structure information of a person existing on both visible and thermal images. The results may demonstrate that the person spatial structure information is more important compared to the color information in VT cross-modality Re-ID.

3. c)

   On the RegDB dataset, $s0$, $s1$, $s2$ and $s3$ achieve comparable performances. On the SYSU-MM01 dataset, $s2$ obtains much better Rank1, mAP and mINP results compared to $s0$ and $s1$. The different performances may be from the different settings of the two datasets. RegDB is collected by a dual-camera system, where the visible image and corresponding thermal image are well aligned. SYSU-MM01 is collected by 6 disjoint cameras deployed at different locations, where the visible image and corresponding infrared image have arbitrary poses and views. Therefore, SYSU-MM01 needs more modality-specific layers to extract the person spatial structure compared to RegDB.

4. d)

   Overall, $s2$ can achieve the best performance, which only sets $stage0$ and $stage1$ as the modality-specific feature extractor with acceptable independent parameters.

To evaluate the effectiveness of part-level feature learning compared to global feature learning, we add the uniform partitioning strategy between the two-stream backbone network and the loss layer, as shown in Fig. 2. Based on the above experimental results, we adopt the $s2$ split as the two-stream backbone network, still with the supervision of identification loss and triplet loss.

Three points are considered: 1) the number of partition strips $p$, 2) the generalized-mean pooling (GeM) layer instead of the traditional average pooling layer or max-pooling layer, and 3) the dimension of each part-level feature $d$ corresponding to the output channel number of $1\times 1$ Conv.

The effect of partition strips. The number of partition strips determines the granularity of a person’s local feature. Fig. 7 shows the results of different partition strips $p$ on the RegDB and SYSU-MM01 datasets. We observe that $p=6$ is the best setting for partition strips to extract the local person feature.

The effect of GeM. This subsection verifies the effectiveness of the generalized-mean pooling (GeM) method compared to the traditional average pooling (Mean) and max-pooling (Max) methods. Table IV lists the results of different pooling methods on the RegDB and SYSU-MM01 datasets. We observe that max-pooling performs better than average pooling, while the generalized-mean pooling method performs the best.

The effect of part-level feature dimension. This subsection shows the effect of part-level feature dimension $d$, corresponding to the output channel number of $1\times 1$ Conv in Fig. 2. The final dimension of the person feature is the product of the part-level feature dimension $d$ and the number of partition strips $p$. Table V lists the results of different dimensions of each part-level feature $d$ on the RegDB and SYSU-MM01 datasets. We find that on SYSU-MM01 dataset, $d=256$ performs the best, while on the RegDB dataset $d=512$ performs the best under the rank1 and mAP criteria, and $d=256$ achieves the best performance under the mINP criterion. considering both the performance and final person feature dimension, we set $d=256$ for both the RegDB and SYSU-MM01 datasets.

In this subsection, we verify the effectiveness of our proposed hetero-center triplet loss $L_{hc\_tri}$ from two aspects. On the one hand, $L_{hc\_tri}$ is compared to traditional triplet loss $L_{bh\_tri}$ to demonstrate the effectiveness of anchor center to all the other centers compared to anchor to all the other samples. On the other hand, $L_{hc\_tri}$ is compared to the learned center loss $L_{lc}$ and hetero-center loss $L_{hc}$ to demonstrate the effectiveness of constraining both the inter-class separability and intra-class compactness.

$L_{hc\_tri}$ vs. $L_{bh\_tri}$. We conducted experiments under the framework shown in Fig. 2 with different triplet losses, $L_{hc\_tri}$ and $L_{bh\_tri}$, fine-tuning the tradeoff parameter $\lambda$ in Eq. (14). The results are listed in Table VI. From the table, we can observe that 1) $L_{hc\_tri}$ outperforms $L_{bh\_tri}$ on both RegDB and SYSU-MM01 datasets, demonstrating the effectiveness of anchor center to all the other centers compared to anchor to all the other samples. 2) With the final loss Eq. (14), those nonconvergent cases on the SYSU-MM01 dataset may show that the anchor to all the other samples of $L_{bh\_tri}$ is truly a strict constraint, demonstrating the effectiveness of the anchor center to all the other centers relaxation operation.

$L_{hc\_tri}$ vs. $L_{lc}$ and $L_{hc}$. We conducted experiments with different center-based losses, including the learned center loss $L_{lc}$, hetero-center loss $L_{hc}$, and our proposed hetero-center triplet loss $L_{hc\_tri}$. The network is in two frameworks: the baseline network that extracts the global person features (Sec. IV-B1) and the part-level local feature learning network (Sec. IV-B2). The results are listed in Table VII. We can observe that in both networks, $L_{hc\_tri}$ outperforms $L_{lc}$ and $L_{hc}$ with large margins. It demonstrates the effectiveness of our proposed $L_{hc\_tri}$ concentrating on both the inter-class separability and intra-class compactness, compared to $L_{lc}$ and $L_{hc}$ which only focus on intra-class cross-modality compactness, ignoring the inter-class separability for both the intra- and inter-modality. It is also illustrated in Fig. 4 through visualizing the features extracted by the baseline model with different center-based losses.

Moreover, to show the effect of every component, we also summarize the corresponding ablation study in Table VIII, whose results are copied from Tables III, VI and VII. It shows that when the component works alone, the performance on the two datasets is different and is not always improved. However, the combination of three components can greatly boost the performance on both datasets.