Unsupervised Domain Adaptive Person Re-ID with Local-enhance and Prototype Dictionary Learning

Since Raia Hadsell et al. [21] proposed the approach of contrastive learning in 2006 which is a supervised contrastive learning method, contrastive learning has received a lot of attention and research. Broad contrastive learning includes supervised contrastive learning [22, 23] and unsupervised contrastive learning, while narrow contrastive learning refers to self-supervised contrastive learning. The contrastive learning mentioned in this paper refers to narrow contrastive learning. Contrastive learning can be broadly divided into two categories: negative-example-based contrastive learning methods [17, 16]and clustering-based contrastive learning methods [24, 19]. The process of negative-example-based contrastive learning can be roughly divided into image enhancement, feature encoding, and loss calculation. Image enhancement is a common method for contrastive learning to construct similar and dissimilar samples by itself. In order to prevent model collapse, SimCLR [17] uses a large batch size, MOCO [16] introduced memory bank to avoid the use of large batch size while improving performance. The process of the cluster-based contrastive learning methods is roughly the same as that of the negative-example-based contrastive learning methods. Instead of using computationally intensive pairwise comparisons, cluster-based contrastive learning obtains feature prototypes from the clustering assignment and contrasts features prototype and sample features. The PCL [19] uses clustering directly to obtain similar and dissimilar sample sets.

To solve the problems of the high cost of labeled data acquisition and poor generalization of person re-identification systems, unsupervised domain adaptive (UDA) methods are introduced. Unsupervised domain adaptive person re-identification can be divided into two main categories: the style transfer methods and the pseudo label estimation methods. Both SPGAN [25] proposed by Deng et al. and PTGAN [26] proposed by Wei et al. utilize the generative adversarial network (GAN) to transfer the style of the source domain image to the style of the target domain while maintaining the original identity label. The style-transformed images and their identity labels are then used to fine-tune the model in a supervised manner. HHL [27] proposed by Zhong et al. learned camera-invariant features by camera-style shifting images. However, the performance of these methods deeply depends on the quality of image generation, and they do not explore the complex relationships between different samples in the target domain. Compared to the former, pseudo label estimation is more effective in capturing the target domain distribution. Pseudo-label estimation methods can be classified into clustering-based methods and instance-feature-similarity-based methods. Yang Fu et al. [13] construct the feature coordinate space with the help of an auxiliary dataset, then embed unlabeled data in the feature space to obtain soft multi-labels of unlabeled data. SSG [13] uses the potential similarity of the whole body and local parts of unlabeled samples to automatically build multiple clusters from different perspectives. Yixiao Ge et al. [14] use soft pseudo labels obtained from the CNN and hard pseudo labels obtained from the clustering method for better feature learning. The pseudo label estimation methods based on clustering have proved to be more effective and maintain the most advanced accuracy [13, 14].

The end-to-end update by back-propagation [28, 17, 29] is a natural mechanism for contrastive learning. However, the approach requires a large mini-batch size to avoid model collapse. And this requires a large GPU memory size[16]. Xiao et al. first used the memory bank mechanism in the task of person search using Online Instance Matching (OIM) loss to solve the training difficulties caused by sparse and unbalanced labeled data samples. Wu et al. introduces a memory bank mechanism for unsupervised feature learning through non-parametric instance discriminations [30]. Instance-level feature vectors are stored in a memory bank. The softmax distribution is approximated by noise-contrast estimation (NCE) to achieve parametric-free multi-classification. He et al. propose the Momentum Contrast (MOCO) method which maintains the key representations in a queue, reducing the capacity of the memory bank while achieving a large capacity dictionary. Yixiao Ge et al. propose the self-paced contrastive learning (SpCL) [15] method that stores both instance-level features and class-level features in a memory bank called hybrid memory which enables mining of data distribution and hard samples.

The framework consists of two main parts: a CNN-based feature encoder and a novel prototype dictionary which is the base of the category-level contrastive learning. Ge et al. proposes the SpCL framework which jointly trains the encoder with all the source-domain class-level, target-domain cluster-level, and target-domain un-clustered instance-level supervisions [15]. We argue that the instance-level supervision is harmful to robust feature learning as the contrastive learning treats each instance as a new class, leading to an increase in the distance of two instances of the same class which, to a certain extent, destroys the certainty of clustering. The training process of PDL is simple and can be roughly divided into two steps: the initialization of the prototype dictionary (Section III-A1) and the update of the prototype dictionary (Section III-A2). After the prototype dictionary is initialized, a mini-batch containing source and target domain data is input to the network, and the feature vector is extracted by the encoder, and then compared to the feature prototype in the prototype dictionary as query, and finally, the prototype dictionary and encoder are updated. The details of the algorithm are presented in Algorithm 1.

Given the source-domain training set $\mathbb{X}^{s}=\{x_{1},\cdots,x_{n^{s}}\}$ and the target-domain training set $\mathbb{X}^{t}=\{x_{n^{s}+1},\cdots,x_{n^{s}+n^{t}}\}$, the encoder $f_{\theta}$ maps $\mathbb{X}^{s}$ and $\mathbb{X}^{t}$ to $V=\{v_{1},\cdots,v_{N}\}$ where $N=n^{s}+n^{t}$. The training goal is to find the optimal $f_{\theta}$ such that $v$ best represents $x$. Traditional contrastive learning, such as InfoNCE loss [32] shown in formula 3 used in SimCLR [17], uses instance-level feature comparison, and does not explore the distribution characteristics of the data set itself. More importantly, instance-level comparison inherently treats each instance as a separate category, inevitably pushing the pair of samples that should be closer together. Saunshi et al. [18] has proven that the phenomenon is harmful to representation learning.

In the formula 3, $N$ represents the number of the training data, $S(\cdot)$ means similarity calculation function. $\tau$ is the temperature. There are $K+1$ features involved in the comparison, of which 1 is a known positive sample of $v_{i}$, such as itself, and the remaining $k$ are regarded as negative samples. It is easy to find that this mechanism can only work when the sample size involved in the comparison is large. That is why SimCLR needs a large batch size and a lot of training epochs.

Instance-level comparisons are easily disturbed by misjudged negative samples, and the clustering model naturally has a good tolerance for misjudged negative samples. We propose the PDL loss which is a kind of InfoNCE-like contrastive loss. The difference is that we use category-level comparisons instead of instance-level comparisons. The formula is as follows:

where $p_{i}^{+}$ is the prototype of the category containing feature vector $v_{i}$, and $p_{j}$ and $p_{r}$ are prototypes of source-domain categories and target-domain categories respectively. In this way, we convert instance-level comparisons into category-level comparisons. The prototypes of the target domain come from the clustering algorithm, which enables PDL to strengthen the mining of the data distribution of the target domain. At the same time, the source-domain prototypes come from their own ground-truth labels, which can help the model to obtain the representative target domain prototypes. Only $n^{s}+n^{t}$ feature prototypes need to be stored and only $n^{s}+n^{t}$ feature prototypes are involved in the comparison. So the training batch size does not need to be large. In fact, the results shown in Table I are obtained at a batch size of 64.

In order to reduce the interference of the domain gap on the model convergence when training with both the source domain data and the target domain data at the same time, and enhance the model’s adaptability to different domains, we propose the Local-enhance Module. X.Huang et al. [20] proposed that the artistic style of the image is the statistical information of each feature channel across the space of the feature map, such as the mean and variance. Our motivation is to change the mean and variance of the feature map by randomly adding disturbances to the feature map so that the model reduces the sensitivity to image style and improves the generalization of the model between different domains.

Person re-ID tasks are affected by posture changes and occlusion problems, and the local features are quite different. Therefore, the person re-ID network will suppress the local features during training, but some useful information will also be discarded. Inspired by the DropBlock [33] mechanism, the local-enhance module select a contiguous region of a feature map. The local-enhance module enhances a contiguous region of the feature maps in a mini-batch instead of dropping them.

We use ResNet50 as the backbone network and insert the local-enhance module between layer3 and layer4. Given the output tensor $\mathcal{F}\in\mathbb{R}^{C\times H\times W}$ of layer3 with the shape of $C\times H\times W$, where $C$ is the number of channels, $H$ and $W$ are the height and width respectively. The width of the enhanced block is the same as the width $W$ of the feature map. The height of the enhanced block is set to one-tenth of the height $H$ of the tensor. Therefore, the randomness of the region is only reflected in the vertical dimension. The enhancement operation can be expressed as follows as the enhancing coefficient is expressed as $\alpha$:

where $e_{c,h,w}$ represents the value of the tensor $\mathcal{F}$ at the coordinate $c,h,w$, and $p$ is the random anchor point on the height of the feature map which is within the range $[0,0.9\cdot H]$. The local-enhance module performs simple linear operations on tensors with zero additional parameters.

Traditional person re-ID systems usually use global average pooling to obtain the final feature vector, which is to reduce the loss of useful information. In order to retain the enhanced feature values effectively, we adopt extra global maximum pooling in the subsequent pooling process. Given the enhanced tensor $\mathcal{F}\in\mathbb{R}^{C\times H\times W}$, the final feature vector is calculated by:

where $C$ is convolution operation of layer4 of ResNet50, $A$ and $M$ respectively represent global average pooling and global maximum pooling, $||\cdot||$ is L2-normalization and $\mathcal{V}$ is the feature vector. Experiments IV-D have also demonstrated the effectiveness of the plug-and-play module.

Market-1501 [34] and DukeMTMC-reID [35] are widely used large-scale re-ID image datasets. We evaluate our model in two mainstream adaptation tasks Market-to-Duke and Duke-to-Market as shown in Tabel I.

Market-1501 [34] was collected on the campus of Tsinghua University, shot in the summer, and released publicly in 2015. It includes 32668 person images of 1501 identities captured by 6 cameras. Each identity is captured by at least 2 cameras, and there may be multiple images in one camera. The training set has 751 people, containing 12,936 images; the test set has 750 people, containing 19,732 images.

DukeMTMC-reID [35] was collected at Duke University in the winter of 2017. The DukeMTMC dataset [35] is a large-scale multi-target multi-camera pedestrian tracking dataset. It provides a new large HD video data set recorded by 8 simultaneous cameras, with more than 7,000 single camera tracks and more than 2,700 independent characters. DukeMTMC-reID, containing 16,522 person images of 702 identities, is a person re-ID subset of the DukeMTMC dataset and provides manually marked bounding boxes.

For re-ID evaluation metrics, cumulative matching characteristics (CMC) [36] and mean average precision (mAP) are the most commonly used evaluation indicators. The former reflects the retrieval precision and the latter reflects the recall. We report the Rank-1, Rank-5, Rank-10, scores to represent the CMC curve.

We adopt ResNet50 with the last classification layer removed as our backbone network for encoder $f_{\theta}$. Then we add the local-enhance module between layer3 and layer4 of ResNet50 which is only enabled during training. At the first beginning, the model is initialized by pre-trained weights on ImageNet [37]. The batch size is set to 64, which contains 4 identities, each with 16 images on tasks Market-to-Duke and Duke-to-Market. All images are resized to regular $256\times 128$ and applied random perturbations to each image, e.g. randomly erasing, cropping, and flipping before being fed into the networks and are encoded into 2048-dimensional feature vectors by encoder $f_{\theta}$. And we adopt an ADAM optimizer with a weight decay of 0.0005 to train the model. The initial learning rate is set to 0.00035 and is decreased to 1/10 of its previous value every 20 epochs in a total of 50 epochs. In all experiments, only the labels of the source domain were used, and no post-processing operations, such as re-ranking, were used.

We utilize DBSCAN [31] for clustering the unlabeled target-domain data during prototype dictionary initialization and Jaccard distance [38] is used as an indicator of the algorithm to measure density. The hyperparameter $eps$ which is the max neighbor distance for DBSCAN is set to 0.6 in task Market-to-Duke and 0.5 in task Duke-to-Market and the domain density threshold $MinPts$ is set as 4. The hyperparameter $K$ which means k-reciprocal nearest neighbors is set as 30. The prototype dictionary is updated by formula 2 where the hyperparameter $m$ which means momentum coefficient is set as 0.1.

In this section, we compare the proposed method with state-of-the-art unsupervised cross-domain methods on the two most representative domain adaptation tasks, Market-to-Duke and Duke-to-Market in Table I. They can be roughly divided into three categories: methods based on feature alignment, such as TJ-AIDL [39], methods based on GAN, such as SPGAN [40], and methods based on pseudo-label prediction, such as SSG [13]. It can be seen that our proposed method outperforms recent works by large margins. Specifically, the mAP of our method surpasses the GAN-based method SPGAN [40] by 49.2% and 61.1% on tasks Market-to-Duke and Duke-to-Market. Moreover, our method surpasses clustering-based SSG [13] by a considerable margin of mAP 18.1% and Rank-1 accuracy 10.1% on Market-to-Duke, while 25.6% and 13.0% on Duke-to-Market. Meanwhile, our PDL only one encoder used is compared with the MMT [14] method that uses the two-stream network, showing a noticeable 6.4% and 14.9% improvements in terms of mAP on Market-to-Duke and Duke-to-Market respectively. Compared to the state-of-the-art method GLT [41] which is based on label transfer, our method leads to 2.3% mAP and 4.4% mAP gain on Market-to-Duke and Duke-to-Market respectively. The results verify the effectiveness of our method.

In this section, we evaluate each component of our proposed framework PDL by conducting ablation studies on Market-to-Duke and Duke-to-Market tasks. The experimental results are shown in Table II.

Effectiveness of the category-level contrastive learning.

The comparison of the performance of the baseline method and the method of baseline without the category-level contrastive learning are specified in Table II. To verify the necessity of category-level contrastive learning, we conduct experiments by replacing instance-level comparisons with category-level comparisons. Baseline w/o cl means experiments without category-level comparisons strategy. As shown in the table, we improve the performance by 3.6% at mAP and 3.2% at rank-1 accuracy on Market-to-Duke and 13.8% at mAP and 7.4% at rank-1 accuracy on Duke-to-Market when the category-level comparisons strategy is adopted.

Effectiveness of the local-enhance module. We use the local-enhance module to reduce the model sensitivity of stytle to data in different fields and reduce the interference of training time domain gap. In Table II, Le and gmp denote to local-enhance module and additional global maximun pooling, respectively. We conduct experiments by Remove the modules added on the baseline in turn. Local-enhance module brings 2.7% and 3.2% improvements in terms of mAP and rank-1 accuracy on Market-to-Duke and 3.0% and 1.1% improvements in terms of mAP and rank-1 accuracy on Duke-to-Market, respectively. Additional global maximun pooling brings 5.3% and 2.6% improvements in terms of mAP and rank-1 accuracy on Market-to-Duke and 1.4% and 0.1% improvements in terms of mAP and rank-1 accuracy on Duke-to-Market, respectively. When both modules are added, the performance is improved by 6.2% mAP and 3.4 rank-1 accuracy on Market-to-Duke. On Duke-to-Market, the improvements in terms of mAP and rank-1 accuracy reached 3.5% and 1.8% respectively.

We also explored the effect of the enhancement coefficients $\alpha$ on loca-enhance modules as shown in Equation 5. The greater the $\alpha$, the greater the change in the mean and variance of the image feature map. Because the local enhancement module is improved in both tasks, we only studied the effect of the enhancement coefficient in task Duke-to-Market, and finally chose 2 as the enhancement coefficient for the two tasks. Table III shows the effect of different $\alpha$ values on performance.