Domain Camera Adaptation and Collaborative Multiple Feature Clustering for Unsupervised Person Re-ID

Extensive supervised methods have been developed on the widely used benchmarks (Zheng et al., 2015, 2017; Ristani et al., 2016), concentrating on discriminative feature representation learning (Li et al., 2018b; Bai et al., 2020), deep metric learning (Yu et al., 2018; Hu et al., 2015), postprocessing procedures (Bai et al., 2017; Zhong et al., 2017; Cao et al., 2020), and other problems such as occlusion (He et al., 2019), various image resolutions (Wang et al., 2018). Although great progress has been observed, these supervised approaches may have a significant performance drop when applied to another unseen domain due to the existence of domain shift.

Unsupervised domain adaptation (UDA) aims to transfer the knowledge on a labeled source domain to the unlabeled target domain. Several unsupervised domain adaptation methods try to reduce the distribution discrepancy between source and target domain on either feature-level (Peng et al., 2016; Hu et al., 2015) or image-level (Deng et al., 2018; Wei et al., 2018). The former focuses on learning domain-invariant representations by aligning the feature statistics, such as the mean and covariance of source domain and target domain feature distributions (Sun et al., 2016), or the adversarial training approach. The latter learns to transform samples in the pixel space from the source domain to the target domain using generative adversarial networks (Bousmalis et al., 2017; Hoffman et al., 2018). Nevertheless, most of the existing unsupervised domain adaptation methods assume that source and target domains share the same set of classes, while in person re-ID task the source and target domains have entirely different classes. Thus, these methods cannot be directly utilized for unsupervised domain adaptation in person re-ID.

Unsupervised person re-ID methods are proposed to utilize the unlabeled target data and large-scale labeled samples. Among them, some works try to address this problem with domain adaptation on feature or image levels. In (Yang et al., 2020a), PPAN is proposed to enforce feature alignment across domains. Peng et al. (Peng et al., 2016) propose to learn based on asymmetric multi-task dictionary learning. Other works (Deng et al., 2018; Wei et al., 2018) attempt to transfer the source domain images to target domain styles using generative adversarial networks (GAN). Deng et al. (Deng et al., 2018) introduce a similarity preserving cycle consistent generative adversarial network (SPGAN) to translate images. However, intra-domain image variations still exist because of the distribution discrepancy at the camera level. HHL (Zhong et al., 2018) considers the intra-domain image style variations caused by different camera configuration. M2MGAN (Liang et al., 2018) takes multiple source and target sub-domains into consideration. Different from these methods, our cross-domain camera style adaptation module explicitly considers the camera-level disparities and transforms images from the source domain to different cameras in the target domain. Moreover, we introduce person identity preserve loss constraint and identity mapping loss constraint to improve the quality of the generated images. Finally, considering the confirmation bias of label generation, a collaborative pseudo re-labeling strategy is proposed.

Beyond the above methods, some approaches focus on label estimation in the target domain. Fan et al. (Fan et al., 2018) propose an unsupervised re-ID approach for iteratively applying k-means clustering. Yang et al. (Yang et al., 2020b) generate labeled virtual data from the target dataset and propose collaborative filtering on unlabeled data. A Self-similarity Grouping (SSG) approach (Fu et al., 2019) iteratively conducts grouping and re-ID model training in a self-paced manner. A self-training method with progressive augmentation (Zhang et al., 2019) jointly captures the local structure and global data distribution. Soft multi-label learning (Yu et al., 2019) mines the soft label information from a reference set for unsupervised learning. Inspired by these methods, we propose a collaborative multiple feature clustering framework (CMFC) to learn identity similarities using multiple features on the target domain. CMFC is a two-branch framework, including global and partial feature branches, which improves the accuracy of internal data similarity learning using multiple features.

To learn feature embeddings of the source domain, we train on the source dataset and denote the obtained model as baseline. We use ResNet50 (He et al., 2016) as backbone network. Given the labeled images in a training batch, we train the baseline model with cross-entropy loss and batch-hard triplet loss (Hermans et al., 2017) simultaneously. The cross-entropy loss is employed with the output of the FC layer by treating the training process as a classification task. Besides, label smoothing (Szegedy et al., 2016) is used to avoid overfitting. Specifically, the cross-entropy loss with label smoothing can be formulated as:

where n_s is the number of images in a batch, $p_{i}(k)\in\left[0,\ 1\right]$ is the predicted probability belonging to class k. q_i(k) is the label distribution and y is the ground truth class label. $\varepsilon$ is a small perturbation term. The triplet loss is employed to enhance intra-class compactness and inter-class separability, which can be written as,

where x_a denotes the anchor. x_p and x_n represent the hardest positive sample and the hardest negative sample in the same batch respectively. m is a margin hyperparameter and D$(\bullet)$ is the Euclidean distance between two features. Thus, the overall loss function is written as follows

where $\lambda_{t}$ can be set to 1 in the experiments for simplicity.

To leverage the knowledge of the source dataset, we try to reduce the distribution discrepancy between the source and target dataset at the image level. Instead of transferring the global style of images, we perform image-image translation by viewing each camera as an individual domain.

Given a dataset X_S from the source domain and a dataset X_T from the target domain where the camera labels are available, our goal is to train a single generator G that learns the mappings among multiple domains. In this way, a given labeled image $\left(x_{s}^{i},y_{s}^{i}\right)$ with camera label c${}_{s}^{i}$ is transferred to another camera style c_t of target domain, while preserving the identity information during the translation. The generated images can then be used to train the person re-ID model. To learn the style mapping between source and target datasets, we employ StarGAN (Choi et al., 2018) and further introduce two loss items during the image translation training procedure to improve the quality of the transferred images.

StarGAN trains a single generator G to translate an input image x into an output image y conditioned on the target domain label c, $G\left(x,\ c\right)\rightarrow y$. In addition, to distinguish real training examples and generated samples from G, an auxiliary classifier is introduced to the discriminator D to control multiple domains. Thus, the discriminator produces probability distributions over both sources and domain labels, $D:x\rightarrow\left\{D_{src}\left(x\right),\ \ D_{cls}\left(x\right)\right\}$. The network architecture of StarGAN is described in Section 4.2.

Adversarial Loss. Adversarial loss is adopted to make the generated images indistinguishable from real images. The generator G tries to minimize the loss while the discriminator tries to maximize it.

Domain Classification Loss. To distinguish the domain labels of a real/fake image, an auxiliary classifier is added on top of D and cross-entropy loss is utilized to optimize both G and D. Specifically, the domain classification loss of real images is used to optimizer D and the domain classification loss of fake images is used to optimize G, that is,

where c^′ is the original domain label of real image x.

Reconstruction Loss. To preserve the content of input images while changing only the domain-related style of images, reconstruction loss is used to formulate forward cycle consistency.

Identity mapping Loss. Apart from adversarial loss, domain classification loss, and reconstruction loss, we introduce identity mapping loss to regularize the generator to be the identity matrix on samples from the target domain. SPGAN (Deng et al., 2018) uses this loss to preserve the color composition between the input and output during translating images from the source dataset to the target dataset. The identity mapping loss is written as

Person Identity Preserve Loss. To utilize the transferred images to supervised person re-ID model training, it is important to preserve the identity of person images while changing the style of images. We introduce the person identity preserve loss by evaluating the variations in the person foreground before and after person transfer. However, the common identity preserve loss form is prone to suffer from perturbation in the foreground masks. To enhance the robustness of $G$, a consistency regularization item is introduced by constraining the output of the original image and the enhanced image to be consistent, which is shown as follows,

where M(x) and $x^{a}$ represents the foreground mask and augmented images of x. In this paper, we use SOLO (Wang et al., 2019) instance segmentation algorithm to extract the foreground mask.

Overall objective function. Stage II in Fig. 2 shows the overview of the training process of the proposed cross-domain camera style adaptation module. Specifically, the objective functions to optimize G and D are written, respectively, as

where $\lambda_{cls}$, $\lambda_{rec}$, $\lambda_{idt}$, $\lambda_{pid}$ are hyper-parameters that control the relative importance of domain classification loss, reconstruction loss, identity mapping loss, and person identity preserve loss, respectively. Empirically, we use $\lambda_{cls}$=1, $\lambda_{idt}$=1, $\lambda_{rec}$=10 and $\lambda_{pid}$=10 in our experiments.

In addition to performing unsupervised domain adaptation, mining potential label information (identity similarity) on the target domain is essential to unsupervised person re-ID. In this section, we introduce a collaborative multiple feature clustering framework (a two-branch network) based on the global features and partial features to mine the similarities of target images to train re-ID model.

Two-branch re-ID model. As illustrated in Stage III of Fig. 2, the two-branch network shares the same backbone with the baseline model. Before training on the target domain, the model is trained on the source dataset and transferred images described in Section 3.2, learning useful representations of person images. However, the model is still not discriminative in the target domain. Afterward, two two-branch networks are trained parallelly with the pseudo labels from each other to alleviate confirmation bias. Our two-branch network includes two branches: the global feature branch and the partial feature branch.

Global feature branch. Given an unlabeled image x${}_{t}^{i}$, we first feed it into the pre-trained model for feature extraction. The feature map of image x${}_{t}^{i}$ denotes as $F_{t}^{i}\in\mathbb{R}^{H\times W\times C}$ (the output of Resnet50 layer5). Next, we employ the global max pooling (GMP) on the feature map to obtain feature vector f${}_{t}^{i}$. For every image in target dataset, we extract the feature vector to form a feature vector set $\left\{f_{t}^{1},\ ...,\ f_{t}^{k},...,f_{t}^{N_{t}}\right\}$. Based on the vector set, an unsupervised clustering algorithm is utilized to divide the target dataset into different groups. In this paper, we use DBSCAN (Ester et al., 1996) algorithm to perform unsupervised clustering. According to the cluster results, each image x${}_{t}^{i}$ is assigned a pseudo label ${\widetilde{y}}_{t}^{i}$. In this way, with the pseudo label of each image in target dataset, a new training dataset $\left\{(x_{t}^{1},\ {\widetilde{y}}_{t}^{1}),\ ...,\ (x_{t}^{i},\ {\widetilde{y}}_{t}^{i}),...,(x_{t}^{N_{t}},\ {\widetilde{y}}_{t}^{N_{t}})\right\}$ is organized.

We then finetune the re-ID model with the new dataset in a supervised manner. Specifically, batch hard triplet loss and cross-entropy loss are used to train the model. Triplet loss is employed with feature vector f${}_{t}^{i}$. Cross entropy loss is employed with the person identity classifier, where the global average pooling (GAP) layer and batch normalization (BN) layer are added before it.

Due to the variation of cluster numbers in the training iterations, the newly added person identity classifier layer should be initialized every time executing DBSCAN (Ester et al., 1996). Following the initialization strategy in (Zhang et al., 2019), we exploit the mean features of each cluster as the initial parameters. Specifically, for each cluster c, we calculate the mean feature $\bar{f_{c}}$ by averaging all the embedding features of its elements. The parameters W_c of the classifier layer are initialized as $W_{c}=\bar{f_{c}}\ (c=1,2,3,\ ...,C)$, where C is cluster number in each iteration. Finally, a novel adversarial erasing based reconstruction branch is proposed. Specifically, the feature maps of images are used to generate activation maps. Afterward, the coordinates with the largest value are recorded. Then rectangular areas of random size centered on these coordinates are generated as the erase areas. Finally, the erased images are fed into another decoder network for the reconstruction of the original images. A reconstruction loss as $\mathcal{L}_{rec}$ is used. With this strategy, the network can mine the key information from other regions of the erased image, so that the network will not over-trust noisy labels.

Partial feature branch. In person re-ID task, part-level feature learning usually achieves better performance on discriminative re-ID model learning by mining discriminative body regions. In this paper, we divide the feature maps of person images into two parts horizontally and each part, containing the information about the upper body or lower body, is used to generate a pseudo label of the image separately. Given an unlabeled image x${}_{t}^{i}$, the feature map $F_{t}^{i}\in\mathbb{R}^{H\times W\times C}$ is divided into two parts, $F_{t\_up}^{i}\in\mathbb{R}^{\frac{H}{2}\times W\times C}$ and $F_{t\_lower}^{i}\in\mathbb{R}^{\frac{H}{2}\times W\times C}$. Next, the global max pooling (GMP) operation is employed in the sliced feature maps to obtain the feature vectors of different body regions. For every image in target dataset, we extract the feature vectors of upper body and lower body to form two feature vector sets, denoted as $f_{t\_up}=\left\{f_{t\_up}^{1},\ ...,f_{t\_up}^{k},...,f_{t\_up}^{N_{t}}\right\}$ and $f_{t\_lower}=\left\{f_{t\_lower}^{1},\ ...,f_{t\_lower}^{k},...,f_{t\_lower}^{N_{t}}\right\}$. The unsupervised clustering algorithm is employed on these two vector sets. Therefore, each image x${}_{t}^{i}$ is assigned two pseudo labels $\left({\widetilde{y}}_{t\_up}^{i},{\widetilde{y}}_{t\_lower}^{i}\right)$.

Based on the pseudo labels, we cross-finetune the re-ID model in a supervised way with triplet loss, similar to the training strategy of the global branch. Specifically, triplet loss is employed with feature vector f${}_{t\_up}^{i}$ and f${}_{t\_lower}^{i}$, respectively.

Global max pooling operation. The procedure of global max pooling and global average pooling is nonparametric. Average pooling calculates the mean of all the pixels within the pooling region while max pooling only considers the maximum response values (Zhao et al., 2020). For unsupervised clustering, max pooling can produce better performance since the regions with higher response values contains more discriminative information about person images. In the ablation experiments, we will evaluate the effect of global max pooling and global average pooling, and demonstrate the superiority of the former (GMP).

Loss function. To fine-tune the re-ID model with the pseudo labels generated from global features and part-level features, the full objective function of the two-branch network is formulated as follows,

where p_t denotes the output sets of the classifier layer in the global branch. $\lambda_{g}$, $\lambda_{up}$, $\lambda_{lower}$ are hyper-parameters that control the relative importance of global feature triplet loss, upper body feature triplet loss, and lower body feature triplet loss, respectively. In this paper, we use $\lambda_{g}$=1, $\lambda_{up}$=1, and $\lambda_{lower}$=0.5 in our experiments. Since the upper body of person images usually contains more discriminative regions than the lower body.

In this section, we describe the overall framework of the proposed method. As shown in Fig. 2, our method can be developed in three stages. The first stage learns a pre-trained re-ID baseline model on a labeled source training dataset (Section 3.1). Based on the baseline, the second stage trains the model with images transferred from the source dataset to the target dataset, performing multi-domain image translation (Section 3.2). In the third stage, we conduct unsupervised clustering and cross-finetune the model with the target training set alternately. Algorithm 1 presents the optimization procedure of our method.

We conduct experiments on two large-scale benchmark datasets, i.e. Market-1501 (Zheng et al., 2015) and DukeMTMC-ReID (Zheng et al., 2017; Ristani et al., 2016).

Market-1501 (Zheng et al., 2015) contains 32,668 images of 1,501 labeled persons from six camera views. Specifically, 12,936 images of 751 identities detected by DPM (Felzenszwalb et al., 2009) are used for training. For testing, in total 19,732 images of 750 identities plus some distractors form the gallery set, and 3,368 manually cropped person regions from 750 identities form the query set.

DukeMTMC-ReID (Zheng et al., 2017; Ristani et al., 2016) contains 1,812 identities captured by 8 cameras. There are 16,522 training images, 2,228 query images, and 17,661 gallery images, with 1,404 identities appearing in more than two cameras. Also, similar to the Market-1501, the rest 408 identities are considered as distractors.

Baseline Model Training. As described in Section 3.1, we first train a baseline model on the source dataset. All training images are resized to $256\times 128$ before being fed into the network. For data augmentation, we employ random cropping and flipping. To meet the requirement of hard batch triplet loss, each mini-batch is sampled with randomly selected P = 16 identities and randomly sampled K = 4 images for each identity from the training set, so that the mini-batch size is 64. In the baseline training stage, the margin hyperparameter of triplet loss is set to 0.5 and $\varepsilon$ is set to 0.1 in label smoothing. We use the Adam (Kingma and Ba, 2015) algorithm with a weight decay 0.0005 to optimize the parameters for 80 epochs. The learning rate is set to 0.00035 and decayed to its 1/10 at the 40th and 70th epochs. During testing, we extract the output of the BN layer as the image descriptor and use the Euclidean distance to compute the similarity between the query and gallery images.

Cross Domain Camera Style Adaptation Model. We follow the same architecture as StarGAN (Choi et al., 2018). Specifically, the generator network composes of two convolutional layers with the stride size of two for downsampling, six residual blocks, and two transposed convolutional layers with the stride size of two for upsampling. PatchGANs (Isola et al., 2017) are leveraged for the discriminator network. The input images are resized to $256\times 128$. We use Adam optimizer with $\beta_{1}$=0.5 and $\beta_{2}$=0.999 for training. The batch size is set to 16. We perform one generator update after five discriminator updates as in (Gulrajani et al., 2017). In the training stage, we train the model with Market-1501 training dataset and DukeMTMC-reID training dataset simultaneously. In the adaptation stage, for each image in the source set, we generate C style-transferred images (C is the number of cameras in the target set). These C fake images are regarded as containing the same person as the original real images. Then we fine-tune the baseline model with the images transferred from the source dataset to the target dataset.

Unsupervised re-ID model training. For unsupervised training with the target dataset, we train the model in a total of 40 epochs. In each epoch, we perform the unsupervised clustering algorithm and the deep re-ID model training alternately. DBSCAN (Ester et al., 1996) clustering method is used to obtain the pseudo labels of each image. Then the new organized dataset is utilized to finetune the re-ID model. For data augmentation, random cropping, random flipping, and random erasing (Zhong et al., 2020) are applied. The margin hyperparameter of triplet loss is set to 0.3. We use the Adam with a weight decay of 0.0005 and a learning rate of 0.00035 to optimize the parameters.

We conduct cross-domain person re-ID evaluation and compare the results with the state-of-the-art methods, including two hand-crafted features, i.e. Bag-of-Words (BoW) (Zheng et al., 2015) and local maximal occurrence (LOMO) (Liao et al., 2015), four unsupervised domain adaptation methods, including PTGAN (Wei et al., 2018), SPGAN (Deng et al., 2018), ARN (Li et al., 2018a) and UDAP (Song et al., 2020), and six unsupervised methods, including PUL (Fan et al., 2018), HHL (Zhong et al., 2018), MAR (Yu et al., 2019), ECN (Zhong et al., 2019), SSG (Fu et al., 2019), MMT (Ge et al., 2020), DCF (Li et al., 2021), GLT (Zheng et al., 2021a) and UNRN (Zheng et al., 2021b). Specifically, we use Market-1501 as the source dataset and DukeMTMC-ReID as the target dataset and report the results on DukeMTMC-ReID test set, and vice versa.

Baseline performance. Here we report the baseline model performance in Table 1. When trained with a labeled dataset and tested on the same dataset, the baseline model achieves a rank-1 accuracy of 92.6% and mAP of 79.6% on Market-1501. However, due to the existence of a domain gap, the performance drops significantly when directly used for another dataset. The rank-1 accuracy declines to 37.6% when tested on DukeMTMC-reID dataset.

Comparison with the state-of-the-art methods on Market-1501 Dataset. Table 2 presents the comparisons when tested on Market-1501 dataset. Compared with four unsupervised domain adaptation approaches, our method outperforms them. The mAP achieves 81.0%, surpassing the UDA method (Song et al., 2020) by 27.3%. Compared with other unsupervised methods, which benefit from initializing the model from the labeled source data and learning with unlabeled target data, our method is superior. Comparing with MMT (Ge et al., 2020), our results are higher by +7.2% in rank-1 and +12% in mAP.

Comparison with the state-of-the-art methods on DukeMTMC-ReID Dataset. A similar improvement can be observed in DukeMTMC-reID dataset. As shown in Table 2, our method achieves rank-1 accuracy = 83.2% and mAP = 71.2%, outperforming all the competing UDA methods. For example, comparing with UDAP (Song et al., 2020), our results are higher by 14.8% in rank-1 accuracy and 22.2% in mAP. When compared with unsupervised methods, our method is still superior to most existing methods. Compared to MMT, our method also shows a great advantage in accuracy.

We also conduct extensive ablation studies to analyze the effectiveness of each component of the proposed method.

Effectiveness of Cross Domain Camera Style Adaptation Module. In the paper, we first train a baseline model on the source dataset. Then we try to reduce the distribution discrepancy at the image level by training a multi-domain image-image generator and transferring images from the source dataset to the target dataset. Specifically, for each image in Market-1501 dataset, we generate eight fake images and assign the same label as the original image. For each image in DukeMTMC-reID dataset, we generate six fake images. Fig. 3 shows generated samples. In this way, we can train the model in a supervised way with the generated images. For example, given Market-1501 as source dataset and DukeMTMC-reID as target dataset, each image in Market-1501 is transferred to eight camera style in DukeMTMC-reID. The generated images with assigned labels are utilized for supervised re-ID training.

We report the results of re-ID model with the transferred images in Table 3. Specifically, when training with the images transferred from Market-1501 to DukeMTMC-ReID, the model achieves rank-1 = 53.8% and mAP = 31.4% on DukeMTMC-ReID test set. Compared to PTGAN (Wei et al., 2018), SPGAN (Deng et al., 2018) (see Table 2), our generation models have better results, which also validates that viewing each camera as an individual domain results in better generation quality. Similarly, when training with the images generated from DukeMTMC-ReID, the model achieves rank-1 accuracy = 66.8% and mAP = 34.0%, surpassing PTGAN and SPGAN. We further conduct experiments to validate the effectiveness of different loss functions. Specifically, when training with the images generated from original StarGAN with adversarial loss, domain classification loss, and reconstruction loss, the model achieves rank-1 accuracy = 53.4% and mAP = 30.8% on DukeMTMC-ReID dataset(see Table 4). Finally, adding identity mapping loss and person identity preserve loss to restrain the training process, our generation models have better results.

Effectiveness of CMFC framework. To verify the effectiveness of our two-branch network and the collaborative pseudo re-labeling strategy for the target domain, we further conduct experiments over baseline. The results are shown in Table 5. Specifically, the model achieves rank-1 accuracy = 83.2% and mAP = 71.2% when tested on DukeMTMC-reID dataset. The model achieves 81.0% and 94.0% on mAP and rank-1 accuracy when tested on Market-1501. Both the proposed two-branch framework and the collaborative re-labeling strategy boost the overall performance.

Specifically, to focus on the contribution of different branches, the ablation studies are evaluated over baseline. Firstly, compared with the baseline model, global and partial feature branches consistently improve the performance, indicating that mining the potential similarity on the target domain is beneficial to discriminative feature learning. The global feature branch obtains higher results than the partial feature branch. We believe that it is because the global feature leads to better clustering quality. Moreover, the partial feature branch can strengthen the representation by mining more information on different body regions. The final two-branch results demonstrate the effectiveness of the combination of global and partial branches. A comparison of visual retrieval results on the Market-1501 dataset between the overall two-branch framework and different branches is shown in Fig. 4. It can be seen that the two-branch framework achieves better results than both global and partial branches. Besides, we can find that although the cross-domain camera style adaptation module provides a higher baseline, the results of the two-branch network yield a marginal performance increase. From this result, we can find that mining identity similarities on the target domain exhibits more potential.

Effect of different pooling operation. We also evaluate the effect of different pooing operations on the two-branch framework. We employ global max pooling (GMP) and global average pooling (GAP) operation on the feature maps to obtain feature vectors, respectively. As shown in Table 6, the model achieves rank-1 accuracy = 91.3% and mAP = 78.0% on DukeMTMC-reID dataset with GMP, surpassing the results with GAP. Similar improvement can be observed when tested on Market-1501. The superiority of GMP probably lies in that max pooling filters out some detrimental signals and focuses on the high response values of feature maps, which benefits the discriminative feature extraction of pedestrian images.

Influences of hyper-parameters. The weight of different loss items in Eq.13 is a key hyperparameter affecting the performance of feature representation learning. $\lambda_{g}$, $\lambda_{up}$, and $\lambda_{lower}$ control the relative importance of the whole body, upper body, and lower body similarity constraint. In the experiments, we set $\lambda_{g}$=1, $\lambda_{up}$=1 and change the value of $\lambda_{lower}$. $\lambda_{lower}$ is set to 0.5 and 1 respectively, and the evaluation results are shown in 7. Specifically, $\lambda_{lower}$=0.5 yields better accuracy. A possible explanation is that the upper body contains more discriminative information about person images than the lower body. However, it still contributes to the learning of features on the target domain.