Unsupervised Person Re-identification via Simultaneous Clustering and Consistency Learning

Clustering tends to focus on low-level features, like color, which is unessential for the objective of semantic clustering. To explore richer visual patterns and obtain semantically meaningful features, we propose ConsLearn and integrate it with the clustering network. The following sections present the cornerstones of our approach, which can encourage clustering network to separate images into semantically meaningful clusters without human annotations.

Dynamic dropblock layer. In this subsection, we present the details of dynamic dropblock layer (DDL). DDL can be applied on any feature layer of network model, and induce model to capture the different semantic information of the object during training. It is worth noting that DDL is deactivated during the testing phase.

The input of DDL is a convolutional feature map $F\in\mathbb{R}^{H\times W\times C}$ where $C$ represents the number of channels, $H$ and $W$ are height and width of feature map, respectively. We first set two main hyperparameters $\alpha$ and $\beta$. They control the height and width ratio of the erased region on map $F$, respectively. The size of region to be dropped increases as $\alpha$ and $\beta$ increases and is adaptive to the size of feature map when the hyperparameters are fixed. Therefore, the dropblock layer is dynamic. The height and width of the erased region vary from task to task. But in our work, the removed region should contain a semantic part of feature map. Specifically, the drop mask $M_{drop}\in\mathbb{R}^{H\times W}$ is generated by setting the pixel to 0 if it is inside the generated erased region, and 1 otherwise. The DDL randomly remove the part of feature map $F$ by multiplying it to the drop mask. In this way, we can produce different views for input feature map, namely different semantic body parts. We visualize the example of each component in the following experiment. The regularization technique of Cutout [10] also studies the application of random erasing. A difference from Cutout is the erased region size of our DDL changes dynamically with the size of feature map, while the removed region size of Cutout is fixed and squared. The other difference is that our DDL can be applied on any feature layer of network instead of only input image as in Cutout. In fact, our DDL can be considered as an extension of random erasing for feature maps and also an extension of Cutout [10] or Dropout [32].

Learning consistency. Consider as input a person images $x$, the pixel input is mapped to a feature space twice by an encoder with DDL, such that $h_{k}=\Phi_{k}(x|\theta)$, $k=1,2$. Here, $k=1,2$ mean the two data flows of the same image, which are fed into network in order. Since the erased area of DDL is given randomly at each layer, the firstly entered network $\Phi_{1}(\cdot|\theta)$ is different from the second network $\Phi_{2}(\cdot|\theta)$. Thus, $k$ is also used to distinguish them. This is also why we use two data flows for the same image. In this way, the network models $\Phi_{k}(\cdot|\theta)$ output two different encoded views $h_{k}$, $k=1,2$, leading to two differently semantic features from the same image. Our aim is to reduce the ability for clustering depending on low-level features and induce the model to learn the inherent visual consistency across different views. Therefore, we want to maximize the agreement between $h_{1}$ and $h_{2}$. A linear layer $g$: $\mathbb{R}^{D}\rightarrow\mathbb{R}^{\Omega}$ follows the feature representations to map them to a latent space, which is defined as

$$\small u_{k}=g(h_{k}),k=1,2.$$

(1)

In this space, our model can better learn a semantically meaningful representation that describes what is in common between $u_{1}$ and $u_{2}$ while discarding instance-specific details.

The encoder model at current iteration can output the encoded views for all the person images, but this rapidly changing encoder also reduces the feature representations’ consistency across different training stages and eventually results in training error amplification. To resolve this issue, we present the momentum-based moving average model as a way of preserving more original knowledge for unsupervised learning with a consistency loss. Formally, let the temporal average model be $\Phi_{k}^{*}(\cdot|\theta^{*})$. Then, its model parameter $\theta^{*(t)}$ at current iteration $t$ can be updated by

$$\small\theta^{*(t)}=\zeta\theta^{*(t-1)}+(1-\zeta)\theta^{(t)},$$

(2)

where $\zeta\in[0,1)$ is a momentum coefficient and the initial temporal average parameters $\theta^{*(0)}$ is $\theta$.

Two momentum-based encoded views of the same images are generated by the temporal average model $\phi$ and the linear layer $g^{*}$. That is, $\tilde{u}_{k}=g_{k}(\Phi_{k}^{*}(x_{k})),k=1,2$. Considering the errors caused by self-variation and large differences in values, the feature distributions $u_{k}$ and $\tilde{u}_{k}$ are scaled up by a softmax operation,

$$\small v_{kl}=\delta(u_{k})=\frac{e^{u_{kl}}}{\sum_{j=1}^{\Omega}e^{u_{kj}}},k=1,2,$$

(3)

$$\small\tilde{v}_{kl}=\delta(\tilde{u}_{k})=\frac{e^{\tilde{u}_{kl}}}{\sum_{j=1}^{\Omega}e^{\tilde{u}_{kj}}},k=1,2,$$

(4)

where $v_{kl}(\tilde{v}_{kl})$ is the $l$-th element of $v_{k}(\tilde{v}_{k})$. Now, the problem is how to turn the consistency information into a learning signal for optimizing network models.

It is well known that the Kullback-Leibler (KL) divergence $\displaystyle KL(\cdot)$ can be thought of as a measurement of how far the distribution $Q$ is from the distribution $P$. For discrete probability distributions $P$ and $Q$ defined on the same probability space $\mathcal{X}$, $P$ typically represents the “true” distribution of data, while $Q$ typically represents a approximation of $P$. In order to find a distribution $Q$ that is closest to $P$, we can minimize $KL$ divergence, which can be rewritten by

	$\displaystyle KL(P\|Q)$	$\displaystyle=-\sum_{x\in\mathcal{X}}P(x)\log(Q(x))+\sum_{x\in\mathcal{X}}P(x)\log(P(x))$		(5)
		$\displaystyle=H(P,Q)-H(P),$

where $H(P,Q)$ is the cross entropy of $P$ and $Q$, and $H(P)$ is the entropy of $P$. Here, $KL(P\|Q)$ is proportional to $H(P,Q)$ if $H(P)$ is a constant.

The embedding features $v_{k}$ and $\tilde{v}_{k}$, $k=1,2,$ can be interpreted as the distribution of a discrete random variable over training samples $x$. Furthermore, $\tilde{v}_{i}$ can be viewed as a constant or “true” distribution since it represents invariant feature distribution obtained by the temporally moving average model. Our aim is that $v_{1}$ should be close to $\tilde{v}_{2}$ and $v_{2}$ should be close to $\tilde{v}_{1}$ as much as possible, which supports visual reasoning between different semantic features. Therefore, $H(P,Q)$ is adopted to turn the consistency information into a learning signal for optimizing our models. The consistency of different encoded views is measured using the following loss, as

	$\displaystyle\mathcal{L}_{co}$	$\displaystyle=H(\tilde{v}_{1},v_{2})+H(\tilde{v}_{2},v_{1})$		(6)
		$\displaystyle=-\sum\tilde{v}_{1}\log(v_{2})-\sum\tilde{v}_{2}\log(v_{1}).$

Unlike pervious clustering techniques [12, 27, 13], our clustering method first generates pseudo labels according to features from a view of the same images and utilizes the same labels for all the encoded views since they share large visual similarity. Its aim is to further enhance the visual consistency between different views based on unsupervised clustering. Specifically, given encoded features $\{h_{1}^{i}\}_{i=1}^{N}$ of images $\{x^{i}\}_{i=1}^{N}$ from a view, clustering algorithm (i.e., DBSCAN [11]) divides these learned features into different clusters as pseudo label $\{\hat{y}\}_{i=1}^{N}$, which represents the image’s membership to one of $M$ possible predefined categories. Meanwhile, the representations are also fed into a linear classification layer, converting the feature vector $h_{1}^{i}$ into a vector of class scores $z_{1}^{i}$. Finally, the softmax operator is performed to map the class scores to class probabilities $p(\hat{y}_{i}|x_{i})$, which is the predicted probability of image $x_{i}$ belonging to the identity $\hat{y}_{i}$. As in [52], the clustering procedure for two encoded views is achieved by minimizing the cross entropy loss,

$$\small\mathcal{L}_{ce}=-\frac{1}{N}\sum_{i=1}^{N}\log p(\hat{y}_{i}|h_{1}^{i})-\frac{1}{N}\sum_{i=1}^{N}\log p(\hat{y}_{i}|h_{2}^{i}).$$

(7)

Furthermore, we introduce a softmax-triplet loss to enforce a positive pair to be closer and a negative pair apart from each other in the representation space. As in [21], let $h_{1}^{(in)}$ and $h_{1}^{(ip)}$ denote the similarity of the hardest negative pair $\{h_{1}^{i},h_{1}^{n}\}$ and the hardest positive pair $\{h_{1}^{i},h_{1}^{p}\}$ for encoded views $\{h_{1}^{i}\}_{i=1}^{N}$, with an anchor $h_{1}^{i}$. The similarity can be computed as $h_{1}^{(in)}=-\|h_{1}^{i}-h_{1}^{n}\|$ and $h_{1}^{(ip)}=-\|h_{1}^{i}-h_{1}^{p}\|$, where $\|\cdot\|$ represents Euclidean distance. In this sense, minimizing the similarity of the hardest negative pair means maximizing the distance between $h_{1}^{(i)}$ and $h_{1}^{(n)}$ and vice versa. Similarity, encoded views $\{h_{2}^{i}\}_{i=1}^{N}$ have similar results. Thus, the softmax-triplet loss for two encoded views can be performed by

	$\displaystyle\mathcal{L}_{st}=$	$\displaystyle-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(h_{1}^{(in)})}{\exp(h_{1}^{(in)})+\exp(h_{1}^{(ip)})}$		(8)
		$\displaystyle-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(h_{2}^{(in)})}{\exp(h_{2}^{(in)})+\exp(h_{2}^{(ip)})}.$

Overall, unsupervised deep clustering alternates between clustering features as pseudo label to update the parameters of the model using Eq. (7) and learning features by collecting and distinguishing informative pairs accurately using Eq. (8). The proposed procedure can be achieved by

$\displaystyle\mathcal{L}_{clu}=\lambda\mathcal{L}_{ce}+\xi\mathcal{L}_{st}$

(9)

The overall loss for network. ConsLearn is implemented with clustering, which not only perceives the current pretext task, but also learns the feature information related to the target task. By adding the clustering loss (Eq. (9)) into the consistency loss (Eq. (6)), the overall loss $\mathcal{L}$ of our final proposed method can be expressed as

$$\small\mathcal{L}=\lambda\mathcal{L}_{ce}+\xi\mathcal{L}_{st}+\eta\mathcal{L}_{co},$$

(10)

where $\lambda$, $\xi$ and $\eta$ are weighting parameters.

Datasets and evaluation protocol. Our proposed model is evaluated on three public person re-identification (re-ID) benchmarks: Market-1501 [50], DukeMTMC-reID [31], and MSMT17 [39]. To evaluate the performance of our method, two evaluation metrics are chosen as the mean Average Precision (mAP) and the cumulative matching characteristic (CMC) top-1, top-5, top-10 accuracies  [50].

Implementation details. All person images are resized to 256$\times$128 before fed into the networks. Then, we adopt random cropping and flipping as the data augmentation. After that, we feed these images into ResNet-50 [19] with the last classification layer removed, initialized by the ImageNet [8]. We use a mini-batch size of 256 in 4 GPUs. The network models are trained for 50 epochs, using ADAM optimizer with learning rate of 0.00035, weight decay of 0.0005. DBSCAN [11] is used for clustering in each epoch. For the momentum coefficient, we set $\zeta$ as 0.999. We empirically set the hyperparameters $\lambda,\xi$ and $\eta$ for loss balance as 0.2, 0.35 and 0.1 for unsupervised re-ID on benchmarks including Market-1501 and DukeMTMC-reID. For other datasets, these parameters may be slightly different.

Unsupervised person re-identification. To verify effectiveness of the proposed method, we compare our method with the state-of-the-art methods by training the re-ID model without any labeled data. Table 1 shows the experimental results on three benchmarks, including Market-1501 (Market), DukeMTMC-reID (Duke), and MSMT17. On Market-1501, under the same setting, our method achieves the best performance among the compared methods with mAP=77.7% and top-1=90.4%, which exceeds the state-of-the-art method (MMT [14]) by large margins (3.0% for mAP and 2.0% for top-1). On DukeMTMC-reID, we also achieve evident 3.7% mAP improvement over the current best approach. For a larger and more challenging dataset (MSMT17), the proposed method has also achieved comparable performance in terms of mAP and CMC. With the obtained results, we validate that our method is able to learn richer representations of person images compared with previous methods.

Unsupervised domain adaptation person re-identification. To further demonstrate its effectiveness, our method is tested on several domain adaptation tasks, which are the representative target tasks to validate the effectiveness of unsupervised representation. These tasks contains Duke-to-Market and Market-to-Duke. We initialize the backbone with the model pre-trained on labeled source domain, and fine-tune on unlabeled target domain.

Table 2 summarizes the experimental results on two adaptation tasks. On Market-to-Duke, compared to the state-of-the-art method, our method obtains 6.2% and 2.8% improvements on mAP and Top-1 accuracy, respectively. Similar results can be observed in Market-to-Duke. The above impressive performances demonstrate the necessity and effectiveness of our proposed method for unsupervised domain adaptation re-ID. More importantly, current advanced techniques (MMT  [14] and MEB-Net [48]) train multiple networks to enhance the discrimination capability of re-ID model, while our method only use a network and thus is not required for the additional learning parameters in training stage compared to MMT and MEB-Net.

In this section, we conduct extensive experiments on DukeMTMC-reID to analyze the effect of our method under different setting.

Dynamic dropblock layer. First, we explore the influence of DDL at different stages by integrating DDL into ResNet-50. During the training phase, we plug DDL into network before the intermediate stages: stage 0, stage 1, stage 2, stage 3, and stage 4. For instance, stage 0 means that DDL is plugged into network before input map is fed into network. Table 3 reports the experimental results. From these results, it can be observed that the best accuracy can be achieved when the DDL is plugged in lower-level layers. However, the performance of DDL becomes worse under higher-level layers. The reason behind is that first-layer features are typically more general (e.g., unrelated to a particular task) while last-layer features exhibit greater levels of specificity (e.g., semantic and appearance diversity). This indicates that DDL tends to work if the features are general, meaning suitable to both pretext task and re-ID task. In addition, as the last line of Table 3, the performance of our method (w/o DDL) is worse than that of our method, which demonstrates that our method benefit from the single visual consistency of the same image’s different views.

Next, we investigate the effect of erased area on accuracy by changing the value of $\alpha$ and $\beta$. The upper part of Table 4 summaries the experimental results. From these results, we obtain the best performance with $\alpha=0.4$ and $\beta=0.3$ for lower-level layers. Meanwhile, when the erased area is greatly reduced (i.e, $\alpha=0.2$ and $\beta=0.1$), 8.5% mAP and 5.1% top-1 decreases are observed. This is because the model hard capture the different semantic information of the object. Furthermore, given that higher-level feature is semantic and appearance diversity, we also reduce the size of erased area when DDL is plugged in high-level layers. The results are reported in the lower part of Table 4, while the findings are still the same as before.

Clustering. We set the model as the baseline when training the network using only the traditional classification component. To verify the effectiveness of our clustering, we conduct extensive ablation studies. From experimental results in the upper part of Table 5, it can be easily observed that the performance of our clustering is worse than that of the baseline model. The main reason is that the clustering (w/o ConsLearn) focus on local detailed feature and thus the learned features from different data flows contains different information, leading to different clustering results. However, our clustering share the same pseudo label for them. When consistency learning is also conducted (the lower part of Table 5), our clustering is greatly improved and surpass the baseline model with ConsLearn (Baseline+ConsLearn) by margins (mAP=1.3% and top-1=0.7%). This indicates that our clustering begins to ignores low-level features and focus on semantic information with the help of ConsLearn. In this way, our clustering further enhance the visual consistency by using the same cluster for two encoded features. Additionally, the softmax-triplet loss also improve the performance of our clustering method, which demonstrates the effectiveness of softmax-triplet loss in our proposed framework.

Pretext task. As showed in Table 5, the baseline model results in much lower performance than our final method. A key reason is that the baseline model mainly focuses on the local detailed features to optimize the target task of re-ID while discarding meaningful semantic feature. As a pretext or base task, consistency learning is integrated into the baseline model, abbreviated as “Baseline + ConsLearn”. Under this setting, the baseline obtain the obvious improvement (mAP=11.5% and top-1=6.5%), which further demonstrates the importance of adopting LearnCons as pretext task.

Momentum update mechanism. The momentum-based averaging model allows us to enforce the temporal consistency over network model at different iterations without additional model parameter. To investigate its effectiveness, we conduct ablation studies by directly use one network s current-iteration output as feature distribution, which is abbreviated as “Ours (w/o M)”. As shown in Table 6, “Ours (w/ M)” consistently improves the results over “Ours (w/o M)”, which represents our method without temporal consistency based on averaging model. The qualitative results demonstrates that the necessary of temporal consistency and the single consistency obtained by model at different stages has different effects on our results.

Projection head. We then study the importance of the linear projections, i.e., $f(\cdot)$ and $g(\cdot)$. Figure 3 reports evaluation results using three different architectures for the projection heads: (1) identity mapping; (2) linear projection; (3) shared linear projection for $f(\cdot)$ and $g(\cdot)$. We observe that two kinds of linear projection are obviously better than no projection (i.e., identity mapping), especially for the large output dimensions (e.g., 2048 in this paper). This indicate that, by using two projections, more information can be formed and maintained in $u_{i}$ and $\tilde{u}_{i}$, $i$=1,2. Furthermore, the performance of shared linear becomes better as output dimensions increases and outperform normal linear with large dimensions (e.g., 1024 and 2048 in this paper), which demonstrates that shared linear projection can obtain more information that is useful for two tasks.

From the perspective of representation learning, our method integrate disparate visual percepts into persistent entities and underlie visual reasoning in feature space. As shown in the left of Figure 4, the network model (w/ DDL) no longer pays attention to some semantic parts at early stages when they are occluded. However, at later stages, the predicted target regions of training image are backtraced to the initial information by using the information of the other view regardless of losing the target on a view. Finally, the heatmap extracted from model almost highlight the entire region of the person. This indicates that the ability of model to predict the missing patch of feature maps are enhanced, inching the two different features toward consistency. On the contrary, the right of Figure 4 shows the model (w/o DDL) only focus on the most discriminative part of object, not the entire object. In addition, unlike the pervious clustering, our clustering treats the same pseudo labels as ground truth labels for two encoded views to bootstrap the discriminative ability of network model.

Our proposed framework is trained with both unsupervised clustering and representation learning. In this way, our method can separate the images into semantic clusters, where images within the same cluster belong to the semantic classes and images in different clusters are semantically dissimilar. Since self-supervised representation learning is task-agnostic during training, they often have to focus on the feature information, which is unrelated to the current target task. Indeed, in our method, the clustering induces network model to capture feature representation that concentrates on the target task of re-ID. Meanwhile, consistency learning encourages clustering to obtain semantically meaningful features rather than relying on low-level features.