Anti-Forgetting Adaptation for Unsupervised Person Re-identification

Given an encoded query $q$ and a set of encoded samples $K$ = {$k_{0}$, $k_{1}$, $k_{2}$, …}, we use $k_{+}$ to denote the positive match of the query $q$. We define a softmax cosine similarity function:

where $q\cdot k$ is the cosine similarity between $q$ and $k$. $\tau$ is a temperature parameter that controls the similarity scale.

Inside an unsupervised lifelong ReID pipeline, our model incrementally learns new knowledge on each domain. For step $s$, $D_{s}=\{x_{1},...,x_{n}\}$ where $n$ is the number of current domain unlabeled images. We first use the momentum encoder $\theta_{m}$ to extract image representations and calculate re-ranking [56] based similarity between each image pair. Based on the pair-wise similarity, the DBSCAN clustering algorithm is utilized to assign pseudo labels $\{y_{1},...,y_{n}\}$ to unlabeled images $\{x_{1},...,x_{n}\}$. Given a current domain image $x_{i}$, $f(x_{i}|\theta)$ and $f(x_{i}|\theta_{m})$ denote respectively the online and the momentum representations. The prototype of a cluster $j$ is defined as the averaged momentum representations of all the samples with a same pseudo-label $y_{j}$:

where $n_{j}$ is the number of images in the cluster $j$.

We use $P$ to denote all the cluster prototypes in the current domain. A cluster prototype adaptation loss maximizes the similarity between a sample $x_{i}$ and the positive prototype in $P$, which can be defined as:

where $\tau_{pa}$ is a temperature hyper-parameter.

The prototype-level adaptation loss $\mathcal{L}_{pa}$ makes samples from the same cluster converge to a common prototype and push them away from other clusters. However, images belonging to the same class can be easily affected by noisy factors, such as illumination and view-point, leading to high intra-class variance. When the adaptation module is only trained with $\mathcal{L}_{pa}$, all the image samples converge at a same pace. Consequently, representations of hard positive samples remain far away after optimization. We use an identity-aware sampler to construct mini-batches. A mini-batch $X$ is composed of $n_{p}$ identities, where each identity has $n_{k}$ positive instances. Thus, the batch-size is $n_{bs}=n_{p}\times n_{k}$. Given an anchor instance $f(x_{i}|\theta)$, we sample the hardest positive momentum representation $f(x_{j}|\theta_{m})$ that has the lowest cosine similarity with $f(x_{i}|\theta)$. To improve the intra-class compactness, we formulate an instance contrastive adaptation loss:

where $f(X|\theta_{m})$ denotes the mini-batch containing one hard positive $f(x_{j}|\theta_{m})$ and $(n_{p}-1)\times n_{k}$ negatives, and $\tau_{ia}$ is a temperature hyper-parameter.

Remark. In our instance-level contrastive adaptation loss, we select the hardest positive to maximally increase the similarity between an anchor and the hard positive. Another possibility is to use the average of top-k hard positives as the contrastive positive view. We compare the positive selection methods in Table II. The results show that the hardest positive slightly outperforms the average of top-k hard positives. We thus choose to select the hardest positive in our method.

By reducing the intra-cluster variance between hard positive samples, the instance-level adaptation loss $\mathcal{L}_{ia}$ allows for purifying the new knowledge on $D_{i}$ before being accumulated into the model $\theta_{m}$. Person ReID is a cross-camera image retrieval task, where the individual camera style is the main factor that causes intra-cluster variance. As shown in ICE [2], using camera labels to reduce camera style variance can further mitigate the noise during the knowledge accumulation. A camera proxy $p_{jk}$ is defined as the averaged momentum representations of all the instances belonging to the cluster $j$ captured by camera $c_{k}$:

where $n_{jk}$ is the number of instances belonging to the cluster $j$ captured by camera $c_{k}$. We form a set of camera prototypes $P_{cam}$, which combines one positive cross-camera prototype and $n_{neg}$ nearest negative prototypes. A camera contrastive loss is the softmax log loss that minimizes the distance between an anchor $f(x_{i}|\theta)$ and each of positive cross-camera prototypes:

where $\tau_{c}$ is a cross-camera temperature hyper-parameter, which is set to 0.07 following [2]. $|\mathcal{P}|$ is the number of cross-camera positive prototypes. We provide extra results with the camera contrastive loss for knowledge accumulation in Section IV.

Technically, person ReID is a representation similarity ranking problem, in which the objective is to have high similarity scores between positive pairs and low similarity scores between negative pairs. However, when a model is adapted into a new domain, the similarity relationship between old domain samples could be affected by the new domain knowledge. As a cluster prototype (the averaged representation of all cluster samples) contains generic information of a cluster, the prototype memory enables the current model to have access to generic old domain cluster information without storing all the images. For the memory buffer of $n_{mem}$ buffered images and prototypes, we first use the buffered cluster prototypes $P^{o}=\{p^{o}_{1},...,p^{o}_{n_{mem}}\}$ as anchors to rehearse the old domain knowledge. As the similarity relationship between images and prototype should be consistent before and after a domain adaptation step, we propose an image-to-prototype similarity loss to ensure the upcoming new knowledge would not change the image-to-prototype similarity.

As the frozen old model $\theta_{s-1}$ from the last domain can be regarded as an expert on the old domain, we calculate the cosine similarity between representations encoded by $\theta_{s-1}$ and prototypes as a reference to regularize the online model $\theta$. Given a mini-batch of old images $\{x_{1}^{o},...,x_{n_{bs}}^{o}\}$ randomly sampled from the memory buffer where $n_{bs}$ is the batch size, the online image-to-prototype similarity between a buffered image $x_{i}^{o}$ and buffered prototypes $P^{o}$ is defined as $S(f(x_{i}^{o}|\theta),P^{o},\tau_{ps})$. For the same mini-batch, we calculate the image-to-prototype similarity with the frozen model $\theta_{s-1}$ as the reference similarity. The reference similarity between the same image-prototype pair is defined as $S(f(x_{i}^{o}|\theta_{s-1}),P^{o},\tau_{ps})$.

We formulate an image-to-prototype similarity consistency loss with a Kullback-Leibler (KL) Divergence between the two similarity distributions:

The buffered prototypes can be regarded as anchors of previously-acquired knowledge. We expect that the new domain knowledge does not change the distance relationship between an image and the prototypes, so that we can retain the previously-acquired knowledge for anti-forgetting adaptation. KL divergence measures the information loss when one distribution is used to approximate another. By minimizing $\mathcal{L}_{ps}$, we encourage the image-to-prototype similarity distribution $S(f(x_{i}^{o}|\theta),P^{o},\tau_{ps})$ calculated with current domain knowledge to be consistent with that calculated with old domain distribution $S(f(x_{i}^{o}|\theta_{s-1}),P^{o},\tau_{ps})$.

Cluster prototypes contain general cluster information, for example, the most salient feature shared by different views of a person. Differently, image instances contain more detailed information in a specific view, which is complementary to prototype-level general information. As the similarity relationship between the same images should be consistent before and after a domain adaptation step, we propose an image-to-image similarity loss that regularizes the similarity relationship updates in a way that does not contradict the old knowledge. Similar to the image-to-prototype similarity, we also use the frozen old model $\theta_{s-1}$ from the last domain as an expert to calculate the reference similarity. Given a mini-batch of old images $X^{o}=\{x_{1}^{o},...,x_{n_{bs}}^{o}\}$ where $n_{bs}$ is the batch size, the image-to-image similarity distribution can be calculated with a softmax function on the cosine similarity between each image pair in the mini-batch. The image-to-image similarity between an old image $x_{i}^{o}$ and a mini-batch $X^{o}$ is calculated with both the online encoder $\theta$ and the momentum encoder $\theta_{m}$, i.e., $S(f(x_{i}^{o}|\theta),f(X^{o}|\theta_{m}),\tau_{is})$. For the same mini-batch, we calculate the image-to-image similarity distribution with the frozen old model $\theta_{s-1}$ as a reference for the constraint. The reference similarity between the same image $x_{i}^{o}$ and the mini-batch $X^{o}$ is $S(f(x_{i}^{o}|\theta_{s-1}),f(X^{o}|\theta_{s-1})),\tau_{is})$.

We formulate an image-to-image similarity constraint loss with the KL Divergence between the two distributions:

By minimizing $\mathcal{L}_{is}$, we encourage the similarity relationship calculated with current domain knowledge $\theta$ to be consistent with that calculated with old domain knowledge $\theta_{s-1}$.

Remark. As shown in Fig.3, we introduce both domain gap and data augmentation perturbations. The domain gap perturbations are introduced by using online and frozen encoders. The data augmentation perturbations are introduced by using two data augmentation settings on same images. Inspired by consistency regularization from semi-supervised learning [57], we use weak data augmentation on reference similarity calculation and strong data augmentation on prediction similarity calculation. As data augmentation can mimic image perturbations, augmentation consistency regularization allows for enhancing the model robustness against perturbations in real deployments.

Our method is implemented under Pytorch [71] framework. The total training time with 4 Nvidia 1080Ti GPUs is around 6 hours. We use an ImageNet [72] pre-trained ResNet50 [73] as our backbone network. For the strong data augmentation, we resize all images to 256$\times$128 and augment images with random horizontal flipping, cropping, Gaussian blurring and erasing [74]. For the weak data augmentation, we only resize images to 256$\times$128.

At each step in the one-cycle full set training, we train our framework 30 epochs with 400 iterations per epoch using a Adam [75] optimizer with a weight decay rate of 0.0005. For the two-cycle subset evaluation, we follow CLUDA [32] to train our model for 60 epochs without specified iterations. The learning rate is set to 0.00035 with a warm-up scheme in the first 10 epochs. No learning rate decay is used in the training. Pseudo labels on the current domain are updated on re-ranked Jaccard distance [56] at the beginning of each epoch with a DBSCAN [55], in which the minimum cluster sample number is set to 4 and the distance threshold is set to 0.55. The momentum encoder is updated with a momentum hyper-parameter $\alpha=0.999$. To balance the model ability on old domains and the new domain, we separately take a mini-batch of current domain images and a mini-batch of buffered images of the same batch size $n_{bs}$, which is set to $n_{bs}=32$ in our experiments. Furthermore, we use a random identity sampler to construct mini-batches to handle the imbalanced images of different identities. Following the clustering setting on the current domain, the 32 current domain images are composed of 8 identities and 4 images per identity. Inside the mini-batch of buffered images, the 32 buffered images are composed of 32 identities and 1 image per identity.

We use grid search to set the optimal temperature and balancing hyper-parameters in our proposed losses. Based on grid search results, we set the temperature hyper-parameters $\tau_{pa}=0.5$, $\tau_{ia}=0.1$, $\tau_{ps}=0.1$ and $\tau_{is}=0.2$. To make rehearsal and adaptation losses on the same scale, we set the balancing hyper-parameters $\lambda_{ia}=1$, $\lambda_{ps}=10$ and $\lambda_{is}=20$. For the memory buffer, we set $n_{mem}=512$, where the total identity number equals 512 and 1 image per cluster. After the whole training, only the momentum encoder is saved for inference.

We re-implement 3 types of unsupervised methods, including unsupervised domain adaptation, unsupervised lifelong and supervised domain generalization methods to compare with our method.

The unsupervised domain adaptive methods include ICE [2], CC [76] and PPRL [22], which are trained sequentially on each seen domain. The lifelong methods include three general-purpose lifelong methods (LwF [42], iCaRL [45] and C$o^{2}$L [47]), one backward-compatible class-incremental method CVS [12] and one lifelong ReID method LSTKC [31]. LwF is a regularization-based method, which does not store old samples for rehearsal. LwF uses a prediction-level cross-entropy distillation [77] between old and new domain models. iCaRL and C$o^{2}$L are rehearsal-based methods. iCaRL conducts prediction-level distillation on new and stored old images for rehearsal. C$o^{2}$L proposes an asymmetric supervised contrastive loss and a relation distillation for supervised continual learning. CVS formulates an inter-session data coherence loss and a neighbor-session triplet loss to enhance model coherence during the incremental learning. For general-purpose methods LwF, iCaRL, C$o^{2}$L and CVS, we combine our unsupervised adaptation module ($\mathcal{L}_{pa}+\mathcal{L}_{ia}$) and the lifelong learning techniques of each paper to convert these methods to person ReID and conduct a fair comparison with our method. For example, in LwF, we combine our contrastive baseline for learning current domain knowledge and the prediction-level distillation for mitigating the forgetting. To convert the supervised lifelong method LSTKC [31] into an unsupervised lifelong method, we combine our unsupervised adaptation module ($\mathcal{L}_{pa}+\mathcal{L}_{ia}$) and the rectification-based knowledge distillation. We also re-implement a supervised domain generalization method ACL [25] to show the upper bound of the generalization ability. The compared lifelong methods, such as LwF [42], iCaRL [45], C$o^{2}$L [47], CVS [12] and LSTKC [31], follow the same data augmentation. For ICE [2], CC [76], PPRL [22] and ACL [25], we directly use their original augmentation setting, including random cropping, flipping and erasing.

To tackle the forgetting problem during the unsupervised domain adaptation, we propose a dual-level joint adaptation and anti-forgetting method. The performance improvement of DJAA over the baseline mainly comes from the combination of the adaptation losses and the rehearsal losses ‘$\mathcal{L}_{pa}$’, ‘$\mathcal{L}_{ia}$’, ‘$\mathcal{L}_{ps}$’ and ‘$\mathcal{L}_{is}$’. To validate the effectiveness of each loss, we conduct ablation experiments by gradually adding one of them onto the baseline. In Table X, when we only use a prototype adaptation loss ‘$\mathcal{L}_{pa}$’, our model tends to lose most of the previous knowledge. The instance-level adaptation loss ‘$\mathcal{L}_{ia}$’ reduces intra-cluster variance, which prevents the noise accumulation in the multi-step adaptation. The prototype-level similarity consistency loss ‘$\mathcal{L}_{ps}$’ and the instance-level similarity consistency loss ‘$\mathcal{L}_{is}$’ respectively make the image-to-prototype and image-to-image similarity relationships consistent to domain knowledge changes. A cluster prototype contains general information of a cluster, while an instance contains specific information from a single view. As the two consistency losses work on different levels, ‘$\mathcal{L}_{ps}$’ and ‘$\mathcal{L}_{is}$’ are complementary to each other. By combining all the above-mentioned losses, our full DJAA framework yields the highest performance on both seen and unseen domains. We draw the forgetting curve in Fig. 7 and the generalization ability curve in Fig. 7, which further validate the effectiveness of our proposed losses.

The backward compatible learning aims at maintaining the consistency of representations after training more data, so that previously extracted representations can be directly compared with newly extracted representations. The backward compatibility is strongly correlated with the anti-forgetting ability of a method. We perform an ablation study to validate the effectiveness of our Rehearsal Module in addressing the backward compatibility problem. The difference between the self-test and the cross-test reflects the backward compatibility. As shown in Table XI, the cross-test performance obviously degrades when we only use the adaptation losses $\mathcal{L}_{pa}$ and $\mathcal{L}_{ia}$. The improvement in the backward compatibility comes mainly from our rehearsal module ($\mathcal{L}_{ps}$ and $\mathcal{L}_{is}$). In particular, the image-to-image similarity consistency loss $\mathcal{L}_{is}$ provides the most significant performance boost.

In our method, the data augmentation includes random horizontal flipping, cropping, Gaussian blurring and erasing. These augmentation techniques are chosen to mimic real-world perturbations, such as viewpoint variance, imperfect detection and occlusion. We use these data augmentation techniques to generate augmented views for contrastive learning. By maximizing the similarity between positive views, our model can learn robust representations. As shown in Table XII, all the chosen augmentation techniques contribute to the adaptation performance. Among these augmentation techniques, the random erasing provides the most significant performance improvement, while the random blurring provides the least. Suitable data augmentation should be diverse but realistic, which means augmentation should not bring in distortions absent in the real dataset. The four augmentations (random flipping, cropping, blurring and erasing) are empirically selected. The results in Table XII show that adding color jitter augmentation achieves similar results. Thus, we do not bother adding color jitter.

In addition to regularizing the consistency between two encoders of different steps, we also introduce data augmentation perturbations to further enhance the model robustness. We report the performance of different data augmentation settings in Table XIII. Different data augmentation settings on the prototype-level similarity consistency loss ‘$\mathcal{L}_{ps}$’ have only a slight influence on the final performance. For the instance-level similarity consistency loss ‘$\mathcal{L}_{is}$’, the influence of the data augmentation setting is more evident. We can observe that data augmentation brings in meaningful perturbations for consistency regularization. Using weakly augmented similarity as reference to regularize the strongly augmented prediction similarity with perturbations is the optimal setting for augmentation consistency regularization.

: We use grid search to set the optimal temperature and weight hyper-parameters to balance our proposed losses. The temperature hyperparameters $\tau_{ps}$ and $\tau_{is}$ control the scale of image-to-prototype and image-to-image similarity. Based on the results in Table XIV, we set the temperature hyperparameters $\tau_{ps}=0.1$ and $\tau_{is}=0.2$ in image-to-prototype and image-to-image similarity rehearsal losses, respectively. The weight hyperparameters $\lambda_{ps}$ and $\lambda_{is}$ balance the importance of contrastive adaptation, image-to-prototype and image-to-image similarity losses. Based on the results in Table XIV, we set the balancing weight hyperparameters $\lambda_{ps}=10$ and $\lambda_{is}=20$. The exponential moving average hyperparameter $\alpha$ control the the speed of the knowledge accumulation from the online encoder to the momentum encoder. Table XIV shows that $\alpha=0.999$ is the optimal setting for our framework. The hyperparameters are tuned in the training order Market$\to$Cuhk-Sysu$\to$MSMT17. The tuned hyperparameters are kept the same for the second training order MSMT17$\to$Market$\to$Cuhk-Sysu, which validates the effectiveness of these hyperparameters.

As datasets are of different scales and diversity, it is meaningful to know whether the training order has a significant influence on the final results. Our primary training order starts from a medium domain Market and ends with the largest domain MSMT17. However, it is hard to control the order of upcoming domains in the real world. We test a second order MSMT17$\to$Market$\to$Cuhk-Sysu that starts from the largest domain MSMT17. In the second training order, it is easier to forget more knowledge on the largest domain MSMT17. Consequently, the performance under the training order #2 is slightly inferior to that of the training order #1. As shown in Table XV, DJAA still significantly outperforms the state-of-the-art methods CLUDA and CVS on seen domains. In the meantime, as shown in Table XVI, DJAA also outperforms state-of-the-art methods on unseen domains. We provide a forgetting curve on the seen domains in Fig. 9 and a generalization ability on unseen domains in Fig. 9. Compared to previous methods, our proposed DJAA shows both strong anti-forgetting and generalization abilities. We further compare backward-compatible ability between state-of-the-art methods and our proposed method DJAA in the second training order. As shown in Table XV, DJAA consistently shows superior backward-compatible ability over previous methods in the second training order.

To update the image memory at the end of each step, our strategy is to first select $n_{new}$ clusters with largest cluster sample numbers. The prototypes of selected clusters are used to update the memory buffer. Then, we select one image that is closest to its cluster prototype. In this way, we select the representative clusters for our image-to-prototype similarity consistency loss. As shown in Table XVIII, we compare three possible memory buffer updating strategies. ‘Random’ refers to randomly sample $n_{new}$ images from the current domain, which neglects the clustering information. ‘ID-wise’ refers to randomly sample $n_{new}$ clusters and one image per cluster, which has better data diversity than ‘Random’. ‘ours’ refers to our proposed strategy in Section III-D that selects representative clusters and credible images, which brings in the best performance on both seen and unseen domains.

We build a hybrid memory buffer to store cluster prototypes and image samples for the rehearsal module of our method DJAA. With a default memory size $n_{mem}=512$, our image memory stores approximately $512$ images $\approx 0.8\%$ of all the training images (Market, Cuhk-Sysu and MSMT17). Meanwhile, our prototype memory stores approximately $512$ prototype vectors (dimension $1\times 2048\times 1\times 1$)$\approx$4.2 MB, which is negligible compared with storing dataset images (for example, MSMT17$\approx$2.5 GB).

To further evaluate the dependency on the memory size, we vary the value of $n_{mem}$ and report the results in Table XIX. If the buffer size is limited to an extremely small number, such as 32 or 64, the results are even lower than our adaptation module baseline. When the buffer size is greater than or equals 128, the results are higher than the baseline. We can observe that the more data we store, the higher performance DJAA can achieve. As shown in Table IV, CLUDA [32] achieves 46.4% mAP and 62.8% mAP with a buffer size of 512. It is worth mentioning that our method achieves higher performance (49.8% mAP and 67.3% mAP) than CLUDA [32], when the buffer size equals 128.

Backward compatible learning aims to maintain the consistency of representations after training more data, so that we do not need to re-extract gallery features after each adaptation step. As shown in Table IX, with Market gallery features extracted at step 1, ICE has a cross-test performance 30.3% mAP / 49.3% Rank1 at step 2 and 3.4% mAP / 6.3% Rank1 at step 3. In such case, Market gallery features need to be extracted at steps 1, 2 and 3, and Cuhk-Sysu gallery features needs to be extracted at steps 2 and 3. In contrast, reinforced by the feature back-compatibility, Market and Cuhk-Sysu gallery features need to be extracted only once in our method, at the 1^st and 2^nd step respectively. For example, with stored Market gallery features, DJAA has a cross-test performance 71.7% mAP / 89.0% Rank1 at step 2 and 65.2% mAP / 85.6% Rank1 at step 3. A real-world lifelong ReID system may involve more adaptation steps, where a backward compatible method can avoid repetitive gallery feature extraction and fasten the inference.