SSKD: Self-Supervised Knowledge Distillation
for Cross Domain Adaptive Person Re-Identification

The identity information of source domain are available, thus the training process of the source data can be treated as a conventional classification problem (Zheng, Yang, and Hauptmann 2016). Source model is optimized by minimizing the cross entropy loss,

where $n_{s}$ is the number of images in source domain. $p(y_{s,i}|x_{s,i})$ is the predicted probability of image $x_{s,i}$ belonging to the identity $y_{s,i}$. At this time, the model has a certain adaptability for cross domain.

Image augmentation. To obtain the cross views of feature representation and prediction results, each image $x_{t,i}$ is augmented three times using random crop, random horizontal flip, and random erasing (Zhong et al. 2020), creating three views of the same image, $\tilde{x}_{t,3i+1}$, $\tilde{x}_{t,3i+2}$ and $\tilde{x}_{t,3i+3}$. The three images are further encoded via three encoder networks with the same architecture, referred to as student models, to generate representation features $h_{t,3i+1}$, $h_{t,3i+2}$ and $h_{t,3i+3}$. These feature encoders are driven to perceive subtle differences of natural characteristic and semantic information among different augmented views which is essential for the representation of precise fine-grained details.

Multi-similarity learning. After obtaining the representation features in the embedding space, we further explore full pair-wise relations between samples in a mini-batch. Triplet loss (Hoffer and Ailon 2015) and contrastive loss (Hadsell, Chopra, and LeCun 2006) are not sufficient to pull similar samples and repel dissimilar ones for re-ID task. These approaches focus on either cosine similarities or relative similarities. To exploiting the multiple similarities for the representation features of augmented views of different images, we employ the multi-similarity loss (Wang et al. 2019), implemented by two iterative steps with sampling and weighting, for pair weighting in the representation space. This allows it to compare the representation of an augmented view with other negative samples, providing a more principled method for mining the fine-grained details existing in the differently augmented views of samples.

Let the similarity of two features be $S_{ij}=<h_{t,i},h_{t,j}>$, where $<\cdot,\cdot>$ defines dot product, yielding a similarity matrix $\bm{S}$ whose element at $(i,j)$ is $S_{ij}$. Multi-similarity (MS) loss is formulated as,

where $n_{t}$ is the number of images in target domain, and $\alpha,\beta$ are hyper-parameters as in (Wang et al. 2019)

Identity learning. Aiming at aggregating the information from different augmented views of the same image, we design a fusion operation to generate an individual class for each training image. To learn richer representations, the operation should effectively confuse different features, while reserving the distinctive information. Under this principle, we sum three features and then average them, formulated as,

Based on these fused features, we perform clustering algorithm (Ester et al. 1996) on them, which leads to every person image assigned with a pseudo label according to clustering results. As a result, we can establish the pseudo label for each image and the number of person identities, denoted as $\hat{y}_{t,i}$ and $m_{t}$ respectively. The $k$-$th$ network model $f(\cdot|\theta_{k})$ is updated by optimizing a cross entropy loss with label smoothing, defined as

where $q_{j}=1-\varepsilon+\frac{\epsilon}{m_{t}}$ if $q_{j}=\hat{y}_{t,i}$, otherwise $q_{j}=\frac{\varepsilon}{m_{t}}$. The overall loss for identity learning can be expressed as $\mathcal{L}_{id}=\sum_{k=1}^{3}\mathcal{L}_{id}^{k}$. Comparing with cross entropy loss, $\mathcal{L}_{id}$ replace the original label distribution with a mixture of the label distribution and the uniform distribution, leading to label-smoothing regularization.

Self-supervised Learning. Due to the domain gap, the pseudo identities generated by clustering suffer from noises. Instead of the clustering-based method treating reliable samples as the same cluster for training, we propose a soft label learning, a complementary part of identity learning, that captures obvious similarities among semantic categories in a smooth way. It is important to note that the soft similarity is not learned from explicit guidance, but from the visual data itself. Specifically, the encoded representations are fed into a linear transformation network (a projection head), resulting in the soft pseudo labels $z_{t,3i+k}$ for data $x_{t,i}$ with the $k$-$th$ encoder network. The proposed learning procedure can be regarded as discriminative perception based on a probability $p^{c}_{3i+k}$ of which the input image belongs to class $c,p^{c}_{3i+k}=\frac{\exp(z_{t,3i+k}^{c})}{\sum_{c=1}^{C}\exp(z_{t,3i+k}^{c})}$, where $z_{t,3i+k}^{c}$ is the $c$-$th$ output of the $k$-$th$ projection head.

$Momentum$ $update$. When the above class predictions are directly served as soft labels to supervise another peer network, we obtain poor results in experiments. Such case is caused by the rapidly changing encoder that produces an error amplification. To enhance the model predictions’ consistency, we propose a momentum-based moving average model for each peer network. Formally, let the parameter of encoder network $f_{k}$ be $\theta_{k}$. The temporal average model is updated by

where $\rho\in[0,1)$ is a momentum coefficient, $t$ is the current iteration, and the initial temporal average parameters $\theta_{k,e}^{(0)}$ are $\theta_{k}$.

Next, we employ the one network’s past evolving model to generate supervised signals for training the other network. As shown in Figure 3, the soft label supervision are generated by the three slowly evolving models as $p_{1}(f(x_{3i+1}|\theta_{1,e})),p_{2}(f(x_{3i+2}|\theta_{2,e})),$ and $p_{3}(f(x_{3i+3}|\theta_{3,e}))$ respectively. The distillation loss with soft label for each network are defined as

The overall distillation loss is

Prediction discriminability and diversity. During training, we want the network could not only predict training samples into reliable classes, but make it acceptable to have a certain ability of prediction discriminability and diversity. BNM (Cui et al. 2020) is proposed to boost the model learning capability under label insufficient scenarios by reducing the ambiguous predictions. Due to its simplicity and effectiveness, BNM is adopted in our work to enhance the discrimination capacity of model. Formally, given a mini batch data with $B$ randomly selected unlabeled images, we define the number of classes as $C$ and introduce the batch prediction output matrix as $A\in\mathbb{R}^{B\times C}$ whose element at $(i,j)$ is $a_{ij}$,

After taking model predictions as pseudo identities, we further strengthen the robustness of prediction on categories without any extra guidance, using the nuclear-norm maximization of batch classification response matrix. Actually, stronger robustness means less uncertainty in the prediction. This achieved by optimizing the following loss of BNM.

Here $A_{k}$ and $A_{k,e}$ stand for the $k$-$th$ network model and its past evolving model respectively. $\|\cdot\|_{*}$ presents the the calculation of nuclear-norm, which is referred as (Cui et al. 2020).

$Overall$ $loss$. By adding the cross entropy loss $\mathcal{L}_{id}^{k}$, the distillation loss $\mathcal{L}_{dt}$, and the BNM loss $\mathcal{L}_{BNM}$ into Eq. (2), the complete loss of our final proposed Self-Supervised Knowledge Distillation (SSKD) can be expressed as

where $\xi\in(0,1)$ controls the importance of IL and SSL, and $\lambda,\eta$ are weighting parameters. The main training process are shown in Algorithm 1, which can be found in supplementary.

Datasets and evaluation metrics. We conduct experiments on three common datasets: Market-1501 (Zheng et al. 2015), DukeMTMC-reID (Ristani et al. 2016), and MSMT17 (Ristani et al. 2016). The proposed method is evaluated on four cross domain person Re-ID tasks, which contains Duke-to-Market, Market-to-Duke, Duke-to-MSMT, and Market-to-MSMT. The mean Average Precision (mAP) and Cumulative Matching Characteristic (CMC) curve are adopted as the evaluation metrics.

Implementation details. Random cropping, flipping and random erasing (Zhong et al. 2020) are employed as the data augmentation. We adopt ResNet-50 (He et al. 2016) as the backbone of our model with the last classification layer removed, and initialize it by the ImageNet (Deng et al. 2009) pretrained model. DBSCAN (Ester et al. 1996) is used for clustering before each epoch. We choose the hyper-parameters (e.g., $\xi,\lambda,\eta$) according to the validation set of the Duke-to-Market adaptation and directly apply them to other three cross domain re-ID tasks. For more experiment details, please refer to the supplementary.

We compare our proposed approach with existing techniques on three benchmarks and give the results in Table 1 and Table 2. Our models achieve state-of-the-art (SOTA) performance in several adaptation tasks.

Dukemtmc-reID and Market-1501. Table 1 reports the quantitative evaluation results on the Dukemtmc-reID and Market-1501. Our SSKD achieves an mAP of 78.7% and a top-1 accuracy of 91.7% for the Duke-to-Market, which surpasses the state-of-the-art method (MEB-Net) by large margins (2.7% for mAP and 1.8% for top-1). Similar results can be observed in another adaptation task. The proposed method obtains an mAP of 67.2% and a top-1 accuracy of 80.2% for Market-to-Duke, and achieves new state-of-the-art accuracy compared with the current techniques. Moreover, our method does not explore how to manually specify the number of identities in the training set of unlabeled target data, comparing with the current best methods (MMT and MEB- Net).

MSMT17. Our proposed method is also evaluated on a larger and more challenging dataset (MSMT17). From experimental results in Table 2, it can be observed that the proposed method clearly outperforms other methods when DukeMTMC-reID and Market-1501 are employed as source domains. For instance, with DukeMTMC-reID as source domain, our approach improves the accuracy by more than 2.5 percentage points over the state-of-the-art accuracy of mAP. Our method has better performance than other existing techniques on all experiments, which further demonstrates the validation of our proposed method. More importantly, our method almost approaches the performances of supervised learning (Luo et al. 2019) without any auxiliary information on the target domain.

Our proposed framework achieves new SOTA performance by using SSKD. To claim the effectiveness of each component in our framework, extensive ablation experiments are conducted under different settings.

Identity learning. We propose the identity learning (IL) to predict the ground truth classes for high-confident samples, which preserve the most reliable clusters for providing stronger supervision. To verify the effectiveness of such a learning manner, we evaluate our framework when removing it. The considerable degrades (e.g., from 78.7% to 51.3% for mAP in Table 3) are observed under this setting (SSKD (w/o $\mathcal{L}_{id}$)), especially for Duke-to-Market adaptation. This indicates the importance of adopting the identity learning.

Soft label learning. As a complementary component of identity learning, soft label learning (SLL) can regard labels as a distribution and enhance the robustness of category prediction. As in Table 3, on Duke-to-Market task, our SSKD beats SSKD (w/o $\mathcal{L}_{dt}$) by 78.7 and 91.7 percentage points on mAP and top-1 respectively. Similarly, for Market-to-Duke, obvious drops on mAP and top-1 are 1.5% and 1.6%, respectively. The large performance gaps demonstrate the effectiveness of the SLL module. Without SLL, images of the same identity from noisy samples are hard to be selected as reliable images because of domain shift and camera variance. In addition, invariance learning has an important effect in this training process by using the momentum update mechanism. We also demonstrates the importance of adopting the momentum update mechanism, which can be referred in the supplementary.

Multi-similarity learning. To investigate the impact of multi-similarity loss, we evaluate the performance of our method without multi-similarity learning (MSL). This modification brings the distinct drops of the performance, e.g., with an over 3% decrease of mAP on Market-to-Duke. MSL allows us to mining the fine-grained details existing in the differently augmented views of the same image. It can integrate the three similarities effectively into a single framework by using only one loss function.

Insufficient learning scenarios. Our learning task is an unsupervised open-set domain adaptation, where some labels are insufficient and even unseen. The methods without the BNM loss result in lower performances. We could see the significant progress and benifit of BNM to our re-ID tasks.

To analyze the effect of different hyper-parameters on adaptation re-ID task, we conduct some comparative experiments by changing the value of one parameter and keep the others fixed. We choose the hyper-parameters on Duke-to-Market task and directly apply them to other three adaptation tasks.

The impact of the hyper-parameter $\bm{\xi}$. We first investigate the impact of the hyper-parameter $\xi$ that balances the effect of the explicit classes and the reliable classes. As shown in Figure 4, when $\xi$ approaches 0 and 1, the model performance continue decreasing. At this time, the model reduces to IL and SLL respectively. The model achieves optimal performances when $\xi=0.5$. The reason is that the weight factor better balances two modules and induces model to predict ground truth classes for clean label and soft similarity for noisy label in a progressive way.

The impact of the hyper-parameter $\bm{\lambda}$ and $\bm{\eta}$. Figure 4 shows how the re-ID performance varies with different values of $\lambda$ and $\eta$, weight factors for BNM and MS losses. When $\lambda$ increases from 0.5 to 1.5, the re-ID performance has a slight increase. When $\lambda$ continually gets larger, we observe a slight decrease on re-ID performance. The weight factor $\eta$ also has the similar result, which further indicates that our method is robust and insensitive to different parameters. Finally, the proposed framework obtains the best result with $\xi=0.5$, $\lambda=1.5$, and $\eta=3$ on Duke-to-Market transfer.

Random cropping, flipping and random erasing (Zhong et al. 2020) are employed as the data augmentation. We adopt ResNet-50 (He et al. 2016) as the backbone of our model with the last classification layer removed, and initialize it by the ImageNet (Deng et al. 2009) pretrained model. DBSCAN (Ester et al. 1996) is used for clustering before each epoch. We choose the hyper-parameters (e.g., $\xi,\lambda,\eta$) according to the validation set of the Duke-to-Market adaptation and directly apply them to other three cross domain re-ID tasks. Specifically, the overall learning process can be trained by two stages: supervised learning for source model and domain adaptation stage. The main training process is shown in Algorithm 1.

Supervised learning for source model. We first pre-train three source models on the source dataset in a supervised manner. These three models has the same architecture: ResNet-50, and are initialized by using parameters pre-trained on the ImageNet. The networks parameters $\theta_{1},\theta_{2},$ and $\theta_{3}$ are optimized independently by minimizing loss. ADAM is employed as our optimizer. We also utilize a mini-batch size of 64 in 4 GPUs, and set an initial learning rate as 0.00035. Finally, 80 epochs are trained with the learning rate multiplied by 0.1 at 40 and 70 epochs.

Cross domain adaptation. Based on the pre-trained network parameters, we update the three networks by optimizing the overall loss with the selected hyper-parameters $\xi=0.5,\lambda=1.5,$ and $\eta=3$. In addition, we also follow the same data augmentation strategy. All the networks are trained for 40 epochs, using ADAM optimizer with learning rate of 0.00035, momentum of 0.0005 and batch size of 64. In each epoch, we use DBSCAN for clustering. It is note that only one model is kept during test.

Momentum update mechanism. The motivation behind SSKD is that we generate self-supervised labels for one student model with the prediction outputs from another network model, leading to transferring rich structured knowledge between different models. Invariance learning has a important effect in this training process by using the momentum update mechanism. To verify its effect, we use one network’s current-iteration predictions as pseudo labels and remove the nuclear-norm maximization of batch classification response matrix in the slowly evolving networks. Such experiments are denoted as SSKD (w/o $\theta_{\cdot,e}$). The large margin of performance declines (e.g., from 78.7% to 46.3% for mAP) can easily be observed in Table 4, which demonstrates the importance of adopting the momentum update mechanism.