Handling Label Uncertainty for Camera Incremental Person Re-Identification

Consider a CIPR problem with several steps, and each incremental step introduces a new camera with a set of classes to learn. Formally, in the $t$-th step, we have the training data $\mathcal{D}_{t}=\{\bm{\mathit{X_{t},Y_{t}}}\}=\{(x^{t}_{i},y^{t}_{i})|^{N_{t}}_{i=1}\}$ with intra-camera annotations $y^{t}_{i}\in\bm{Y_{t}}$ captured by the newly installed camera $c_{t}$, and $N_{t}$ is the number of classes in $c_{t}$. We note that the training data $\mathcal{D}_{t}$ can contain overlapping classes in $\mathcal{D}_{t-1}$, while the old training data $\mathcal{D}_{t-1}$ are not available due to the privacy concern. Hence, we first need to identify the real number of extensions to correctly learn new classes. The goal of CIPR is to learn a robust ReID model that can be generalized to unseen classes from all encountered cameras.

We first present a straightforward baseline for CIPR task. Basically, in the $t$-th step ($t>1$), the feature extractor $F(\theta_{t})$ initialized by $F(\theta_{t-1})$ is updated to learn a set of classes employing $\mathcal{D}_{t}$, and the classifier $G(\phi_{t})$ is also extended to the corresponding new dimension (Hou et al., 2019), which is expected to predict all the classes seen so far. As a common incremental learning baseline, in addition to ReID loss (He et al., 2020) (e.g. ID loss $\mathcal{L}_{\text{ID}}$+ triplet loss $\mathcal{L}_{\text{Triplet}}$ (Hermans et al., 2017)), knowledge distillation (KD) loss $\mathcal{L}_{\text{KD}}$ is employed to prevent catastrophic forgetting, which can be formulated as:

where $KL(\cdot)$ is the Kullback Leibler (KL) divergence, $p^{o}_{i}$ and $p^{n}_{i}$ denote the logit output of the old and new models, respectively.

To discriminate the seen and unseen identities without accessing the old data, a straightforward method is to leverage the softmax prediction score. We assume that samples belonging to unseen classes will produce smooth probability distributions since they are equally wrong and ambiguous. Therefore, we can treat an image as the seen class if the maximum softmax score is above a threshold $T$. For samples identified as a new class, we add a new ID based on the existing old classes. For samples classified into old classes, we use the model predict as its pseudo label. Then we can minimize the cross entropy with the global pseudo labels. The loss function can be formulated as:

where $\bm{Y^{\prime}_{t}}$ is the pseudo label of samples $\bm{X_{t}}$, $\mathcal{L}_{CE}$ is the cross-entropy loss function.

Overall, the optimization objective of the baseline CIPR model can be formulated as:

The graphical illustration of our framework is depicted in Fig. 2. We replace the linear classifier with the non-parametric memory bank to alleviate the over-confidence issue  (Bendale and Boult, 2016), which stores the moving average of the cluster prototypes. The old model is fixed, and the new model is updataed via Exponential Moving Average (EMA) scheme during the optimization. Then we elaborate our ExtendOVA in three parts to tackle CIPR problem. The first technical novelty comes from taking advantage of the One-vs-All (OVA) detector for instance-wise seen class identification before training (section 4.2). Then the samples are assigned with global pseudo labels via our criteria and an early learning regularization term $\mathcal{L}_{\text{Aux}}$, as to be detailed in section 4.3. Finally, in section 4.4, surrogate features are sampled based on the memory bank to guide the cross-camera distillation objective.

In this section, we elaborate on the process of instance-wise seen class identification. We first describe the training of the One-vs-All detector before describing the remaining methods.

Although the OVA detector is effective and more robust for instance-level prediction, it may still introduce noisy labels, especially for hard samples. As a result, two images of the same class may be paradoxically predicted as a new class and an old one, which can affect the class-level prediction. Furthermore, we argue that due to the domain gap, the latent space trained for each class could not effectively represent the data from future cameras, as illustrated in Fig. 2. To this end, we propose an ID-wise pseudo label generation module (IPLG) to correct noisy labels and associate the samples with the same local label to the same pseudo-global label. We shall detail the operation of this module below.

where $W_{k}\in\mathcal{R}^{d}$ stands for the $k$-th column of ID-wise prototype in memory bank $\bm{W}\in\mathcal{R}^{d\times N_{t-1}}$, $d$ is the feature dimension. Note that we only choose the most frequently predicted one if there are multiple $k$ for one class. Correspondingly, we expand the memory bank to $\bm{W}\in\mathcal{R}^{d\times N_{t}^{\prime}}$, $N_{t}^{\prime}=N_{t-1}+|C_{uc}|$.

the key is to keep the old prototypes fixed throughout the process, serving as a boilerplate for learning domain-invariant features.

Additionally, the hope is that samples belonging to unseen classes can also output the second largest probability to the potential nearest prototype, which is constrained by the following auxiliary loss:

Here, $\tilde{y}_{i}\in[1,N_{t-1}]$ is obtained using the same way as in Eq.5. The motivation behind this approach lies in the consensus  (Ren et al., 2018; Grandvalet and Bengio, 2004) that seen classes are usually clustered to form high-density regions in the latent space. Hence, this regularizer encourages these samples to be closed to a shared and real prototype. On the other hand, the unseen classes are often distributed in low-density regions, leading to optimization conflicts where the samples struggle to simultaneously approach the current prototype and the nearest old prototype.

After the early-stage regularization, we would employ the selection criterion again to obtain refined pseudo labels, which would continue to be used for model updates. Meanwhile, those new prototypes that were previously created by falsely identifying as unseen classes would be removed from the memory bank.

In incremental learning, models are susceptible to forgetting previous learned knowledge without relearning the old data. To address this, previous work (Lu et al., 2022) has proposed to use a GAN (Goodfellow et al., 2020) to reconstruct old data in the image space. In our method, we generate substitute samples in the feature space instead. Concretely, to estimate the distribution of the old data, we assume a class-conditioned multivariate Gaussian distribution denoted as $F(x_{i}^{t-1}|y_{i}^{t-1}=k)\sim\mathcal{N}_{k}^{t-1}(\mu_{k}^{t-1},{\textstyle\sum_{k}^{t-1}})$. Here, $\mu_{k}^{t-1}$ is the mean of the Gaussian distribution and can be approximated using our prototypical memory bank. To estimate the covariance matrices, we utilize the statistics of BatchNorm (BN) layers. During training, a BN layer normalizes the features, which implicitly captures the means and variances of the data (Yin et al., 2020), thereby enabling the estimation of the covariance matrices of the old data. Overall, we estimate the distribution of the data in previous step by:

Then we can sample surrogate features $\tilde{f_{k}}\sim\tilde{\mathcal{N}}_{k}(W_{k},\text{BN}(var))$. Based on these surrogate features, we present a cross-camera distillation loss that serves to regularize forgetting, by ensuring that the cosine distance is maintained across different cameras. Formally, given a batch of samples $\bm{\mathcal{X}}=\{(x^{t}_{i},\hat{y}_{i})|^{B}_{i=1}\}$ along with a batch of sampled surrogate features $\bm{\tilde{F}}=\{\tilde{f_{k_{1}}},\tilde{f_{k_{2}}}...\tilde{f_{k_{B}}}\}$, the loss can be calculated by:

where $\cos(a,b)=\frac{a\cdot b}{\left\|a\right\|_{2}\left\|b\right\|_{2}}$ denotes the cosine similarity . The distillation loss $\mathcal{L}_{\text{CD}}$ improves stability that is commensurate with the ability of previous data to maintain past structure.

In summary, the overall objective function for our ExtendOVA framework is formulated as

where $\lambda_{1}$ and $\lambda_{2}$ are coefficients. To enhance the model’s stability during optimization, we utilize the Exponential Moving Average (EMA) technique (Tarvainen and Valpola, 2017), wherein the student model’s parameters are initially shared by the teacher model $F(\theta_{t})$. Once this iteration is complete, the student model is updated using the EMA parameters computed from the teacher model’s parameters by

where $\alpha$ is a smoothing factor typically set to a value 0.99 (Xu et al., 2021). In the test phase, we will use the student model to extract feature representations. By incorporating EMA into the training process, the updates to the student model’s parameters are smoothed, leading to improved stability and better generalization performance.

We use the widely adopted ResNet-50 (He et al., 2016) as the backbone network. To obtain 2048-dimensional features, a Batch Normalization (BN) layer  (Ioffe and Szegedy, 2015) is placed after the last layer of the network. The batch size is set to 64, comprising of 16 identities with 4 images per identity. The Adam optimizer with a learning rate of $3.5\times 10^{-4}$ is used for optimization at the initial step and the learning rate of the backbone is set to $lr/10$ during the incremental learning. The model is trained for 40 epochs per step, and the early-stage learning regularization is performed during the first ten epochs. The hyper-parameter $T$, $\lambda_{1}$, and $\lambda_{2}$ are set to $0.5,0.9$ and $0.6$ (see hyper-parameter analysis in the supplemental material).

For comparative experiments, we reproduce the state-of-the-art methods, i.e., data-free methods LwF (Li and Hoiem, 2017), AKA (Pu et al., 2021), AGD (Lu et al., 2022), PatchKD (Sun and Mu, 2022) , the replay-based methods iCaRL (Rebuffi et al., 2017), PTKP (Ge et al., 2022), and a distribution alignment methods MMD (Long et al., 2015) on our setting. It is noteworthy that these methods are based on a class-disjoint setting, and they do not match our setting. Therefore, to implement them in our setting, they can only treat old classes as new ones. For more extensive assessment, we design some other comparative methods, including the baseline described in section 3.2, the fine-tune method that fine-tunes the model on new data, the Joint-T that denotes an upper-bound by training the model on all data seen so far.

We compare our ExtendOVA with the current state-of-the-art. The evaluation is conducted on all cameras encountered so far and the final-step results are reported in Tabel 3 with both the general and exceptional setups. We summarize the results as follows:

• •

  Our proposed ExtendOVA outperforms the current state-of-the-arts by a clear margin, and is the closest to the upper bound Joint-T. Notably, it even achieves comparable or better results than the replay-based methods, validating the effectiveness of our proposed solutions.

• •

  Previous methods, designing for the non-overlapping setting, still achieve poor performance. We attribute this poor performance to two aspects: Firstly, in the absence of cross-camera labels, these methods fail in learning cross-view representations. Secondly, turning old classes into new ones results in a false-positive prediction of the spurious classes, leading to accumulated errors.

• •

  Interestingly, our proposed baseline outperforms the data-free methods LwF, AKA and AGD, demonstrating the potential improvement in addressing the class-overlap issue.

To further observe the impact of early regularization learning on the ID-wise predictive performance, Fig. 5 shows the training curves of ID loss and model accuracy of seen classes during the training process. We can observe that during the training process, both models show a decreasing trend in loss and eventually converge. In the initial iterations of training, the accuracy of both models shows an increasing trend, indicating that the models have not yet started fitting to the noise. After a certain number of iterations, the accuracy of the model without early regularization starts to gradually decrease, while our method corrects the noise in the early stage, resulting in an increase in accuracy.

Effectiveness of the different components. We conduct ablation studies in the three-step exceptional setup to evaluate the effectiveness of each module in each step. To evaluate the effectiveness of our ID-wise pseudo label generation module, we conduct experiments where we disable the $\mathcal{L}_{\text{Aux}}$ components and also compare the results to a baseline method, i.e. LwF, which is trained using ReID loss supervised by intra-camera labels. First, as shown in Table  2, removing $\mathcal{L}_{\text{Aux}}$ will decrease the final performance by $0.9\%$ to $3.7\%$ in mAP. Second, the combination of $\mathcal{L}_{\text{Aux}}$ and $\mathcal{L}_{\text{ID}^{*}}$ bring the gain in the range of $3.3\%$ to $8.2\%$ in mAP compared with the baseline. This suggests that simply optimizing the cross-entropy loss with intra-camera supervision is not sufficient.

To evaluate the contribution of the EMA scheme, we conduct experiments by removing it. Without the EMA technique, the performance drop ranges between $2.1\%$ and $4.3\%$ in the second step in terms of mAP, and the degradation becomes more significant as the incremental training phase proceeds. This clearly indicates that incorporating this design is crucial for overall performance. While the effect of $\mathcal{L}_{\text{CD}}$ is not as pronounced as the EMA scheme, it still has an obvious impact on the performance. When both terms are eliminated, there is a significant decline in performance, suggesting that the $\mathcal{L}_{\text{CD}}$ plays a role in maintaining the overall performance.

Compared with different distillation loss. Table 3 compares the performance of our method with different distillation losses in the general setup. Specifically, we evaluate the impact of knowledge distillation (KD) and cross-camera distillation (CD) loss on the performance of the baseline model trained with ID loss ($\mathcal{L}_{\text{ID}^{*}}$) and the auxiliary losses ($\mathcal{L}_{\text{Aux}}$). From the table, we observe that incorporating distillation losses, either $\mathcal{L}_{KD}$ or $\mathcal{L}_{CD}$, improves the performance of the baseline model in terms of mAP and Rank-1 accuracy. Notably, adding $\mathcal{L}_{CD}$ achieves higher performance than adding $\mathcal{L}_{KD}$, indicating that $\mathcal{L}_{CD}$ is more effective in preserving the structure-wise knowledge.

We evaluate the anti-forgetting properties of our proposed method by measuring the performance on the test-set of the first step after each step. Fig. 6 plots the forgetting trend on DukeMTMC and MSMT17 in the general setup. We found that our method showed superior anti-forgetting properties, with no performance degradation and even a slight improvement on the previous tasks. Our baseline model exhibits less forgetting when compared to data-free methods, clearly indicating that class-overlap is an issue to be addressed. AGD employs the DeepInversion (Yin et al., 2020) to generate synthetic exemplars from previously learned classes, however, it still suffers from catastrophic forgetting. This is due to its reliance on a unified classifier, treating overlapping classes as new ones can result in the generation of a significant amount of noise.