Dual-Tuning: Joint Prototype Transfer and Structure Regularization for Compatible Feature Learning

Due to the increasing size of gallery sets and the heavy workload of re-extracting gallery features, feature compatible learning was proposed recently. Previous works [25, 12] aimed to learn a separate mapping model to translate features from a source model to a target model. However, an additional feature re-representation process is needed before the feature can be compared with the stored gallery features. Recently, Shen et al. [19] proposed BCT, a backward-compatible training framework with a regularization function by using the old classifier in training the new embedding model, without additional feature re-generation process. However, when the classifiers/loss functions of new and old models are different, BCT faces severe performance drops. Differently, our Dual-Tuning can explicitly align embedding features by transferring the prototype information, and implicitly constrain the feature intrinsic structure in a bi-directional manner. In this way, we can better achieve feature compatibility even the old embedding space is not ideal and has distribution discrepancy with the new space.

Domain adaptation [31, 27] aims to learn a model for the unlabeled target domain by leveraging knowledge from a labeled source domain. A group of methods uses various metrics to mitigate the distribution difference caused by domain gap, such as maximum mean discrepancy (MMD) [5] and its variants [32, 18]. Another typical line aims to design a feature space such that confusion between the source and target distributions in that space is maximal [6, 16, 42]. These methods can be used to align the (marginal) distribution of the new and old classes. The problem we focus on differs in that we aim to achieve feature-level compatibility between any feature pair of new and old models.

Knowledge distillation aims to guide the learning of a small network using the knowledge from a large network. Hinton et al. [11] proposed using the probabilistic outputs of a larger network as soft targets to supervise the smaller network, and KL Divergence loss is adopted. Besides, some variants were proposed to mimic the intermediate feature maps [33], attention maps [36], or second-order statistics (Gram matrix) [34] from the teacher network. However, knowledge distillation is not applicable in compatible learning. Since the existing teacher model (old model) is unsatisfactory, mimicking a “weak” teacher will affect new models to get better discrimination capability.

In this work, we focus on feature compatible learning, which requires the learned features of a new (updated) model to be directly compared with features of its old version. Typically, a network model can be split into two components, an embedding backbone $\phi(\cdot;\omega_{\phi})$ and a task head $h(\cdot;\omega_{h})$, where $\{\omega_{\phi},\omega_{h}\}$ are the network parameters. For simplicity, those two components are also denoted as $\phi(\cdot)$ and $h(\cdot)$. In visual retrieval, the classification network is widely used to obtain the embedding feature [15]. The embedding module $\phi:\mathcal{X}\rightarrow\mathcal{F}$ maps an input $x\in\mathcal{X}$ to an embedding space $\mathcal{F}$, and the head module $h:\mathcal{F}\rightarrow\mathcal{P}$ further projects the features to the classifier’s hypothesis space $\mathcal{P}$. Then the category probabilities can be obtained by $p=h(\phi(x;\omega_{\phi});\omega_{h})$, as shown in Fig. 2(a). In feature compatible learning, the old model can be represented as $\phi_{O}$ and $h_{O}$ trained on the old training set $\tau_{O}$. As model structure evolves or data increases, a new model with $\phi_{N}$ and $h_{N}$ will be generated based on the new training set $\tau_{N}$. In $\tau_{N}$, the overlapping classes with $\tau_{O}$ is denoted as $\tau_{N^{\prime}}$.

After model updating, some of the gallery features stored in the database are still derived from the old model, while the queries are extracted from the new model. The compatible feature learning makes it possible to conduct retrieval between the new query features and old gallery features. The expected compatible features derived from the new model should satisfy two criteria. First, it should be compatible with the old features. Second, compatible learning should not affect the new model optimization to get better performance from updated data or network designs.

The distance relationship in the compatible embedding space can be described as,

where $a$ denotes the query sample, and $p$, $n$ denote positive and negative galleries. $y$ is the class label, and $d(.)$ is the Euclidean distance. This formulation can be described as, no matter if the gallery features are from the old or new embedding space, the distance between the positive pair $(a,p)$ should be consistently smaller than the negative pair $(a,n)$.

Besides the feature-level optimization, we also design a component-level mutual structural regularization to further improve the compatibility, as shown in Fig. 2(c). Works[39, 15] in other fields demonstrate it is effective to use embedding and classification spaces simultaneously(e.g., triplet+softmax). Therefore, if the head is available, our mutual structural regularization can further provide knowledge in classifier to get more enhanced results.

Given a network, it can be split into two components, i.e., an embedding module for feature extraction and a head for target task. Such a head can be treated as the “rules”, which contains the intrinsic structure of embedding space. Take a classifier head as an example, classifier head formulates classification rule, and according to this rule, the embedding in feature space $\mathcal{F}$ can be mapped to category probabilities in classifier’s hypothesis space $\mathcal{P}$, $h_{\omega_{h}}:\mathcal{F}\rightarrow\mathcal{P}$.

Therefore, if features from two models can conduct mutual matching, the features derived from one model can also produce good predictions when passing through the other model’s classifier. This inspires us to design a component interoperation for mutual structural regularization. Specifically, the distribution structure of the new features should satisfy the “rules” of the old classifier, and correspondingly, the new classifier should also predict correctly based on the old features.

For a classification task, the widely used cross-entropy loss can be expressed as,

where $y_{i}$ is the one hot vector label of sample $x_{i}$. For training a single new or old model, the optimization targets are $\mathcal{L}_{\text{CE}}({\phi_{N}},{h_{N}};\tau_{N})$ and $\mathcal{L}_{\text{CE}}({\phi_{O}},{h_{O}};\tau_{O})$, correspondingly.

The component interoperation means that, for the new and old models, we can recombine the embedding module of one model and classifier head of the other model while still maintaining good prediction performance. Thus, the optimization targets of mutual structural regularization $\mathcal{L}_{\text{stru}}({\phi_{N}},{h_{N}};\tau_{N},\tau_{N^{\prime}})$ can be expressed as,

Different from BCT [19] which only uses the old classifier head to supervise the new embedding module, our component interoperation can provide dual supervision. The old “rules” will guide the training of the new embedding module. At the same time, the old features will also assist in formulating the embedding rules of the new classifier.

Finally, to simultaneously exploit the prototype knowledge in the embedding space and conduct mutual structural regularization by old components, the final optimization objective for Dual-Tuning is,

where $\mathcal{L}_{\text{CE}}$ is the supervision loss for the target task, which is not limited to cross-entropy loss and does not need to be the same form as the old model. $\mathcal{L}_{\text{proto}}$ and $\mathcal{L}_{\text{stru}}$ are illustrated in Eq. 5 and Eq. 7, respectively.

Dataset. To comprehensively evaluate the compatible learning approaches, besides two widely used large scale person ReID datasets Market1501 [38] and MSMT17 [28], we also conduct experiments on two million-scale classification datasets ImageNet [2] and Place365 [40] used in [19]. We evaluate the compatible retrieval performance on the validation or testing sets of these four datasets. For Market1501 and MSMT17, the standard person ReID testing process is performed [28]. For ImageNet and Place365, each image is considered as a query image, and all other images are considered as gallery images, the same setting as in [19].

Implementation details. For compatible model learning, we adopt two widely used architectures ResNet18 and ResNet50 [10] as our backbones, whose feature dimensions are 512 and 2048 respectively. For Market1501 and MSMT17, we employ the bag-of-tricks scheme [15] for ReID model training, and the input size is 256$\times$128. For ImageNet and Place365, we adopt the training scheme in [19], and the image input size is 224$\times$224. For the memory bank, the queue size is set to 4096. The supervision loss ($\mathcal{L}_{\text{CE}}$ in Eq. 8) for target task is cross-entropy loss. We optimize the model with Adam optimizer and use 1 GPU for network training. The model is trained for 120 epochs with a start learning rate of $3.5\times 10^{-4}$ and performs learning rate decay $1/10$ in the $40{th}$ and $70{th}$ epochs.

Experiment setting. We conduct “Cross-test” and “Self-test” experiments for compatible model evaluation:

• •

  Cross-test: In retrieval, the query and gallery features are extracted from the new and old model respectively, and the Euclidean distances are directly computed between the new and old features. If the dimensions of these two types features are different, zero padding will be used for dimension alignment [19].

• •

  Self-test: In retrieval, the query and gallery features are both extracted from the new (learned) model.

The mean Average Precision (mAP) and Top-K accuracy are used as evaluation metrics.

To emulate the practical case that a new model is generated when a new larger training set is available, half (50%) of the classes are used for training an old model, and all (100%) classes are used for training a new model. In our table presentation, the top part shows the performances of the independent models. The bottom part provides the performances of feature compatibility learning.

The results of Market1501, MSMT17, Place365, and ImageNet are shown in Table 2, 3, and 4. The “Res18 v.s. Res18” or “Res50 v.s. Res18” in setting (new v.s. old) column means the new model is with ResNet18 / ResNet50 backbone, and the old model is ResNet18. Several comparison methods are adopted, including a naive $\mathcal{L}2$ loss [35], knowledge distillation method KL Divergence [11], metric-based Asymmetric triplet loss [1], and compatible learning method BCT [19].

Independently trained new and old model. We first conduct a simple experiment to test the cross-test performance between two independently trained models, i.e., Independent (new) and Independent (old) in Table 2. The cross-test performance between these two independent models is almost 0% accuracy in most experiments, demonstrating its bad compatibility. In addition, the cross-test between Res18-50% and Res18-100% models on Market1501 is 13.55% mAP, since they use the same initialization parameters, which is helpful for feature compatibility.

A naive $\mathcal{L}2$ loss. Given a sample, the $\mathcal{L}2$ loss aims to minimize the Euclidean distance between its new and old features [35]. However, such a straightforward approach is too local and strict for new feature learning. It not only fails to achieve feature compatibility, but also interferes with the training of new models, as shown in Table 2 and 4.

Knowledge distillation - KL Divergence. We also compare with the classic KL Divergence [11] for knowledge distillation, which constrains the probabilistic outputs of the new model by those of the old model. However, the new model (student) should achieve better performance than old model (teacher). Therefore, using the “bad” teacher to guide the learning of a new model will affect obtaining a better embedding space. As shown in Table 2(a), the independently trained ResNet18 model obtains 80.43% mAP on Market1501, while supervised by KL Divergence, it only obtains 73.90% mAP on self-test.

Metric-based asymmetric triplet loss. For the asymmetric triplet, we adopt the triplet loss to optimize the distances between new and old features, and the triplet unit is $(\phi_{N}(a),\phi_{O}(p),\phi_{O}(n))$. Such asymmetric triplet is first used for knowledge transfer learning in [1]. On the MSMT17 dataset (Table 2(a)), the asymmetric triplet loss can only achieve 26.60% cross-test mAP. Such performance illustrates that when the scale of the dataset becomes larger, our prototype-based global optimization can achieve significant advantages over pair-based local optimization.

Backward-compatible learning - BCT. BCT [19] uses the old classifier head to supervise the new embedding module to achieve feature compatibility. BCT and Asymmetric triplet loss achieve comparable performances. For example, on the Market1501 datasets (Table 2(a)), the asymmetric triplet loss achieves better performance on self-test (79.67% v.s. 77.59% mAP), and slightly worse performance on cross-test, while on the Place365 and ImageNet datasets (Table 3, 4), opposite results are obtained.

By comparison, our Dual-Tuning achieves the best performances on all datasets. Impressively, the cross-test performance of our Dual-Tuning beats the old model by a remarkable margin, e.g., on Market1501 Res18 v.s. Res18 setting, 8.04% mAP improvement can be obtained (69.53% v.s. 61.49%). Such a performance gain indicates that the prototype loss is able to push the new query features close to the old feature centers of the same ID. Besides, on the Market1501, MSMT17, and Place365 datasets, the self-test performances even surpass the performance of independently trained new model, demonstrating that the old model information is also useful for new model learning.

In this section, we first perform the ablation study, then explore the feature compatibility under different supervision losses and model architectures. Without additional explanation, both the new and old models use ResNet18 backbone, and are supervised by softmax (cross-entropy) loss.

Ablation study. Our method contains two parts: a compatible prototype loss and a mutual structural regularization. In Table 5, only using the compatible prototype loss can get 80.72% mAP on the self-test, and even surpasses the independently trained new model 80.43% mAP (see Table 2). This can be attributed to that prototype loss can exploit the prototype-based manifold information in old embedding space, and meanwhile minimize the impact of noise and disordered distribution derived from the old space. Moreover, with additional mutual structural regularization (Mutual-Reg), Dual-Tuning can further boost the cross-test performance, and also obtain good performance on the self-test setting.

For the ablation study of compatible prototype loss, incorporating new prototypes for a mixed retrieval can achieve better representation than only using the old prototypes (Center-based Prototype Loss). Besides, for the mutual structural regularization, the O2N-Reg in Table 5 indicates using the old embedding features to train the new head (i.e., $\phi_{O}$ + $h_{N}$), and the N2O-Reg uses the old classifier to supervise the new embedding training (i.e., $\phi_{N}$ + $h_{O}$). The results demonstrate the mutual structural regularization can achieve better performance than only using N2O-Reg or O2N-Reg.

Performances on the mixed gallery. In the gallery, there exist the old features and continuously added new features. To simulate this scenario, a hybrid gallery retrieval experiment is performed. As shown in Fig. 4, we change the ratio of the old and new features in the gallery and report the results of different methods. The performances of asymmetric triplet loss and contrastive loss [1] extremely degrade in the mixed gallery settings. Under the setting of 80% old data and 20% new data in the gallery, asymmetric triplet can only achieve 50.96% mAP, which is much inferior to our 70.02% mAP. Such great performance advantages strongly demonstrate the robustness of our method.

Results of different supervision losses. Different supervision losses will generate different feature distributions. We test using softmax and triplet loss [8] for the old model training, and adopting softmax, softmax+triplet [8], and circle loss [21] for the new model. As shown in Table 6, BCT [19] works poorly when the old model is supervised by triplet loss without a classifier. Additionally, BCT suffers huge performance drops in circle v.s. softmax setting, since the circle loss has a different classification mechanism from softmax. Differently, our Dual-Tuning achieves impressive results, and beats it by a remarkable margin. The superior results indicate the designed compatible prototype loss can mitigate the affection by the distribution discrepancy caused by different supervision losses.

Results of different model architectures. We also conduct experiments that the new model with different architectures and feature dimensions from the old model. First, we test the new model with ResNet50 of 2048 feature dimension, and old model with ResNet18 of 512 feature dimension. As shown in Table 2, 3 and 4, during retrieval with old gallery features, using new ResNet50 feature as query can achieve better performance than the old ResNet18 feature.

More results of different architectures are shown in Table 7. We test 5 different architectures, including the extremely lightweight networks Osnet384 and Osnet512 [41], the latest split-attention network ResNeSt50 [37], and the widely used networks ResNet18 and ResNet50. The feature dimensions for Osnet384, Osnet512, and ResNeSt50 are 384, 512, and 2048, respectively. We use zero padding [19] to align features with different dimensions. Under different architectures, the cross-test between new and old models can also achieve impressive performances.

Towards multi-model and sequential compatibility. We simulate the scene with three model versions on Market1501 dataset. These three models are sequentially updated using 25%, 50%, and 100% training data, respectively. The version 2 model $\phi_{2}$ is compatible to version 1 $\phi_{1}$. The version 3 model $\phi_{3}$ only conducts supervision from $\phi_{2}$ to achieve compatibility. The sequential compatible results are shown in Table 8. We can find that, $\phi_{3}$ can achieve feature compatibility with $\phi_{1}$, even $\phi_{1}$ is not directly involved in training $\phi_{3}$. Compared with BCT, we can better solve the sequential updated models, e.g., 60.51% v.s. 53.64% mAP on the cross-test between $\phi_{3}$ and $\phi_{1}$.

Center-based asymmetric loss. After obtaining the old centers, we can also conduct an asymmetric center loss, whose triplet unit is $(\phi_{N}(x^{c}),m_{o}^{c},m_{o}^{c}{}^{\prime})$. The positive representation is the old prototype $m_{o}^{c}$ with the same class as $x^{c}$, and the negative is an old prototype of other classes $m_{o}^{c}{}^{\prime}$. As shown in Table 9, the center-based asymmetric loss can achieve better performance than triplet loss, while worse than our compatible prototype loss. The results further demonstrate the necessity of using the whole structure information in embedding space.