Covariance-based Space Regularization for Few-shot Class Incremental Learning

Class Incremental Learning (CIL) aims to learn new classes from a sequence of classification tasks without access to previously encountered data. The primary objective of CIL is to effectively learn new classes while minimizing the forgetting of old classes. Recent research in CIL can be broadly categorized into three approaches. The first and most straightforward method is to retain old data or knowledge during the learning process. Recent works [26, 12, 27, 52, 50] propose to mitigate the forgetting issue by rehearsing and generating previously retrained class data. Another common approach involves identifying key model parameters associated with previously learned classes and dynamically updating only the remaining parameters during incremental sessions [14, 41, 45, 20]. The third category focuses on addressing the bias inherent in CIL methods, which tend to favor the most recently learned classes [40, 11, 2].

Few-Shot Learning (FSL) aims to develop a classification model with very limited data. To generalize on few-shot classes, metric-based methods focus on learning a similarity metric that can effectively distinguish between classes with minimal examples [37, 28, 34, 46]. Hallucination-based approaches utilize data augmentation techniques, such as geometric transformations, style transfer and statistical augmentations, to increase the amount of training data [39, 43, 3, 8].

Few-Shot Class Incremental Learning (FSCIL) aims to learn new classes with limited incoming data in an incremental manner [32, 47, 51, 49, 44]. TOPIC [32] first introduces this setting and employs a neural gas algorithm to preserve the topology in the embedding space. To address the challenge of limited data in incremental sessions, prototype learning [28, 42, 19] has been widely adopted in FSCIL to enhance the model’s generalization to new or unseen data. To mitigate the catastrophic forgetting of old classes, recent studies [47, 10, 44] propose freezing the feature extraction backbone after training the base sessions and computing new class prototypes during incremental sessions. To learn more representative prototypes, Zhu et al. [53] introduces a self-promoted prototype refinement mechanism to develop extensible feature representations in the base session. LDC [21] utilizes a recurrent calibration module to learn new prototypes from sampled data, though this can be inefficient due to its recurrent nature. Unlike previous methods, our approach boosts FSCIL by regularizing the feature space for the few-shot new classes.

Our framework follows the two-stage learning procedure in most of recent works [18, 10, 44], which begins with pretraining the model using the base classes. The pretraining stage is called the base session, where we integrate the proposed covariance constraint loss (CCL) to constrain the learned representations of base classes, as shown in Fig. 2(a). During the incremental sessions, we propose a semantic perturbation learning (SPL) approach, aiming to enlarge the separation between old and new classes by learning the perturbed new data, which is shown in Fig. 2(b).

Recent wisdom [55, 5, 29] has demonstrated a good pretrained model benefits the learning of incoming few-shot new classes. The common attempt in recent works is to learn very compact base class representations by pushing data to the learned prototypes via fantasizing new classes or metric-based classification losses [18, 49, 29]. However, simply pushing the data close to the class centroid does not explicitly constrain the span of each class distribution, which may lead to representation confusion when learning new classes (Fig. 1(a)). Hence, we apply a covariance constraint to each base class during training, ensuring distinct means but identical covariance across classes, as shown in Fig. 2(a).

In order to constrain the covariance of each distribution, it is vital to estimate base class distributions $p(x,y)$. However, it is generally computational intractable. To solve this issue, previous works [16, 15, 6] adopt the variational inference, which involves a parametric posterior function $q_{\phi}({z}|x,y)$ to approximate the true posterior by maximizing the evidence lower bound (ELBO):

		$\displaystyle\log p_{\theta}(x,y)$		(4)
		$\displaystyle=\log p_{\theta}(y\mid x)+\log p_{\theta}(z)+\log\frac{p_{\theta}(x\mid z)}{p_{\theta}(z\mid x)}$
		$\displaystyle=\int q_{\phi}(z\mid x,y)\{\log p_{\theta}(y\mid x)-\log\frac{q_{\phi}({z}\mid{x,y})}{p_{\theta}(z)}+$
		$\displaystyle\log\frac{q_{\phi}(z\mid x,y)p_{\theta}(x\mid z)}{p_{\theta}(z\mid x)}\}dz$
		$\displaystyle\geq\mathbb{E}_{q_{\boldsymbol{\phi}}\left({z}\mid{x,y}\right)}\left[\log p_{\theta}\left({y}\mid{x}\right)\right]-D_{KL}\left[q_{\boldsymbol{\phi}}\left({z}\mid{x,y}\right)\|p_{\theta}({z})\right],$

where $\phi$ and $\theta$ are modeled by neural networks. The first term in Eq. 4 aims to maximize the likelihood by improving the confidence of the prediction. When optimizing the first term, the optimizing process can be written as

		$\displaystyle\underset{q_{\phi}(z\mid x,y)}{\arg\max}\int_{x}\sum_{y}p(x,y)\int_{z}q_{\phi}(z\mid x,y)\log p_{\theta}(y\mid x)$		(5)
	$\displaystyle=$	$\displaystyle\underset{q_{\phi}(z\mid x,y)}{\arg\max}\int_{x}p(x)\int_{z}q_{\phi}(z\mid x,y)\sum_{y}p(y\mid x)\log p_{\theta}(y\mid x).$

As $q_{\phi}({z}\mid{x,y})$ is a probability distribution, the integral is upper bounded by $\max\sum_{y}p(y\mid x)\log p_{\theta}(y\mid x)$, which is equivalent to minimizing the cross entropy loss in Eq. 1. The second KL divergence minimizes the divergence between the variational posterior distribution $q_{\boldsymbol{\phi}({z\mid x})}$ and the prior distribution $p_{\theta}{(z)}$. The third term actually can be calculated by $D_{KL}\left[q_{\boldsymbol{\phi}}\left({z}\mid{x,y}\right)\|p_{\theta}({z\mid x})/p_{\theta}({x\mid z})\right]$, which is a non-negative value and we eliminate this item. In order to explicitly contain the covariance of all distributions, the remaining problem lies in how to formulate the posterior distribution $q_{\boldsymbol{\phi}({z\mid x})}$ and the prior distribution $p_{\theta}(z)$.

We first assume that $z$ satisfies multivariate Gaussian with a mean and diagonal covariance. The $p_{\theta}(z)$ can be formulated as $\mathcal{N}(\mu_{c},\mathbf{I})$ with mean $\mathbf{\mu_{c}}$ as the class center and fixed eye matrix $\mathbf{I}$, respectively. As we aim to constrain the covariance, we adopt the fixed covariance as the prior. In order to learn the latent distribution quickly from a single instance during training, we adopt linear layers $P_{\mu}(\cdot)$ and $P_{\sigma}(\cdot)$ after the feature extraction network to form a statics prediction pipeline to estimate $q_{\phi}(z|x,y)$ following previous works [16]

	$\displaystyle q_{\phi}(z|x,y)$	$\displaystyle=\mathcal{N}(z|\hat{\mu_{c}},\hat{\sigma_{c}}^{2})$		(6)
		$\displaystyle=[P_{\mu}(x),P_{\sigma}(x)]_{y=c}.$

After obtaining the estimated statistical information, we can then formulate the second KL divergence term into the covariance constraint loss by

		$\displaystyle-D_{KL}\left[q_{\boldsymbol{\phi}}\left({z}\mid{x,y}\right)\|p_{\theta}({z})\right]$		(7)
		$\displaystyle=\int q_{\phi}({z}|{x,y})\log\frac{p_{\theta}({z})}{q_{\phi}({z}|{x,y})}d{z}$
		$\displaystyle=\int\mathcal{N}\left({z};\hat{\mu},\hat{\sigma}^{2}\right)\log\frac{\mathcal{N}({z};{\mu_{c}},\mathbf{I})}{\mathcal{N}({z};\hat{\mu_{c}},\hat{\sigma_{c}}^{2})}d{z}$
		$\displaystyle=\frac{1}{2}\sum_{c=1}^{\mathcal{C}^{0}}\sum_{i=1}^{d}\left(1+\log\hat{\sigma}_{c,i}^{2}-\hat{\sigma}_{c,i}^{2}-(\hat{\mu_{c}}-\mu_{c})^{2}\right)$
		$\displaystyle\leq\frac{1}{2}\sum_{c=1}^{\mathcal{C}^{0}}\sum_{i=1}^{d}\left(1+\log\hat{\sigma}_{c,i}^{2}-\hat{\sigma}_{c,i}^{2}\right),$

where $d$ represents the feature dimension. We drop the final term to formulate the covariance constraint loss as $\mathcal{L}_{ccl}=\frac{1}{2}\sum_{c=1}^{\mathcal{C}^{0}}\sum_{i=1}^{d}\left(1+\log\hat{\sigma}_{c,i}^{2}-\hat{\sigma}_{c,i}^{2}\right)$, which constrains the covariance of each feature distribution. Eq. 7 shows that the covariance constraint loss can be viewed as the upper bound of $-D_{KL}$. Thus, by deploying covariance constraint loss, we are able to obtain a tighter upper bound than Eq. 4, which can be easier to optimize. The overall learning objective for the base session training is

$$\mathcal{L}_{Base}=\mathcal{L}_{ce}+\gamma\mathcal{L}_{ccl},$$

(8)

where $\gamma$ is a positive hyperparameter that controls the degree of constraint.

In order to prevent the model from overfitting the few-shot new classes during incremental sessions, recent approaches [18, 10, 29] attempt to freeze the feature extractor and learn the new class prototypes by averaging the few-shot samples. Though the overfitting problem is mitigated by fixing the previously learned feature space, the new class distributions can only account for small parts of the feature space compared to base classes due to the scarcity of data. Under this ill-divided space dilemma, as shown in Fig. 1(b), new class prototypes may lie very close to base class distributions or new class prototypes, which may further lead the model to misclassify new classes.

In this paper, we propose to address this issue from two perspectives. Firstly, we intend to expand the new class feature distributions by assigning the fixed covariance, which is the same as the base distributions and allows the new class distributions to take up more feature space. Secondly, to facilitate better separation among classes, we push the few-shot samples to be away from semantically similar distributions and then retrain the classifier to distinguish the perturbed new class samples from other classes. We achieve these two goals by forming the semantic perturbation learning framework, where we aim to learn a perturbation distribution from which any perturbations can perturb new data to be away from those close distributions but within a fixed range. The perturbation distribution can also be learned by formulating a similar variational inference in Eq. 4 but with a different prior distribution. Specifically, we form the prior distribution as the Gaussian distribution $p_{\theta}({\tilde{z}})$ as $\mathcal{N}(\tilde{\mu},\mathbf{I})$, where $\tilde{\mu}$ is the linear combination of other class prototypes. The weight is calculated by the similarity score over classes. For each incremental session, we first initialize the classifier by computing the new class prototypes by averaging the few-shot samples. Given new data sample $\boldsymbol{x}_{i}^{t}$ with label $\boldsymbol{y}_{i}^{t}$, we compute the similarity score over other classes:

$$S_{i,c}=\frac{\mathbb{I}_{c\neq{y}_{i}^{t}}\exp\left(cos({\boldsymbol{w}_{c}},{x}_{i}^{t})\right)}{\sum_{c=1}^{|\mathcal{C}^{t}|-1}\exp\left(cos({\boldsymbol{w}_{c}},{x}_{i}^{t})\right)},$$

(9)

where $S\in\mathbb{R}^{K\times(|\mathcal{C}^{t}|-1)}$ and $\mathbb{I}(\cdot)$ is an indicator function. For each sample, we can get the prior mean by multiplying the confidence score with other class prototypes

$$\tilde{\mu}=\sum_{c\neq{y}_{i}^{t}}^{|\mathcal{C}^{t}|-1}S_{i,c}\boldsymbol{w}_{c}.$$

(10)

We also adopt a linear layer to predict the mean and covariance of each sample, the same as Eq. 6. To be noted, we do not reuse the linear layer in the base session so that this method can be directly applied to other methods in a plug-and-play manner. After predicting the statistics, the overall objective during the incremental training is:

	$\displaystyle\mathcal{L}_{incre}$	$\displaystyle=\mathcal{L}_{ce}({x}_{i}^{t},y_{i}^{t})+\mathcal{L}_{ce}(\hat{\mu}+\hat{\sigma}\odot{x}_{i}^{t},y_{i}^{t})$		(11)
		$\displaystyle-\alpha D_{KL}\left[q_{\boldsymbol{\phi}}\left({\tilde{z}}\mid{x}_{i}^{t}\right)\|p_{\theta}(\tilde{z}\mid{x}_{i}^{t})\right],$

where $\odot$ represents the element-wise multiplication. The first and second term is the classification loss for the few-shot data and the perturbed few-shot data. The third term aims to learn the perturbation distribution by minimizing the KL divergence between the predicted feature distribution of each sample and the prior distribution. The positive hyperparameter $\alpha$ controls the strength of the perturbation. The KL term is formulated as:

		$\displaystyle-D_{KL}\left[q_{\boldsymbol{\phi}}\left({\tilde{z}}\mid{x}_{i}^{t}\right)\|p_{\theta}(\tilde{z}\mid{x}_{i}^{t})\right]$		(12)
		$\displaystyle=\frac{1}{2}\sum_{i=1}^{d}\left(1+\log\hat{\sigma}_{i}^{2}-\hat{\sigma}_{i}^{2}-(\hat{\mu_{i}}-\tilde{\mu_{i}})^{2}\right).$

We conduct our experiments on three FSCIL benchmarks, i.e., CIFAR100, MiniImageNet, and CUB200, and compare our approach with the baseline models and other recent FSCIL methods. We also compare our method with classical CIL methods such as ICARL [26], NCM [11], and FSCIL methods CEC [18], C-FSCIL [10] and meta-learning based approaches [4, 51]. We report the numerical results of MiniImageNet in Tab. 1, and more results on CIFAR100 (Tab. A4) and CUB200 (Tab. A5) can be founded in Appendix C.

For baseline models, adding our proposed loss and semantic perturbation learning boosts the model performance for incremental sessions. After 8-session incremental learning, our method can improve the final model performance by at least 0.53% on MiniImageNet and 0.56% on CIFAR100, respectively. It is worth noticed that our method has a more profound effect on SAVC, which learns representations by pushing the data to the class center using contrastive learning. As SAVC does not consider the span of each class distribution explicitly, constraining the class distribution using our proposed methods boosts the base class performance and obtains better performance when learning new classes.

We further demonstrate the comparisons of harmonic performances of all classes after incremental sessions and comparisons of new class performance in each incremental session in  Fig. 3. As shown in Fig. 3(a), compared with baseline models, our approach boosts the harmonic performance by at least 9.80%. The CE model enjoys an improvement of 11.45% from our approach. The significant boost in harmonic performance results from the improvement of new classes. As illustrated in Fig. 3(b), the performance of new classes demonstrates a substantial improvement at each incremental session. Overall, our approach boosts the new class performance by at least 5.65%. Benefiting from the covariance constraint and distribution expansion strategy, we are able to learn more separable representations of new classes, boosting the performance of new classes.

We conduct the ablation experiments of the proposed CCL and SPL on the CIFAR100 dataset, as shown in Tab. 2. Following previous works [49, 29, 38], we verify the effectiveness of our approach by six metrics, i.e., the performance of the base model, the performance on the base classes after incremental sessions, the performance of all new classes, the average performance of all incremental sessions, the performance drop of the base classes ($PD=Accuracy_{base}-Accuracy_{old}$), and the harmonic accuracy of base and new classes after the incremental learning. When we do not deploy the SPL for incremental sessions, the model adopts the prototype-based incremental learning approach mentioned in Sec. 3.1.2.

As shown in Tab. 2, by deploying our proposed covariance constraint loss $L_{ccl}$, our model is able to achieve higher base class performance compared to the baseline model. By constraining the base class distributions, the drop in PD rate and the gain of new class performance on three baseline models validate that the covariance constraint loss allows the model to learn the new classes better while obtaining less forgetting of old classes.

We also conduct experiments of SPL on both baseline models and $\mathcal{L}_{ccl}$ trained models. The results demonstrate that directly deploying SPL can boost the performance of new classes and harmonic accuracy by at least 0.25% and 0.27%, respectively. It is worth noticing that simply using SPL on the baseline models leads to a higher PD. As we expand the distribution of new classes during incremental sessions without constraining the base class distributions, it may lead the model to confuse the old classes with new classes. If SPL is deployed on the $\mathcal{L}_{ccl}$ trained model, the model is able to achieve the highest performance on new classes and the highest harmonic accuracy, validating the effectiveness of the combination of proposed components. We also conduct experiments on comparing SPL with other incremental update methods in Appendix B.

We compare the different approaches by visualizing the learned feature embedding on the CIFAR100 dataset. In Fig. 4(a), we compare the base model trained with and without covariance constraint loss. We visualize the feature embedding of 8 different base classes, demonstrating that by adding the constraint on the covariance of the feature distribution, the learned feature distribution becomes more separable and compact, leaving more feature space for the incoming new classes. In Fig. 4(b), we demonstrate the effectiveness of SPL by showing the learned new class features before SPL, and new class features along with all perturbed data after SPL during incremental sessions $(t>0)$. As shown in the left side of Fig. 4(b), the new class features (colored triangles) tend to lie close with each other and mix with base classes due to the scarcity of the training data. By deploying SPL during the incremental sessions, our approach is able to push the few-shot training samples away from base classes and facilitate better separation among classes. The generated perturbed data in each epoch (denoted by “$\times$” in the figure) expands the original small new class feature distribution, which allows the new class distributions to take more feature space, benefiting the new class prediction.