Few-shot continual learning has gained increasing attention in recent years, with the goal of enabling a model to learn continuously from a sequence of few-shot labeled data. Several approaches have been proposed to address the challenges in this task, including forgetting, over-fitting, and data imbalance. For example, Tao et al. (Tao et al. 2020a) proposed to utilize a neural gas network to learn and maintain the topological structure of different category features for few-shot continual learning. A bilateral distillation structure (Zhao et al. 2023) was proposed to alleviate catastrophic forgetting of knowledge by considering the over-fitting issue. Inspired by the phenomenon of neural collapse, Yang et al.(Yang et al. 2023) addressed the misalignment dilemma in few-shot continual learning. Also, Zhuang et al. (Zhuang et al. 2023) adopted a Gaussian kernel embedded analytic learning method to balance the preference between old and new tasks. Zhang et al. (Zhang et al. 2021) introduced a graph attention network based evolving classifier to facilitate information propagation between classifiers. FACT (Zhou et al. 2022) generates virtual classes and preserves knowledge of old classes to address the catastrophic forgetting problem. A bi-level optimization method (Chi et al. 2022) was proposed to reduce forgetting of learned knowledge and adapt to new classes based on meta-learning. Self-supervised stochastic classifiers were also proposed in (Kalla and Biswas 2022) to address few-shot class incremental learning using a self-supervision mechanism. The semantic-aware knowledge distillation method (Cheraghian et al. 2021) treats the word embedding as additional information and incorporates knowledge distillation terms to mitigate the issue of forgetting. Michael et al. (Hersche et al. 2022) proposed the constrained few-shot class-incremental learning method, which utilizes hyper-dimensional embedding to enable the continual learning of more classes beyond the fixed number of dimensions in the feature space, for the purpose of balancing the accuracy and the computational-memory cost of learning novel classes. Further, Zhu et al. (Zhu et al. 2021) proposed an incremental prototype learning scheme to achieve continual learning by fine-tuning class prototypes using known prototypes and few-shot samples from new classes to train the model, and updating the prototypes based on this.

Inspired by the achievements of big-models of the world such as (CLIP) (Radford et al. 2021a), SAM (Kirillov et al. 2023), GPT-3 (Dale 2021) and Dino (Liu et al. 2023), a series of research have involved the pre-training of vision-language models on large-scale image-text datasets (Radford et al. 2021a; Jia et al. 2021). Among these approaches, the contrastive language-vision pre-training (CLIP) (Radford et al. 2021a) demonstrates impressive results across diverse downstream tasks. The CLIP model consists of an image encoder and a text encoder. During the pre-training phase, contrastive learning is employed, where a matched image-text pair serves as a positive example, while different image-text pairs are treated as negatives. Many studies have proposed different training methods to improve the performance of visual language models in downstream tasks. For instance, CLIP-FSAR (Wang et al. 2023) introduces data augmentation methods or architectural changes to enhance the generalization performance of CLIP models on different tasks. CLIP-Adapter (Gao et al. 2021) is optimized for specific tasks by adding adapters at different layers in the model. Thengane et al. (Thengane et al. 2022) demonstrated that the zero-shot prediction of the CLIP model achieves the highest level of performance in continual learning even without any training. ZSCL (Zheng et al. 2023) strengthens the correlation between images and text by improving the contrast learning strategy, leading to better performance in zero-shot transfer. Besides, Zhuang et al. (Ding et al. 2022) adopted an adapted variant of the learning without forgetting technique which uses randomly generated sentences to calculate distillation losses.

Few-shot continual learning focuses on optimizing the model to efficiently learn and adapt to new knowledge with limited samples while retaining learned knowledge. There are two severe challenges we need to address: i) Scarce training data for novel classes, which hampers effective knowledge acquisition for these classes; ii) The necessity to alleviate catastrophic forgetting and preserve previously learned knowledge while undergoing the continual learning. In fact, humans can quickly learn each new task without forgetting previous ones, while artificial neural networks do not have such extraordinary abilities. After investigating the cognitive mechanisms of human brain, the studies (Flesch et al. 2018; Kudithipudi et al. 2022) find that augmenting large-scale neural networks with abundant supervised/unsupervised experience could learn a good embedding of the stimulus space (similar to that observed in humans), and further reduce the catastrophic forgetting in the continual learning process.

Inspired by this above findings, we attempt to leverage the powerful understanding capability of existing big-models of the world (such as CLIP (Radford et al. 2021b)), which can be understood as the empirical knowledge that humans have learned to a certain extent. Here we address the FSCL task from the following two aspects. Firstly, we can perform the big-model driven transfer learning to employ the powerful encoding capability of the pre-trained big-model, which can adapt the continual model to a few newly added samples while alleviating the over-fitting and catastrophic forgetting problems. Secondly, considering that the big-model and the continual model may have different perception results even for the same thing, we introduce an instance-level adaptive decision to provide high-level inference support adjusted to varying image samples. Similar to the effective communication between people, the final decisions can well consider the inference strengths of both the big-model and the continual model under the guide of the instance-level adaptive mechanism. Furthermore, the results can be used to gradually optimize the parameters of the continual model, performing the knowledge distillation from the big-model of the world in the continual learning process.

Fig.1 illustrates the pipeline of our proposed B-FSCL framework. Different from existing FSCL methods, the entire B-FSCL is continuously learning under the guidance/traction of the world’s existing big-models, which have the powerful embedding and reasoning capabilities for the data to be processed. First, by taking full advantage of the world’s big-model (denoting as $\Psi^{*}$), we design a big-model driven embedding transfer module to enhance the encoding ability of the continual model (denoting as $\Theta$), especially for the few-shot learning sessions. Specifically, we maximize the mutual information of two distributions between the convolutional feature $F(x,\Theta_{conv})^{(l)}$ of the continual model and the corresponding feature $F(x,\Psi^{\ast})^{(l)}$ of big-model on multiple scales (i.e., $l=1,2,\dots,L$). Through the big-model driven transfer learning, we can quickly learn new concepts by adapting the continual model to new few-shot samples, while alleviating the over-fitting and catastrophic forgetting problems of the FSCL task. Furthermore, considering potential disparities in understanding capability between the big model and the continual model for the identical sample, we introduce the instance-level adaptive decision module to give a more reasonable prediction for different input images. Here we construct an $\alpha$-Net to learn an adaptive decision parameter $\alpha$ with $F(x,\Psi^{\ast})$ from the big-model and $F(x,\Theta_{conv})$ from the continual model as the input. Under the guidance of the learned parameter $\alpha$, we can obtain the final classification result by considering the decision probabilities of the big-model and the continual model (i.e., $\mathbf{p}_{bg}$ and $\mathbf{p}_{cmt}$). We use the same training strategy described above for both base and new sessions in our proposed B-FSCL framework.

The limited learning capacity of the continual model often hampers its ability to grasp concepts from a few newly added samples. In most existing FSCL methods(Zhang et al. 2021; Zhou et al. 2022), a common strategy to mitigate forgetting is to only update the last fully-connected layer while freezing the encoding convolutional layers. However, these approaches make it difficult to learn new concepts from few-shot samples. Inspired by the cognitive mechanism of the human brain (Flesch et al. 2018), we attempt to leverage the strong capabilities of a pre-trained big model that possesses a well-embedded stimulus space. Here we perform the big-model driven embedding transfer to enhance the encoding capability of the continual model, thereby alleviating the issues of catastrophic forgetting and over-fitting to a certain extent.

Under the traction of the world’s big-model, we hope that the learned knowledge of the continual model can be more inclined to the big-model. Specifically, we maximize mutual information of multi-scale feature distributions as follows:

$\displaystyle\mathop{\arg\max}\limits_{\Theta_{conv}}\sum_{l=1}^{L}\mathcal{I}_{\Theta_{conv}}(F(x,\Theta_{conv})^{(l)},F(x,\Psi^{\ast})^{(l)}),$

(1)

where $\Theta_{conv}$ denotes the convolutional parameters of the continual model, $\Psi^{\ast}$ refers to the fixed big-model, and $L$ represents the total number of convolutional scales involved. $F(x,\Theta_{conv})^{(l)}$ denotes the convolutional feature of the input image $x$ in the $l$-th layer of the continual model, while $F(x,\Psi^{\ast})^{(l)}$ represents the corresponding convolutional feature of the big-model. Since the mutual information of discrete features is difficult to calculate, we employ the lower-bound to the mutual information based on the Kullback-Leibler divergence (Donsker and Varadhan 1975). The big-model driven embedding transfer process in Eqn. (1) can be further represented as:

	$\displaystyle\mathop{\arg\max}\limits_{\Theta_{conv},\varphi}\sum_{l=1}^{L}\mathbb{E}_{\mathbb{J}}[T_{\varphi}(F(x,\Theta_{conv})^{(l)},F(x,\Psi^{\ast})^{(l)})]$		(2)
	$\displaystyle-\log\mathbb{E}_{\mathbb{M}}[e^{T_{\varphi}(F(x,\Theta_{conv})^{(l)},F(x,\Psi^{\ast})^{(l)})}],$

where $T_{\varphi}(\cdot)$ is a discriminant function between features of different models. $\mathbb{E}_{\mathbb{J}}$ represents the expectation of the joint distribution of $F(x,\Theta_{conv})^{(l)}$ and $F(x,\Psi^{\ast})^{(l)}$, while $\mathbb{E}_{\mathbb{M}}$ denotes the expectation of the margin distributions of $F(x,\Theta_{conv})^{(l)}$ and $F(x,\Psi^{\ast})^{(l)}$. Through gradually optimizing the network parameters $\Theta_{conv}$ and $\varphi$, the features of the continual model $F(x,\Theta_{conv})$ would stably converge towards the embedding of the world’s big-model $F(x,\Psi^{\ast})^{(l)}$, which also make the continual model adapt to these incremental image samples.

In the continual learning process, another major challenge is that it is difficult to learn the model and obtain accurate prediction results with only few-shot samples. Considering that big-model and continual model may possess varying inference abilities for the same sample, we introduce an instance-level adaptive decision mechanism to provide the effective inference adjusted to different samples. By promoting effective communication between models, the final decision can comprehensively integrate the reasoning advantages of both the big-model and the continual model, thereby enabling more accurate decision-making. To guide the instance-wise adaptive decision, we build an $\alpha$-Net to learn the weight coefficient $\alpha$ for each input, formally,

$$\alpha=\mathrm{H}(F(x,\Theta),F(x,\Psi^{\ast}),\omega_{\alpha}),$$

(3)

where $F(x,\Theta)$ and $F(x,\Psi^{\ast})$ represent the embedding features of the image $x$ from the continual model and the big-model, respectively. $\omega_{\alpha}$ denotes the parameters of $\alpha$-Net to be learned. The weight $\alpha\in[0,1]$ is achieved by considering the feature embeddings of two models (i.e., $F(x,\Theta)$ and $F(x,\Psi^{\ast})$). Under the guidance of the learned weight $\alpha$, the instance-level adaptive decision can be performed as follows:

$$\mathbf{p}=\alpha*\mathbf{p}_{ctm}+(1-\alpha)*\mathbf{p}_{bg},$$

(4)

where $\mathbf{p}_{ctm}$ and $\mathbf{p}_{bg}$ represent the probability distributions achieved from the continual model and the big-model, respectively. $\mathbf{p}$ refers to the probability distribution of the final decision.

To optimize the network parameters of the continual model and $\alpha$-Net (i.e., $\Theta$ and $\omega_{\alpha}$), we minimize the cross-entropy value between the ground truth and the probability of our final decision:

$$\mathop{\arg\min}_{\Theta,\omega_{\alpha}}-\sum_{i}^{|X_{t}|}y_{i}\log p_{i},$$

(5)

where $i$ is the index of the training set $X_{t}$. For each input image $x_{i}$, $y_{i}$ is its ground truth and the final decision probability $p_{i}$ is the maximum value in the probability distribution of $\textbf{p}_{i}$. The instance-level adaptive decision allows the comprehensive integration of inference advantages between the big-model and the continual model in the ultimate decisions, achieving more effective classification results. Meanwhile, this process can selectively distillate knowledge from the world’s big-model to the continual model, which can make the continual model adapt to these new-added samples.

Datasets: We evaluate the performance of B-FSCL on three publicly available datasets: CIFAR100 (Krizhevsky 2009), miniImageNet (Russakovsky et al. 2015), and CUB200 (Wah et al. 2011). CIFAR100 (Krizhevsky 2009) consists of 100 classes with a total of 60,000 RGB images. miniImageNet (Russakovsky et al. 2015) is a subset of ImageNet, comprising 100 classes. CUB200 (Wah et al. 2011) is a fine-grained classification dataset with 11,788 images distributed among 200 classes. For the CIFAR100 and miniImageNet datasets, we use all training data of the 60 base classes to train a base network. The 40 new classes are then used for eight 5-way 5-shot continual learning tasks. We divide the 200 classes of CUB200 into 100 base classes and 100 new classes. The 100 new classes are used for ten 10-way 5-shot continual learning tasks. To ensure a fair comparison, we follow the same split setting on the three datasets, as in FSCIL (Tao et al. 2020b).

To assess the superiority of our proposed B-FSCL framework, we conduct an evaluation on the CUB200 dataset (Wah et al. 2011) and compare its performance with other state-of-the-art methods. The detailed results are presented in Table 1, and the accuracy curves across continual learning sessions are illustrated in Fig. 3(c). Our B-FSCL framework shows an impressive performance, significantly outperforming all other compared methods. Notably, our B-FSCL demonstrates significant improvements in each stage compared to the second-best approach CABD (Zhao et al. 2023), which validates the superiority of our method. In the final continual stage, our approach significantly outperforms CABD by a $\Delta$Final of 5.53%. This result highlights the consistent learning capability of B-FSCL throughout the continual stage. We confirm that the big-model driven embedding transfer module can effectively transfer the powerful encoding ability of big-model to the continual model, thereby addressing the issue of insufficient learning capability for new classes and improving adaptability to new tasks with few-shot samples. Compared to the current best method (i.e., CABD(Zhao et al. 2023)), our proposed B-FSCL achieves 4.70% in terms of the Avg. metric. It indicates that our B-FSCL has superior performance across all continual learning stages. This finding further confirms that the proposed B-FSCL can effectively mitigate the lack of the learning capacity of the continual model. Impressive results in Avg and $\Delta$Final metrics further verifies the superiority of our B-FSCL in learning new knowledge and alleviating catastrophic forgetting.

Our experimental results consistently demonstrate that B-FSCL outperforms all other methods, surpassing the best CABD(Zhao et al. 2023) method with 4.28% in the KR metric. The KR metric measures the knowledge ratio between the final continual stage and the base stage, where a higher KR value indicates better retention of learned knowledge in continual learning stages. Moreover, these results that the B-FSCL can effectively enhance the few-shot classification accuracy by integrating the big-model driven embedding transfer and instance-level adaptive decision modules. Our proposed B-FSCL demonstrates outstanding performance by integrating the reasoning advantages of both the big-model and the continual model, resulting in more accurate decision-making. Additionally, the accuracy changing curves depicted in Fig. 3(c) visually validate the superiority of our B-FSCL framework. Besides, we incorporate distinct accuracy curves for the miniImageNet and CIFAR100 datasets in Fig. 3(a) and Fig. 3(b), respectively. The evaluation results clearly demonstrate the superior performance of our B-FSCL method across different datasets. This further emphasizes the effectiveness of our proposed big-model driven embedding transfer and instance-level adaptive decision modules in acquiring knowledge of novel classes while addressing the issue of catastrophic forgetting.

All ablation studies are conducted on the CUB200 dataset to assess the efficacy of the proposed modules.

In this work, we propose a Big-model driven Few-shot Continual Learning (B-FSCL) framework, which can well employ the strong embedding and reasoning capabilities of the existing big-model to address two key challenges in the FSCL task. Under the knowledge traction of the world’s big-model, we introduce the big-model driven embedding transfer learning to enhance the convolutional encoding capability of the continual model, which can also mitigate the catastrophic forgetting and over-fitting problems in the continual learning process. By facilitating effective communication between the models, we especially design an instance-level adaptive decision module to obtain more accuracy recognition results, which can be used to guide the knowledge distillation for the continual model. Extensive experiments have proved the effectiveness of our proposed B-FSCL. In future work, we will further employ the advantages of the world’s big-model to promote the continual learning ability of artificial neural networks.