Towards a General Framework for
Continual Learning with Pre-training

To cope with real-world dynamics, continual learning has received widespread attention, especially in the context of pre-training. Through adapting the pre-trained knowledge effectively to downstream tasks, it provides not only positive knowledge transfer but also robustness to catastrophic forgetting [10, 8, 16, 20]. An emerging direction is the implementation of parameter efficient fine-tuning (PEFT) techniques (e.g., Prompt [3], Adapter [11], LoRA [2], FiLM [9], etc.), which usually freeze a pre-trained transformer backbone and employ additionally a few parameters to steer representation learning. In particular, recent prompt-based approaches [19, 18, 17, 12, 15] focus on construction and inference of appropriate prompts for each task, and achieve outstanding performance under strong supervised pre-training. However, existing methods usually degrade in performance with challenges in upstream knowledge (e.g., different pre-training paradigms) and downstream tasks (e.g., out-of-distribution and fine granularity), with generality left to be desired.

In this work, we provide an in-depth theoretical analysis of the continual learning objective in the context of pre-training, which can be decomposed into hierarchical components such as within-task prediction, task-identity inference and task-adaptive prediction. By leveraging the well-distributed pre-trained representations, we then propose an innovative approach applicable to various PEFT techniques to optimize explicitly the hierarchical components. We perform extensive experiments on downstream continual learning to demonstrate the superiority and generality of our approach, and further explore the applicability of PEFT techniques in upstream continual learning. We also provide neurological insights into the proposed framework for acquisition of open-world knowledge.

Continual learning aims to master a sequence of tasks represented by their respective training sets $\mathcal{D}_{1},...,\mathcal{D}_{T}$ and excel on their corresponding test sets. Each training set $\mathcal{D}_{t}=\{(\boldsymbol{x}_{t,n},y_{t,n})\}_{n=1}^{N_{t}}$, where $|\mathcal{D}_{t}|=N_{t}$ denotes the size of $\mathcal{D}_{t}$. $\boldsymbol{x}_{t,n}\in\mathcal{X}_{t}$ and $y_{t,n}\in\mathcal{Y}_{t}$ indicate the sample and label elements, respectively. Consider a neural network model with a backbone $f_{\theta}$ parameterized by $\theta$, and an output layer $h_{\psi}$ parameterized by $\psi$. This model seeks to learn the projection from $\mathcal{X}=\bigcup_{t=1}^{T}\mathcal{X}_{t}$ to $\mathcal{Y}=\bigcup_{t=1}^{T}\mathcal{Y}_{t}$, aiming to predict the label $y=h_{\psi}(f_{\theta}(\boldsymbol{x}))\in\mathcal{Y}$ of an unseen test sample $\boldsymbol{x}$ drawn from previous tasks. The backbone function $f_{\theta}$ is assumed to be pre-trained with a substantial quantity of additional training samples external to each $\mathcal{D}_{t}$. There are commonly three distinct settings for continual learning [13]: task-, domain-, and class-incremental learning (TIL, DIL, and CIL). Specifically, $\mathcal{Y}_{1},...,\mathcal{Y}_{T}$ are identical for DIL while disjoint for TIL and CIL. The task identity is provided for TIL at test time but is not available for DIL and CIL.

Here we take CIL as a typical scenario for theoretical analysis, where $\mathcal{Y}_{t}\cap\mathcal{Y}_{t^{\prime}}=\emptyset$, $\forall t\neq t^{\prime}$. Let $\mathcal{X}_{t}=\bigcup_{j}\mathcal{X}_{t,j}$ and $\mathcal{Y}_{t}=\{\mathcal{Y}_{t,j}\}$, where $j\in\{1,...,|\mathcal{Y}_{t}|\}$ indicates the $j$-th class in task $t$. Now assume we have a ground event denoted as $\mathcal{D}=\{\mathcal{D}_{1},...,\mathcal{D}_{t}\}$ and a pre-trained model $f_{\theta}$. For any sample $\boldsymbol{x}\in\bigcup_{k=1}^{t}\mathcal{X}_{k}$, a general goal of the CIL problem is to learn $P(\boldsymbol{x}\in\mathcal{X}_{i,j}|\mathcal{D},\theta)$, where $i\in\{1,...,t\}$ and $j\in\{1,...,|\mathcal{Y}_{i}|\}$. This can be decomposed into two probabilities, including task-identity inference (TII) and within-task prediction (WTP), denoted as $P(\boldsymbol{x}\in\mathcal{X}_{i}|\mathcal{D},\theta)$ and $P(\boldsymbol{x}\in\mathcal{X}_{i,j}|\boldsymbol{x}\in\mathcal{X}_{i},\mathcal{D},\theta)$, respectively. Based on Bayes’ theorem, we have

$\displaystyle P(\boldsymbol{x}\in\mathcal{X}_{i,j}|\mathcal{D},\theta)=P(\boldsymbol{x}\in\mathcal{X}_{i,j}|\boldsymbol{x}\in\mathcal{X}_{i},\mathcal{D},\theta)P(\boldsymbol{x}\in\mathcal{X}_{i}|\mathcal{D},\theta).$

(1)

Let $\bar{i}\in\{1,...,t\}$ and $\bar{j}\in\{1,...,|\mathcal{Y}_{i}|\}$ be the ground truth of an $\boldsymbol{x}$ w.r.t. the task identity and within-task index. Eq. (1) shows that if we can improve either the WTP performance $P(\boldsymbol{x}\in\mathcal{X}_{\bar{i},\bar{j}}|\boldsymbol{x}\in\mathcal{X}_{\bar{i}},\mathcal{D},\theta)$, the TII performance $P(\boldsymbol{x}\in\mathcal{X}_{\bar{i}}|\mathcal{D},\theta)$, or both, then the CIL performance $P(\boldsymbol{x}\in\mathcal{X}_{\bar{i},\bar{j}}|\mathcal{D},\theta)$ would be improved. However, such an improvement is limited since it is upper-bounded by WTP or TII. To further improve the CIL performance, we propose a hierarchical decomposition of its objective. That is, besides the improvement of $P(\boldsymbol{x}\in\mathcal{X}_{\bar{i},\bar{j}}|\mathcal{D},\theta)$, we also need to improve the performance of task-adaptive prediction (TAP), denoted as $P(\boldsymbol{x}\in\mathcal{X}^{y}|\mathcal{D},\theta)$, where $\mathcal{X}^{y}$ represents the domain of class $y$ in all previous tasks, and $y=\mathcal{Y}_{\bar{i},\bar{j}}$ is the ground truth label of $\boldsymbol{x}$. Then the final goal of CIL is formulated as a multi-objective optimization problem, i.e., $\max[P(\boldsymbol{x}\in\mathcal{X}_{\bar{i},\bar{j}}|\mathcal{D},\theta),P(\boldsymbol{x}\in\mathcal{X}^{y}|\mathcal{D},\theta)]$. Notice that the WTP probability is a categorical distribution over all observed tasks $\{1:t\}$, while the TAP probability is over all observed classes $\bigcup_{k=1}^{t}\mathcal{Y}_{k}$.

To resolve the problems above, we derive the sufficient and necessary conditions in the context of the widely-used cross-entropy loss. Specifically, we define

	$\displaystyle{H}_{\rm{WTP}}(\boldsymbol{x})$	$\displaystyle=\mathcal{H}(\boldsymbol{1}_{\bar{j}},\{P(\boldsymbol{x}\in\mathcal{X}_{\bar{i},j}|\boldsymbol{x}\in\mathcal{X}_{\bar{i}},\mathcal{D},\theta)\}_{j}),$		(2)
	$\displaystyle{H}_{\rm{TII}}(\boldsymbol{x})$	$\displaystyle=\mathcal{H}(\boldsymbol{1}_{\bar{i}},\{P(\boldsymbol{x}\in\mathcal{X}_{i}|\mathcal{D},\theta)\}_{i}),$		(3)
	$\displaystyle{H}_{\rm{TAP}}(\boldsymbol{x})$	$\displaystyle=\mathcal{H}(\boldsymbol{1}_{\bar{c}},\{P(\boldsymbol{x}\in\mathcal{X}^{c}|\mathcal{D},\theta)\}_{c}),$		(4)

where ${H}_{\rm{WTP}}$, ${H}_{\rm{TII}}$, and ${H}_{\rm{TAP}}$ are the cross-entropy values of WTP, TII, and TAP, respectively. The operation $\mathcal{H}(p,q)\triangleq-\mathbb{E}_{p}[\log q]=-\sum_{i}p_{i}\log q_{i}$. $\boldsymbol{1}_{\cdot}$ is a one-hot encoding function.

We now present the first theorem under the CIL scenario:

With the use of cross-entropy, the continual learning performance tends to be better as the bounds are tightened. In Theorem 1 we have shown that good performances of WTP, TII and TAP are sufficient to guarantee a good performance of CIL. For completeness, we now study the necessary conditions of a well-performed CIL method in Theorem 2.

Theorem 2 suggests that if a continual learning model is well trained (i.e., with low loss), then the WTP, TII and TAP for sequential tasks are always implied to be small.

Motivated by these theoretical insights, we propose to optimize explicitly the hierarchical components (i.e., WTP, TII and TAP) for continual learning with pre-training. Our proposal stems from two particular advantages of pre-training: (1) the representations can be effectively adapted to downstream tasks through PEFT techniques, and (2) the distributions of unadapted and adapted representations (denoted as $\hat{\mathcal{G}}_{c}$ and $\mathcal{G}_{c}$ for each class $c\in\mathcal{Y}_{i},i=1,...t-1$, respectively) can be effectively preserved through their statistical information. For efficiency and generality, here we employ multiple centroids obtained from K-Nearest Neighbor (KNN) and add Gaussian noise as a specific implementation.

First, we improve WTP through effectively incorporating task-specific knowledge from each $\mathcal{D}_{t}$. Specifically, we construct task-specific parameters $\boldsymbol{e}_{t}$ with a PEFT technique (e.g., Prompt [3], Adapter [11], LoRA [2], FiLM [9], etc.), and optimize ${H}_{\rm{WTP}}$ with cross-entropy (CE). $\boldsymbol{e}_{1},...,\boldsymbol{e}_{t-1}$ are frozen to avoid catastrophic forgetting, while $\boldsymbol{e}_{t}$ is initialized with $\boldsymbol{e}_{t-1}$ to transfer knowledge. Besides, the adapted representations of $\boldsymbol{e}_{t}$, although allowing the new task to be performed well, may overlap with that of the old tasks and thus affect TAP. To overcome this issue, we preserve statistics of adapted representations collected by $f_{\theta}$ and $\boldsymbol{e}_{i},i=1,...,t-1$, where for classification we calculate the mean $\boldsymbol{\mu}_{c}$ of $\mathcal{G}_{c}$ for each class $c\in\mathcal{Y}_{i}$, and design a contrastive regularization (CR):

$$\mathcal{L}_{{\rm{CR}}}(\boldsymbol{e}_{t})=\sum_{\boldsymbol{h}\in\mathcal{H}_{t}}\frac{1}{\sum_{i=1}^{t-1}|\mathcal{Y}_{i}|}\sum_{i=1}^{t-1}\sum_{c\in\mathcal{Y}_{i}}\log\frac{\exp(\boldsymbol{h}\cdot\boldsymbol{\mu}_{c}/\tau)}{\sum_{\boldsymbol{h}^{\prime}\in\mathcal{H}_{t}}\exp(\boldsymbol{h}\cdot\boldsymbol{h}^{\prime}/\tau)+\sum_{i=1}^{t-1}\sum_{c\in\mathcal{Y}_{i}}\exp(\boldsymbol{h}\cdot\boldsymbol{\mu}_{c}/\tau)},$$

(5)

where $\mathcal{H}_{t}$ is the embedding transformation of $\mathcal{D}_{t}$ with $f_{\theta}$ and $\boldsymbol{e}_{t}$. $\tau$ is the temperature coefficient, which is insensitive and set to 0.8 in practice. Then, the loss function of WTP can be defined as

$$\mathcal{L}_{{\rm{WTP}}}(\psi,\boldsymbol{e}_{t})=\mathcal{L}_{{\rm{CE}}}(\psi,\boldsymbol{e}_{t})+\lambda\mathcal{L}_{{\rm{CR}}}(\boldsymbol{e}_{t}).$$

(6)

Therefore, the adapted representations of new classes can be well distinguished for WTP while avoiding overlap with the previous ones. $\lambda$ is a hyperparamter to balance the impact of old classes.

Second, we improve TII and TAP through leveraging the approximated distributions of unadapted and adapted representations, respectively. For ${H}_{\rm{TII}}$, we construct an auxiliary output layer $\hat{h}_{\omega}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{T}$ parameterized by $\omega$, learning explicitly the projection from unadapted representations to task identity via cross-entropy:

$$\mathcal{L}_{{\rm{TII}}}(\omega)=\frac{1}{\sum_{i=1}^{t}|\mathcal{Y}_{i}|}\sum_{i=1}^{t}\sum_{c\in\mathcal{Y}_{i}}\sum_{\hat{\boldsymbol{h}}\in\hat{\mathcal{H}}_{i,c}}-\log\frac{\exp(\hat{h}_{\omega}(\hat{\boldsymbol{h}})[i])}{\sum_{j=1}^{t}\exp(\hat{h}_{\omega}(\hat{\boldsymbol{h}})[j])},$$

(7)

where $\hat{\mathcal{H}}_{i,c}$ is constructed by sampling an equal number of pseudo representations from $\hat{\mathcal{G}}_{c}$ for $c\in\mathcal{Y}_{i}$ and $i=1,...,t$. Similarly, the final output layer $h_{\psi}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{|\mathcal{Y}|}$ is further optimized for ${H}_{\rm{TAP}}$:

$$\mathcal{L}_{{\rm{TAP}}}(\psi)=\frac{1}{\sum_{i=1}^{t}|\mathcal{Y}_{i}|}\sum_{i=1}^{t}\sum_{c\in\mathcal{Y}_{i}}\sum_{\boldsymbol{h}\in\mathcal{H}_{i,c}}-\log\frac{\exp(h_{\psi}(\boldsymbol{h})[c])}{\sum_{j=1}^{t}\sum_{c^{\prime}\in\mathcal{Y}_{j}}\exp(h_{\psi}(\boldsymbol{h})[c^{\prime}])},$$

(8)

where $\mathcal{H}_{i,c}$ is constructed by sampling an equal number of pseudo representations from $\mathcal{G}_{c}$ for $c\in\mathcal{Y}_{i}$ and $i=1,...,t$. As $\omega$ and $\psi$ are usually light-weight, the optimization of TII and TAP is computationally efficient. At test time, our approach predicts the task identity $i=\hat{h}_{\omega}(f_{\theta}(\boldsymbol{x}))$ and then the label $y=h_{\psi}(f_{\theta}(\boldsymbol{x};\boldsymbol{e}_{i}))$.

Experimental Setup: We consider two CIL benchmarks that are widely used for downstream continual learning [19, 18, 12], such as Split ImageNet-R [5] of 200-class natural images and Split CUB-200 [14] of 200-class bird images, randomly split into 10 incremental tasks. After learning multiple incremental tasks, we further evaluate upstream continual learning with the ability of few-shot learning, i.e., adapting the backbone to a N-way K-shot task [1] randomly sampled from subsequent unseen classes. We consider supervised and self-supervised pre-training on ImageNet-21K, denoted as Sup-21K and iBOT-21K, respectively.

Experimental Result: We implement two representative PEFT techniques as the task-specific parameters in our approach, such as Prompt [3] (adjusting intermediate inputs through prepending a short sequence of learnable prompt parameters) and LoRA [2] (adjusting backbone parameters through adding a learnable low-rank parameter matrix). We first evaluate the performance of downstream continual learning with different pre-training paradigms and CIL benchmarks. As shown in Table 1, the performance of state-of-the-art prompt-based approaches degrades remarkably under self-supervised pre-training (e.g., iBOT-21K) and fine-grained classification (e.g., Split CUB-200), while both versions of our approach outperform them significantly.

On the other hand, a potential limitation of prompt-based methodologies is that, the pre-trained knowledge in backbone parameters cannot be updated and enriched from incremental tasks, which has been rarely discussed in previous literature. Motivated by this, we then consider upstream continual learning, i.e., the ability of accumulating knowledge in backbone parameters. Specifically, after downstream continual learning of multiple incremental tasks, we evaluate the performance of the backbone to perform few-shot learning of an additional task randomly sampled from subsequent unseen classes. As shown in Table 2, the backbone adapted by the LoRA version of our approach acquires strong improvements in few-shot learning, compared to the unadapted backbone of the Prompt version. In addition to Split ImageNet-R and Split CUB-200 that split all tasks from the same dataset, we further consider a mixture of tasks sampled from CUB-200 [14] and Cars-196 [4] datasets, where the improvements remain significant. These results demonstrate the importance and feasibility of synchronizing upstream and downstream continual learning.

In this work, we propose a general framework for continual learning in the context of pre-training, with decomposing the objective into three hierarchical components (i.e., WTP, TII and TAP) and optimizing them explicitly with PEFT techniques and representation statistics. Through extensive experiments, we demonstrate the superiority and generality of our approach in downstream continual learning, and further elaborate on the importance of upstream continual learning, which requires updating the backbone parameters rather than instructing only (intermediate) inputs. Interestingly, the proposed framework requires sequential invocation of the unadapted and (task-specific) adapted representations for inference, which is consistent with recent advances in biological learning and memory [7, 6] that the activation of non-memory cells and memory cells (as well as their specific sub-populations) is internally switched. This connection potentially bridges the intrinsic mechanisms of biological and artificial intelligence in acquisition of open-world knowledge.

This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106302), NSFC Projects (Nos. 62061136001, 92248303, 62106123, 61972224), BNRist (BNR2022RC01006), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. L.W. is also supported by Shuimu Tsinghua Scholar, and J.Z. is also supported by the XPlorer Prize.