Incorporating neuro-inspired adaptability for continual learning in artificial intelligence

A central challenge of continual learning is to resolve the mutual interference between new and old tasks due to their distribution differences. The functional advantages of the $\gamma$MB system suggest that stability protection and active forgetting are both important richards2017persistence ; shuai2010forgetting ; dong2016inability ; zhang2018active ; davis2017biology ; gao2019genetic ; mo2022age (Fig. 1b), although current efforts mainly focus on the former to prevent catastrophic forgetting mccloskey1989catastrophic ; mcclelland1995there . Here we formulate this process with the framework of Bayesian learning, which has been hypothesized to well model biological synaptic plasticity by tracking the probability distribution of synaptic weights under dynamic sensory inputs aitchison2021synaptic ; schug2021presynaptic . We briefly describe a simple case of two tasks (Fig. 2a) and leave the full details to Methods.

Let’s consider a neural network with parameters $\theta$ continually learning tasks $A$ and $B$ from their training data $\ignorespaces D_{A}$ and $\ignorespaces D_{B}$ in order to perform well on their test data, which is called a “continual learner”. From a Bayesian perspective, the learner first places a prior distribution $p(\theta)$ on $\theta$. After learning task $A$, the learner updates the belief of the parameters, resulting in a posterior distribution $p(\theta|D_{A})\propto p(D_{A}|\theta)p(\theta)$ that incorporates the knowledge of task $A$. Then, task $A$ can be performed successfully by finding a mode of the posterior: $\theta_{A}^{*}=\arg\max_{\theta}\log\,p(\theta|D_{A})$. For learning task $B$, $p(\theta|D_{A})$ becomes the prior and the posterior $p(\theta|D_{A},D_{B})\propto p(D_{B}|\theta)p(\theta|D_{A})$ will further incorporate the knowledge of task $B$. Similarly, the learner needs to find $\theta_{A,B}^{*}=\arg\max_{\theta}\log\,p(\theta|D_{A},D_{B})$, corresponding to maximizing both $\log p(D_{B}|\theta)$ for learning task $B$ and $\log p(\theta|D_{A})$ for remembering task $A$.

However, due to the differences in data distribution, remembering old tasks precisely can increase the difficulty of learning each new task well. Inspired by the biological active forgetting, we introduce a forgetting rate $\beta$ and replace $p(\theta|D_{A})$ with

where $p(\theta)$ is a non-informative prior without incorporating old knowledge wang2021afec . $Z$ is a $\beta$-dependent normalizer that keeps $\hat{p}$ a normalized probability distribution (Supplementary Section 1.1). $\hat{p}$ tends to forget task $\ignorespaces A$ when $\beta\rightarrow 1$, while be dominated by $p(\theta|D_{A})$ with full old knowledge when $\beta\rightarrow 0$. For the new target $p(\theta|D_{A},D_{B},\beta)\propto p(D_{B}|\theta)\hat{p}(\theta|D_{A},\beta)$, we derive the loss function

$\mathcal{L}_{B}(\theta)$ is the loss function of learning task $B$. $\lambda_{\rm{SP}}$ and $\lambda_{\rm{AF}}$ are hyperparameters that control the strengths of two regularizers responsible for stability protection and active forgetting, respectively. The stability protection part is to selectively penalize the deviance of each parameter $\theta_{m}$ from $\theta_{A,m}^{*}$ depending on its “importance” for old task(s), estimated by the Fisher information $F_{A,m}$.

The optimization of active forgetting can be achieved in two equivalent ways, i.e., AF-1 and AF-2 (Fig. 2b). They both encourage the network parameters $\theta$ to renormalize with an “expanded” set of parameters $\theta_{e}$ when learning task $B$. For AF-1, $\theta_{e,m}=0$ is “empty” with equal selectivity $I_{e,m}=1$ for renormalization, where the active forgetting term becomes the $L_{2}$ norm of $\theta$. The hyperparameters $\lambda_{\rm{AF}}\propto\beta$ and $\lambda_{\rm{SP}}\propto 1-\beta$, indicating that the old memories are directly affected. For AF-2, $\theta_{e,m}=\theta_{B,m}^{*}$ is the optimal solution for task $B$ only, obtained from optimizing $\mathcal{L}_{\rm{B}}(\theta_{e})$, and $I_{e,m}=F_{B,m}$ is the Fisher information. The forgetting rate is fully integrated into $\lambda_{\rm{AF}}\propto\beta/(1-\beta)$ and is independent of $\lambda_{\rm{SP}}$. In the absence of active forgetting ($\beta=0$), the loss function in Eq. (2) is left with only $\mathcal{L}_{\rm{B}}(\theta)$ and the stability protection term, which is (approximately benzing2022unifying ) equivalent to regular synaptic regularization methods such as EWC kirkpatrick2017overcoming . In particular, since the loss functions of these methods kirkpatrick2017overcoming ; aljundi2018memory ; zenke2017continual ; chaudhry2018riemannian typically have a similar form and differ only in the metric for estimating the parameter importance benzing2022unifying (Methods Eq. (11)), our proposal can be naturally combined with them by plugging in the active forgetting term.

For biological neural networks, active forgetting is able to remove outdated information and provide flexibility for adapting to a new memory shuai2010forgetting ; dong2016inability ; richards2017persistence . This strategy is essential for Drosophila to cope with the interference of previous tasks shuai2010forgetting ; dong2016inability ; bouton1993context ; zhang2018active . Here we theoretically analyze how this benefit is achieved in our computational model. First, an appropriate forgetting rate $\beta$ is able to improve the probability of learning new tasks well through attenuating old memories in $\theta$ (Methods Eq. (5)), which can be empirically determined by a grid search of $\lambda_{\rm{AF}}$ and/or $\lambda_{\rm{SP}}$. Second, when $\theta$ moves to the neighborhood of an empirical optimal solution, the active forgetting term in Eq. (2) can minimize the upper bound of generalization errors for continual learning, especially for new tasks (Methods Proposition 2).

Now we evaluate the efficacy of active forgetting on three continual learning benchmarks for visual classification tasks. They are all constructed from the CIFAR-100 dataset krizhevsky2009learning of 100-class colored images but with different degrees of overall knowledge transfer wang2021afec . As shown in Fig. 2c, the proposed active forgetting can largely enhance the average accuracy of all tasks, using EWC kirkpatrick2017overcoming as a baseline for preserving memory stability. Then we analyze the benefits of active forgetting on learning plasticity and memory stability with the metrics of forward transfer and backward transfer, respectively, where the former is clearly dominant. Similar results are observed when plugging the active forgetting term in other synaptic regularization methods that only preserve memory stability (Supplementary Fig. 1). In contrast, active forgetting fails to improve the joint training performance (Supplementary Fig. 4a), suggesting that its benefits are specific to continual learning. From visual interpretation of the latest task predictions in Fig. 2d, active forgetting can indeed eliminate the past conflicting information, leading to better recognition of the object itself.

After demonstrating the benefits of active forgetting together with stability protection for a single continual learner (SCL), we turn to investigate the organizing principles of the $\gamma$MB system where new memory forms and active forgetting happens shuai2010forgetting ; cohn2015coordinated ; waddell2016neural ; modi2020drosophila ; gao2019genetic ; berry2012dopamine ; berry2018dopamine ; handler2019distinct ; shuai2015dissecting . Specifically, there are five compartments that process sequential experiences in parallel. The outputs of these compartments are integrated in a weighted sum fashion to guide adaptive behaviors. Inspired by this, we design a $\gamma$MB-like architecture consisting of multiple parallel continual learners (Fig. 3a). Each learner employs a parameter space to learn all tasks, but its dedicated output head is removed and the weighted sum of the previous layer’s output is fed into a shared output head, where the output weights of each learner are incrementally updated.

In such a $\gamma$MB-like architecture, the relationship between learners is critical to the performance of continual learning. When the diversity of their expertise is properly coordinated, the obtained solution can provide a high degree of compatibility with both new and old tasks. Here we present a conceptual illustration via performing tasks $A$, $B$ and $C$ with different similarities (Fig. 3c). Since it is difficult to find a shared optimal solution for all tasks, the SCL has to converge to a high error region for tasks $A$ and $C$. In contrast, the multiple continual learners (MCL) with appropriate diversity allow for division of labor to address task discrepancy and complement their functions as parameter changes. Then, the output weights can integrate respective expertise of each learner into the final prediction.

In general, each learner’s expertise is directly modulated by its target distribution, where the proposed functional strategy can naturally serve this purpose. With the formulation of active forgetting, the target distribution $p(\theta|D_{A},D_{B},\beta)\propto p(D_{B}|\theta)\hat{p}(\theta|D_{A},\beta)$ tends to be different for learners with different forgetting rates $\beta$, and vice versa (Fig. 3b). Therefore, we implement the forgetting rates adaptively for these learners to coordinate their relationship. Since the target distribution also depends on $p(D_{B}|\theta)$, we further propose a supplementary modulation that constrains explicitly the differences in predictions between learners, corresponding to adjusting the learning rules for each new task. We refer to the $\gamma$MB-like architecture with these two modulations as Collaborative continual learners with Active Forgetting (CAF), and provide a formal definition in Methods Eq. (12).

For multiple learners with identical network architectures and similar forms of learning objectives, the priority is to obtain adequate differentiation of their expertise. In this case, the forgetting rates serve to diversify these learners, similar to the neurological strategy of decaying old memories differentially in each compartment aso2016dopaminergic ; handler2019distinct ; mccurdy2021dopaminergic ; berry2012dopamine ; berry2018dopamine . In practice, the differences of learners can also arise from their innate randomness, such as the use of dropout and different random initializations, leading to sub-optimal solutions with moderate performance. This potentially corresponds to the anatomical randomness of KCs receiving olfactory signals in Drosophila chen2023ai ; caron2013random ; endo2020synthesis . At this point, the modulations of forgetting rates and learning rules can provide finer adjustments, e.g., by constraining excessive differences.

Given the same network width for each learner, using more learners (i.e., more parameters) generally results in better performance. However, there is an intuitive trade-off between learner number and width under a limited parameter budget. We verify that this trade-off is independent of training data distributions (Supplementary Section 1.3) and is relatively insensitive over a wide range (Supplementary Table 5). Therefore, we simply choose five learners ($K=5$) corresponding to the five biological compartments, which employs approximately 5 $\times$ parameters, and then reduce the network width accordingly to keep the total amount of parameters similar to that of the SCL. To evaluate the effect of innate diversity, we construct a low-diversity background by removing the dropout and using the same random initialization for each learner, and a high-diversity background by maintaining these randomness factors, where the overall diversity of expertise is evaluated by the average cosine or Euclidean distance between learners’ predictions.

As shown in Fig. 3e,f, adaptive implementations of either active forgetting (AF-1) or its supplementary modulation (AF-S) can greatly enhance the performance of MCL, where the degree of improvement varies with the effectiveness of increasing inadequate diversity or reducing excessive diversity in the two backgrounds, respectively. In response to different degrees of innate diversity, the respective advantages of AF-1 and AF-S are combined to achieve consistently better performance. Such modulations enable the multiple learners to effectively divide and cooperate in continual learning (Fig. 3d, Supplementary Fig. 4b,c, Supplementary Fig. 5). They exhibit a clear diversity of task expertise with several experts collaborating on each task, validating the conceptual model in Fig. 3c. Accordingly, the performance of CAF is largely superior to that of the SCL, averaging the predictions of five independently-trained continual learners, or using a separate learner for each task (Supplementary Fig. 6a,b). In particular, CAF can improve the SCL by a similar magnitude under different parameter budgets, indicating its outstanding scalability (Supplementary Fig. 7a, Supplementary Fig. 6c).

According to our theoretical analysis in Methods Proposition 1, the performance of a shared solution for new and old tasks depends on the discrepancy of task distributions and the flatness of loss landscape around it. With respect to these two aspects, we delve more deeply into the benefits of our approach. As for the former, we evaluate the discrepancy of task distributions in feature space via the difficulty of distinguishing them after continual learning long2015learning ; wang2022coscl (Fig. 4a-c). The large increase in discrimination error suggests that our approach can successfully reconcile this discrepancy. As for the latter, the solution obtained by ours enjoys a clearly flatter loss landscape (Fig. 4d-f), indicating that it is more robust to modest parameter changes in response to dynamic data distributions. Therefore, CAF can update parameters more flexibly than the SCL (Fig. 4g-i), with the performance of new and old tasks simultaneously improved (Supplementary Fig. 7b,c).

Finally, we evaluate CAF under the setting of task-incremental learning van2022three and compare it to a range of representative methods kirkpatrick2017overcoming ; aljundi2018memory ; zenke2017continual ; riemer2018learning ; schwarz2018progress ; jung2020continual ; cha2020cpr . We first consider visual classification tasks with different particular challenges. Besides the overall knowledge transfer, we employ additionally four benchmark datasets such as Omniglot lake2015human for long task sequence with imbalanced class numbers, CUB-200-2011 wah2011caltech and Tiny-ImageNet delange2021continual for larger scale images, and CORe50 lomonaco2017core50 for smoothly-changed observations. As shown in Fig. 5, the performance of all baselines varies widely across experimental settings, while CAF achieves consistently the strongest performance in a plug-and-play manner. We further experiment with Atari reinforcement tasks, where an agent incrementally learns to play several Atari games (Fig. 6a). The overall performance is evaluated by the normalized accumulated reward (NAR) jung2020continual ; wang2021afec ; cha2020cpr , where the rewards obtained for all tasks ever seen are normalized with the maximum reward of fine-tuning on each task, and then accumulated. Likewise, CAF can greatly enhance the performance of baseline approaches (Fig. 6b,c) through improving both learning plasticity and memory stability (Fig. 6d,e).

To improve learning plasticity, we introduce a forgetting rate $\beta$, and replace $p(\theta|D_{A})$ in Eq. (3) with $\hat{p}(\theta|D_{A},\beta)=\frac{p(\theta|D_{A})^{(1-\beta)}p(\theta)^{\beta}}{Z}$ as Eq. (1), where $\beta\in[0,1]$ is deterministic and $Z$ is a $\beta$-dependent normalizer. Correspondingly, the target distribution $p(\theta|D_{A},D_{B})$ becomes $p(\theta|D_{A},D_{B},\beta)$. $\ignorespaces\hat{p}$ has a nice property that it follows a Gaussian distribution if $p(\theta|D_{A})$ and $p(\theta)$ are both Gaussian (Supplementary Section 1.1), so we can compute $\hat{p}$ in a similar way as we compute $p(\theta|D_{A})$. A certain value of $\beta$ can maximize the probability of learning each new task well through forgetting the old knowledge, validating the motivation of introducing the non-informative prior in Eq. (1):

With the implementation of active forgetting, the learner needs to find

When $\beta=0$, the loss function in Eq. (9) degenerates to a similar form as regular synaptic regularization methods that only preserve memory stability kirkpatrick2017overcoming ; aljundi2018memory ; zenke2017continual ; chaudhry2018riemannian :

Since these methods differ mainly in the metric $\xi_{1:t-1}$ of estimating the importance of parameters for performing old tasks benzing2022unifying , the active forgetting term can be naturally combined with them. We discuss in more depth the motivation and implementation of active forgetting in Supplementary Section 1.2, including the choice of an appropriate $\beta$, the connections of two equivalent versions, and the technical details of derivation.

The $\gamma$MB-like architecture of multiple continual learners (MCL) adopts $K$ identically structured neural networks $f_{\phi_{i}}(\cdot),i=1,...,K$, corresponding to $K$ continual learners ${\rm{L}}_{i},i=1,...,K$ with their own parameter spaces. We remove the dedicated output head of each learner, and feed the weighted sum of the previous layer’s output into a shared output head $h_{\varphi}(\cdot)$ to make predictions, where the output weight of each learner $g_{i},i=1,...,K$ is updated incrementally. Then, the final prediction becomes $p(\cdot)=h_{\varphi}(\sum_{i=1}^{K}g_{i}f_{\phi_{i}}(\cdot))$, where the optimizable MCL parameters $\theta_{\rm{MCL}}$ include $\bigcup_{i=1}^{K}\phi_{i}$, $\bigcup_{i=1}^{K}g_{i}$ and $\varphi$. The proposed MCL is applicable to a wide range of loss functions for continual learning. By default, here we focus on the synaptic regularization methods kirkpatrick2017overcoming ; aljundi2018memory ; zenke2017continual ; chaudhry2018riemannian as defined in Eq. (11). In order to coordinate the diversity of learners’ expertise, we implement the proposed active forgetting in each learner. We further regularize differences in their predictive distributions, quantified by the widely-used Kullback Leibler (KL) divergence. Therefore, the loss function for the full version of our approach is defined as:

Continual learning aims to find a solution $\theta$ that can generalize well over a new distribution $\mathbb{D}_{t}$ and a set of old distributions $\mathbb{D}_{1:t-1}:=\{\mathbb{D}_{k}\}_{k=1}^{t-1}$. Let $\mathcal{E}_{\mathbb{D}_{t}}(\theta)$ and $\mathcal{E}_{\mathbb{D}_{1:t-1}}(\theta)$ denote the generalization errors. The training set and test set of each task follow the same distribution $\mathbb{D}_{k}$ ($k=1,2,...,t$), where the training set $D_{k}=\{(x_{k,n},y_{k,n})\}_{n=1}^{{N}_{k}}$ includes $N_{k}$ data-label pairs. Then we define $\mathcal{E}_{\mathbb{D}_{t}}(\theta)=\mathbb{E}_{(x,y)\sim\mathbb{D}_{t}}[\mathcal{L}_{t}(\theta\mathchar 24635\relax\;x,y)]$ and $\mathcal{E}_{\mathbb{D}_{1:t-1}}(\theta)=\frac{1}{t-1}\sum_{k=1}^{t-1}\mathbb{E}_{(x,y)\sim\mathbb{D}_{k}}[\mathcal{L}_{k}(\theta\mathchar 24635\relax\;x,y)]$, where $\mathcal{L}_{k}(\theta)$ can be generalized for any bounded loss function of task $k$. To minimize $\mathcal{E}_{\mathbb{D}_{t}}(\theta)$ and $\mathcal{E}_{\mathbb{D}_{1:t-1}}(\theta)$ without the use of old training samples $D_{1:t-1}:=\{D_{k}\}_{k=1}^{t-1}$, a continual learning model can only minimize an empirical risk over the current training samples ${D_{t}}$ in a parameter space $\Theta$ applicable to old tasks wang2022coscl ; knoblauch2020optimal , denoted as $\min_{\theta\in\Theta}\hat{\mathcal{E}}_{D_{t}}(\theta)$ where $\hat{\mathcal{E}}_{D_{t}}(\theta)=\frac{1}{N_{t}}\sum_{n=1}^{N_{t}}\mathcal{L}_{t}(\theta\mathchar 24635\relax\;x_{t,n},y_{t,n})$. In practice, sequential learning of each task by $\min_{\theta\in\Theta}\hat{\mathcal{E}}_{D_{t}}(\theta)$ can find multiple solutions with different generalizability for $\mathcal{E}_{\mathbb{D}_{t}}(\theta)$ and $\mathcal{E}_{\mathbb{D}_{1:t-1}}(\theta)$, where a solution with flatter loss landscape typically acquires better generalizability and is therefore more robust to catastrophic forgetting deng2021flattening ; cha2020cpr ; mirzadeh2020understanding .

Accordingly, we define a robust empirical risk for the current task as $\hat{\mathcal{E}}_{D_{t}}^{b}(\theta):=\rm{max}_{\lVert\Delta\rVert\leq b}\hat{\mathcal{E}}_{\textit{D}_{t}}(\theta+\Delta)$ by the worst case of parameter perturbations $\Delta$, where $\lVert\cdot\rVert$ denotes the $L_{2}$ norm and $b$ is the radius of perturbations around $\theta$. Likewise, a robust empirical risk for the old tasks is $\hat{\mathcal{E}}_{D_{1:t-1}}^{b}(\theta):=\rm{max}_{\lVert\Delta\rVert\leq b}\hat{\mathcal{E}}_{\textit{D}_{1:t-1}}(\theta+\Delta)$. Then, $\min_{\theta\in\Theta}\hat{\mathcal{E}}_{{D}_{t}}^{b}(\theta)$ can find a flat minima over the current task. However, parameter changes that are much larger than the “radius” of the old minima can interfere with the performance of old tasks, while staying around the old minima can interfere with the performance of new tasks. Therefore, it is necessary to find a solution that can properly balance memory stability with learning plasticity, while being adequately compatible with the observed distributions. Formally, we analyze the generalization errors of a certain solution for continual learning with PAC-Bayes theory mcallester1999pac , so as to provide the objective for computational modeling of neurological adaptive mechanisms. We leave technical details to Supplementary Section 1.3 and present the main results below:

From Proposition 1, the generalization gaps over new and old tasks, corresponding to learning plasticity and memory stability, are uniformly constrained by the loss flatness and task discrepancy, which further depend on the cover of parameter space, i.e., $C_{1}$ and $C_{2}$. In particular, we have $C_{1}=2\sqrt{\frac{d\ln(N_{1:t-1}/d)+\ln(2/\delta)}{N_{1:t-1}}}$ and $C_{2}=2\sqrt{\frac{d\ln(N_{t}/d)+\ln(2/\delta)}{N_{t}}}$ for $K=1$. When $K>1$, $C_{1}<2\sqrt{\frac{d\ln(N_{1:t-1}/d)+\ln(2/\delta)}{N_{1:t-1}}}$ and $C_{2}<2\sqrt{\frac{d\ln(N_{t}/d)+\ln(2/\delta)}{N_{t}}}$ due to $d_{i}<d$ for $i\in[1,K]$. This means that employing multiple continual learners (i.e., $K>1$) compared to a single continual learner (i.e., $K=1$) can tighten the generalization bounds and thus benefit the performance of continual learning. Likewise, the benefits of active forgetting can be explained from a similar perspective:

Proposition 2 externalizes the two generalization bounds in Proposition 1 by defining the VC dimension with $L_{2}$ norm for parameters under some technical assumptions. Notably, the claim in Proposition 1 regarding the benefits of using multiple continual learning still holds in such a tightened version. In particular, optimization of the active forgetting term in Eq. (2) and Eq. (12), which takes the form of minimizing a weighted $L_{2}$ norm regarding all parameters, can contribute to tightening the two generalization bounds in Proposition 2, especially through optimizing $C_{1}$ and $C_{2}$. Besides, $N_{1:t-1}$ becomes larger as $t$ increases, while $N_{t}$ remains constant in general. This means that $C_{1}$ will decrease more rapidly around the empirical optimal solution $\hat{\theta}_{1:t}^{b}=\arg\min_{\theta\in\Theta}\hat{\mathcal{E}}_{D_{t}}^{b}(\theta)$ as more tasks are introduced, resulting in more pronounced improvements in learning plasticity.

We mainly consider the setting of task-incremental learning van2022three to perform continual learning experiments, with task identities provided in both training and testing. We split representative benchmark datasets for visual classification tasks. The CIFAR-100 dataset (krizhevsky2009learning, ) includes 100-class colored images of size $32\times 32$. We split it based on different principles to evaluate the effect of overall knowledge transfer. Specifically, R-CIFAR-100 and S-CIFAR-100 are constructed by splitting CIFAR-100 into 20 tasks, depending on random order or superclasses defined by semantic similarity, respectively. R-CIFAR-10/100 includes 2 tasks randomly split from the CIFAR-10 dataset (krizhevsky2009learning, ) of 10-class colored images, followed by the 20 tasks of R-CIFAR-100. The Omniglot dataset lake2015human includes 50 alphabets for a total of 1623 classes of characters, where each class contains 20 hand-written digits of size $\ignorespaces 28\times 28$ . We split each alphabet as a task consisting of a different number of classes. The CUB-200-2011 dataset wah2011caltech includes 200-class bird images of size $224\times 224$, and the Tiny-ImageNet dataset delange2021continual includes 200-class natural images of size $\ignorespaces 64\times 64$ , both split randomly into 10 tasks. The CORe50 dataset lomonaco2017core50 includes 50 handheld objects with smoothly-changed observations of size $128\times 128$, randomly split into 10 tasks pham2021contextual . We construct a sequence of Atari reinforcement tasks for continual learning, i.e., DemonAttack - Robotank - Boxing - NameThisGame - StarGunner - Gopher - VideoPinball - Crazyclimber, using the same PPO algorithm schulman2017proximal to learn each task. The network architecture and training regime are described in Supplementary Sections 2.1 and 2.2, respectively.

To ensure generality in realistic applications, we restrict the old training samples to be unavailable in continual learning, and compare with representative methods that follow this restriction. Specifically, EWC kirkpatrick2017overcoming , MAS aljundi2018memory and SI zenke2017continual are synaptic regularization methods that selectively penalized parameter changes to preserve memory stability. AGS-CL jung2020continual took advantages of parameter isolation and synaptic regularization to prevent catastrophic forgetting. P&C schwarz2018progress adopted an additional active column on the basis of EWC kirkpatrick2017overcoming to improve learning plasticity. CPR cha2020cpr encouraged convergence to a flat loss landscape, which can be combined with other baseline approaches. The hyperparameters for continual learning are determined with a comprehensive grid search. We construct a different task sequence (e.g., different class splits, data shuffling, task orders and random seeds) from the actual experiments and run it for once. Then we use the best combinations of hyperparameters to perform the actual experiments for multiple runs, as described in Supplementary Section 2.3.

We consider three evaluation metrics for visual classification tasks, i.e., average accuracy (AAC), forward transfer (FWT) and backward transfer (BWT) lopez2017gradient ; wang2023comprehensive :

where $\ignorespaces{\rm{A}}_{t,i}$ is the test accuracy of task $\ignorespaces i$ after continual learning of task $\ignorespaces t$ , and $\ignorespaces\tilde{a}_{i}$ is the test accuracy of each task $\ignorespaces i$ learned from random initialization. ACC is the average performance of all tasks ever seen, which evaluates the overall performance of continual learning. FWT evaluates the average influence of remembering old tasks to new tasks for learning plasticity. BWT evaluates the average influence of learning new tasks to old tasks for memory stability.

The diversity of learners’ predictions is quantified by the average cosine (Cos) or Euclidean (Euc) distance:

where $p_{i}$ and $p_{j}$ denote the predictions of learners $i$ and $j$, respectively.

The discrepancy of task distributions in feature space is evaluated by the difficulty of distinguishing them, which is an empirical approximation of the $\mathcal{H}$-divergence long2015learning ; wang2022coscl in Proposition 1. We train a simple discriminator consisting of a fully-connected layer and use binary cross-entropy to measure the average discrimination error on test sets.

The performance of Atari reinforcement tasks is evaluated by the normalized accumulated reward (NAR), normalized plasticity (NP) and normalized stability (NS), corresponding to the overall performance, learning plasticity and memory stability, respectively:

where ${\rm{R}}_{t,i}$ is the reward for task $i$ obtained in the test step after learning task $t$, and $r_{i}$ is the maximum reward for task $i$ obtained in each test step of fine-tuning on the task sequence.