Continual Learning with Strong Experience Replay

Rehearsal-based methods [26, 29, 3, 39, 43] reduce catastrophic forgetting by replaying a subset of previously seen samples stored in a memory buffer. Experience Replay (ER) [27] is a simple but effective rehearsal-based method that jointly trains the model with current data and a mini-batch of randomly selected old samples. Besides, it applies a reservoir sampling strategy to update the memory buffer over time. Based on the core idea of ER, recent CL approaches further reduce forgetting with various techniques. For example, GSS [1] stores optimally chosen examples in the memory buffer. GEM [24] and AGEM [12] leverage episodic memory to avoid forgetting and favor positive backward transfer. ERT [8] adopts a balanced sampling method and bias control. MER [27] considers experience replay as a meta-learning problem to maximize transfer and minimize interference. iCaRL [26] uses a nearest-mean-of-exemplars classifier and an additional memory buffer for model training. Besides, it adopts a knowledge distillation method to reduce forgetting further. RM [3] develops a novel memory management strategy based on per-sample classification uncertainty and data augmentation to enhance the sample diversity in the memory. DER++ [7] mixes rehearsal with knowledge distillation and regularization. CLS-ER [2] adopts a dual-memory experience replay method to maintain short-term and long-term semantic memories. LVT [38] designs a vision transformer for continual learning with replay. SCoMMER [31] enforces activation sparsity along with a complementary semantic dropout mechanism to encourage consistency. Different from these methods that distill past experiences from a limited memory buffer only, we also explore future experiences mimicked on the current training data, which improves the model generalization on previous tasks.

Regularization-based methods usually incorporate an additional penalty term into the loss function to prevent model changes in parameter or prediction spaces [17]. Elastic Weight Consolidation (EWC) [19, 32], Synaptic Intelligence (SI) [44], and Riemmanian Walk (RW) [10] prevent the parameter changes between the new model and the old one. LwF [23] and PASS [46] mitigate forgetting with task-specific knowledge distillation. Without any previous data for replay, recent studies [7, 25] show that these methods usually fail to handle class-incremental scenarios. Therefore, the regularization-based strategy is often simultaneously used with other CL approaches for better performance, such as iCaRL [26] and DER++ [7, 6]. In this work, besides memorizing ground truth labels of previous samples, we also seek consistent logit distributions. Therefore, we incorporate two consistency losses as model regularization to reduce forgetting.

The early CL methods usually adopt a task identity to learn task-specific knowledge. For example, PNN (Progressive Neural Networks) [30] instantiates new networks incrementally and adopts a task identity at both training and inference times. However, identity might be unknown in inference time, limiting its applications. Recently, DER [42] expanded a new backbone per incremental task without using task identities during the testing time. FOSTER [37] adds an extra model compression stage to maintain limited model storage. L2P and Dualprompt [41, 40] adopt a prompt pool to learn and extract task-specific knowledge. In addition, for rehearsal-based methods, storing raw samples is impossible sometimes due to privacy concerns, some generative approaches [33, 34] attempt to use a data generator (such as GAN [13]) to produce synthetic data for replay. But it needs a long time for model training and the generated data might be unsatisfactory for complex datasets. Additionally, to distill more knowledge from old tasks, DMC [45] and GD [22] sample external data to assist model training. Although much progress has been achieved in recent years, efficient CL with less forgetting remains a challenging problem.

Formally, given a sequence of $T$ nonstationary datasets $\{\mathcal{D}_{1},\cdots,\mathcal{D}_{T}\}$, $(x,y)\in\mathcal{D}_{t}$ represents the samples $x$ with corresponding ground truth labels $y$ of the $t$-th task. At a time step, $t\in\{1,\cdots,T\}$, the goal of a CL method is to sequentially update the model on $\mathcal{D}_{t}$ without forgetting the previously learned knowledge on $\{\mathcal{D}_{1},\cdots,\mathcal{D}_{t-1}\}$.

In general, when all training data is available in advance, the loss function of conventional learning can be formulated as:

where $\ell_{ce}$ denotes cross-entropy loss and $f(x;\theta)$ denotes the network’s output logits, $\theta$ represents the model parameter to be optimized.

By contrast, for the $t$-th task with training data $\mathcal{D}_{t}$ in CL, given a trained model with the initialized parameter $\theta_{t-1}$, the conventional model finetuning can be achieved by minimizing a classification loss function as:

However, when the previous data is not accessible, simply updating the model with Equation 2 usually leads to a drastic performance drop on previous datasets $\{\mathcal{D}_{1},\cdots,\mathcal{D}_{t-1}\}$, i.e. catastrophic forgetting.

ER [27] is a classical rehearsal-based strategy that alleviates forgetting by replaying a subset of previously seen data stored in a permanent memory buffer $\mathcal{M}$. By jointly training the model on current training data $\mathcal{D}_{t}$ and previous data stored in $\mathcal{M}$, the training loss is computed as:

where $\mathcal{L}_{cls}^{m}$ is a classification loss function on the samples stored in the memory buffer and it is used to preserve the acquired knowledge as:

Theoretically, the larger the memory buffer, the more knowledge would be preserved. For the extreme case, when all the previous data is stored in $\mathcal{M}$, it is equivalent to the joint training in one task. Unfortunately, due to privacy or storage concerns, the previous data might be unavailable in practice, storing a subset of previous data in $\mathcal{M}$ reduces the difficulty of CL.

To leverage the limited memory buffer effectively, ER adopts a reservoir sampling method [36] to update the buffer over time. In this way, when we randomly select $|M|$ samples from the input stream, each sample still has the same probability $\frac{|M|}{|S|}$ of being stored in the memory buffer without knowing the length of the input data stream $|S|$ in advance. Although ER alleviates the forgetting issue effectively and outperforms many elaborately designed CL approaches [7], its performance may drastically drop when few samples are available [35].

To further mitigate the forgetting issue, various ER extensions [7, 41, 2, 40] are developed to preserve learned knowledge when learning new tasks. However, these methods may overfit the memory buffer when a few samples are available. In this work, we mix the rehearsal-based strategy with model regularization and we extend ER by incorporating two consistency losses, which helps the CL model consistently evolve during training. Ideally, if an updated model still has the same outputs as its original ones when learning new tasks, it can be considered that the previously learned knowledge is not forgotten. However, this goal cannot be achieved in practice because of disjoint class labels in class-incremental scenarios. Instead, we aim to obtain approximate output logits between the new model and the old one.

Our method consists of a backward consistency loss and a forward consistency loss. Specifically, the backward consistency $\mathcal{L}_{bc}^{m}$ is used to distill the model’s past experiences on the data stored in the memory buffer. On the other hand, the forward consistency $\mathcal{L}_{fc}^{t}$ is used to distill the model’s future experiences on the current training data, as the new data is from the future for the old model. An overview of the proposed SER method is illustrated in Figure 2. Unlike the backward consistency used in DER++ [7], forward consistency can leverage all training data and it can propagate consistent predictions to new tasks over time. Details of these two consistency losses are as follows.

Backward consistency. For the $t$-th task, given a trained model with initialized parameter $\theta_{t-1}$, we measure the backward consistency between two models on the memory buffer $\mathcal{M}$ with a Mean Square Error (MSE) loss:

As discussed in DER++ [7], the optimization of Equation 5 is equivalent to minimizing the KL divergence loss in knowledge distillation [15]. Instead of using smoothing softmax responses, directly using the logits with an MSE loss can avoid information loss occurring in prediction spaces due to the squashing function (e.g., softmax).

In addition, to reduce the computational cost, we store the logits of the network outputs when updating the memory buffer [7]. Let $z=f(x;\theta)$ denote the output logits of past experience on sample $x$, by replacing $f(x;\theta_{t-1})$ with $z$, then Equation 5 can be rewritten as:

To clearly illustrate the core idea of the proposed method, we directly use $f(x^{m};\theta_{t-1})$ in Figure 2, instead of $z$ stored in the memory buffer. Based on the classification loss on the samples stored in the memory buffer and backward consistency loss, the updated model will mimic its training trajectory and its output logits will approximate its original ones, reducing the forgetting issue.

Forward consistency. Intuitively, the previous data stored in the memory buffer $\mathcal{M}$ only contains a small subset of $\{\mathcal{D}_{1},\cdots,\mathcal{D}_{t-1}\}$. When a few previous samples are available, using a backward consistency alone cannot properly make the outputs of the new model approximate its original ones. As a result, preserving the learned knowledge on a limited memory buffer suffers from the risk of overfitting the $M$, leading to the forgetting problem.

To address this issue, we design an additional consistency loss to distill future experiences on $\mathcal{D}_{t}$. In particular, we first compute the output logits of the old model on $\mathcal{D}_{t}$, and then employ an MSE loss as in Equation 6 to measure the new consistency as:

Since data in $\mathcal{D}_{t}$ is from a future task for the old model, we simply refer to $\mathcal{L}_{fc}^{t}$ as a “forward consistency” loss.

As illustrated in Figure 1, the backward consistency retains a subset of past experiences stored in $\mathcal{M}$. In contrast, the forward consistency can propagate the future experiences mimicked on $\mathcal{D}_{t}$ to new tasks over time. Therefore, forward consistency can effectively leverage the complete training data. Moreover, during the model training, the parameters of the old mode are frozen, seeking consistent predictions on the current training data enhances the updated model has a larger overlap with the old model, improving the model generalization on old tasks and further preserving the acquired knowledge.

Total training loss. Based on the above analysis, the proposed SER method combines two classification losses and two consistency losses during model training. Formally, the total loss used in SER is represented as:

where $\alpha$ and $\beta$ are the balancing parameters.

The core challenge in CL is the stability-plasticity dilemma. For the $t$-th task, $\mathcal{L}_{cls}^{t}$ focuses on the plasticity by finetuning the model on the new dataset $\mathcal{D}_{t}$; and the other losses focus on the stability by memorizing the experiences of the old model. Specifically, $\mathcal{L}_{cls}^{m}$ and $\mathcal{L}_{bc}^{m}$ preserve past experiences on the memory buffer $\mathcal{M}$. $\mathcal{L}_{fc}^{t}$ retains the future experiences on the current training data $\mathcal{D}_{t}$. Therefore, compared to the classical ER method, the two consistency losses used in SER effectively improve the stability of the model, and they prevent model changes in prediction spaces and further reduce forgetting.

Simple Implementation. The proposed SER algorithm is described in Algorithm 1, where “aug” denotes the data augmentation. During model training, we randomly sample one batch of samples from the memory buffer and the augmented data is used for classification and backward consistency. After each training batch, a reservoir sampling method is employed to update the memory buffer. To reduce the computational cost, we store the original training data with its output logits in the memory buffer. It can be seen that SER does not need any complex computation, which makes it easy to implement. Besides, notice that we do not use the consistency losses for the first task, as there is no previous knowledge to be preserved.

Datasets. The proposed method is evaluated on 5 image classification benchmark datasets. The CIFAR100 dataset contains 100 classes and each class has 500 images for training and 100 images for testing. CIFAR10 dataset has 10 classes, each consisting of 5000 training images and 1000 test images. The image size of CIFAR10 and CIFAR100 is $32\times 32$. TinyImageNet has 200 classes and it includes 100,000 training images and 10,000 test images in total, and the image size is $64\times 64$. Both Permuted MNIST [19] and Rotated MNIST [24] are built upon the MNIST dataset [21], which has 60,000 training images and 10,000 test images, and the image size is $28\times 28$. Permuted MNIST applies a random permutation to the pixels and Rotated MNIST rotates the digits by a random angle in the interval $[0,\pi)$. In our experiments, CIFAR10, CIFAR100, and TinyImageNet are used to evaluate the performance of Class-IL and Task-IL. Permuted MNIST and Rotated MNIST are used for Domain-IL.

Baselines. We compare the proposed SER method with three rehearsal-free methods, including oEWC [32], SI [44], and LwF [23]. We also compare SER with multiple rehearsal-based methods: iCaRL [26], ER [27], GEM [24], A-GEM [12], FDR [4], GSS [1], HAL [11], DER++ [7], Co2L [9], ERT [8], RM [3], LVT [38], CLS-ER [2], and TAMiL [5]. Notice that CLS-ER [2] updates three models (a plastic model, a stable model, and the current training model) during training. In contrast, SER only updates the current training model (the old model is frozen). To show the effectiveness of CL methods, we further provide two baselines without CL techniques: SGD (a lower bound) and Joint (an upper bound). The results of baselines are obtained from the published papers or source codes. Besides, the best parameter configurations of baselines are used for comparison.

Metrics. We evaluate the continual learning methods with two metrics: final average accuracy and average forgetting. Let $a_{T,t}$ denote the testing accuracy on $t$-th task when the model is trained on $T$-th task, and the final average accuracy $A_{t}$ on all $T$ tasks is computed as:

Besides, the average forgetting on $T$ tasks is defined as:

Architectures. We employ the same network architectures used in DER++ [7]. For CIFAR10, CIFAR100, and Tiny ImageNet, we adopt a modified ResNet18 without pretraining. For the MNIST dataset, we adopt a fully connected network with two hidden layers each one comprising 100 ReLU units.

Augmentation. For fairness, we use the same data augmentation as in DER++ [7], which applies random crops and horizontal flips to both stream and buffer examples. It is worth mentioning that small data transformations enforce implicit consistency constraints [7].

Training. Following the training strategy in DER++ [7], we employ a Stochastic Gradient Descent (SGD) optimizer in all experiments. For the CIFAR10 dataset, we train the model with 20 epochs, and CIFAR100 with 50 epochs, respectively. For the Tiny ImageNet dataset, we increase the number of epochs to 100. Besides, we only use one epoch for MNIST variants. The learning rate and batch size are the same as in DER++ [7]. In addition, we employ a MultiStepLR to decay the learning rate by parameter 0.1. The milestones are as follows. CIFAR10: [15]; CIFAR100: [35, 45]; TinyImageNet: [70, 90].

Due to the dynamic learning scenarios in continual learning, it is very difficult to adjust the hyper-parameter automatically. Therefore, we follow the previous methods [7, 2, 5] that tune the balancing parameters for each dataset. In our experiments, the balancing parameters are as CIFAR10 ($\alpha=0.2$, $\beta=0.2$), CIFAR100 ($\alpha=0.5$, $\beta=0.5$), and TinyImageNet ($\alpha=0.2$, $\beta=1$), P-MNIST and R-MNIST ($\alpha=0.2$, $\beta=0.2$). Other parameter configurations may obtain better performance.

Evaluation on CIFAR100. We follow the experimental setting in LVT [38] to evaluate the proposed method in Class-IL and Task-IL scenarios. Table 1 reports the performance of each CL method on CIFAR100 with 5, 10, and 20 tasks and each task has disjoint class labels. It can be seen that all the rehearsal-free methods fail to reduce forgetting on the CIFAR100 dataset when compared to SGD (a low-bound). Although LwF seeks consistent predictions with knowledge distillation, without the help of previous samples for replaying, the learned model severely drifts over time. In contrast, by storing a small number of previously seen samples in a memory buffer, rehearsal-based methods can effectively preserve the learned knowledge.

Among the rehearsal-based methods, SER outperforms DER++ by a large margin on all tasks. For example, given a memory buffer with 200 samples for Class-IL, SER outperforms DER++ by 20.5% on 5 tasks and 9.2% on 20 tasks. Besides, for Task-IL with 10 tasks, SER achieves a 16.3% accuracy improvement. Figure 3 and Figure 4 plot the average accuracy of ER, DER++, and SER on CIFAR100 when learning each task. It can be seen that SER substantially improves the accuracy in both Class-IL and Task-IL for each learning stage. By adding a forward consistency loss, DER++ surpasses ER by a noticeable margin in both Class-IL and Task-IL. Moreover, with the help of a forward consistency loss, SER substantially outperforms DER++ on CIFAR100 with different lengths of tasks.

Even compared to the most recent LVT method in Class-IL, SER surpasses it by a large margin of 8.3% on 5 tasks and 3.0% on 20 tasks when 200 previous samples are stored. Moreover, SER obtains at least 6% accuracy improvement in Task-IL. Furthermore, given a larger memory buffer with 500 samples, it can be observed that SER also obtains consistent accuracy improvement. These results demonstrate that SER can effectively preserve preserves the learned knowledge with a strong experience replay.

Evaluation on CIFAR10. Table 2 shows the performance of each CL method on the CIFAR10 dataset (5 tasks). Similar to the results observed in Table 1, rehearsal-based methods surpass the rehearsal-free methods by a large margin and our SER achieves the best performance among compared algorithms. CIFAR10 is a small dataset that contains 10 classes. When the memory size is 200, the average number of stored samples for each class is 20. In contrast, for the CIFAR100 dataset with 100 classes, there are only 2 samples for each class. Therefore, by storing more samples for each class, the average accuracy on CIFAR10 is much better for all CL approaches.

Compared to DER++, SER further reduces the forgetting issue and boosts performance with the forward consistency loss. For example, SER outperforms DER++ by 5% and 3.7% in Class-IL when the memory size is 200 and 500, respectively. Notice that CLS-ER is a dual memory experience replay architecture, which computes a consistency loss on the data stored in the memory buffer and maintains short-term and long-term semantic memories with episodic memory. By contrast, our SER incorporates a forward consistency loss on current training data only. Compared to CLS-ER on CIFAR10, SER improves the accuracy by 3.7% when the memory size is 200. Furthermore, although SER does not employ the task identities during model training, it slightly outperforms the most recent method TAMiL by 1%.

Evaluation on TinyImageNet. In Table 2, the TinyImageNet data is divided into 10 tasks and each task has 20 classes. TinyImageNet is more challenging than CIFAR10 and CIFAR100 for CL when a few previous samples are available. As the results reported in Table 2, given a limited memory buffer with a buffer size of 200, which indicates that there is only 1 sample for each class on average. Compared to the low-bound baseline SGD (7.92% in Class-IL), all rehearsal-free methods and most rehearsal-based methods cannot effectively reduce forgetting with a noticeable margin. Because there are a few samples for each class only, it is very difficult to retain the acquired knowledge from limited past experiences. Therefore, these rehearsal-based methods fail to reduce forgetting.

By mimicking the future experiences of the old model on current training data, SER can distill additional knowledge and further reduce forgetting. From Table 2, we can observe that SER substantially improves the performance on TinyImageNet when compared to most previous CL methods. In particular, the average accuracy of SER is much better than DER++ in Class-IL, i.e. 28.50% vs. 10.96%. For Task-IL, SER significantly outperforms DER++ with an absolute improvement of 29.25%. Compared to another recent method Co2L, SER also obtains 14.6% accuracy improvement in Class-IL and more than 27.7% in Task-IL. Moreover, SER surpasses CLS-ER and TAMiL by a noticeable margin of 5.03% and 8.04% in Class-IL, respectively. These results suggest that SER can effectively preserve the learned knowledge when few samples are available for experience replay.

Evaluation on P-MNIST and R-MNIST. To evaluate the performance of the proposed method in Domain-IL, we follow the experimental settings in DER++[7], which adopts 20 tasks for Permuted MNIST (P-MNIST) and Rotated MNIST (R-MNIST). As the average accuracy reported in Table 2, based on a memory buffer with 500 samples, most rehearsal-based methods outperform the rehearsal-free methods (except A-GEM) on both two datasets. However, by reducing the memory size to 200, the performance of ER is slightly worse than oEWC on P-MNIST. In contrast, DER++ and SER are also effective in Domain-IL by seeking consistent predictions. Besides, CLS-ER achieves the best performance on P-MNIST and R-MNIST and it is comparable to SER and DER++ on both two datasets.

Based on these observations, it can be concluded that the proposed forward consistency is more effective in Class-IL and Task-IL than Domain-IL. The reason is that the category spaces in Class-IL and Task-IL are disjoint while it is the same in Domain-IL. Therefore, seeking consistency with the current training data may overwrite the previous knowledge and it does not further mitigate forgetting.

Average Forgetting. Besides average accuracy, forgetting is another important metric to measure the performance of a CL method. Table 3 shows the average forgetting of ER, DER++, and our SER. It can be observed that SER achieves the lowest forgetting in all the comparison experiments, including both Class-IL and Task-IL on two datasets. These results indicate that SER effectively reduces the forgetting issue, verifying the effectiveness of our proposed forward consistency loss.

To further analyze the effectiveness of each loss in our method, we report the ablation study results in Table 4. Without storing any previous samples, simply finetuning the model with $\mathcal{L}_{cls}^{t}$ on current training data nearly forgets all previous knowledge in Class-IL, which is only 19.26% on CIFAR10 and 4.73% on CIFAR100. By replaying a small number of previous samples, ER effectively reduces the forgetting issue on both two datasets with $\mathcal{L}_{cls}^{t}+\mathcal{L}_{cls}^{m}$. Besides, based on the backward consistency loss $\mathcal{L}_{bc}^{m}$, DER++ further reduces forgetting by seeking consistency predictions on the data stored in the memory buffer. Furthermore, fusing two classification losses and two consistency losses, i.e. $\mathcal{L}_{cls}^{t}+\mathcal{L}_{cls}^{m}+\mathcal{L}_{bc}^{m}+\mathcal{L}_{fc}^{t}$, SER achieves the best performance on all the comparison experiments.

It is worth mentioning that simply extending ER with a forward consistency loss $\mathcal{L}_{fc}^{t}$ alone also surpasses DER++. For example, from the results on CIFAR10 reported in Table 4, the average accuracy is improved from 64.88% to 69.04% on Class-IL and 91.92% to 95.79% on Task-IL. These observations indicate that seeking consistent predictions on current training data can also help to preserve the acquired knowledge. Moreover, the forward consistency can be computed on the complete training data and it can propagate the consistent distributions to new tasks over time. Therefore, the forward consistency loss demonstrates better performance when a small memory buffer is given.

To better understand how well different CL methods strike a balance between stability and plasticity, we report the performance of several CL methods after sequentially learning 5 tasks on the CIFAR10 dataset in Figure 5. In terms of average accuracy, DER++, CLS-ER, and SER outperform ER with the help of consistency loss, showing the effectiveness of stability. On the other hand, ER achieves better accuracy on the current training data, and it outperforms other methods on plasticity. Based on the proposed forward consistency, SER improves the model generalization on previous tasks and it maintains a better balance. Compared to DER++ and CLS-ER, SER preserves more knowledge of previous tasks and it also achieves good performance on the recent task. Therefore, SER outperforms the compared methods with the metric of average accuracy, verifying its effectiveness.

Based on the same backbone, the computational cost of different CL methods relies on the efficiency of model training. Since we adopt the less or the same training epochs as the baseline methods [7, 2], we mainly analyze the computational cost by comparing the model design. As illustrated in Algorithm 1, besides the basic model optimization on the current training data, we draw one batch of samples from the memory buffer for both the experience replay and backward consistency at the same time. In contrast, DER++ [7] draws two batches of samples from the memory buffer, where one batch of samples is for the experience replay while the other is for backward consistency. Therefore, although we incorporate a forward consistency loss by computing the output logits of the old model on the current training data, SER does not increase the computational cost when compared to DER++. By contrast, CLS-ER [2] adopts a working model for experience replay, and it additionally computes the consistency with the memory buffer by a stable model and a plastic model. Compared to the computational cost of CLS-ER, the proposed SER method is more efficient.

The proposed SER method mixes the experience replay and model regularization in CL. Compared to the previous rehearsal-based methods, we additionally design a forward consistency loss to improve the model generalization on previous tasks. Since forward consistency can leverage more data to reduce drastic model changes during training and propagate the past logit distributions to new tasks over time, the forgetting issue is effectively reduced. Moreover, the proposed method is easy to implement, and its computational cost is not expensive when compared to the state-of-the-art methods, e.g., DER++ and CLS-ER. Therefore, the proposed SER method is simple and effective. We hope it can improve the incremental learning ability of other deep models in the future.