ER-FSL: Experience Replay with Feature Subspace Learning
for Online Continual Learning

Since continual learning generally exists in various scenarios (Hu et al., 2022; Zhang et al., 2022c; Yang et al., 2022; Zhang et al., 2022a) of deep neural networks, its related research is quite extensive. In addition to the work related to the innovation and application of continual learning methods, the analysis (Pham et al., 2021; Mirzadeh et al., 2020) and overview (Masana et al., 2022; Mai et al., 2022) work also have attracted much attention.

Continual learning (De Lange et al., 2021; Masana et al., 2022), also known as lifelong learning (Liu and Mazumder, 2021) or incremental learning (Wang et al., 2023b), is a machine learning paradigm that trains models on a continuous stream of new data. It ensures the model’s generalization ability (Ghunaim et al., 2023; Brahma and Rai, 2023) and anti-forgetting ability (Dong et al., 2023, 2021) at the same time. Existing methods can be generally divided into three categories. 1) Architecture-based methods (Qin et al., 2021; Yan et al., 2021; Hu et al., 2023) overcome CF problem by dynamical networks or static networks. dynamic networks imply that the model’s network structure gradually expands with the increasing number of samples during the continual learning process, while static networks maintain their structure unchanged and allocate parameters selectively. 2) Regularization-based methods (Wang et al., 2023a; Pelosin et al., 2022; Akyürek et al., 2021) constrain the optimization process of the model using an additional regularization term. The design of this regularization term can be based on variations in parameters during the training process or on knowledge distillation. 3) Replay-based methods save true old data (Lopez-Paz and Ranzato, 2017; Sun et al., 2023; Luo et al., 2023; Zhou et al., 2022) or generate pseudo old data (Cui et al., 2021; Qi et al., 2022) to replay with new data. And some feature replay methods (Toldo and Ozay, 2022) are proposed. The proposed ER-FSL in this work is a novel replay-based method.

OCL is a specialized area within continual learning that emphasizes effective learning from a single pass through an online data stream, where tasks or information are introduced incrementally over time. It plays a critical role in scenarios requiring continual knowledge evolution and adaptation to new information (Mai et al., 2022).

Existing OCL methods are mainly based on replaying ways except AOP (Guo et al., 2022a). A variety of ER-based methods have been proposed for OCL due to the effectiveness of a replay-based method called Experience replay (ER) (Rolnick et al., 2019). Some approaches are proposed to select more valuable samples for storing (Aljundi et al., 2019b; Jin et al., 2021; He and Zhu, 2021) and replaying (Aljundi et al., 2019a; Shim et al., 2021; Wang et al., 2022; Prabhu et al., 2020). Other approaches (Mai et al., 2021; Caccia et al., 2022; Zhang et al., 2022b; Guo et al., 2022b; Gu et al., 2022; Lin et al., 2023b; Chrysakis, 2023; Wei et al., 2023; Guo et al., 2023; Wang et al., 2023c; Liang and Li, 2023; Lin et al., 2023a) belong to the model update strategy and focus on improving the training process of samples. Both SS-IL (Ahn et al., 2021) and ER-ACE (Caccia et al., 2022) propose different cross-entropy loss functions for learning new data and reviewing old data to alleviate catastrophic forgetting. Subsequently, PCR (Lin et al., 2023b) analyzes and integrates these two types of methods from the perspective of gradient propagation, while LODE (Liang and Li, 2023) approaches the integration from the angle of decomposing the loss function. Both of them significantly enhance the performance of the original methods. Furthermore, the performance of the latest methods (Guo et al., 2022b; Wei et al., 2023) depends on multiple data augmentation operations, since data augmentation helps improve the performance of the model (Zhu et al., 2021).

The proposed ER-FSL introduces a novel model update strategy for OCL. Different from existing strategies that learn and replay within the same feature space, the proposed ER-FSL embeds features in different spaces for current samples and buffered samples. This differentiation allows for a more effective enhancement of the model’s anti-forgetting capabilities compared to existing methods.

Taking a class-incremental scenario as an example, OCL generally considers a single-pass data stream and divides it into a sequence of $T$ learning tasks as $\mathcal{D}=\{\mathcal{D}_{1},...,\mathcal{D}_{T}\}$, where each task $\mathcal{D}_{t}=\{\bm{x},y\}_{1}^{N_{t}}$ contains $N_{t}$ labeled samples. $y\in\mathcal{C}_{t}$ is the class label of sample $\bm{x}$, where $\mathcal{C}_{t}$ is the set of task-specific classes. Different tasks contain unique classes, and all of the learned classes are denoted as $\mathcal{C}_{1:t}=\bigcup_{k=1}^{t}\mathcal{C}_{k}$. The model is a neural network, consisting of a feature extractor $\bm{z}=h(\bm{x};\bm{\theta})$ and a classifier $f(\bm{z};\bm{W})=\bm{W}\cdot\bm{z}$ for the sample $\bm{x}$. $\bm{z}=[z_{1},z_{2},...,z_{d}]$ is a $d$-dimensional feature vectors, $\bm{\theta}$ and $\bm{W}=[\bm{w}_{1},\bm{w}_{2},...,\bm{w}_{c}]$ are learnable parameters, and $\bm{w}_{c}=[w_{1}^{c},w_{2}^{c},...,w_{d}^{c}]$ is a $d$-dimensional prototype vector for class $c$. OCL aims to train a unified model on data seen only once while performing well on both new and old classes.

At the beginning, the model can only access each mini-batch current samples ${\mathcal{B}\subset\mathcal{D}_{t}}$ once in the training process of each task. Such a training strategy is known as finetune, where the model learns without any anti-forgetting operations. Based on $\bm{z}=h(\bm{x};\bm{\theta})$, its objective loss function can be denoted as

Subsequently, a memory buffer $\mathcal{M}$ is utilized to store a small subset of observed data for replay-based methods, such as ER (Rolnick et al., 2019). To alleviate the forgetting problem, a mini-batch of buffered samples $\mathcal{B}_{\mathcal{M}}\subset\mathcal{M}$ is drawn from the memory buffer and then trained alongside current samples. Therefore, the loss function defined as Equation (1) can be improved to

Finally, for each unknown sample $\bm{x}$, the model categorizes it as the class with the highest prediction probability

To further address the forgetting problem of the model, we analyze existing methods, explore their shortcomings, and find corresponding solutions. Specifically, we divide the CIFAR10 dataset that contains 10 classes into two tasks, each containing 5 classes, and conduct OCL analysis experiments. The model used in the experiments is Resnet18 (He et al., 2016), where the dimension of $\bm{z}$ is 512. All analysis results are demonstrated in Figure 2.

The imbalanced data, through gradient descent, results in the model focusing more on features of distinguishing new classes and ignoring features of recognizing old classes. In the class-incremental scenario, the CF phenomenon of the model is manifested as the biased prediction $\bm{W}\cdot\bm{x}$, where the learned model tends to classify most samples into new classes. Specifically, when training a sample $\bm{x}$ of class $y$, the gradient of feature extractor can be expressed as

where $c\neq y$. It makes the feature $\bm{z}$ of the sample to be closer to the prototype $\bm{w}_{y}$ of class $y$ while keeping away from the prototypes $\bm{w}_{c}$ of other classes. As a result, the inner-product $\bm{w}_{y}\cdot\bm{z}$ will be larger than others $\bm{w}_{c}\cdot\bm{z}$ in the prediction. By decomposing $\bm{w}_{c}\cdot\bm{z}$ into $[w_{1}^{c}z_{1},w_{2}^{c}z_{2},...,w_{d}^{c}z_{d}]$, we find that the value of each dimension represents a certain feature and its importance to the sample. The greater the importance of a feature, the higher its corresponding value, and consequently, the greater its contribution to the prediction. When training the model with Equation (1), all gradients are generated by the new classes. The model can only focus on features that distinguish new classes and ignore features related to old classes. Although this situation can be alleviated using Equation (2), the changing of features related to old classes is inevitable. Since the number of current samples is still higher than the number of buffered samples, the gradient is primarily influenced by new classes within the same feature space.

To validate this view, we calculate the decomposed inner-product between the features of buffered samples with the prototypes of old classes (red) and new classes (blue). Figure 2 (a) shows the results after the model learning the first task. It can be seen that the values of the decomposed inner-product for old classes (red) are larger, and the model can recognize most of the buffered samples. Besides, Figure 2 (b) and (c) show the results of completing the learning of the second task in a finetune way (Equation (1)) and in an ER way (Equation (2)), respectively. If the model learns the second task using finetune, the gradient is produced by current samples from new classes. The original features (red) the model learned are changed, and the model pays more attention to the features related to new classes (blue). Hence, most of the buffered samples belonging to old classes are classified into new classes due to the biased prediction. Although existing replay-based methods such as ER can improve the values for old classes (as seen in Figure 2 (c)), the average value for new classes (0.0013) is still higher than the one for old classes (-0.0016). Therefore, simply learning and replaying in the same feature space is not beneficial for the model to address the forgetting problem. This motivates us to investigate a new question: Why not learn current samples and replay buffered samples in different feature spaces?

Analysis for Learning. Learning current samples across the feature whole-space is not necessary. As illustrated in Figure 2 (a) and (b), only a subset of the features are useful for new classes within the feature whole-space. This is due to $\bm{w}_{c}\cdot\bm{z}=\sum_{i=1}^{d}w^{c}_{i}z_{i}$, where the significance of these features will be smoothed, potentially impacting model performance, especially in larger feature spaces. To address this issue, we introduce a scaling factor to regulate the dimensions of feature and prototype vectors in a fine-tuned manner. The model’s performance on new classes for each task is reported in Figure 2 (e). The findings show that reducing the dimensions of the feature space through factors of different scales, the model’s performance on new classes will be significantly improved. Meanwhile, the smaller the scale of the feature space, the smaller the fluctuation of the model.

Analysis for Replaying. Replaying buffered samples with a larger feature space than the one for learning can improve the ability of anti-forgetting for the model. As seen in Figure 2 (b) and (c), due to imbalanced data, features related to old classes generally exhibit a phenomenon of weaker importance. With larger feature space, the model can memorize more features associated with old classes in the additional space, further improving the performance of old classes. We use different feature subspaces for the model to learn current samples while replaying in the feature whole-space, and the results are stated in Figure 2 (f). The results demonstrate that with a higher dimensional feature space, the accuracy of the model on old classes has been effectively improved. As the scale changes, there is little room for improvement in this performance. This is because, for old data, too many features are not necessary either. Furthermore, we also calculate its decomposed inner-product for buffered samples when the scale is 0.3 and report the results in Figure 2 (d). As seen in the left part, these features (indexes 1-150) are used for distinguishing new classes, since the values for new classes are larger. In the right part, these additional features (indexes 151-512), which are used to replay buffered samples, tend to correctly categorize most buffered samples as old classes.

Summary of Analyses. After conducting these analyses, we have synthesized our key findings as follows: (1) Learning current samples and replaying buffered samples within the same feature space is adverse to overcoming the forgetting problem. (2) Learning current samples within a feature subspace is sufficient to ensure the generalization ability of the model. (3) Replaying buffered samples within a larger feature space can leverage more features associated with old classes, thereby improving the anti-forgetting ability of the model. Therefore, adopting a strategy of learning within a feature subspace while replaying within a larger feature space presents a viable approach to enhancing the model’s performance.

The setting of a memory buffer ($\mathcal{M}$) is critical for the model’s performance in OCL. First, the size of the memory buffer is fixed throughout the entire training process of OCL. Second, reservoir sampling is used to screen current samples and determine whether they are stored in the memory buffer. A random sampling algorithm can extract a portion of samples from a large set and ensure that the probability of selecting each sample is equal. Third, random sampling retrieves buffered samples from the buffer for replaying.

The training phase of ER-FSL plays a crucial role in maintaining the model’s generalization ability and anti-forgetting capability. The model can not only quickly learn novel knowledge from current samples, but also ensure the retention of historical knowledge using buffered samples as much as possible. Its objective function is

where $\gamma$ is a scale factor to balance a learning component $L_{c}$ and a replaying component $L_{b}$. It encompasses a balanced optimization approach that incorporates learning novel knowledge while preserving historical knowledge.

The learning component $L_{c}$ is a general cross-entropy loss function only associated with current samples. It learns current samples of new classes using a feature subspace, where the novel knowledge of distinguishing new classes is saved in the subspace. Its loss function can be denoted as

Here, $\bm{z}^{s}=\bm{z}\cdot\bm{S}$ is the embedding of sample $x$ and $\bm{w}_{c}^{s}=\bm{w}_{c}\cdot\bm{S}$ is the prototype of class $c$ in the feature subspace (the blue elements in Figure 3). $\bm{S}$ is the diagonal matrix as

It means that ER-FSL divides a $d$-dim feature space into $T$ $k$-dim subspaces, where each subspace is used to learn task $t$.

However, the overall size of the model’s feature space is typically fixed, even as the number of new tasks increases. After a certain number of new tasks, the model cannot allocate a blank subspace for learning additional tasks. Hence, it is necessary to select a portion of space from the previously learned space for new tasks.

Based on the classifier $\bm{W}\cdot\bm{z}$, the contribution of the parameters $\bm{W}$ for the $d$-th dimension is $[w_{d}^{1}z_{d},w_{d}^{2}z_{d},...,w_{d}^{c}z_{d}]$. It equals to $[w_{d}^{1},w_{d}^{2},...,w_{d}^{c}]\cdot z_{d}$, where $[w_{d}^{1},w_{d}^{2},...,w_{d}^{c}]$ is the $d$-th dimension of $\bm{W}$. If the variance of $[w_{d}^{1},w_{d}^{2},...,w_{d}^{c}]$ is larger, the features on this dimension can provide richer information to distinguish between sample classes. It means that the subspaces on dimensions with small variances contribute less and can be selected for learning new tasks. Hence, we denote the subspace reuse mechanism to select the subspace when there is no blank subspace as

where $\mathcal{K}$ is a subset of feature space indexes. For all elements $\mathcal{K}[j]$ ($j\in[1,k]$) in the $\mathcal{K}$, the $\mathcal{K}[j]$-th dimension of $\bm{W}$ has the top $k$ smallest variance.

The replaying component $L_{b}$ is also a general cross-entropy loss function only related to buffered samples. It replays buffered samples using an accumulated feature space, which consists of all learned feature space. The model uses the additional space to store features related to old data, improving the model’s memory level of old knowledge. The loss function is denoted as

where $\bm{z}^{a}=\bm{z}\cdot\bm{A}$ is the embedding of sample $x$ and $\bm{w}_{c}^{a}=\bm{w}_{c}\cdot\bm{A}$ is the prototype of class $c$ in the accumulated feature space (the green and blue elements in Figure 3). $\bm{A}$ is the diagonal matrix as

The testing component predicts testing samples by the learned model. Similar to the replaying component, each testing sample obtains its class probability distribution in the used feature whole-space. And it can be classified as

The process of this framework is described in Algorithm 1. To begin with, a fixed-size memory buffer is used to save current samples (line 10) and replay previous samples (line 5). Then, the continual training module (lines 5-9) overcomes the forgetting problem by learning current samples and replaying previous samples in different feature spaces. Finally, the continual testing module (lines 14-16) predicts unknown instances by the accumulated feature space.

For clarity, we illustrate the subspace and accumulated space for different tasks in Figure 4. For the first task, $\bm{z}^{s}$ and $\bm{z}^{a}$ are shown as the green elements. Similarly, for the second task, $\bm{z}^{s}$ is shown as the blue elements, and $\bm{z}^{a}$ is shown as the concatenation of the green and blue elements. Besides, given the third task, $\bm{z}^{s}$ is shown as the gray elements, and $\bm{z}^{a}$ is shown as the concatenation of the green, blue, and gray elements. Finally, given the fourth task, ER-FSL adopts a subspace reuse mechanism since no new subspace is available for new data. Hence, $\bm{z}^{s}$ is shown as the orange elements.

Evaluation Datasets. We conduct experiments on three datasets. Split CIFAR10 (Krizhevsky et al., 2009), which is divided into 5 tasks, with each task comprising 2 classes; Split CIFAR100 (Krizhevsky et al., 2009), and Split MiniImageNet (Vinyals et al., 2016), both organized into 10 tasks, each consisting of 10 classes.

In this section, we conduct experiments to compare the overall performance of ER-FSL with various state-of-the-art baselines. We aim to gain insights into the strengths and weaknesses of ER-FSL.

Table 1 demonstrates the final average accuracy for three datasets. All reported scores are the average score of 10 runs with a 95% confidence interval. The results evidence that our proposed ER-FSL achieves the best overall performance. Specifically, ER-FSL achieves the best performance under 10 of the 12 experimental scenarios. It has the most outstanding performance on Split CIFAR100 and Split MiniImageNet. For example, ER-FSL outperforms the strongest baseline PCR with a gap of 1.6%, 0.9%, 1.6%, and 2.2% on Split CIFAR100 when the size of the memory buffer is 500, 1000, 2000, and 5000, respectively. We note that ER-FSL is not optimal on Split CIFAR10 when the buffer size is smaller. Since there are fewer classes in this dataset and fewer samples in the buffer, and OCM addresses this issue using additional data augmentation operations.

We also compare ER-FSL with CBA (Wang et al., 2023c) using the experimental setup described in CBA(Wang et al., 2023c). The learning rate is set at 0.03, which differs from the 0.1 used in our work. As shown in Table 2, the results indicate that ER-FSL performs significantly better than CBA.

Figure 5 describes the performance for some effective approaches at each task on all datasets. In the learning process, ER-FSL consistently outperforms other baselines. Especially on Split CIFAR100 and Split MiniImageNet, the performance of ER-FSL becomes increasingly evident as the number of tasks increases. For instance, ER-FSL does not surpass PCR in the initial few tasks but achieves the best in the remaining tasks as shown in Figure 5 (b) and (c).

We conduct ablation experiments to analyze the contribution of various components and choices made in ER-FSL, and the results are stated in Table 3. First, the learning component $L_{c}$ is necessary to ensure the model’s generalization ability. Without it (Index 2), the model’s overall performance is significantly decreased. Besides, the replaying component $L_{b}$ overcomes the forgetting issue, as the accuracy is low when the replaying component is removed (Index 3). Finally, if the model uses the same subspace as $\bm{z}^{s}$ when testing, the model can not memorize anything of old data (Index 4). Therefore, the model should predict unknown instances in the space as $\bm{z}^{a}$.

In the meantime, some subtle settings are also important. First, it is necessary to assign different feature subspaces to each task. If we use a fixed $S$ to select the same subspace, the performance of the model will decrease (Index 5). Second, the spaces used for replaying and learning cannot be inversed. Since the inversion version (Index 6, replaying in a feature subspace and learning in a larger space) performs worse than the original one (Index 1). Third, the scale $\gamma$ is vital to balance the novel and historical knowledge. Without it, the Equation (5) becomes $L_{c}+L_{b}$, which can not achieve the best results (Index 7). Moreover, we report the performance of the model with different $\gamma$ in Figure 6. The results indicate that a suitable $\gamma$ can enable the model to perform better on both old and new data.

Furthermore, we even conduct experiments on Split CIFAR100 with different subspace sizes and report the results in Table 4. Firstly, the performance of ER-FSL improves as the subspace size increases since a larger space can capture more useful features. Secondly, the results show that the best subspace size is 100, which triggers the subspace reuse mechanism. It means that the results of ER-FSL in Table 1 could be better; however, for a fair comparison with other methods, we have to limit our method to choosing a feature subspace of size 51 to fill the entire feature space (size = 512). Thirdly, although the performance tends to decrease when the size further increases, the decline is not significant.

The final comparison between the existing methods and ER-FSL regarding computation and memory complexity is illustrated in Table 5. We compare the computation complexity using $C_{\mathcal{D}}$ (Ghunaim et al., 2023), which is determined by the relative training FLOPs. For example, we set the computation complexity of ER as 1. Since ER-FSL, ER-ACE, and PCR straightly modify the loss of ER, their computational complexities are equivalent to 1. However, OCM heavily relies on massive data augmentation operations, making its relative complexity 16. Meanwhile, we set the relative memory as the memory complexity. For instance, the memory complexity of ER is set as 1. OCM has a complexity of 2 due to the need to save an additional model for knowledge distillation, while OBC uses two classifiers, making its complexity greater than 1. The memory complexity of ER-FSL is 1. Thus, ER-FSL is also excellent in terms of complexity.