Online Continual Learning via the Knowledge Invariant and Spread-out Properties

With the development of deep learning, current deep models can perform well on various computer vision problems[1, 2, 3, 4], such as object detection [5], image classification [2, 6], semantic segmentation [7], and so on. The standard learning process of current deep models is underpinned by the assumption that the data is independently and identically distributed (i.i.d). However, in our ever-changing world, the training data often be obtained from an open environment [8], which is with stream format [9]. To solve this challenge, the model should learn new tasks incrementally instead of restarting the training process from scratch. A naïve method is to finetune the model on the incoming new tasks, however, it suffers the forgetting of past tasks when new ones are trained, a phenomenon known as catastrophic forgetting [10]: due to the absence of previous data, the performance on prior tasks drops significantly. Continual learning [11, 12, 13] aims to learn a sequence of tasks by leveraging the knowledge obtained in the past while not forgetting how to perform previous tasks. Fig.1 illustrates the setup of continual learning. As is shown in Fig.1, when the first task arrives, the model is required to classify dogs and spiders. After that, the model is incrementally updated with another new task, i.e. , bag and box, and it needs to correctly classify both the past task (dogs and spiders) and the current new task (bag and box), simultaneously. That means, when the new task arrives sequentially, the agent is required to learn a new task while should not forget how to address the previous tasks.

Recently, several works have attempted to resolve catastrophic forgetting for a continual learning scene, which can be coarsely grouped into two categories according to whether maintain a memory. The methods without memory [14, 15, 16, 17] constrain the updating of the model parameters to resist catastrophic forgetting and the key difference for this category of approach lies in the way of parameter importance measurement, such as knowledge distillation loss [14], the Fisher information matrix [16kirkpatrick], and so on. Other works selectively save some examples to prevent forgetting [18, 19, 20, 21]. Compared to other methods, the approaches based on episodic memory can effectively prevent forgetting and achieve superior performance. Thus, in this paper, we also maintain an episodic memory which is used to jointly train to prevent forgetting. However, in this continual learning setting, it is worthy to note that since the budget for storing the examples from the previous tasks is limited for practical continual learning, which can create a bias towards the current task. Such bias makes the model more prone to the current tasks by which worsens forgetting. Recalling the existing researches on this issue, some approaches [22, 23] attempt to apply an additional subset of examples to alleviate this problem, however, this kind of method requires the whole data of current task which is unavailable for the online continual learning setup. Furthermore, the knowledge distillation strategy used in LWF [14] is to minimize the KL divergence between the soft-max scores (input called logit) of the old and the new tasks. Others [24, 25] distill knowledge from the penultimate layer. Zhou et.al [26] directly leverage all previous model snapshots which means that directly matches the outputs of current network with those from the corresponding previous task models. However, we find that they treat all logits of the intermediate feature levels as separation. That means, these methods only require the current model to mimic the previous model’s logit output values, thus neglecting the important structural knowledge existing in the previous tasks, i.e., similarities or dissimilarities between the logits. We anticipate that fully transferring the important intrinsic structural knowledge underlying in the old tasks is benefit to current task learning, thus further resisting forgetting. In addition, in recent studies, existing works mainly focus on the situations where the whole task’s training data arrives at each step and allow the agent to learn the current tasks for many epochs. However, this assumption is not suitable for many real-world scenarios where the training data is observed sequentially in a stream form. Based on this fact, in this paper, we consider a more realistic but difficult scenario, called online continual learning where the agent only has one look at each instance. It is obvious that continual learning is closer to animal learning[27].

In response to the above problems, motivated by positive concentrated and negative separated in the self-supervised field [28], we propose a new framework under the online continual learning setting, called Online Continual Learning via the Knowledge Invariant and Spread-out Properties (OCLKISP). Particularly, we propose a new method to transfer the important structural knowledge of the previous tasks via knowledge invariant and spread-out properties for the continual learning setting. In detail, facing the incoming of new tasks, the model should not forget previous knowledge learned from past tasks. That means, the inter-instance space location relationships should keep invariant, in other words, the embedding features of episodic memory learned from the previous model and current model respectively should maintain invariant or the highest similarity as much as possible. However, the embedding features of different examples are separated, which can be beneficial to learn the similarities or dissimilarities of the output distribution among different examples in episodic memory and then better maintain the structure of the embedding features as the task arrives. Here, we transform the multi-class classification problem to a binary classification problem and apply maximum likelihood estimation (MLE) to optimize the continual learning effect.

In summary, the major contributions of this work are as follows:

• $\bullet$

  We proposed a novel online continual learning method via the knowledge invariant and spread-out properties (OCLKISP), which can transfer the structural knowledge from previous tasks and then improve the overall performance of the model.

• $\bullet$

  We analyze the learning bias of the replay-based method in online continual learning from the angle of embedding features and the classification accuracy respectively. Then, the forgetting problem due to the learning bias can be alleviated via our OCLKISP method.

• $\bullet$

  We evaluate our proposed method on four popular datasets, and the experimental results illustrate the effectiveness of our OCLKISP method in a continual learning setting. The ablation studies are conducted to further understand the contribution of KISP.

The remaining of this paper is organized as follows. In Section 2, we give a survey of the related works. In Section 3, we illustrate the problem of continual learning. In Section 4, we introduce the proposed approach of this work. In Section 5, contrast experiments and ablation studies are performed to demonstrate the effectiveness of our proposed method. Finally, conclusions and future researches are given in Section 6.

1) Continual learning

Continual learning is now a popular topic in the field of machine learning [15], [29, 30, 31, 32]. The ultimate goal of continual learning is to achieve good performance on the current task while not forgetting how to perform past tasks. According to whether maintain a memory, the method of continual learning can be coarsely grouped into two groups: Non-memory-based strategy and Memory-based strategy.

Non-memory-based strategy This line of works avoids preserving raw inputs and alleviating memory requirements. We can further divide these methods into parameter-based, distillation-based, and architecture-based methods. Firstly, the methods of parameter-based such as EWC, SI, et al, attempt to use fixed-capacity models and add a regularization term on the loss function to control the changes on significant parameters. The differences among these works lie in the way to estimate the weight importance. For example, a representative method called Elastic Weight Consolidation(EWC) [16] applies the Fisher information matrix to estimate the important parameters and then prevent the large change for the important parameters. In Synaptic Intelligence (SI) [15], the importance of weight is computed by the cumulative change of distance after learning new tasks, where the importance is estimated in an online manner during train. Compared to EWC, SI is more intuitive and persuasive in terms of the measure about the importance. In addition, Memory Aware Synapses (MAS) [33] applies the sensitivity of the learned function instead of loss to measure the importance of weights. Another line of work is distillation-based methods and a typical model is Learning Without Forgetting (LWF) [14], which constrains the change of the model predictions on past tasks via a distillation loss [34]. Although the methods lessen the catastrophic forgetting, they usually fail to solve the forgetting problem well and perform non-favorably when the sequence of tasks is long [35]. Others focus on the network structure which tries to train a separate model for each incoming task to resist catastrophic forgetting. For example, Progressive Neural Network (PNN) [36] alleviates catastrophic forgetting by fixing the parameters of the network from past tasks while learning new tasks by expanding the network. Other relevant works [37, 38, 39, 40] consider adaptively selecting the network structure based on the correlation of knowledge and then improving the efficiency of the network capacity.

Memory-based strategy This line of work saves prototype instances of former tasks in episodic memory and then a rehearsal or pseudo-rehearsal process is used to prevent catastrophic forgetting when learning a new task. Incremental Classifier and Representation Learning (ICARL) [40] combines knowledge distillation and feature learning to resist forgetting. Specifically, the examples closest to the feature mean of every class are picked and stored in fixed memory. During training, the distillation loss between targets obtained from previous model predictions and current model predictions is applied to alleviate forgetting. Gradient Episodic Memory (GEM) [18] and Average-GEM (AGEM) [41] apply an inequality constraint via episodic memory to prevent the forgetting for former tasks. Hou et al [24] propose to learn a unified classifier via a cosine linear layer and then incrementally learn new knowledge in a continual learning setting. Wu et al [23] and Pham et al [19] utilize the samples to build an extra validation set and then further improve the generalization of the model. Along with the development of the generative model, such as auto-encoder [42], VAE [43], or GAN [44], another direction is to use a generative model to mimic the samples of former tasks and then we prevent forgetting by replaying the generated data of previous tasks [23, 45, 46]. Compared to the methods of Non-memory based, the major drawback of this kind of method is to require additional computation and storage of raw input samples.

2) Data imbalance

Recently, the method with experience replay has achieved significant improvements in continual learning setting. However, in a replay approach, the data from prior tasks are limited compared to current task which would cause the data imbalance between the previous tasks and current task and then create a learning bias. Many researchers have attempted to resolve this issue in batch continual learning. Francisco et al. [22] attempt to apply an additional balanced subset of examples to finetuning the model. Yue et al. [47] apply a bias correction layer to further avoid the strong bias towards current task. Bowen et al. [48] attempt to correct the bias via a weight alignment approach in the fully-connected layer after each update. However, these methods require that the data of current task is available and then train multiple epochs or many learning stages is applied to alleviate the learning bias, which is not appropriate for online continual learning where the data of each task arrives in the data stream. We also note that some distillation-based methods [19, 26, 34] can be used to alleviate this problem, such as requiring the corresponding layer in the previous model to be exactly the same as the current model. However, these methods only require the current model to mimic the previous output value of each layer in episodic memory, which is not beneficial to capture the structure knowledge of previous tasks. Based on the above study, in this work, we design a new loss function to deal with the learning bias.

3) Contrastive Learning

Recently, contrastive learning [49, 50, 51] has shown great potential to learn general representations by constructing positive and negative examples. There are some prior works in this area. Typically, a random data augmentation, such as scaling, crop, and flip, is employed to every image and then leverages the augmented image as a positive instance. That means we aim to learn more concentration features that should be invariant under different data augmentations. Then, a small batch of randomly selected examples can be considered as negative examples of each image. Under this assumption, some works [51, 52, 53] attempt to separate each instance from other negative instances which leads to a spread-out property. Xiao et al.[54] consider local features of training inputs and different inputs as positive and negative examples respectively. Tan et al. [55] proposed hyperspherical consistency regularization (HCR) to regularize the classifier which would enforce data points on hyperspheres to have similar structures.

Contrastive representation learning [56, 52] has also shown superior downstream task performance via the positive concentrated and negative separated properties. In the Multiview setting, Tian et al. [56] consider that every view is noisy and incomplete, however, the important information should be shared among all views. Based on this assumption, different views of input are recognized as the positive instance and then the model can further capture the shared information. Lin et al.[57] adopt this property to video representation learning. Contrastive learning is now also widely applied to other computer vision fields, e.g., objective detection [58], natural language domain [59], and signature verification [60]. In this work, inspired by the positive concentrated and negative separated properties, we mainly apply this property to the continual learning setting based on our novel observation.

Continual Learning Before diving into the details of our approach proposed in this work, we first introduce the definition of continual learning followed by a classical memory-based continual learning scenario. Continual learning studies the problem of learning from the streaming tasks, with the goal of incrementally gaining knowledge from data of tasks. Assume there is a sequence of task $\left\{{1,2,\cdots,t,\cdots,{\rm T}}\right\}$, where ${\rm T}$ denotes the total number of training tasks. Each task $t$ has its own dataset $D^{t}{\rm{=}}\left\{{\left({x_{i}^{t},\,y_{i}^{t}}\right)}\right\}_{i=1}^{N_{t}}$,where $x_{i}^{t}$ is the i-th training instance and $y_{i}^{t}$ is the corresponding ground truth label. It is worth noting that there is no overlapping class among different tasks. During the training process of task $t$, the previous data $D^{1},D^{2},\cdots,D^{{\kern 1.0pt}t{\rm{-}}1}$ are no longer available. Our goal at every step is not only to learn the knowledge from the current task, but also to retain the knowledge learned from former tasks. After ending every task, we evaluate the performance of the trained model on all currently seen tasks.

The crux problem is that when a new task $t$ arrives, the model should incrementally obtain the ability to correctly classify the new class and all currently seen classes, simultaneously. Here, we split the trained model into two parts: embedding function $\phi\left(\cdot\right)$ and a linear classification head $W^{t}$. When the training instance $x_{i}^{t}$ is observed, firstly, the embedding feature $f_{i}^{t}{\text{ = }}\phi\left({x_{i}^{t}}\right)$ is generated by the embedding function. After that, we compute its classification probability for class $y_{i}^{t}$ by $\sigma\left({\left[{W^{t}}\right]^{T}f_{i}^{t}}\right)$ where $\sigma\left(\cdot\right)$ is the soft-max operator. Similarly, the classification probability about the all currently seen class label set is calculated as $\sigma\left({\left[W\right]^{T}f_{i}^{t}}\right)$, where $W{\text{ = }}\left[{W^{1},W^{2},\cdots,W^{t}}\right]$ denotes the concatenation of all the current classification heads, i.e., the set of logits for tasks $1,\cdots,t$. As aforementioned, normally, memory-based approaches prevent catastrophic forgetting via a tiny episodic memory. A typical way is to compute cross-entropy using these examples from episodic memory and current task respectively:

where $M{\kern 1.0pt}^{t}$ is episodic memory. We optimize the above cross-entropy loss about all previous replay examples in episodic memory and current novel examples, thus learning new knowledge while alleviating the forgetting.

Eq.(1) describes a way to prevent forgetting for continual learning by memory replay. Recently, a lot of literature has shown that compared with other state-of-the-art methods, methods based on experience replay can achieve competitive performance in the online continual learning setting. However, there are some intractable problems which should be solved for these approaches. Firstly, note that the training model only observes none or a few examples of previous tasks due to the small episodic memory but substantially more for the current task. Under this circumstance, as new tasks are learned sequentially, the training process can create a bias toward the current task, thus leading to the forgetting of knowledge learned from previous tasks.

Here, to more closely and clearly show what the problem is, we make a quantitative analysis. Specifically, we measure the change in cosine distance for embedding features of episodic memory in the buffer after each update when a new task comes. As is illustrated in Fig. 2, due to unbalance data and learning bias, as new knowledge is continuously learned, the change of the embedding features in the buffer gradually increases. Moreover, in Fig. 3, we plot the evolution of task classification accuracy accordingly. We observe that the task classification accuracy performs poorly as new tasks are learned.

From Figs. 2 and 3, we found that the knowledge learned from previous tasks changes. Thus, the forgetting uninterruptedly and deterioratingly occurs as new knowledge is learned sequentially. So, we must further take some actions to maintain the knowledge invariant and then prevent forgetting due to the learning bias.

Fortunately, we find that there is some side information neglected in this setting. Specifically, maintaining the structured knowledge in episodic memory (i.e., similarities or dissimilarities between the embedding features $f_{i}^{\,t}$ and $f_{j}^{\,t}$) as the new task sequentially arrives is useful means. By means of preserving the structured knowledge in the continual learning setup it is promising to alleviate the learning bias.

In this section, we introduce our proposed OCLKISP for online continual learning detailly. In subsection 4.1, we start the approach discussion with a brief introduction to the framework of OCLKISP. Then, we present the details of our proposed OCLKISP in subsection 4.2. Lastly, we further make the rationale analysis for our proposed approach in subsection 4.3.

In this section, we compare our OCLKISP with other baselines on several datasets used in continual learning to verify the effectiveness of our proposed method. Then, we introduce the metrics and implementation details used in our experiments. Finally, we also perform ablation studies to validate the improvement of my proposed method.

Continual learning aims to prevent catastrophic forgetting when new tasks are added. In this paper, we have studied the potentials and limitations of current popular continual learning paradigms. The experimental results illustrate the effectiveness of some state-of-the-art approaches, especially memory-based approaches. However, the learning bias between the current and previous tasks during experience replay is also a key problem, which can enhance forgetting. Furthermore, existing methods also neglect the transfer of structural knowledge in episodic memory which would be harmful to learning tasks sequentially. Inspired by this, this work introduces a novel continual learning method called Online Continual Learning via the Knowledge Invariant and Spread-out Properties (OCLKISP). The proposed approach can further transfer the structural characteristics of instances via knowledge invariant and spread-out properties when continually learning a sequence of tasks, which can also be beneficial to overcome forgetting caused by the learning bias. Finally, comprehensive experiments are conducted on various benchmark datasets to validate the effectiveness of our proposed paradigm.

In the future, there are several directions worth considering. Firstly, in this paper, we have illustrated the feasibility of our OCLKISP approach in classification settings and we can consider further extending our proposed method to more avenues of computer vision, such as continual face recognition and continual semantic segmentation. Besides, we can also consider some more realistic problems during the practical application, such as concept drift and abnormal detection. Furthermore, note that experience replay is an important means to overcome catastrophic forgetting, thus it would be interesting to study how to select the more representative samples which can better preserve latent decision boundaries for previously observed tasks. Another interesting extension to this work would be to use generative models to generate new examples which are used instead of episodic memory.