Class-incremental Learning using a Sequence of Partial Implicitly Regularized Classifiers

A straightforward solution to prevent catastrophic forgetting is to revisit the previous tasks. Rehearsal methods accomplish this by storing a small portion of the seen examples for later retraining. When the model is faced with new training data, it augments it with memory samples to reduce the forgetting of previous knowledge.

iCaRL[21] uses the nearest-mean-of-exemplars classification approach, classifying items to the class with the nearest center. It uses a heuristic approach for updating the memory buffer, prioritizing items based on their proximity to their class mean. Together with network distillation, it has been able to learn in class-incremental learning scenario. Gradient Episodic Memory (GEM)[22] is designed for task-incremental learning from streaming data. In this setting, the model receives a series of tuples $(x,t,y)$ consisting of input, task label, and target, respectively. GEM uses the memory samples not for replay but to serve the inequality constraints that prevents the loss for the past tasks to decrease. Replay using Memory Indexing (REMIND) [23] applies quantization to the extracted feature maps from deeper layers of a CNN and stores their indices in memory. This results in a compressed representation and allows for much more number of stored examples. REMIND’s architecture consists of a frozen feature extractor followed by a trainable classifier. REMIND was shown to outperform existing approaches in a streaming setup on the datasets ImageNet, and CORe50[24].

These methods impose a type of regularization that helps retain the learned knowledge of past tasks. They do it by adding additional loss terms to the loss function.

Learning without forgetting (LwF)[29] presented a modification to the standard fine-tuning. A new network is initialized as a copy of the old one with an extension of the output layer(called multihead approach) for the new task. A distillation loss is added between old and new task heads so the new network is reminded of the old tasks indirectly. In backward pass, only the new network is updated.

Elastic Weight Consolidation (EWC)[30] and Synaptic Intelligence (SI) [31] both try to approximate the importance of each of the parameters for previous tasks and selectively apply regularization to limit their change. Regularization methods alone, as several papers demonstrate, are not sufficient for proper class-incremental learning[16, 32].

Architectural methods try to manipulate weights, neurons, layers, or architecture of the network to protect the learned knowledge while acquiring a new one. They either use fixed or dynamic architectures. These methods have the advantage of completely eliminating interference between tasks while allowing knowledge transfer between them. However, practically, they are coupled with scalability issues as the number of tasks grow.

“Progressive Neural Networks”(PNN)[33] inherently target the task-incremental learning. Their architecture grows laterally, each time adding a new neural network, called “columns”, for a new task. The new columns have connections to other layers of previous networks and thus, highly benefit from knowledge transfer. Once a column is trained, it will be kept frozen making the PNNs completely immune to catastrophic forgetting. A big issue with PNNs is that they keep growing too large, limiting their use in practice to only a small number of tasks. “Compacting, Picking and Growing” (CPG) [34] tries to overcome this issue by dynamically controlling the architecture of the network in an efficient way. The weights of the network are grouped into “compact” and “free”, where the newly added weights are considered free until they are trained and then compacted. For each new task, a learnable binary mask is created and applied to the compact weights to select a subset of them. Then this task is learnt using this subset of compact weights and the free weights. The network is grown during the training if the performance does not reaching a satisfactory level, providing more free weights. The training is followed by pruning to compact the newly learned weights. This method is useful in scenarios with less number but larger tasks.

As the memory should be kept updated to include new samples, a sampling strategy has to decide which examples should be selected or removed. We adopt the greedy sampling approach described in [20]. It randomly replaces some of the old examples in the memory with the new ones, while trying to satisfy the balancing constraint, that is, to maintain an equal number of examples for each class.

Training consists of multiple sessions. At the beginning of each training session, a classifier is constructed which has the output size equal to the number of classes in the training data. Training objective is to learn the parameters of this new classifier. We train it by mini-batch gradient descent where each mini-batch contains an equal number of samples from the current dataset and the memory buffer. At the end of each training session the memory buffer is updated with new samples.

As the number of examples per class in the memory keeps decreasing over time, the model becomes prone to overfitting. Therefore, an early stopping mechanism is essential. We propose that a portion of the data should be held for validation. Apart from splitting the incoming data into train and validation parts, we also dedicate a part of the memory for validation samples. We make sure that the part of the memory for training gets updated only with the samples in the training set, and similarly the validation part of the memory gets updated only with the samples in the validation set.

Since the mini-batches contain nonequal number of samples from the old and new classes, it poses a class imbalance problem. This problem has been first addressed in [26] and the authors proposed a simple yet effective solution, called Bias Correction (BiC). It is a layer containing only two parameters and applies a linear transformation to the output logits belonging to new classes $\mathcal{Y}_{i}$ as follows:

where $\alpha$ and $\beta$ are the bias parameters and $o_{k}$ is the output logits of the final layer. The BiC layer is trained separately at the end of each training session with a small amount of data as it contains only two parameters.

For feature extraction, we use a EfficientNet-B0 network[37] that is pretrained on ImageNet. We remove from it the final convolution layer, the output layer and the final MBConv block. Since this feature extractor accepts inputs of size $224\times 224$, we resize the CIFAR100 images which are $32\times 32$ to match the input size. The feature extractor is kept frozen in all experiments of our method and the baselines. For classifiers, we use a single $1\times 1$ convolution layer followed by global average pooling, and a dense layer. In all experiments the classifiers have the same architecture except the number of filters in the convolution layer and the number of units in the final dense layer. Overall, we have made sure that our method does not use more trainable parameters in total than the baselines (See table II).

In all experiments, we stop a training session when the validation loss does not improve for 10 consecutive epochs. We have dedicated 10% of the data for validation. For all methods we start by learning rate of 0.01. For our method we decrease the learning rate when the validation loss does not improve for 3 consecutive epochs. For GDumb and ER we apply exponential decay to learning with rate 0.95. At the end of each training session, just before testing, we train the BiC layer to remove the prediction bias towards later classes. Since the whole purpose of BiC is to remove prediction bias, we will train on it only on the validation part of the memory buffer(after it has been updated to include all seen classes) which the model itself has never trained on.

GDumb and ER use the same architecture, they only differ in training. Gdumb only trains on memory samples. It updates the memory buffer at the beginning of each training session and then trains only on the memory buffer discarding the rest of the data. We also train a BiC layer at the end of each training session of ER. BiC layer is not necessary for GDumb due to the fact that it trains on balanced dataset, i.e., the memory buffer.

We test the models at the end of each training session over all classes that have been encountered so far. In table I we report the accuracy at the end of the training in three different settings with varying memory buffer sizes (500, 1000, and 2000) and dataset splitting into 5 (Fig. 2), 10 (Fig. 3), and 20 (Fig. 4) parts. In all settings we see a large gap in accuracy between our method and the second best method, ER. ER always outperforms GDumbs, as expected, because it trains on all available data and the early stopping mechanism that we introduced here prevents it from unnecessary training which causes more forgetting of past data.

We observe that the gap in accuracy tends to get larger when we shrink the memory buffer size. Our method still reaches a remarkable performance of 56.45% when memory size is 500 leaving a gap of 14% relative to the next method, ER. On the other hand, we see a large performance drop of GDumb. This is mainly because it is highly dependent on the memory size as it trains only on the memory samples. However, GDumb shows the least difference in performance when training with different dataset splittings, meaning that it might have an advantage in settings more close to online learning.

We conduct experiments to see if all components of our method are indeed important. The first component is that we freeze the classifiers after we train them so that in this way we believe the problem of “forgetting” is eliminated. Therefore we have conducted an experiment where we let our method to keep updating all classifiers. The second component is the additional BiC layer. Looking at the confusion matrices, we have observed a prediction bias towards the last group of classes which was a signal to incorporate BiC layer. We also present the results here where our method does not use a BiC layer. These experiments are conducted in the case of CIFAR100 split into 10 classes, and a memory buffer size of 2000. The reported results are in Table III. We can see the benefit of using BiC layer, which confirms that our method would have some prediction bias without it. On the other hand, we can see a large drop in accuracy (65.79% $\to$ 57.04%) when we allow updating our classifiers. Nevertheless, still the performance stays above the other two methods (looking at Table I, 54.68% and 48.74% of ER and GDumb, respectively).