Subspace Distillation for Continual Learning

A variety of methods has been proposed to alleviate catastrophic forgetting for class-incremental classification problems [4, 12, 13, 20, 21, 22]. These methods for continual learning are classified primarily into three categories, such as regularization, dynamic architecture and memory replay based methods  [23].

To address the security concern, few recent approaches employed generative model to produce samples belonging to old classes [40] instead of storing subset of old samples in memory. Deep Generative Replay (DGR) [40] proposed a dual-model architecture, one for generating pseudo samples and another for solving tasks by replaying pseudo samples together with new samples. To reduce the memory footprint of storing real samples, compressed feature have been stored in [41, 42]. REMIND [41] used product quantization method to quantize latent representation and stored indices in memory that were used later to decode the representation for reply. However, because of incremental update in model, stored latent representation also requires adaptation. To fit the stored representation into current latent space, Iscen et al. [42] employed a multi-layer perceptron to map corresponding old and new feature map generated for images of current task.

Recently, Dark Experience Replay (DER++) [43] proposed to store both logit, and label for corresponding sample and used the reservoir sampling strategy to select samples from data stream for memory buffer. In DER++, knowledge distillation is performed by mapping output logit from current model with corresponding memory logit.

In recent time, a growing number of works have emerged into continual semantic segmentation. In the literature, CSS methods primarily fall into two major categories: (i) regularization and (ii) Replay based model.

Following the success of regularization methods in CL, several works have proposed mechanism for controlling update of neuron weights to mitigate catastrophic forgetting in CSS [15, 11, 16, 44]. ILT [15] investigates a regularization-based technique by freezing weights of the encoder network after learning the first task and applying knowledge distillations for the upcoming tasks. ILT is also equipped with the masked cross-entropy loss and masks on the output of the current model to consider only the seen classes. However, this approach doesn’t resolve the background shift problem of CSS properly. Since the background in CSS may include previously seen objects, MiB [11] proposed to preserve the knowledge of an old model with knowledge distillation. However, the difference compared to standard knowledge distillation is that the probabilities of the background and the novel classes are appended such that the non-overlapping prediction of a novel class as the output of a current model can be aligned with the output of a previous model. As knowledge distillation shows a promising direction for CSS, PLOP [16] introduced pseudo-labelling to tackle the background shift problem and adapt Pooled Out Distillation (POD) [13] to preserve the previously learned knowledge. Since POD is developed for a classification problem in classical continual learning settings with reliance on global statistics, thus PLOP uses a multi-scale version of POD to integrate local and global statistics at different intermediate layers. SDR [44] leveraged contrastive learning together with novel sparsity constraint and prototype matching strategy to efficiently learn novel tasks and mitigate forgetting of prior knowledge. In SDR, to organize geometric structure of feature representation, clusters of data are described using prototypes that are forced to be closer in consecutive learning steps and far apart from one another by using prototype matching and repulsive force. Additional sparsity constraint imposed on feature representation of same classes helped to construct well-separated and tight cluster as well as create space for accommodating novel classes.

Distilling knowledge from old model to the current model has shown promises to mitigate catastrophic forgetting of neural network in a CL setting [12, 13]. Here, the old model with the knowledge of already observed tasks acts as a teacher model and the purpose is to distill knowledge from teacher model to student model (i.e., current model) such that prediction of these two models matches for previously seen tasks samples. Knowledge distillation is often performed on the probability space to relate the temperature smoothed probability distribution of old to the new model [12, 11]. Feature distillation on the other hand is performed on the feature space extracted from the intermediate layers of neural network to match corresponding local or global statistics of feature maps between old and new model [44, 15, 13, 16].

Assume that the extracted feature and the prediction using teacher model from step $t-1$ for input ${\mathbf{X}}$ are ${\mathbf{f}}^{t-1}$ and $\hat{y}^{t-1}$, respectively. Similarly, let ${\mathbf{f}}^{t}$ and $\hat{y}^{t}$ be the feature and prediction respectively for the same input using the student model at step $t$. The knowledge distillation is performed between teacher and student model by minimizing the following loss function

Feature distillation strategy applies constraint on the similarity between old and new feature representation and is performed by minimizing a notion of distance between corresponding features of ${\mathbf{f}}^{t-1}$ and ${\mathbf{f}}^{t}$, such as the distance induced by the $\ell_{1}$ norm as

For the task of continual classification, we distill the subspace constructed across samples in a mini-batch. For a mini-batch $\big{(}\mathcal{X}_{B},\mathcal{Y}_{B}\big{)}$ of size $b$, let ${\mathbf{f}}_{j}\in\mathbb{R}^{d}$ be the latent representation for input ${\mathbf{X}}_{j}\in\mathcal{X}_{B}$. The formulation below can be applied to any layer in a DNN and hence we drop the layer index for the sake of simplicity. Assume there are $p=\lfloor b/|\mathbb{C}^{t}|\rfloor$ samples per class in the mini-batch. In SD, we propose to compute class-wise subspaces using latent features generated from both old and new model. More specifically, from the mini-batch, we form $\{{\mathbf{F}}^{t-1}_{k}\}_{k=1}^{|\mathbb{C}^{t}|}$ and $\{{\mathbf{F}}^{t}_{k}\}_{k=1}^{|\mathbb{C}^{t}|}$ with ${\mathbf{F}}^{t-1}_{k},{\mathbf{F}}^{t}_{k}\in\mathbb{R}^{d\times p}$ by stacking corresponding samples in class $k$ into a matrix (i.e., ${\mathbf{F}}^{t}_{k}=\big{[}{\mathbf{f}}_{k,1},\cdots,{\mathbf{f}}_{k,p}\big{]}$, where $\mathcal{Y}_{B}\ni y_{k,i}=k$). We note that stacking ordering is not important in forming ${\mathbf{F}}^{t-1}_{k},{\mathbf{F}}^{t}_{k}$ as we are merely interested in subspace spanned by the samples. Next, we represent each class $k$ from the teacher and the student model by its low-dimensional subspace ${\mathbf{P}}^{t-1}_{k},{\mathbf{P}}^{t}_{k}\in\mathbb{R}^{d\times m},m\leq p$. This is achieved by applying thin SVD to ${\mathbf{F}}^{t-1}_{k},{\mathbf{F}}^{t}_{k}$ and picking up the top left singular vectors. With the above, the SD loss is defined as

For class-incremental semantic segmentation problem, we propose to compute subspace across the channel dimension of intermediate feature map for each sample in a batch. To reduce the memory overhead while performing subspace distillation, instead of computing subspace from all feature maps at a layer, we split the full feature maps into several smaller group $G$ with $p$ channels. To compute a subspace from each group, we need to form a matrix representation from the feature map. To do so, for a particular input, we form $\{{\mathbf{F}}^{t-1}_{g}\}_{g=1}^{G}$ and $\{{\mathbf{F}}^{t}_{g}\}_{g=1}^{G}$ with ${\mathbf{F}}^{t-1}_{g},{\mathbf{F}}^{t}_{g}\in\mathbb{R}^{d\times p}$ from the feature maps. Here, $d=wh$, where $h$ and $w$ denote the height and width of the feature map. We encode the geometry of ${\mathbf{F}}^{t-1}_{j},{\mathbf{F}}^{t}_{j}$ by low-dimensional subspaces ${\mathbf{P}}^{t-1}_{g},{\mathbf{P}}^{t}_{g}\in\mathbb{R}^{d\times m}$ via SVD. The subspace distillation loss is defined as

Since image pixels belonging to prior classes are labeled as background in CSS, old model is employed for distilling knowledge. The idea of knowledge distillation is a crucial step and widely adapted in preserving prior knowledge for CSS. In CSS, we apply subspace distillation loss at intermediate layers of model to maintain consistency in geometric structure of latent features. Additionally, we use output distillation to ensure that the current model mimics the output of prior model. In other words, subspace distillation is combined with classification loss and output distillation loss to train CSS model at any step $t$. Bellow, we briefly discuss the classification loss and output distillation loss for CSS.

We analyze the efficacy of subspace distillation in continual learning setting for classification problem and present the result in Table 1. In comparison, we consider two knowledge distillation based methods (LwF [12], iCARL [29]), two regularization methods (SI [26], oEWC [63]) and three memory replay based methods (ER [20], DER [51], DER++ [51]). Additionally, we provide upper and lower bounds, that reflect to jointly training on tasks one by one or on all tasks with Stochastic Gradient Descent (SGD) without any specialized strategy designed for CL.

The results suggest that regularization methods performs poorly on class incremental learning setting as those methods are developed focusing on task incremental learning setting. This observation indicates that regularization towards the set of old parameters is not suitable for tackling catastrophic forgetting because of the local information modeling weight importance [51]. Overall, replay based methods outperform regularization methods with big margin across the datasets. Regularization methods performs good on MNIST datasets for task incremental settings while they perform notably worse on comparatively complex datasets, e.g. CIFAR10, Tiny-Imagenet. We observe that our method outperforms state-of-the-art replay base methods in task incremental learning setting on all datasets. Noticeably, for class incremental learning setting on split CIFAR-10 dataset our method performs noticeably better by 15% and 8% percentage points compared to iCARL and ER methods, respectively. However, DER method performs slightly better than subspace distillation method as DER relies on storing additional information (i.e., logits) together with corresponding image and label from past tasks in the memory for the distillation that requires additional memory requirements. We observe that by combining dark experience replay method DER, we can further improve our result. For instance, we can achieve about 2% and 5% performance gain on Tiny-Imagenet and CIFAR10 datasets respectively with medium size memory buffer.

We evaluate our model in a more constrained setting where each sample at a task is presented to model once. In other terms, we train our model for a single epoch at each task given a tiny buffer size of 100 samples. Overall, because of the complex setting, continual learning model faces underfitting problem and struggles to mitigate catastrophic forgetting. Replay-based distillation methods also suffer from poor performance when there are insufficient exemplars available, especially for previously observed classes where no memory exemplars exist. For instance, in S-Tiny Imagenet, during the final task, with just 100 memory exemplars, only one example is stored for the 100 previously seen classes out of a total of 180 prior classes, leading to a subpar performance of replay-based distillation methods. We observe that our method considerably improves on baseline ER method across the datasets and performs significantly better than DER. For example, our method enjoys around 2% improvement on both CIFAR-10 and Tiny-Imagenet datasets than state-of-the-art DER method for class incremental setting. However, we think that this scenario deserves further investigation because of its complex nature.

In task-incremental setting, we notice better performance compared to class-incremental setting for all methods across the datasets. The reason behind such observation is the presence of task identifier at the test time in task-incremental learning that makes the problem easier. We see that SI and oEWC perform badly on relatively complex datasets such as CIFAR10, and Tiny-Imagenet though regularization based methods are designed particularly for task-incremental scenario and our method outperforms both methods with unbridgeable gap in performance. Subspace distillation method performs remarkably better than iCARL across the settings with different memory size. For example, our method outperforms iCARL by 15% percentage points with large memory buffer on Tiny-Imagenet, and by 24% percentage points with medium size memory on CIFAR10 dataset. We also note that our method shows competitive performance with other memory replay based methods. We see around 2% improvement on ER for task incremental setting on Tiny Imagenet dataset while the performance gain is around 15% on CIFAR10 dataset for both small and medium size memory. Furthermore, by combining our method with DER, we see 4%, 5% and 3% performance improvements on DER with small, medium and large size memory buffer for task-incremental setting on CIFAR10 dataset.

Table 2 reports the IOU of different baseline strategies and subspace distillation method for three scenarios: (i) 19-1, (ii) 15-5 and (iii) 15-1 overlap scenarios on Pascal-VOC dataset. The results suggest that our proposed subspace distillation based method outperforms state-of-the-art methods in 19-1 and 15-5 task settings by a significant margin. Though our method does not match the performance of PLOP in 15-5s task setting, we observe about 13% IOU improvement on MiB at last task. Furthermore our method outperforms ILT by around 30% in IOU of last task. We note that subspace distillation shows consistency in retaining prior knowledge across the settings. To show the efficacy of subspace distillation method in tacking catastrophic forgetting, we combine SD with ILT and PLOP methods, and see improvements on IOU in different settings. We observe that by combining SD with ILT the overall performance (i.e., mIOU) of ILT method improves by about 3% and 2% for 19-1 and 15-5 task settings respectively while performance remains similar for 15-5s setting. Similarly, we notice 2% performance improvement of PLOP methods after combining SD for 19-1 and 15-5s settings which suggests that SD imposes additional constraint on distillation strategy using multi-scale POD that helps learning complimentary information and tackling catastrophic forgetting better together. Though our method shows little lower plasticity in case of learning new classes, subspace distillation shows promises in retaining prior knowledge and tackling forgetting old classes.

Fig. 4 reports the continual evolution of mean IOU for 15-5s (6 tasks) overlap setting on Pascal-VOC dataset. As depicted in the figure, the performance of MiB drops significantly throughout the learning steps while PLOP shows strength in preserving already learned knowledge and at each learning step PLOP performs significantly better than MiB. We observe considerably improved performance across the learning steps when subspace distillation is added with MiB. SD facilitates MiB to preserve previously learned knowledge and therefore we note about 9%, and 12% improvement of mean IOU at task 5, and 6 respectively after adding SD with MiB. Similarly, SD helps PLOP tackling catastrophic forgetting better and we see improvements on mean IOU at the end of all learning tasks.