On robustness of generative representations against catastrophic forgetting

First, to test our hypothesis, we look at the problem of catastrophic forgetting from the accuracy perspective, as it is the most widely adopted measure of the catastrophic forgetting phenomenon. To measure the robustness of representations to the catastrophic forgetting, we train the models $\mathcal{D}$, $\mathcal{G}_{AE}$ and $\mathcal{G}_{VAE}$ on identical data sequences $T_{N}$ and collect their representations from penultimate layer for all data splits. These representations are used to train a set of linear classifiers. Although we collect the representations for all data splits throughout the training, we train respective classifiers only when the corresponding task is active. This means that we train only the first classifier during the first task, and the rest remain untrained. After finishing the first task, we freeze the first classifier for the rest of the training. This way, degradation of its performance can be attributed exclusively to forgetting the representations for the first task, and we can directly measure the amount of forgetting in the model.

More precisely, after each epoch of training, for task data of the form $\langle\mathbf{x}^{k},\mathbf{y}^{k}\rangle$, where $k$ denotes the respective data split, we collect the representations for model $\mathcal{D}$ from its penultimate layer $\mathcal{A}^{k}_{\mathcal{D}}(\mathbf{x}^{j})$ for $j=k$ and train softmax regression classifier $\mathcal{C}_{\mathcal{D}j}$ on dataset of the form $\langle\mathcal{A}^{k}_{\mathcal{D}}(\mathbf{x}^{k}),\mathbf{y}^{k}\rangle$, where $k$ denotes the present task. Next, the trained softmax classifiers are evaluated on the validation dataset after extracting features with respective backbones. Note that in the above approach, classifiers $\mathcal{C}_{\mathcal{D}j}$ for $j>k$, remain untrained. Since the classifiers $\mathcal{C}_{\mathcal{D}j}$ are frozen after completing $j$-th task, the loss of performance may only be attributed to the drift of representations from the penultimate layer. Therefore, the smaller the loss of the classifier’s performance, the more resilient the features are to the catastrophic forgetting. The same procedure is applied to the models $\mathcal{G}_{AE}$ and $\mathcal{G}_{VAE}$ as shown in Fig 2.

Fig. 3 depicts the results of this experiment. Columns represent results for $\mathcal{D}$, $\mathcal{G}_{AE}$, $\mathcal{G}_{VAE}$ respectively. Starting from the top, the rows present the results for MNIST, FashionMNIST, and CIFAR. Different colors present the accuracy for different tasks. As one can see in the left column of Fig. 3, for all three benchmarks, the classifier’s performance trained with representations of discriminative model suddenly drops after introducing new tasks and degrades further to the region of random guessing (depicted as dashed line). In contrast, for the generative case for both models, the performance is stable throughout the whole sequence of training, suggesting that the representations of tested generative models are almost immune to the problem of catastrophic forgetting, especially in the case of simpler datasets as MNIST and FashionMNIST. In the case of CIFAR-10, the amount of forgetting is considerable for the case of $\mathcal{G}_{AE}$. However, the degradation of the performance is more stable than in the case of the discriminative model. This suggests that forgetting in generative models has different nature than forgetting in discriminative models. The right column shows that the representations of $\mathcal{G}_{VAE}$ model suffer the least in the case of CIFAR-10. Explaining this difference requires further analysis, and we foresee it as future work.

Surprisingly, in MNIST and FashionMNIST, achieving an average accuracy above 90% without any dedicated mechanism to overcome catastrophic forgetting is possible. This raises the question of whether these datasets and evaluation protocols should be used to benchmark novel methods in continual learning.

The above experiment proves the validity of the first hypothesis through the lenses of accuracy as it is a widely adopted measure to estimate catastrophic forgetting. However, such an approach may not be fully informative as generative representations can have poor discriminative abilities. To address this limitation, we propose to measure catastrophic forgetting through the index of representations similarity.

To further examine the validity of our hypothesis that representations learned by generative models are less susceptible to catastrophic forgetting, we investigate the evolution of network representations in time. Since the above experiment is directly linked to the discriminative abilities of the representations, here we use a task-agnostic measure to directly estimate the drift of the representations in continual learning training. For that purpose, we use the well-established method of Centered Kernel Alignment (CKA) [9], defined as:

$$\operatorname{CKA}(X,Y)=\frac{\left\|X^{T}Y\right\|_{F}^{2}}{\left\|X^{T}X\right\|_{F}\left\|Y^{T}Y\right\|_{F}},$$

(1)

where $X\in\mathbb{R}^{n\times m_{x}}$ and $Y\in\mathbb{R}^{n\times m_{y}}$. CKA takes values from 0 to 1, where 1 means identical representations. Using CKA, it is possible to estimate the similarity of representations obtained for different datasets or different dimensionality. We measure the similarity between representations of the same network collected at different moments of the training of the neural network. Specifically, to analyze the evolution of representations from model $\mathcal{D}$, we collect the reference representations $\mathcal{A}^{k}_{\mathcal{D}}(\mathbf{x}^{k})$ just after finishing task $k$. Then, to measure the relative drift of representations on task $k$, we compute the CKA index between the reference and current representations $\operatorname{CKA}(\mathcal{A}^{k}_{\mathcal{D}}(\mathbf{x}^{k}),\mathcal{A}^{j}_{\mathcal{D}}(\mathbf{x}^{k}))$, for $j\geq k$. We follow this procedure for each task in training sequence $T_{N}$. The analogous procedure is applied to the representations of models $\mathcal{G}_{AE}$ and $\mathcal{G}_{VAE}$.

Fig. 4 presents the results of the experiment. In the case of discriminative representations (top row), we can see an abrupt change of representations just after introducing the new task on all tested datasets. This is in line with the results from the previous experiment, which show that most of the performance is lost during the next task. In the following tasks, the representations stabilize, but these representations are no longer valuable since classifying with them is not better than random guessing (see Fig. 3). The results from the middle row of Fig. 4 show that representations from $\mathcal{G}_{AE}$ slightly evolve during the training on consecutive tasks and remain similar to the reference representations. These results directly support hypothesis 1. The representations of $\mathcal{G}_{VAE}$ changes significantly on the first two tasks of simpler datasets (MNIST and FashionMNIST). These results indicate that it takes more time for the VAE model to learn stable features. In CIFAR-10, the amount of forgetting on VAE representations is almost equal to the amount of forgetting on AE representations which in both cases is almost negligible. This may suggest that the more complex the data is, the more general features the generative models learn.

Additionally, the dynamics of changes are less chaotic for the generative model, as the similarity of representations monotonically degrades with each task. This is in stark contrast to the discriminative representations which evolve chaotically. For instance, in the FashionMNIST dataset (middle column, top row), representations of the first task drastically change during the second and third task, then stabilize on the fourth task and unexpectedly become more similar during the last task. This may suggest that although the forgetting occurs in the generative case, it has a gradual form and is more predictable than in the discriminative part.