Future-Proofing Class-Incremental Learning

While Class-Incremental Learning may be applied to various tasks and settings such as semantic segmentation [9, 10] or federated learning [11, 12], in this work we only consider the image classification task which is the most extensively studied. Most approaches for Class-Incremental Learning [13, 14] leverage knowledge distillation [3, 15, 16, 17], a rehearsal memory and a method to mitigate the bias toward the new classes [15, 18, 19, 20] induced by the limited size of the replay memory. Recent works [21, 22] proposed to extend the feature extractor at each incremental step to learn the new classes while preserving past knowledge by freezing previously learned weights.

Exemplar-Free Class-Incremental Learning is a more challenging setting in which there is no rehearsal memory, making the replay of samples from the previously encountered classes impossible. Similarly to approaches designed for Class-Incremental Learning, numerous methods for Exemplar-Free Class-Incremental Learning also continuously finetune the feature extractor and the classifier [23, 24, 25, 26, 27]. Several authors have highlighted the risk of overfitting on the current task and the importance of having general and transferable feature representation for EFCIL. To address this concern, approaches often rely on self-supervised learning [23, 26] or by creating augmented classes using Mixup [28]. Recently, methods relying on a fixed feature extractor have gained more attention [6, 29]. Those methods either rely on a pre-trained model [30, 31, 32, 8] or by freezing the feature extractor after learning it during the initial step [6, 7, 33]. Petit et al. [7] proposed FeTrIL which achieves state-of-the-art performances by relying on a frozen feature extractor and a linear classifier trained using actual features of new classes and pseudo-features of past classes generated by a geometric translation of new class features. FeCAM [33] is a recently proposed approach for Exemplar-Free Class-Incremental Learning with a fixed feature extractor that rely on the Mahalanobis distance combined with Tukey’s transformation, covariance shrinkage, and correlation normalization. Those methods exhibit numerous advantages including a less computationally expensive training procedure that allows for for a deployment on edge devices. However, they also depend on the initial training strategy used [34, 35] and may achieve poor performance if the quantity of data available for training the feature extractor during the initial step is too limited.

While few attempts have been made using GAN-based models [36, 37], the use of synthetic images for training deep learning models has recently gained momentum following the advances in diffusion models [38, 39] and their wide availability. Diffusion models generate synthetic samples using an iterative denoising process conditioned on an input, usually a text prompt. Imagen [40] performs the denoising task in the pixel space while Stable Diffusion [41] and DALL-E2 [42] perform it in the latent space in order to decrease the computational cost of the generative process. In one of the pioneer works, Sarıyıldız et al. [43] studied the use of the pre-trained text-to-image diffusion model Stable Diffusion to generate a synthetic replacement for the original training dataset. Their experiments highlighted that while the model trained using the real train dataset reaches a higher accuracy on the real test dataset compared to the one trained on synthetic images, they perform on par for transfer learning tasks. Concurrently, Zhang et al. [44] proposed to expand real datasets using synthetic samples generated with various diffusion models and achieved significant improvement on various image classification datasets. Several authors proposed to use synthetic images generated by pre-trained diffusion models for different problems of computer vision such as zero-shot and few-shot learning [45], long-tailed image classification [46], incremental learning [47, 48], out-of-distribution detection [49], or semantic segmentation [50]. Tian et al. [51] showed that the representation learned using contrastive learning on synthetic images generated by Stable Diffusion outperforms state-of-the-art methods trained on real datasets.

Several authors [52, 53, 54, 55, 56] have considered the use of additional data for Class-Incremental Learning by leveraging datasets of real images belonging to classes different from the ones that the model has to learn. Most of these approaches assume that the additional dataset is unlabeled, restricting the use of those additional data samples to the distillation loss [52, 53, 56]. Jodelet et al. [47] first proposed to use large pre-trained text-to-image diffusion models for Incremental Learning in order to generate additional synthetic images for the classes previously encountered by the model. Using a textual prompt based on the class name, this method generates labeled samples that can be used for the distillation loss but also for any supervised losses such as the classification loss. More recently, pre-trained diffusion models were also used for Class-Incremental semantic segmentation [57] and object detection [58]. However, there remains a significant distribution gap between the real samples and the synthetic ones [47] whose impact on the classifier makes the use of pre-trained diffusion model for replaying samples from past classes especially difficult for Exemplar-Free Class-Incremental Learning.

While most of the methods for Class-Incremental Learning only focus on how to prevent the forgetting of past data, some authors have studied how to better prepare the model for future updates. Douillard et al. [59] took inspiration from zero-shot learning and proposed to incorporate into the classifier feature vectors of unseen future classes during training. Those possible future vectors, named ghost features, are produced by a generator using actual features of seen classes and prior information about all the classes. [60] proposed to use a pre-allocated Regular Polytope Classifier which can exploit future unseen classes as negative examples. Zhou et al. [61] proposed to reserve embedding space for future classes by pre-assigning virtual prototypes and by forecasting future classes using manifold Mixup [28]. Bellitto et al. [55] proposed to use an auxiliary dataset of real images containing classes unrelated to the main task in order to prepare the model for the future. Those additional data samples are used to learn more general and stable representation that can be leveraged when learning future classes.

Compared to previously mentioned methods, in this work, we aim to leverage pre-trained text-to-image diffusion models not for generating additional past data but for generating synthetic samples for possible future classes. We study how those synthetic images of future classes can be used during the initial incremental step and to what extent they can help to better prepare Exemplar-Free incremental learning methods relying on a frozen feature extractor.

In order to create a future-proof model, as illustrated in Figure 1, we introduce Future-Proofing Class Incremental Learning (FPCIL), a straightforward approach which can be combined with any method for Class-Incremental Learning relying on a frozen feature extractor. During the first incremental step, when training the feature extractor, in addition to the initial step dataset $\mathcal{D}_{1}$, an additional dataset $\mathcal{S}$ containing $\lvert\mathcal{Y}_{\mathcal{S}}\rvert$ different classes is used. While any labeled datasets could be used for $\mathcal{S}$, we propose a novel approach which consists in using images from classes that will be encountered by the model during the future incremental steps. In detail, the objective of the initial incremental step is extended from solely learning the initial classes $\mathcal{Y}_{1}$ into learning $\mathcal{Y}_{1}\cup\mathcal{Y}_{\mathcal{S}}$ where $\mathcal{Y}_{\mathcal{S}}=\bigcup_{i=2}^{T}\mathcal{Y}_{i}$ are the future classes. The feature extractor is jointly trained with a classifier on $\mathcal{D}_{1}\cup\mathcal{S}$. At the end of the initial step, the classifier is restricted to the base classes $\mathcal{Y}_{1}$ by dropping the weights corresponding to the future classes from the auxiliary dataset $\mathcal{S}$, and the feature extractor is frozen. Then, the training procedure continues according to the baseline method for the remaining incremental steps. Our proposed method is detailed in Algorithm 1.

In the initial step, our proposed method FPCIL trains on both the initial training dataset $\mathcal{D}_{1}$ and an additional dataset $\mathcal{S}$ which contains training data for predicted future classes. As only the data for the current step are available in the Class-Incremental Learning setting, we propose to use synthetic data instead of the real ones. We use a pre-trained text-to-image diffusion model to generate synthetic images from the future classes during the initial step. This means that only the labels of the future classes are necessary. Compared to recent approaches for Incremental Learning relying on a pre-trained diffusion model [47, 57, 58] for generating synthetic samples of past classes, our proposed method aims at generating synthetic samples of future classes instead. By primarily using the synthetic data to train the feature extractor, our proposed method circumvents the impact of the distribution gap between real and synthetic samples generated by the diffusion model, making it applicable to Exemplar-Free Class-Incremental Learning.

While the requirement of having access to the label of future classes may seem counter to the concept of Class-Incremental Learning, we argue that it is practical and compatible with most real-world applications of Class-Incremental Learning. In general, during the developing a deep learning model meant for continuously accumulating new knowledge while achieving a given task in an ever-evolving environment, both the task and the environment are known. Therefore, it is reasonable to assume that it is possible, based on this initial knowledge of the environment and task, to predict their possible future evolution. However, acquiring beforehand real data corresponding to these future states of the environment and task may be difficult, expensive, or even impossible in some cases which lead us to use synthetic samples instead. As a concrete example, we propose to consider the case of an agent that would be deployed on an embedded device for assisting its user inside his home. The agent would have to be able to learn new concepts based on data directly collected from its environment depending on the particular need of its user. It would have to rely on incremental learning as frequent retraining from scratch would not be possible on either a cloud server as uploading user’s data would be difficult due to privacy reasons, or in local due to the limited computational and memory capacity of the embedded device. Therefore, using our proposed method it is possible to train an initial model precisely tailored for each user (or group of users) before deployment, by relying on available knowledge about the user and its environment (interests, location, particular requests, etc.) to preemptively learn specific concepts that may appear in the future in addition to generic concepts. Since building a specific dataset of real samples for all possible needs of all possible users would not be tractable due to the significant cost in time and resources of the data collection process, the use of synthetic data appears as a faster and cheaper alternative. As a second example, we consider the case of an agent deployed on an embedded device for performing a task in a hard to access and potentially hostile environment where the agent has to be autonomous due to unreliable remote control possibility (deep sea, space, etc). The nature of the environment intrinsically limits the capacity to collect large training datasets in advance but its properties may be known. Using our proposed method, the agent could be trained before its deployment using synthetic data from generative diffusion models, or more realistically in this scenario using simulation-based tools. The agent would then be able to incrementally learn and adjust itself to the environment using real data collected while performing the task.

It should also be noted that based on our experiments detailed in the ablation study, the classes in the auxiliary dataset $\mathcal{S}$ do not need to be perfectly predicted to have a positive impact on the performance. For clarity, we refer to FPCIL as “FPCIL-Oracle” when the list of future classes has been perfectly predicted and “FPCIL-Partial (k%)” otherwise, with $k$ the percentage of future classes correctly predicted.

Predictions of the future classes can be done through expert knowledge or through automated methods. Here, we introduce a simple approach for automatically predicting the future classes based on recently popular large language models. Our proposed method relies on the GPT model [62] “GPT-3.5 Turbo Instruct” used through the OpenAI API. The large language model is queried for a completion task using the prompt “The dataset contains the following 100 classes: $c_{1}$, […], $c_{k}$,” where $c_{1}$ to $c_{k}$ are the names of the $k$ classes from the initial step, with $k$ equals to $\lvert\mathcal{Y}_{1}\rvert$. The number “100” is used here to encourage the model to generate a longer list of possible future classes as initial experiments without specifying a number would result in only a few number of guesses. In practice, the output of GPT has to be parsed to extract the predicted future classes. The completion function of GPT often continues the prompt as if it was part of a website or a scientific paper by appending imaginary details such as the number of images per class in the dataset, nonexistent download url for the dataset, or architecture of the model. Due to the random nature of the output from GPT, the prompt is queried 10 times and the predicted classes are ordered according to the number of times they have been predicted. The auxiliary dataset $\mathcal{S}$ is then restricted to the classes which were the most often predicted. We evaluated three versions with an increasing level of restriction: “FPCIL-GPT”, “FPCIL-GPT R1”, and “FPCIL-GPT R2”. Those approaches based on GPT also serve to imitate a more realistic application of Future-Proof Class-Incremental Learning by predicting various number of future classes with various accuracy.

Leveraging recent advances in diffusion models and the wide availability of generative text-to-image models pre-trained on natural images, we propose to use them for generating the synthetic images dataset of future classes following the procedure initially defined by Sarıyıldız et al. [43].

The auxiliary dataset $\mathcal{S}$ is generated using a pre-trained diffusion model $G$ conditioned solely on a textual prompt. As proposed in [43], the default prompt used is “$c,d_{c}$” with “$c$” the name of the class and “$d_{c}$” a simple description of the class. Adding the description of the class into the prompt is intended to ensure that the generated images belong to the targeted class and not to any homonyms: for example “crane” in the ImageNet dataset, which may refer to either the bird or the machine used to move heavy objects. The definition “$d_{c}$” also tends to restrain the domain of the generated images even though the domain could also be enforced by adding it to the prompt. To automatically create the prompts, we rely on WordNet [63] and use the lemmas of the synset as “$c$” and the definition of the synset as “$d_{c}$”. For each class, $n$ synthetic images are generated with $n$ equals to the number of images per class in the target dataset. Compared to traditional pre-training and concurrent method [55], our proposed method does not require the costly and fastidious process of collecting, curating, and annotating a dataset of real images. Examples of synthetic images generated for CIFAR100 are displayed in Figure 2.

The guidance scale is an important hyper-parameter of diffusion models relying on classifier-free guidance [64]. To some extent, it can be used to control the trade-off between the diversity and the quality of the generated synthetic images. For Stable Diffusion, the default recommended value for the guidance scale is often $7.0$ but following [47, 43] we use the lower value of $2.0$ as a higher diversity is beneficial for pre-training.

While this generation procedure allows our proposed method to achieves significant improvement compared to baselines, there remains an important distribution gap between the real and synthetic data [47]. Reducing this distribution gap and its impacts on the training of the feature extractor would further improve the performance of our proposed method. As possible future research direction, the diffusion model could be finetuned using the data from the first incremental step, a more sophisticated prompting mechanism could be implemented, or a selection process could be used to discard the synthetic samples too far away from the distribution of the target dataset.

Average incremental accuracy and final accuracy of our proposed method FPCIL are reported for CIFAR100 and ImageNet-Subset in Table 2 and Table 3 for both the Learning From Scratch and Learning From Half procedures.

Our proposed approach FPCIL considerably improves the average incremental accuracy of the state-of-the-art method FeTrIL [7], from 5.39 percentage points (p.p.) up to 22.12p.p. on CIFAR10 and from 4.24p.p. up to 33.00p.p. on ImageNet-Subset. This improvement is notably higher for the more difficult Learning From Scratch procedure which is particularly challenging for FeTrIL due to the limited quantity of data available during the first incremental step. Compared to using an auxiliary dataset of real images from classes different from those in the target dataset, our method also achieves significantly higher accuracy. This highlights the fundamental advantage of our proposed method: while the standard pre-training procedure improves the generalization capability of the feature extractor by leveraging additional data belonging to classes different from those in the target dataset, our proposed method aims to learn a feature extractor specifically designed for the incoming task by relying on data from future classes instead. Even if those data from future classes are synthetic, it allows to learn a feature extractor more suitable for the future tasks resulting in a significantly higher accuracy compared standard pre-training on real data. Moreover, on ImageNet-Subset, our proposed method FPCIL even outperforms a two times larger auxiliary dataset of real images, demonstrating that our proposed algorithm is more data-efficient that standard pre-training. Furthermore synthetic datasets can be rapidly generated whereas auxiliary datasets of real data require a long and costly collection, curation, and annotation process. Finally, even in the case where the future classes have not been accurately predicted, the auxiliary dataset of synthetic images performs on par with the standard pre-training approach relying on real datasets: for the Learning From Half procedure, the difference of average incremental accuracy between using ImageNet-Compl or its synthetic counterpart is at most 0.29p.p.. This makes our proposed method beneficial in every setting.

On CIFAR100, our more practical approach FPCIL-GPT also reaches a higher accuracy than the baseline method FeTrIL yet slightly lower than the FPCIL-Oracle. On CIFAR100 in the Learning From Scratch procedure with 10 steps of 10 classes (B0 Inc10), the less restricted FPCIL-GPT predicted a total of 150 classes with 53 that actually are among the 90 future classes while the most restricted FPCIL-GPT R2 predicted only 53 classes with 30 among the 90 future classes. All the details regarding future class predictions using GPT are reported in Table 1. While the more restricted version FPCIL-GPT R1 and R2 tend to predict a lower number of classes that are not among the future classes, the less restricted version predicts a higher number of future classes, achieving higher performances. There is a trade-off between the precision and recall in the prediction of future classes, while predicting more classes improve the performance it also increase the computational cost of the first incremental step due to the cost of generating synthetic images and training the feature extractor. In the ablation study, we further analyze the impact of the correctness of the prediction of future classes.