LVP-CLIP: Revisiting CLIP for Continual Learning with Label Vector Pool

We consider a sequence of tasks $S=\{S^{1},\cdots,S^{M}\}$, where $M$ is the total number of tasks. Each task is a set, $S^{t}=\{(x_{1}^{t},y_{1}^{t}),\cdots,(x_{n^{t}}^{t},y_{n^{t}}^{t})\},t\in[1,...,M]$, where $(x_{i}^{t},y_{i}^{t}),i\in[1,...,n^{t}]$, is a pair containing the input $x_{i}^{t}\in X$ and its corresponding label $y_{i}^{t}\in Y$ and $n^{t}$ is the total numbers of samples. The goal of continual learning is to train a model $f(\theta):X\rightarrow Y$ continuously over time on a set of tasks, arriving sequentially, such that it learns the new tasks without forgetting the old tasks.

Task-, Domain- and Class-Incremental Learning (TIL, DIL, CIL) are the three main scenarios of continual learning. Domain-incremental learning assumes that the number and labels of classes remain consistent across tasks, and the only difference among tasks is the distribution of the input data. Both Task- and Class-incremental learning address the scenario, where each task introduces a distinct set of new classes to be learned. Task-incremental learning assumes a known task identity at inference time, whereas the class-incremental learning does not make such assumption.

CLIP is a vision-language model trained on text-image pairs. It consists of a text encoder $g_{\theta_{t}}(\cdot)$ and an image encoder $f_{\theta_{i}}(\cdot)$. Given a sentence $txt$ and an image $img\in\mathbb{R}^{H\times W\times C}$, the text and image embeddings are $g_{\theta_{t}}(txt)=T$ and $f_{\theta_{i}}(img)=I$ respectively, where $T,I\in\mathbb{R}^{D}$ with $D$ denoting the dimension of the embeddings. Given a dataset containing $K$ classes, the $txt$ is a phrase like “a photo of a [$y$]”, where $y\in[1,...,K]$ is the index of the label, and [$y$] denotes the class name of the label. The probability of labeling the test image $img$ with the class $y$ is computed as:

where $\langle\cdot,\cdot\rangle$ denotes the similarity function. Three commonly used similarity functions are L1 norm, L2 norm and cosine similarity, denoted by $\langle\cdot,\cdot\rangle_{L1},\langle\cdot,\cdot\rangle_{L2},\langle\cdot,\cdot\rangle_{Cos}$, respectively. To classify an image, the class with the highest probability is chosen, i.e.

As discussed in Sec. 3.2, CLIP classifies an input image by comparing the distance between the image embedding and a list of text embeddings that represent class labels. The query vector with unknown-ID is compared with a group of key vectors with known IDs, and the query vector is assigned the ID of the key vector with the highest similarity. Following this concept, we define label/labeled vector $L\in\mathbb{R}^{D}$ as a vector with known ID, such as the class or domain name, or task ID, etc. A LVP for class $k$ is a set of label vectors with ID $k$, denoted as $L^{k}=\{L^{k}_{1},L^{k}_{2},\cdots,L^{k}_{P^{k}}\},\text{where }L^{k}_{i}\in\mathbb{R}^{D},i\in[1,..,P^{k}],k\in[1,...,K]$, and $P$ is the pool size. The similarity between an image embedding $I$ and $L^{k}$ is computed as the maximum similarity between $I$ and all instances in $L^{k}$:

The probability that a testing image $img$ belongs to class $y$ can be computed as:

As can be seen, Eq. 1 is a special case of Eq. 4, where the LVP contains only one labeled vector, which is the text embedding of the class name, i.e., $L^{k}=\{T^{k}\},\text{and }P^{k}=1$. We denote the computational complexity for classifying one image as $O$, and represent it using the number of times the similarity function is calculated, i.e., $O=\sum_{k=1}^{K}{P^{k}}$. If all classes have the same pool size $P$, then the complexity simplifies to $O=\sum_{k=1}^{K}{P^{k}}=P\times K$.

Our LVP-based approach is a general framework for similarity-based classification and extends CLIP in two ways: (1) the test image is no longer restricted to comparison with text embeddings, it can be compared with any labeled vectors with matching dimensions; (2) each class can be represented by one or multiple labeled vectors, which may be obtained from different modalities.

The training of CLIP ensures that matching images and text phrases will be mapped to nearby locations in the feature space. Intuitively, we expect that embeddings of inputs from the same modality (e.g., image-to-image or text-to-text) will be closer to each other than those from different modalities, as illustrated in Fig. 2. As discussed in LABEL:sec:\labelpool, the task of classification is to identify the labeled vector most similar to the query. This raises an interesting question: can we use image embeddings by themselves in the LVP instead of text embeddings?

We verified this idea on CIFAR100 [14] dataset by using the embeddings of the training images as the LVP. The experimental results are shown in Tab. 1. When 30% of the training data is used as the LVP, the classification accuracy already surpasses that achieved by using the text labels as LVP. With the entire set of training data used as the LVP, testing accuracy improves by almost 5% compared to using text labels. Since we used CLIP as the foundation model and leveraged its powerful embedding capability, we refer to our approach as LVP-CLIP. The drawback of LVP-CLIP is that the computational and memory complexity increases as the pool size $P$ grows, which will be addressed next.

To address the computational and memory complexity while maintaining performance, we propose three different designs using LVP: (i) LVP-CLIP-I uses only image embeddings in the LVP, and reduces the pool size $P$ to 1, so that its inference complexity is the same as the zero-shot learning while providing better accuracy; (ii) LVP-CLIP-IT extends the LVP by combining text embeddings with image embeddings; (iii) Unlike LVP-CLIP-I and LVP-CLIP-IT, LVP-CLIP-C does not use the similarity function to compare the test image with LVP. It trains a classifier using information stored in LVP and applies it for classification.

The overall framework for the three variations of LVP-CLIP is shown in Fig. 3. The traditional CLIP-based classifier compares the encoded test image with the text-embedding of the class label. This can be considered as a special case of LVP, where the label vector is the text embedding. Therefore, we also refer to it as LVP-CLIP-T.

Reducing LVP Size (LVP-CLIP-I). As noted in Sec. 3, using the whole training set as the LVP is effective, yet expensive in terms of computational and memory requirements. If we can use only one vector in the LVP, which one would be the best representative of the whole training set for each class? Each image embedding $I^{k}$ is a set of image features, $I^{k}=\{E^{k}_{1},E^{k}_{2},\cdots,E^{k}_{D}\},\text{where }E^{k}\in\mathbb{R}$. We plot the distributions of those features in the same class and compare such distributions across different example classes in Fig. 4. In the figure, the superscript of the features indicates class index, and subscripts represent the feature index. As can be seen, within a class, each feature roughly follows a Gaussian distribution. Across classes, features have different combinations of means. Therefore, we use their mean value as the representative label vector for each class,

where $\tilde{I}^{k}_{i}\in\mathbb{R}^{D}$ is a training image embedding of class $k\in[1,K]$ and the total number of training images from class $k$ is $\tilde{N}^{k}$. As shown in Tab. 2, by using the average embedding of the whole training data in each class as the LVP, we not only reduce the inference complexity but also improve the performance by about 2% with LVP-CLIP-I.

Enrich Image Embedding Representation with Text (LVP-CLIP-IT). The success of CLIP indicates that text embeddings and image embeddings follow similar distributions and have very high correspondence. In cases, where the distribution of training images is not similar to that of the testing images, information from another modality, i.e., text, may serve as an unbiased prompt. Therefore, with LVP-CLIP-IT, we design the LVP as a weighted combination of the text embedding and the average embedding $\bar{\tilde{I}}^{k}$ of the training images as follows:

where $\alpha^{k},\beta^{k}\in\mathbb{R}^{D},$ are trainable vectors to balance the text embedding and image embedding, respectively. Experimental results in Tab. 2 show that combining image embedding and text embedding improves the performance by another 2% over LVP-I.

With LVP-CLIP-I and LVP-CLIP-IT, the embedding vectors belonging to difference classes are stored, and classification is performed by comparing the distance between the embedding of a test image against each stored labeled vector using a similarity function. This process treats all feature dimensions as equally important. To handle cases, where some features should have higher importance than others in the classification process, we build a simple linear classifier that maps an embedding vector to a class prediction, $f_{\theta}(\cdot):\mathbb{R}^{D}\rightarrow[0,1]^{K}$ and train it using data only from the LVP. For a test image $\hat{I}$, the prediction is:

Among the three variations of LVP, LVP-CLIP-I has no trainable parameters. It solely relies on the averaging of the embeddings of training images. LVP-CLIP-IT and LVP-CLIP-C, on the other hand, have a small set of parameters that are trainable.

For LVP-CLIP-IT, we set $\alpha,\beta$ as task-specific parameters to avoid forgetting, and use the entropy loss to train them. Given task $t\in[1,M]$ and class $k^{t}\in[1,K^{t}]$, where $M$ and $K^{t}$ are the total number of tasks and the number of classes in task $t$, respectively, the loss function used to train $\alpha$ and $\beta$ is:

For each new task, a new set of $\alpha$ and $\beta$ will be optimized.

The parameter $\theta$ in LVP-CLIP-C is shared among all tasks. Since it is trained using labeled vectors stored in LVP, every time a new task is added, it will be trained again with the current LVP. The process is similar to experience replay. In this way, it also does not suffer from the forgetting. Again, entropy loss is used for the training. Given the LVP $L^{k},k\in[1,K]$ of each class, the loss function can be written as:

For all three LVP variants, the size of the LVP grows as the new tasks are learned, albeit at a very slow rate. For each class, we only need one or a few labeled vectors based on the difficulty and distribution of each dataset. Taking ImageNet100 as an example, each labeled vector $L\in\mathbb{R}^{D}$ is of size $D=768$, which is only $0.5\%$ of the size of an image $(3\times 224\times 224)$. Even for 100 classes, the overall size of LVP is $76800$, equivalent to the size of 0.5 images. The same amount of memory is required to store the text embeddings for classes, if CLIP is used to classify input images.

Furthermore, LVP-CLIP is task-order invariant, since each LVP is generated independently. As a result, its performance is not affected by task order. Additionally, since new label vectors do not modify the existing ones, LVP-CLIP exhibits minimal forgetting. The independence of the label vectors allows for parallel learning of different classes and making it simple to merge multiple classes —a distinct advantage over existing approaches. Finally, the LVP pool is not a rehearsal buffer, since it does not store raw images, making it less vulnerable to user privacy leakage.

We first evaluate our method on two popular 2D image datasets namely CIFAR100 [14] and ImageNet100 [7]. Following the setup in [32], we select 100 classes from the original ImageNet. Both CIFAR100 and ImageNet100 are divided into 10 tasks, with each task consisting of 10 classes.

We compare our methods with two rehearsal-based methods, iCaRL [24] and ARI [31], and three CLIP-based methods, CoOp [45], Continual-CLIP [29] and AttriCLIP [32]. For a fair comparison, all methods are implemented using ViT-L [8], except for iCaRL and ARI, which are implemented with ResNet [10].

In Tab. 3, the best and 2nd-best performances are shown in bold and with underline, respectively. For ImageNet100, all variants of LVP-CLIP outperform other CLIP-based and experience-replay methods with significant margins. Specifically, LVP-CLIP-IT and LVP-CLIP-C provide 9.2% and 9.3% improvement, respectively, over the best performing CLIP-based method AttriCLIP. On CIFAR100, LVP-CLIP-IT has 0.6% higher accuracy than AttriCLIP.

For these experiments, we use two popular public datasets, namely DomainNet [22] and CORe50 [18]. DomainNet includes objects from 345 classes. Each object is represented by images spanning six domains: clipart, infograph, painting, quickdraw, real-world and sketch. Each domain has its own dedicated training and testing datasets. Therefore, it has 6 training tasks and 6 testing tasks.

The CORe50 dataset, on the other hand, consists of 50 classes across 11 domains. Of these, 8 domains are used for training data, presented sequentially one at a time as 8 tasks, while 3 domains are reserved for testing. Importantly, the testing domains are not part of the training process, making CORe50 also suitable as a domain adaptation dataset.

The performance is measured as the average accuracy over all testing domains. For every class, we generate a label vector for each trained domain, resulting an LVP of size 6 for DomainNet and 8 for CORe50.

We compare our method with two regularization-based approaches, EWC [13] and LwF [16], a rehearsal-based method, ER [9], and two prompt-pool-based methods, L2P [36] and S-Prompts [33]. For fair comparison, all methods are implemented with ViT-L [8]. As seen in Tab. 4, for both DomainNet and CORe50 datasets, the variants of LVP-CLIP provide the top and second-best performances outperforming all the baselines.

We present the Cross-Task Incremental Learning (CTIL) experimental setting, which uses an ID-UNKNOWN multi-dataset for task-incremental learning. Here, a task refers to the unit for continual learning, encompassing both class-incremental and domain-incremental learning across dataset. Our motivation is to show how well a method can perform in a more realistic setting to continuously learn new tasks. CTIL presents a significant challenge in continual learning, as it requires a model to perform both CIL and DIL. We conduct experiments on a four-dataset CTIL benchmark, which combines CIFAR100, ImageNet100, DomainNet and Core50, resulting in a total of 595 (100+100+345+50) distinct object classes, divided into 34 (10+10+6+8) training tasks and 29 (10+10+6+3) testing tasks. The 34 training tasks are processed sequentially in a random order.

Since this setting is too challenging for most of the existing incremental learning frameworks, we chose only L2P [36] and DualPrompt [35] as the baseline methods for comparison. We also present the performance of our methods and the baselines on each individual dataset. In the second-to-last column of Tab. 5, we show the “Ideal” performance of each technique in the combined dataset. This is calculated as the weighted average of their accuracies on individual datasets, adjusted by the number of test samples in each. For example, the “Ideal” score for LVP-CLIP-I is calculated as $(80.2*10+91.8*10+70.1*6+86.1*3)/(10+10+6+3)=82.7$. The Difference between the Ideal and the actual accuracy is shown in the last column. As a reference, the upper-bound of the classification accuracy is also provided, which is obtained by training a classifier using all the training data with a ViT-L-based backbone.

It is not surprising that the actual accuracy is consistently lower than the ideal accuracy. This is due to two main reasons: (i) as the number of classes grows, their separation in the feature space diminishes, making it harder to distinguish them; and (ii) as tasks are trained sequentially, earlier tasks are increasingly susceptible to forgetting. Since LVP-CLIP-I does not modify previously learned knowledge, it largely avoids forgetting. The 1.8% drop in accuracy is mainly due to more congested feature distributions. LVP-CLIP-C shows a slightly higher degradation (4.0%), likely because its simple linear classifier does not work well in a congested feature space. Overall, the LVP-CLIP-based approaches closely approximate the ideal performance, indicating that the object classes in these four datasets remain separable in the high-dimensional feature space. This also indicates that the significant 42% accuracy drop observed in L2P and DualPrompt is primarily attributed to forgetting.

Although prompt-pool-based approaches show slightly better performance on certain datasets (e.g., CIFAR100 and DomainNet), they greatly suffere from forgetting as the number of classes increases. Moreover, DualPrompt and L2P require roughly twice as much computation at inference compared to LVP-CLIP. In addition, LVP-CLIP-based learning only requires forward propagation, whereas L2P and DualPrompt rely on backpropagation, making LVP-CLIP far more efficient in terms of learning complexity.

Ablation studies are conducted to assess the effect of various design variables on performance. All studies were performed on CIFAR100.

Text Prompt Quality. The impact of different text prompts is shown in Tab. 6. Here, we use LVP-CLIP-T to represent the traditional CLIP-based classification, where the text (T) embedding serves as the label vector for similarity. Two slightly different text phases are used for LVP-CLIP-IT and LVP-CLIP-T and their performances are compared. As can be seen, tradiational CLIP is highly sensitive to text prompt quality, making it essential to optimize the prompt. However, the proposed LVP-CLIP-IT is less susceptible to the text quality, since it uses text as supplementary information alongside image features.

Initialization of $\alpha$. The performance of LVP-CLIP-IT is sensitive to the initial value of $\alpha$ as shown in Tab. 7. It is a good practice to set $\alpha$ to 0.5 when text quality is good, and decrease it as text quality declines.

Training dataset size. The effect of training dataset size on LVP-CLIP is shown in Tab. 8. Since the labeled vector is the average embedding of the training images, a larger training set generally improves the label vector’s approximation of the distribution mean, provided the data is randomly sampled. However, it appears that around 150 training images sufficient to obtain a relatively good estimation of the mean. As expected, with a small number of training images, the LVP-CLIP-IT significantly outperforms LVP-CLIP-I and LVP-CLIP-C, due to the additional information provided by the text embedding. For more experimental results and analysis, please refer to the Suppl. file.