DesCLIP: Robust Continual Adaptation via General Attribute Descriptions for Pretrained Vision-Language Models

Continual Learning (CL) investigates how deep models can incrementally learn knowledge. Existing CL research can be categorized into several types based on the strategies they employ. Among these, regularization-based methods [11, 31, 32] introduce regularization terms during model training to penalize forgetting old knowledge. These regularization terms can either focus on protecting model parameters [31] or on output distributions [11, 33] (e.g., knowledge distillation). Dynamic network-based methods [34, 35, 36, 37] aim to learn the model by introducing new structures for new tasks while preserving old knowledge, although this incurs substantial overhead as model parameters increase with the number of tasks. Recently, replay-based methods have become increasingly common. Data replay methods [38, 37] assist models in retaining old knowledge by recalling a small number of real samples. Additionally, some methods [39, 40, 41] recall old knowledge by storing sample features and the distributions of these features. However, replay-based methods introduce storage costs and require repetitive computation for old data.

In recent years, studies such as [42, 43, 44, 45] have predominantly focused on integrating additional components for incremental tasks, such as learnable prompts [44, 45, 46, 47] and adapters [42, 43], into pretrained models. This integration necessitates the development of methods for selecting and evaluating the relevance of these components to ensure both their appropriateness and compatibility with the pretrained model. However, a significant limitation arises as the number of tasks increases: the associated computational and storage costs grow substantially, posing challenges to scalability and efficiency.

With advancements in pre-training techniques, large-scale foundation models [1, 48, 49, 50, 51] have significantly impacted the industry. For instance, Vision-Language Models such as Contrastive Language-Image Pretraining (CLIP) [1] and Adversarially Learned Inference for Image-Text Matching (ALIGN) [2] have demonstrated remarkable zero-shot capabilities for general tasks. However, despite being pre-trained on over 400 million image-text pairs, CLIP still face challenges in specific downstream tasks, such as accurately identifying certain types of vehicles and lizards.

To better adapt VLMs for downstream tasks, various text prompt-based fine-tuning methods [3, 4, 6, 7, 52] have been proposed, which can enhance VLM performance on specific tasks. In more complex scenarios, learnable prompts can be inserted into intermediate layers [8] to incorporate more sophisticated general knowledge. Additionally, the integration of adapter structures [53, 19, 20] has also been shown to be an effective strategy. Other approaches [54, 55, 56, 57] focus on the representation alignment of VLMs and aim to improve the transfer of general knowledge. Although these methods demonstrate excellent performance in CLIP transfer tasks, they are inherently unsuitable for incrementally learning, as the additional learnable structures cannot effectively mitigate catastrophic forgetting.

Investigating the continual learning and adaptation of VLMs for diverse downstream tasks holds significant value, as it reduces data storage requirements and computational redundancy while addressing the challenge of inaccessible previous data. It is crucial to protect the model’s gerenic pretrained knowledge and previously learned knowledge. The full fine-tuning strategies discussed in II-A will lead to significant forgetting of pre-trained knowledge, which is a notable distinction between pre-trained foundation models (e.g., CLIP) and small-scale deep models. Additionally, frameworks such as CoOp [3] and CoOpOp [4] have been shown to have limited adjustment capabilities for VLMs in incremental tasks due to their reliance on shared structures and contextual prompts across all tasks, leading to forgetting old knowledge during the process of fitting new knowledge. To solve this, Wang et al. [16] introduced AttriCLIP, which establishes a shared attribute bank for all tasks and selects suitable contexts based on visual images to bridge the gap between images and text. Yu et al. [18] proposed using a mixture of experts (MoE) framework to adapt knowledge for different tasks, decoupling the model’s zero-shot capabilities from its specialized task abilities. From the perspective of parameter sparse updating, efforts from SPG [22], SparseCL [21], and SPU [10] have aimed to update VLM parameters selectively by employing appropriate “important parameter” selection patterns; for example, SPU selects more important parameters for updates based on the gradients accumulated by batches. Additionally, Zheng et al. [9] and Yu et al. [23] proposed the use of additional reference datasets to facilitate knowledge distillation in a VLM, effectively mitigating the forgetting of generic knowledge.

1) Continual Learning Formulation: A sequence of task datasets is denoted as $\{\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{T}\}$. During training on task $\mathcal{D}_{t}$ ($t\in\{1,2,\ldots,T\}$), access to data from previous tasks $\{\mathcal{D}_{1},\mathcal{D}_{2},\ldots,\mathcal{D}_{t-1}\}$ is either highly restricted or entirely unavailable. In class-incremental learning (CIL), datasets for different tasks are introduced sequentially. Each task $t$ is associated with a unique set of classes $\mathcal{C}_{t}=\{C_{t,1},C_{t,2},\ldots,C_{t,|\mathcal{C}_{t}|}\}$, where $|\mathcal{C}_{t}|$ denotes the number of classes in task $t$. The classes associated with different tasks are disjoint:

2) CLIP for Incremental Tasks: CLIP [1] comprises an image encoder $\mathcal{F}_{\Theta}(\cdot)$ and a text encoder $\mathcal{T}(\cdot)$. Specifically, an image $x\in\mathbb{R}^{H\times W\times 3}$ and a text prompt, referred to as the rudimentary prompt $\mathbf{RP}_{y}$, are input into $\mathcal{F}_{\Theta}(\cdot)$ and $\mathcal{T}(\cdot)$, respectively, producing a visual embedding $\mathbf{z}\in\mathbb{R}^{D}$ and a text embedding $\mathbf{w}_{y}\in\mathbb{R}^{D}$:

Here, $\mathbf{RP}_{y}$ is derived from hand-crafted prompts, typically following a template such as “A photo of a [CLS],” where [CLS] represents the specific class name. The probability of classifying a test image $x$ as class $y_{i}$ is computed using the softmax function:

where $\tau$ is the temperature parameter, $\mathbf{w}_{y_{k}}$ is the class text embedding derived from the rudimentary prompt of the $k$-th class, and $K$ denotes the total number of downstream classes.

Building on this architecture, ContinualCLIP [58] tackles the challenge of continual learning with a training-free approach. For each new task $t$, the text embedding set $\mathbf{W}_{t}$ is expanded to incorporate embeddings for the new task’s classes. At task $t$, the updated text embedding set is defined as:

where $\mathbf{w}_{j,k}$ denotes the text embedding for the $k$-th class of task $j$ encountered so far. Consequently, the prediction for a test image $x$ after task $t$ is computed as:

The overall architecture of our proposed framework is shown in Fig. 3. Within our framework, the CLIP’s textual encoder $\mathcal{T}(\cdot)$ remains fixed and comprises two input branches: one processes rudimentary prompts derived from basic class names, while the other generates a diverse pool of general attribute (GA) description candidates via a language assistant. To obtain highly relevant visual-GA text pairs, we introduce the anchor-based embedding filter (AEF). The AEF identifies the most relevant attribute description embeddings from the candidate pool with respect to the current visual features. These filtered embeddings are then paired with visual features to compute a class-agnostic instance-matching loss, which is utilized to fine-tune the visual encoder $\mathcal{F}_{\Theta}(\cdot)$. Concurrently, the text embeddings are gradually calibrated to align with shared attribute embeddings, further enhancing the consistency among representations of vision, GA, and downstream classes.

CLIP establish robust visual-textual associations during the pre-training phase through instance-level contrastive learning. However, most existing research overlooks this foundational capability, conventionally relying on fixed, hand-crafted templates combined with class names as prompts to derive “prior” classification weights via the textual encoder. Although Wang et al. [16] introduced an “attribute bank” to enable attribute sharing across different task instances, this approach lacks intrinsic relevance to specific classes. For instance, attributes such as “white” and “grass” fail to provide meaningful distinctions between classes like “cat” and “dog”.

To address this limitation, we propose the use of a language assistant to generate rich, contextually relevant attribute descriptions for specific classes. The language assistant utilizes an advanced large language model (LLM) with a generalized understanding of downstream task entities. Drawing inspiration from [25, 26, 28], we design a variety of describe-request prompts (DRPs) to guide the language assistant in generating visually relevant attribute descriptions. Examples of basic DRPs include:

• •

  Q: Describe what does a/an [CLS] look like?

• •

  Q: Describe a/an [CLS]’s attribute features.

• •

  Q: Describe a/an [CLS]’s outlook features.

Additionally, more complex prompts are designed to produce discriminative attribute descriptions, such as:

• •

  Q: Describe what are some attribute characteristics of [CLS] compared with other [P-CLS], visually.

• •

  Q: Describe what kind of [P-CLS] a/an [CLS] is, visually.

Here, [P-CLS] refers to the parent class of [CLS]. Fine-grained DRPs are also employed for tasks with a known general scope, such as identifying objects within the “birds” supercategory:

• •

  Describe a/an [CLS]’s attributes from its beak, eyes, body, belly, tail, wings, breast, etc.

The DRP-guided general attribute description generation are illustrated in Fig. 4. The language assistant generates $n_{dsc}$ attribute description candidates (DCs) for the $k$-th class of incremental task $t$, denoted as:

where $k\in\{1,2,\ldots,|C_{t}|\}$. These DCs are then embedded using the textual encoder $\mathcal{T}(\cdot)$ to produce attribute embedding candidates (ECs):

Each element of $\mathbf{EC}_{\{t,k\}}$ has the same dimension as the rudimentary text embedding $\mathbf{w}_{t,k}$.

The generated embedding candidates of GAs do not always align with visual representations due to potential domain discrepancies or unrelated information in the text descriptions produced by the language assistant. To establish robust vision-GA associations, we propose an anchor-based embedding filter (AEF) mechanism to refine the embedding candidates. This mechanism identifies candidates that sufficiently match the visual representations, enabling the construction of approximate image-text pairs tailored to the specific requirements of the task.

For a training sample $(x_{i},y_{i})$, where $y_{i}=c=\{t,k\}$ is assumed, the label $c$ is considered to correspond to the $k$-th class of incremental task $t$, with $k\in\{1,2,\dots,|C_{t}|\}$. As illustrated in Fig. 3, the inputs to AEF include the visual features $\mathbf{z}_{i}=\mathcal{F}_{\Theta}(x_{i})$, the rudimentary text embedding $\mathbf{w}_{c}=\mathcal{T}(\textbf{RP}_{y_{i}})$, and the embedding candidates $\mathbf{EC}_{c}$. The cosine similarity between the visual features $\mathbf{z}_{i}$ and the rudimentary text embedding $\mathbf{w}_{c}$ is calculated as:

Subsequently, the similarity scores between the visual features and each embedding candidate in $\mathbf{EC}_{c}$ are calculated:

where $j\in\{1,2,\dots,n_{dsc}\}$. To mitigate the risk of overfitting in the CLIP’s visual encoder, visual features with low relevance to either class text or GA descriptions should be filtered out. Hence, a condition $\chi(\mathbf{z}_{i})$ is defined to filter visual features as:

Here, $\delta_{d}$ is a predefined threshold. The similarity between retained visual features and the rudimentary text embedding is used to define the anchor threshold $AT_{i}^{c}$:

We posit that embedding candidates in $\mathbf{EC}_{c}$ exhibiting a similarity score surpassing a threshold $\gamma$ (relative to the anchor threshold $AT_{i}^{c}$) are more consistent with the visual features of the current sample. These candidates are filtered as follows:

where $j\in\{1,2,\dots,n_{dsc}\}$, and sorted with descending order according to $\textbf{EC\_S}_{c}^{i,j}$. To reduce the potential influence of domain discrepancies arising from contextual descriptions, we further restrict the selection process by exclusively retaining attribute description sentences in $\mathbf{DC}_{c}$ that explicitly include the class name as a noun (i.e., [CLS]+[GA]).

This section introduces the methodology for optimizing the visual and textual branches of CLIP in incremental tasks, leveraging the filtered embeddings identified for relevant training samples. Within the CLIP architecture, the optimization focuses on the visual encoder $\mathcal{F}_{\Theta}(\cdot)$ (specifically, the initial MLP layers within each Transformer block [10]) and the rudimentary text embeddings introduced for the current task.

1) Instance Matching: To align the visual features with highly relevant embedded GA descriptions, we select the most closest textual representation $\mathbf{h}_{i}$ as the paired text embedding:

The instance matching loss $\mathcal{L}_{\text{IM}}$ is computed across the batch as:

where

Here, $\tilde{\tau}$ denotes an elevated temperature scaling factor, defined as $\tilde{\tau}=10\tau$. The set $P$ represents the indices of valid samples within a batch of size $B$ and is specified as:

where $\chi(\mathbf{z}_{i})$ is the condition defined in III-D. For each incremental task $t$, we tune the model using a contrastive learning framework similar to the pretraining strategy of CLIP [1]. To mitigate forgetting, we adhere to a “nearest matching” principle, aligning visual features with GA description embeddings exhibiting higher correlations. This approach minimizes the risk of overfitting visual features to specific class text embeddings, maintaining both generalization and previous knowledge.

2) Text Embedding Calibration: General attribute descriptions play a pivotal role in guiding the calibration of text embeddings to achieve better alignment with visual representations. This alignment is particularly crucial because the original rudimentary text embeddings often exhibit weak correlations with visual features in “unfamiliar” downstream tasks. Such misalignment can lead to overfitting in the visual branch of the VLM to class labels, thereby exacerbating forgetting. To mitigate this issue, we propose a weight-shifting mechanism that calibrates the rudimentary text embeddings for the classes in task $t$. This mechanism repositions the text embeddings toward representative attributes shared across the corresponding visual features, fostering stronger alignment between shared general attributes and class-specific text embeddings. Specifically, we define a shifting weight $\mathbf{s}_{t,k}\in\mathbb{R}^{D}$, and a shift transformation $\Psi(\cdot,\mathbf{s}_{t,k})$ for the calibration of rudimentary text embedding $\mathbf{w}_{t,k}$, where $\{t,k\}$ representing the $k-$th class of incremental task $t$, $k\in\{1,2,...,|C_{t}|\}$. The calibrated text embedding $\mathbf{w^{\prime}}_{t,k}$ can be obtained as:

The key to text embedding calibration is to ensure a strong correlation with the visual representations of the class while preventing an excessive focus on any single attribute text. Therefore, $\mathbf{w}^{\prime}_{t,k}$ should be aligned with $\mathbf{FE}^{i}$, which is filtered based on the visual features $\mathbf{z}_{i}$ of the class $c=\{t,k\}$:

where $\beta$ is a parameter that determines whether the alignment of text embeddings places greater emphasis on their absolute distance in the text space (e.g., Euclidean distance) or on the directional consistency. Text alignment loss $\mathcal{L}_{\text{TA}}$ of the current batch is: $\mathcal{L}_{\text{TA}}=\sum_{i\in P}\mathcal{L}_{\text{TA}}^{i}.$

Since, in the context of continual learning, data from previous tasks cannot be revisited during the current task, the weights $\mathbf{w^{\prime}}_{t}=\{\mathbf{w^{\prime}}_{t,1},\mathbf{w^{\prime}}_{t,2},...,\mathbf{w^{\prime}}_{t,|C_{t}|}\}$ are calibrated solely within the scope of the current task $t$. Consequently, only the shifting weights $\mathbf{s}_{t}=\{\mathbf{s}_{t,1},\mathbf{s}_{t,2},...,\mathbf{s}_{t,|C_{t}|}\}$ associated with the current task are learnable, whereas the shifting weights $\mathbf{s}_{0:t-1}=\{\mathbf{s}_{0},\mathbf{s}_{1},...,\mathbf{s}_{t-1}\}$ from previous tasks remain fixed.

3) Reconstructed Intra-task Classification: To ensure alignment between text embeddings and the visual branch during optimization, we reconstruct the classification loss for task $t$. This loss constrains the calibration of text embeddings to remain within the low-loss region of the classification space for the current task, which is a critical prerequisite. Specifically, for each visual feature $\mathbf{z}_{i}$ in a batch, its similarity to all calibrated text embeddings for the current task is computed to generate predicted logits. These logits are aligned with the ground truth label $y_{i}$, and the prediction classification loss is calculated as:

where $\mathbf{w^{\prime}}_{y_{i}}$ represents the calibrated text embedding for the class corresponding to $\mathbf{z}_{i}$.

4) Optimization: To achieve optimal performance, the total loss for batch optimization is defined as:

where $\lambda_{\text{IM}}$, $\lambda_{\text{TA}}$, and $\lambda_{\text{RIC}}$ are balancing factors that control the contributions of the respective loss terms.

The GA descriptions and embeddings we introduce do not participate in the inference stage of the CLIP after each training phase, which effectively avoids additional storage overhead and eliminates any increase in inference time. Consequently, the model obtained through our method maintains identical parameter size and inference time during the inference stage as the original zero-shot CLIP. In inference stage after task $t$, we leverage the calibrated text embeddings of all seen classes: $\mathbf{W^{\prime}}_{t}=\bigcup_{j=1}^{t}\bigcup_{k=1}^{|C_{j}|}\mathbf{w^{\prime}}_{j,k}.$ Hence, the probability of predicting the testing image $x$ as the class $y_{i}$ can be expressed as:

1) Datasets: The evaluation experiments for continual learning are conducted on CIFAR100 [29], ImageNet-Subset [24] and CUB-200 [30]. ImageNet-full [24] is utilized as a control set to evaluate the retention of the pretrained generalization knowledge in CLIP.

• •

  CIFAR100 [29] introduced a dataset comprising 60,000 images, each with a resolution of $32\times 32$, distributed across 100 distinct classes. Each class includes 500 training samples and 100 testing samples. CIFAR100 has become a widely recognized benchmark for evaluating continual learning methods. Its low-resolution images pose a significant challenge for VLMs that are pre-trained on high-resolution datasets, highlighting the difficulty of adapting to such datasets in continual learning scenarios.

• •

  ImageNet-Subset [24]. Following the class configuration proposed in [32], we select the challenging ImageNet-Subset benchmark for method evaluation. This subset consists of fine-grained animal images from 100 categories that are unfamiliar to VLMs and prone to misclassification due to their subtle differences.

• •

  Caltech-UCSD Birds-200 (CUB-200) [30] comprises 11,788 bird images distributed across 200 categories. The subtle variations among bird images pose significant challenges for VLMs to achieve accurate identification. Moreover, the relatively small size of the dataset further intensifies the difficulty, particularly in few-shot scenarios.

• •

  ImageNet-full [24] is a large-scale image dataset containing over 1.2 million images across 1,000 classes, widely used for pretraining and evaluating visual representation learning models.

2) Metrics: To evaluate the continual learning performance of classification models, we employ two primary metrics: ‘Last’ and ‘Avg’. ‘Last’ denotes the average accuracy across all classes after the model has completed training on the final task. ‘Avg’ represents the mean incremental accuracy calculated over all tasks the model has learned thus far. In addition, The control set accuracy ‘C.’ [10] is employed to evaluate the retention of CLIP’s zero-shot generalization knowledge after continual learning, assessed on ImageNet-full [24] (1000 classes in total).

3) Competitors: We compare our method against baseline and state-of-the-art methods for continual learning VLMs. These include ContinualCLIP [58] under zero-shot conditions; non-continual adapting methods such as CoOp [3] and CoOpOp [4]; and conventional continual learning method such as LwF [11] and EWC [31]. Additionally, we evaluate VLM-specific continual learning methods, including AttriCLIP [16], TaskRes-CL [5], SPU [10], MoE-Adapter [18], and RAPF [60]. For a broader comparison, we also incorporate techniques tailored to address continual learning in visual-only pre-trained models, such as L2P [44], DualPrompt [45], and SLCA [59].

4) Implementation Details: All experiments are conducted on two NVIDIA GeForce RTX 3090 GPUs. Following [16], we adopt the pre-trained CLIP ViT-L/14 [1] as the backbone. To ensure a fair comparison, no data replay strategies are used during the continual learning process of the VLM.

The model is trained for 10 epochs on the datasets for each incremental task. Stochastic Gradient Descent (SGD) [61] is employed as the optimizer, using a cosine learning rate decay schedule with a batch size of 32. The learning rate for the MLPs’ weights in the CLIP’s visual encoder is set to $1\times 10^{-5}$, while the learning rate for the text shifting weights is set to 0.1. The coefficient $\alpha$ for shifting weight addition is set to 0.1. Loss balancing factors are set as $\lambda_{\text{IM}}=2.0$, $\lambda_{\text{TA}}=0.5$, and $\lambda_{\text{RIC}}=1.0$ by default. For the fine-grained CUB-200 dataset [30], $\lambda_{\text{IM}}$ is adjusted to 15.0. To filter out visual features that fail to intuitively reflect attribute or class information, $\delta_{d},\gamma$ are estimated based on downstream tasks and set to 0.20, 0.015; 0.25, 0.03; and 0.30, 0.015 for CIFAR100 [29], ImageNet-Subset [24], and CUB-200 [30], respectively. Additionally, we generate $n_{dsc}=30$ attribute descriptions for each class using the language assistant GPT-4 [50].

The accuracy results for continual learning, including ‘Last’ and ‘Avg’, are summarized in Table I. These results are derived from comprehensive experiments conducted on multiple datasets under varying incremental task settings. Additionally, Fig. 5 depicts the forgetting curves, comparing recent methods and providing a detailed evaluation of accuracy after each incremental task.

1) Performance on Coarse Dataset: CIFAR100 [29] is selected as the coarse dataset, where VLMs generally perform well on common classes but may struggle with blurred images. As shown in Table I, our method consistently achieves superior performance across all incremental task stages, obtaining an average accuracy (‘Avg’) of 90.1% under the setting of $T=10$. Notably, our approach surpasses ContinualCLIP [58] by a significant margin of 11.6% on the ‘Last’ metric. Moreover, it outperforms TaskRes-CL [5], MoE-Adapter [18], and SPU [10] by +3.3%, +2.4% and +1.6%, respectively.

2) Performance on Fine-grained Datasets: Table I demonstrate that our method consistently achieves the best performance across all incremental task stages on fine-grained datasets ImageNet-Subset [24] and CUB-200 [30]. On ImageNet-Subset, our method outperforms TaskRes-CL [5], SPU [10], and AttriCLIP [16] by +1.7%, +1.8%, and +5.7%, respectively. For CUB-200, our approach achieves the highest task accuracy at every stage. With $T=10$, our method surpasses SLCA by +4.5%, SPU by +2.2%, and TaskRes-CL by +3.4% on the ‘Last’ metric.

3) Performance with CLIP ViT-B/16: Table II presents the continual learning performance across 10 tasks utilizing the efficient backbone CLIP ViT-B/16 [1]. Without relying on replay mechanisms or incorporating additional architectural components, our method almost outperforms state-of-the-art approaches on both CIFAR100 [29] and CUB-200 [30]. Notably, while achieving outstanding performance on CIFAR100 and CUB-200, our method maintains accuracies of 62.0$\%$ and 62.5$\%$ on the control set ImageNet-full [24], closely matching the original pretrained CLIP accuracy of 63.6$\%$.

4) Zero-shot Degradation or Improvement? We propose a metric to evaluate the relative accuracy changes for all classes with respect to the zero-shot CLIP model after incremental tasks. Specifically, we calculate the class-wise accuracy difference between the zero-shot CLIP and the trained CLIP model after half of the incremental tasks (50 classes), denoted as $\Delta_{50|(50,zs)}$, and present the results in Fig. 6 (Upper). Our method achieves the highest averaged $\Delta_{50|(50,zs)}$, reaching +9.3. This demonstrates that our approach experiences the least zero-shot degradation while achieving superior performance, attributed to the more robust visual-textual relationships established during continual learning compared to the original vision-class text relationships.

Furthermore, we calculate the accuracy difference between the ‘after all-class incremental learning’ state and the ‘only learning on the first-50 classes’ state, denoted as $\Delta_{50|(100,50)}$, as shown in Fig. 6 (Bottom). Our method achieves an averaged $\Delta_{50|(100,50)}$ of +0.02, indicating effective knowledge transfer from previous tasks. This improvement can be attributed to the establishment of a strong connection between general visual-textual knowledge during continual learning.

5) Few-shot Performance: Fig. 7 illustrates the continual learning performance in few-shot scenarios on the CUB-200 dataset [30]. When the number of training samples per task is reduced from ‘full’ to ‘1/5’, our method maintains a competitive advantage, with an average performance drop of only -6.4. This decline is significantly smaller compared to TaskRes-CL’s -7.3 and SLCA’s -18.0, showcasing the robustness of our approach under challenging conditions.

1) Effectiveness of Each Component: We conduct detailed ablation studies on various components of our DesCLIP framework. As shown in Table III, the zero-shot model is used as the baseline, and the relative improvements or declines are reported. It is evident that a fully fine-tuned visual encoder (V.E.(full)) suffers from catastrophic forgetting. Without data replay, merely fine-tuning the visual encoder (V.E.) partially (tuning the first MLPs in each Transformer block) proves inadequate for continual learning [10]. Based on V.E., by integrating instance matching (IM) with visually filtered attribute description embeddings, we achieve a relative improvement of 2$\sim$3% over the zero-shot baseline. This improvement highlights the effectiveness of IM in mitigating forgetting, which typically arises from overfitting specific downstream task classes. Furthermore, the impact of text embedding calibration (TEC) is notable. The combination of V.E.+TEC+IM+TA, leveraging the filtered attribute description embeddings, yields the best performance, showcasing the synergy between these components.

2) Parameter Selection: Table IV presents the 10-task performance under different parameter configurations. Our analysis reveals that the effectiveness of the instance matching strength, $\lambda_{\textbf{IM}}$, is highly sensitive to the specific task context. Stronger instance matching proves advantageous in scenarios requiring finer granularity, such as CUB-200 [30]. On the other hand, the filtering of attribute description embeddings plays a crucial role. Excessively conservative filtering (e.g., $\gamma=0$) or overly stringent filtering (e.g., $\gamma=0.05$) leads to noticeable performance degradation.

3) Types of Text Embedding Calibration: We demonstrate the comparisons of different types of TEC in Table V. “$\Psi(\mathbf{w},\mathbf{s})=\mathbf{w}/\|\mathbf{w}\|+\alpha\cdot\mathbf{s}$” is the optimal solution, which is an calibration of the rudimentary text embedding in cosine space, which can best approximate the alignment of the representative shared attribute description embedding.

4) Generated Attribute Descriptions: We analyze the impact of the generated description amount on the performance of continual learning, as shown in Fig. 8. For coarse CIFAR100 [29], we only employed the auxiliary prompt “Please maintain diversity between descriptions” to generate $n_{dsc}$ attribute descriptions for each class. Our findings indicated that the model performed optimally when $n_{dsc}$ is set to 30. In contrast, for the fine-grained CUB-200 [30], we have discovered that incorporating additional auxiliary prompts, “Describe a/an [CLS]’s attributes from its beak, eyes, body, belly, tail, wings, breast, etc.” results in improved performance (red star in Fig. 8) even with a smaller $n_{dsc}$. This approach enhances the quality of the descriptions, provided that the language assistant possesses sufficient knowledge of the object’s details.

1) Description of Compliance and Noncompliance: In Fig. 9, we present examples visual image along with attribute description sentences that match and do not match the attribute descriptions filtered by AEF. In particular, different instances on the left, also from ‘Cardinal’, select the similar attribute feature description “a stout, bright orange beak” Furthermore, the highly matched attribute “crimson red feathers that cover its entire body” filtered by the first instance is not represented in the second instance.

2) GA Descriptions Closest to Calibrated Text Embeddings: In Fig. 10, we present the descriptions that are most closely aligned with the calibrated text embeddings. For each class, we showcase three of the top five closest attribute descriptions across several classes. Notably, some common visual attribute features are reflected in these descriptions, such as “slender, pointed wings; streaked brown and white feathers; a clean white belly; and darker hues on the wings and tail” for the “Brown Creeper” category.

3) Filtered-out Visual Instances: As shown in Fig. 11, it can be observed that, compared to the retained visual instances, the filtered visual instances lack relevance to both general attributes and class-specific information.

4) t-SNE Visualization: Fig. 12 presents the t-SNE visualizations of CLIP’s visual representations and text embeddings of each class. We employed distribution alignment to transform text embeddings into the visual representation space, rather than using the original text embedding space. Intuitively, compared to Zero-shot, TaskRes-CL [5] adjusts text embeddings in downstream incremental tasks without optimizing visual representations; SPU [10] optimizes visual representations but struggles to align visual-text representations for unfamiliar classes. In contrast, our method achieves superior alignment of visual-text representations in incremental tasks.