Retaining and Enhancing Pre-trained Knowledge
in Vision-Language Models with Prompt Ensembling

Grouped Prompts and Auxiliary Prompts We introduce Group-wise Prompt Ensemble (GPE), designed to preserve CLIP’s inherent zero-shot capabilities while integrating domain-specific knowledge. As illustrated in Figure 2, we divide the prompts into three sets: $P_{v}=\left[P_{v}^{1};P_{v}^{2};P_{v}^{\prime}\right]$ and $P_{t}=\left[P_{t}^{1};P_{t}^{2};P_{t}^{\prime}\right]$, with a total length of $K_{\text{total}}$. The prompts $P^{1}$ and $P^{2}$, each with an equal length of $K$, are initialized with [CLS] or [EOS] as in previous work [21]. Each prompt token serves as the special token, optimized for downstream task prediction. $P^{1}$ is referred to as the ’first group’, and $P^{2}$ is denoted as the ’second group’. Together, they form the ’main prompt’.

Masked Attention First, similar to RPO[21], we use read-only prompts that prevent the original features from being altered by the learnable prompt embeddings, thus preserving the model’s pre-trained knowledge and minimizing internal representation shifts. Second, we introduce masked attention within the framework of prompt grouping. The first group is restricted to reading the input only without modifying it. For the second group, masking allows attention to both the input tokens and auxiliary prompts. The auxiliary prompts can attend to itself, while all other features remain masked except for itself. Our masked attention strategy not only minimizes internal shifts but also allows multiple prompt groups to be trained independently. This leads to both greater prompt diversity, and more robust ensemble effect, improving overall performance.

Group-wise Ensemble As shown in Figure 2, for $K$ prompts of the same group in each modality, we compute $K\times K$ logits for each group based on cosine similarity given a single image $x$ and class label $y$. We define the prediction probability $p(y=y_{k}|x)=\text{softmax}\left(\{\operatorname{sim}(x,y_{k^{\prime}})/\tau\}_{k^{\prime}}\right)_{k}$ for each group of prompts by averaging intra-group logits independently as:

where $\tau$ denotes the temperature parameter. Finally, prompts are optimized with the group-wise cross-entropy loss $L_{\text{CE-Group1}}$ and $L_{\text{CE-Group2}}$.

Covariance Regularization To further enhance the ensemble process, we introduce a covariance regularization. This technique promotes diversification among the feature embeddings during training, ensuring that each prompt contributes unique and complementary information to the final prediction. By doing so, we not only mitigate the risk of information redundancy but also significantly enhance the model’s adaptability to new domains, providing a sense of reassurance.

Ensemble Inference Our inference method is fundamentally based on CLIP’s inference approach, which computes the similarity between special tokens $v_{0}$ and $t_{0}$ in Eq (1). To fully leverage the knowledge in both the main prompts and the original CLIP model, we introduce an ensemble inference technique that combines all main prompts in each encoder and utilizes each pair as a classifier, as shown in Figure 2:

where $\operatorname{sim}_{i,j}(x,y_{k})=\frac{v_{i}\cdot t_{j}}{\left|v_{i}\right|\left|t_{j}\right|}$.

Special Tokens for Zero-shot Performance As shown in Eq 3, our ensemble approach utilizes both groups’ prompts and special tokens during inference. Specifically, we employ a special token from CLIP to guide the model in predicting with its pre-trained knowledge. Leveraging these special tokens, we enhance the ensemble mechanism by integrating newly acquired insights into the generalizable knowledge special tokens represent. This configuration is strategically designed to preserve zero-shot capabilities while fostering collaboration between the first and second prompt groups. Such collaborative effort proves particularly beneficial in complex scenarios.

Dataset For base-to-new generalization and cross-dataset evaluation, we use the 11 image recognition datasets as in [43]. The datasets include ImageNet [7] and Caltech101 [11] for classification on generic objects; OxfordPets [25], StandfordCars [20], Flowers102 [24], Food101 [3] and FGVCAircraft [23] for fine-grained classification; SUN397 [35] for scene recognition; UCF101 [30] for action recognition; DTD [6] for texture classification; and EuroSAT [14] for satellite images. For domain generalization benchmark, we use ImageNet [7] as a source dataset and ImageNetV2 [28], ImageNet-Sketch [33], ImageNet-A [16] and ImageNet-R [15] as target datasets.

Implementation Details All experiments are based on the vision transformer backbone ViT-B/16 on CLIP [10]. Following [21], we use a total of 24 prompts, with 9 prompts allocated for each of the first/second prompt groups, along with an additional 6 auxiliary prompts$(K=9,K^{\prime}=6)$ for each encoder. We train the model with a batch size of 4 and utilize the SGD optimizer, using a learning rate of 0.01. The epochs for each experiment are set as follows: For the base-to-new setting, ImageNet is trained for 15 epochs, while all other datasets are trained for 30 epochs. In extended cross-dataset experiments and domain generalization, all datasets, including ImageNet, are trained for 15 epochs. During training, we use 16 samples per class and set $\lambda$ = 500 to weight the covariance loss. We report accuracy averaged over 3 runs, using the model from the last epoch for evaluation. We conduct overall experiments using a single RTX 3090 GPU.

Setting We equally split the datasets into base and novel classes, training the model on the base classes in a few-shot setting and evaluating it on both base and novel categories. This benchmark aims to evaluate our approach’s within-dataset generalization capabilities. We evaluate GPE against baselines including CoOp [44], CoCoOp [43], ProGrad [45], MaPLe [18] and RPO [21], which served as an inspiration for the masked attention mechanism.

Result As shown in Table 1, GPE emerges as a standout performer in both base and novel class categories, securing the top harmonic mean compared to baselines.

Setting We evaluate our model’s ability to generalize across different datasets while preserving pretrained knowledge. After training on all classes of one source dataset, we test the model on the remaining 10 target datasets. Instead of previous approach with a single source dataset, such as ImageNet [7], we extend cross-dataset evaluation by varying the source dataset, each with diverse characteristics and classes. Our extended cross-dataset setting includes fine-grained datasets to assess the model’s resilience and its ability to retain general knowledge even after training on specialized data. This setup can be extended into a continual learning framework [42, 31, 38], where the model retains its zero-shot capabilities while preventing the forgetting of knowledge learned from previous datasets and simultaneously acquiring new knowledge from additional datasets.

Result In Table 2, we compare GPE’s performance with CLIP zero-shot performance, CoOp, CoCoOp, RPO, and MaPLe. The original CLIP performance on the target dataset serves as the upper bound, indicating that there is a trade-off between preserving zero-shot capability and adapting to new dataset. GPE achieves the highest cross-dataset transfer performance, showing minimal loss in zero-shot capabilities after fine-tuning. Our extended evaluation highlights significant performance drops in other models when fine-tuned on specialized datasets. CoOp and CoCoOp, for example, show substantial declines in performance when fine-tuned on datasets like Flowers102 and FGVCAircraft and evaluated on other datasets. In contrast, GPE, even when fine-tuned on niche datasets, maintains near zero-shot performance, demonstrating its ability to retain knowledge across various training scenarios.

Setting In this evaluation framework, our primary objective is to assess the model’s ability to effectively handle out-of-distribution data. The source model is trained on ImageNet  [7] and then evaluated on four distinct ImageNet variants, which introduce various types of domain shifts.

Result As presented in Table 3, all methods show improved performance on the out-of-distribution dataset compared to zero-shot CLIP, indicating that the distribution shift is less pronounced than in section 4.3 setting. Thus, our strategy, which leverages the knowledge of the original CLIP model, does not show dramatic improvements. However, the results reflect our goal of effectively utilizing existing knowledge while adapting to new datasets, as evidenced by the performance enhancement on the source dataset and comparable performance on the target dataset.

Analysis on Auxiliary Prompts To assess the impact of auxiliary prompts, we conduct an ablation study by varying the number of prompt tokens. The results summarized in Table 5 show that integrating auxiliary prompts enhances performance compared to using only 18 main prompts without auxiliary prompts ($K=9$). For a fair comparison, we also test 24 prompts without auxiliary prompts ($K=12$). This results in performance deterioration(77.98), highlighting the need for the auxiliary prompts.

The Effect of Group-wise Ensemble We compare our group-wise ensemble strategy explained in section 3.3 with pair-wise training where 18 pairs of main prompts from the text and image encoders generate classifiers, as in RPO. The key difference between pair-wise and group-wise training is that, in pair-wise training, only specific pairs of prompts are used, while in group-wise training, all possible combinations within each group are considered. Pair-wise training results in a performance decline, as shown in Table 5. This is due to fewer classifiers available for ensemble learning compared to our group-wise strategy.

Role of the Special Tokens during Inference We examined the model’s performance without the special token, using only 18 main prompts for prediction. Although base class performance showed a slight improvement, new class performance declined, leading to a lower harmonic mean. This highlights the importance of the special token during inference for improving zero-shot generalization.

GPE to Single Modality Our experiments show that applying GPE solely to the text encoder nearly matches dual-modality performance, while using it only on the image encoder reduces it to 75.32%, as shown in Table 5. This underscores the critical role of text-based prompting and the importance of balanced prompting across modalities for optimal performance.

Number of Groups We conducted experiments to assess the impact of increasing the number of groups in GPE. Specifically, we created two groups of auxiliary prompts and split the main prompts into three groups: the second group interacts with the first auxiliary group, and the third group interacts with the second auxiliary group. We analyze that expanding the number of groups does not sufficiently diversify the learning trajectories between prompts to improve performance, as shown in Table 5.

Prompt Length As shown in Table 5, we observe a slight performance degradation in both base and novel classes when reducing the total prompt length $K_{\text{total}}$. Increasing the prompt length resulted in a more significant performance drop, particularly in the novel classes. This degradation is likely due to overfitting to specific domain knowledge as the prompt length increases, reducing the model’s ability to generalize effectively for zero-shot learning. Through extensive experiments with various prompt lengths, we determined that setting $K_{\text{total}}$ to 24 is optimal for balancing performance.