Training-Free Unsupervised Prompt for Vision-Language Models

CLIP is trained on 400 million image-text pairs with a language-image contrastive loss function. Specifically, the structure of CLIP consists of two components: visual encoder, denoted as $\mathrm{F}(\cdot)$, for converting the input images into visual features and text encoder, denoted as $\mathrm{G}(\cdot)$, for transforming input texts into text representations. Then image and text features are projected into the same embedding space through joint training. After pre-training on a large dataset, it has excellent classification performance in zero-shot scenarios, demonstrating the vision-language model’s excellent understanding of open-set concepts. Consider an image classification task that is defined as classifying a given test image $\mathrm{x_{test}}$ into one of $\mathrm{C}$ categories. Since the text description used in pre-training is different from the labels of the downstream recognition tasks, CLIP places all category names into the “[CLASS]” token of a pre-defined textual template such as “a photo of a [CLASS]”. We denote the text prompts obtained by converting the original labels as $\mathrm{P_{text}}$. Then, we obtain $d$-dimensional visual features $\mathrm{f_{test}}\in\mathbb{R}^{1\times d}$ and text features $\mathbf{F}_{\rm text}\in\mathbb{R}^{C\times d}$ by

	$\displaystyle\mathrm{f_{test}}$	$\displaystyle=\mathrm{F}(\mathrm{x}_{\mathrm{test}}),$		(1)
	$\displaystyle\mathbf{F}_{\rm text}$	$\displaystyle=\mathrm{G}(\mathrm{P_{text}}),$		(2)

where $\mathrm{f}_{\mathrm{test}}$ and $\mathbf{F}_{\rm text}$ are L2-normalized visual and text features, respectively. We can then obtain the classification $\mathrm{logits}\in\mathbb{R}^{1\times C}$ by calculating the cosine similarity of visual and textual features,

$$\mathrm{logits}=\operatorname{softmax}(\mathrm{f}_{\mathrm{test}}\mathbf{F}_{\rm text}^{\top}),$$

(3)

where $\mathrm{logits}$ denote the prediction probabilities for the $\rm C$ categories. We can easily identify the prediction $\hat{y}=\underset{c}{\operatorname{argmax}}\,(\mathrm{logits})$.

Given the pre-trained CLIP [29] model, we aim to leverage the unlabeled training set, denoted as $\mathrm{M}$, for unsupervised classification. For each training image, denoted as $\mathrm{x_{i}}$, we utilize the CLIP’s visual encoder to extract its $d$-dimensional visual feature, denoted as $\mathrm{f}_{\rm i}=\mathrm{F}(\mathrm{x_{i}})$. Then, we can use Eq. (3) to generate the prediction probability, denoted as $\mathrm{logits_{i}}$. Finally, we view CLIP outputs the maximum one as the prediction and the corresponding index as the pseudo label $\mathrm{L_{i}}=\mathrm{OneHot}(\underset{c}{\operatorname{argmax}}\,(\mathrm{logits_{i}}))$. For all training samples, we denote their visual features and corresponding pseudo-labels as $\mathbf{F}_{\rm train}\in\mathbb{R}^{M\times d}$ and $\mathbf{L}_{\rm train}\in\mathbb{R}^{M\times C}$. To create the key-value cache, we view visual features $\mathbf{F}_{\rm train}$ as keys and the corresponding pseudo-labels $\mathbf{L}_{\rm train}$ as values. Therefore, the unsupervised dataset $\mathrm{M}$ is transformed into a key-value cache database $\mathbf{S}_{\rm train}=\langle\mathbf{F}_{\rm train},\mathbf{L}_{\rm train}\rangle$.

For creating a standard Feature Cache Model (FCM), we first employ a confidence filter to filter out high-confidence samples and their corresponding pseudo-labels from the unsupervised dataset $\mathbf{S}_{\rm train}=\langle\mathbf{F}_{\rm train},\mathbf{L}_{\rm train}\rangle$. Specifically, we select top-K most confident samples for each class, instead of keeping all samples with confidence scores higher than a pre-defined threshold, to preserve a balanced distribution of pseudo-labeled data. Then, we denote the high confidence visual features and corresponding pseudo-labels as $\mathbf{F}_{\rm confi}\in\mathbb{R}^{K\times d}$ and $\mathbf{L}_{\rm confi}\in\mathbb{R}^{K\times C}$. Consequently, we obtain the high confidence pseudo-label dataset $\mathbf{S}_{\rm confi}=\langle\mathbf{F}_{\rm confi},\mathbf{L}_{\rm confi}\rangle$.

However, these high-confidence images are not necessarily representative samples. To further refine the confident pseudo-label data into representative samples, we propose a prototype filter which calculates the cosine similarity between each confidence sample $\mathrm{f_{i}}\in\mathbf{F}_{\rm confi}$ and other confidence samples by

$$\mathrm{S_{i}}={\sum_{\mathrm{j=1}}^{\mathrm{K}}\cos\left(\mathrm{f_{i}},\mathrm{f_{j}}\right)},$$

(4)

where $\mathrm{S_{i}}$ denotes the prototype score of the $\mathrm{i}$-th image. Higher value of $\mathrm{S_{i}}$ means higher potential to be category prototypes. We select the $N$ highest scoring samples for each category as prototype samples. Then, we obtain the prototype visual features and corresponding labels as $\mathbf{F}_{\rm proto}\in\mathbb{R}^{N\times d}$ and $\mathbf{L}_{\rm proto}\in\mathbb{R}^{N\times C}$. Finally, we denote the prototype pseudo-label dataset as $\mathbf{S}_{\rm proto}=\langle\mathbf{F}_{\rm proto},\mathbf{L}_{\rm proto}\rangle$.

Based on the constructed feature cache model, we calculate the similarity between test image and prototype samples as the weights of the corresponding labels. We then combine different sample labels into a similarity-base prediction probability according to the weights. The current similarity measure only considers the degree of similarity at the feature level. Differently, we propose a Multi-level Similarity Measure (MSM) which considers both feature-level and semantic-level similarity to measure the similarity between test samples and prototype samples. Specifically, for each test image, denoted as $\mathrm{x_{test}}$, we utilize the CLIP’s visual encoder to extract its $d$-dimensional visual feature, denote as $\mathrm{f}_{\rm test}=\mathrm{F}(\mathrm{x_{\rm test}})$. Then, we calculate the multi-level similarity between test image and prototype samples as the weights of the corresponding labels. The MSM consists of Feature Similarity Measure (FSM) and Semantic Similarity Measure (SSM). For the FSM, we calculate the cosine similarity between the test image features $\mathrm{f}_{\mathrm{test}}\in\mathbb{R}^{1\times d}$ and the prototype features $\mathbf{F}_{\rm proto}\in\mathbb{R}^{N\times d}$ by

$$\mathrm{W_{cont}}=\mathrm{f}_{\mathrm{test}}\mathbf{F}_{\rm proto}^{\top},$$

(5)

where $\mathrm{W_{cont}}$ represents the feature similarity score. In addition, the feature-level similarity measure only considers the similarity of the overall information of the image and ignores the consistency of the sample categories. To this end, we introduce SSM which calculates the KL-divergence between the test image prediction probabilities, denoted as $\mathrm{logits}_{\mathrm{test}}\in\mathbb{R}^{1\times C}$, and the prototype prediction probabilities, denoted as $\mathrm{logits}_{\rm proto}\in\mathbb{R}^{N\times C}$, by

	$\displaystyle\mathrm{\mathrm{logits}_{\mathrm{test}}}$	$\displaystyle=\operatorname{softmax}(\mathrm{f}_{\mathrm{test}}\mathbf{F}_{\rm text}^{\top}),$		(6)
	$\displaystyle\mathrm{\mathrm{logits}_{\rm proto}}$	$\displaystyle=\operatorname{softmax}(\mathbf{F}_{\rm proto}\mathbf{F}_{\rm text}^{\top}),$		(7)
	$\displaystyle\mathrm{W_{sem}}$	$\displaystyle=1-\operatorname{softmax}({\mathrm{KL}(\mathrm{logits}_{\mathrm{test}},\mathrm{logits}_{\rm proto})}),$		(8)

where $\mathrm{W_{sem}}$ represents the semantic similarity score and $\mathbf{F}_{\rm text}$ denotes textual features. $\mathrm{KL}$ computes the KL-divergence, i.e., $\mathrm{KL}(P,Q)=\sum_{i}P_{i}\log\frac{P_{i}}{Q_{i}}$. To emphasize the criticality of the two measures, we combine CSM and SSM by

$$\mathrm{W_{fsm}}=\mathrm{W_{cont}}\circ\mathrm{W_{sem}},$$

(9)

where $\circ$ denotes hadamard product and $\mathrm{W_{fsm}}$ represents the ultimate multi-level similarity score. Then, we combine different prototype sample pseudo-labels $\mathbf{L}_{\rm proto}$ into a similarity-base prediction probability according to the multi-level similarity scores $\mathrm{W_{fsm}}$ by

$$\mathrm{logits}_{\rm sim}=\mathrm{W_{fsm}}\cdot\mathbf{L}_{\rm proto},$$

(10)

where $\cdot$ denotes matmul product and $\mathrm{logits}_{\rm sim}$ denotes the predicted probability based on feature cache model. After that, we obtain the final prediction probability of the test image, denote as $\mathrm{logits}_{\rm all}$, by adding the similarity-base prediction probability $\mathrm{logits}_{\rm sim}$ to the original prediction probability $\mathrm{\mathrm{logits}_{\mathrm{test}}}$,

$$\mathrm{logits_{all}}=\mathrm{\mathrm{logits}_{\mathrm{test}}}+\mathrm{logits}_{\rm sim},$$

(11)

where $\mathrm{logits_{all}}$ presents the ultimate prediction probability.

The performance of TFUP can be further boosted by parameter efficient fine-tuning. Following the standard Parameter-Efficient Fine-tuning (PEFT) methods [7, 40], we attach image and text adapters to the image and text encoders, denoted by $\mathrm{T_{x}(\cdot)}$ and $\mathrm{T_{t}(\cdot)}$, respectively. Each adapter consists of two layers of linear transformations. Referring to ReZero [2], we further employ two constant values $\alpha$ and $\beta$ as residual ratio to adjust the proportion of downstream knowledge and model inherent knowledge to enhance the robustness of the model and prevent over-fitting. Mathematically, the new knowledge captured via fine-tuning is added with the original features via residual connections,

	$\displaystyle\mathrm{f_{x}}^{\star}$	$\displaystyle=\alpha\mathrm{T_{x}}(\mathrm{f_{x}})+(1-\alpha)\mathrm{f_{x}},$		(12)
	$\displaystyle\mathrm{\mathbf{F}_{\rm text}}^{\star}$	$\displaystyle=\beta\mathrm{T_{t}}(\mathbf{F}_{\rm text})+(1-\beta)\mathrm{\mathbf{F}_{\rm text}},$		(13)

where $\mathrm{f_{x}}^{\star}$ and $\mathrm{\mathbf{F}_{\rm text}}^{\star}$ denote the merged image and text features, respectively. Then, we can use Eq. (6) to generate the prediction probabilities of the test image, denote as $\mathrm{Logits}_{\mathrm{test}}\in\mathbb{R}^{1\times C}$.

In the paradigm of unsupervised prompt tuning, it is crucial to seek appropriate supervision to train adapters on unlabeled data. Thanks to the effectiveness of our training-free strategy, we first generate pseudo-labels on unlabeled data via our TFUP. Thus we can utilize Eq. (11) to generate the prediction probabilities of the unlabeled instances, denote as $\mathrm{Logits}_{\mathrm{all}}\in\mathbb{R}^{1\times C}$. We then view the maximum output of the CLIP model as the prediction and the corresponding index as the pseudo label $\mathrm{L}_{\mathrm{all}}=\mathrm{OneHot}(\underset{c}{\operatorname{argmax}}\,(\mathrm{logits}_{\mathrm{all}}))$. After that, we train the adapters via a standard cross-entropy loss,

$$\mathcal{L}_{ce}={\mathbf{1}}\left(\max\left(\mathrm{Logits}_{\mathrm{test}}\right)\geq\theta\right)\mathrm{CE}\left(\mathrm{L}_{\mathrm{all}},\mathrm{Logits}_{\mathrm{test}}\right),$$

(14)

where $\mathbf{1}$ denotes retaining only the high-confidence predictions higher than pre-defined threshold $\theta$. However, all these pseudo-label-based methods [17, 25] focus solely on the instance-level constraint, ignoring the significance of the global prediction statistic on unlabeled data. Though high-confidence filtering can effectively select the more convincing pseudo-labels, constraining only on individual prediction can inevitably introduce prediction bias [43], considering the different learning difficulties of distinct classes, especially with a huge category space. To this end, inspired by recent studies in mutual-information maximization [16, 32, 22], we further introduce a marginal distribution entropy loss to constrain the model from a global perspective,

$$\mathcal{L}_{md}={\log\rm{C}}-[-\sum_{c}\mathrm{h}_{c}\log\mathrm{h}_{c}],$$

(15)

where $\log\rm{C}$ is the maximum value of distribution entropy and $\mathrm{h}_{c}$ denotes the marginal distribution of the prediction probabilities of the test images on the class index $c,c\in\{0,1,...,C-1\}$ .

Tab. 1 reports the results comparing TFUP and TFUP-T with unsupervised prompt tuning and few-shot prompt learning methods across 6 domains on the Domain-Net [27] datasets. It is obvious that our TFUP achieves the new state-of-the-art (SOTA) performance without any training compared with previous unsupervised prompt tuning methods. Specifically, TFUP outperforms original CLIP with prompt engineering by 3.2%. In addition to prompt tuning, TFUP-T not only achieves an average accuracy improvement of 3.3% compared to the current SOTA POUF of unsupervised methods, but also obtains improvement by 1.2% compared to the SOTA KgCoOp of few-shot approaches. The above experiment demonstrates the efficiency and effectiveness of TFUP, as well as the superiority of the training-base TFUP-T.

Tab. 2 illustrated the results comparing TFUP and TFUP-T with unsupervised prompt tuning and few-shot prompt learning methods across 4 domains on the Office-Home [36] datasets. Our TFUP achieves new SOTA performance without any training compared with unsupervised tuning methods and outperforms original CLIP with prompt engineering by 2.2%. For prompt tuning, TFUP-T boosts the performance of the POUF by 2.0% and demonstrates clear advantages over few-shot methods.

Tab. 3 provides the results comparing TFUP and TFUP-T with unsupervised prompt tuning and few-shot prompt learning methods across 3 domains on the Office-31 [31] datasets. Our TFUP has significant advantages without any training compared with Tent and UPL. Although the accuracy of training-free TFUP is slightly lower than the trainable POUF, TFUP-T achieves new SOTA on all domains. Due to the smaller amount of unsupervised data in the Office-31 dataset compared to the other datasets, the performance achieved by our method is limited.

Tab. 4 summarizes the results compare TFUP and TFUP-T with unsupervised prompt tuning and few-shot prompt learning methods on the VisDA-2017 [28] datasets. Obviously, the training-free TFUP demonstrates competitive performance compared to the other unsupervised methods. By efficient fine-tuning, TFUP-T further improves the performance of model, confirming the effectiveness and versatility of our methods.

To evaluate the effectiveness of various components, we conduct ablation experiments on Office-Home and Domain-Net datasets, as reported in Tab. 5. We can clearly see that each component significantly improves the performance of the model. For training-free TFUP, FCM + FSM increases the average accuracy of Office-Home and Domain-Net datasets by 2.2% and 3.2%, respectively. By efficient fine-tuning, $\mathcal{L}_{ce}$ further improves the performance of the model. Additionally, the pseudo-labeling strategy mainly focuses on rectifying individual predictions. We introduce a global prediction constraint $\mathcal{L}_{md}$, which increases the average performance of CLIP from 57.7% to 63.5% on the most challenging Domain-Net. It demonstrates the effectiveness of each component, and the superiority of our TFUP and TFUP-T.

As presented in Tab. 6, we compare several common sample filter strategies to assess the effectiveness of our approach. i) Confidence Filter Strategy. Select top-K high confidence samples by pseudo labels. ii) Prototype Filter Strategy. Select top-K representative samples by prototype score. iii) Double Filter Strategy. Combine the confidence filter and prototype filter strategy to obtain the representative samples. Compared with the overall unlabeled data, both of the filter strategy designs could improve the final results by large margins. It demonstrates that the key factor of the filter strategy to achieve significant performance is reducing the introduction additional noisy samples. In addition, we find that combining confidence and prototype filter can both improve the results compared with using either of them. It indicates that only selecting the high confidence samples is not enough, and prototype scores are needed to screen representative samples.

As summarized in Tab. 7, we provide in-depth analysis about the similarity measure strategies. i) Feature Similarity Measure (FSM) leverages cosine similarity to measure the similarity of the overall information of the images. ii) Semantic Similarity Measure (SSM) leverages KL-divergence to measure the distance of the image prediction probabilities. iii) Multi-level Similarity Measure (MSM) considers both feature-level and semantic-level similarities between images. We observe that FSM and SSM improve the average accuracy by 1.4% and 1.9% respectively, and the combination of them improves the results by 2.2%. It demonstrates that the effectiveness of considering both feature-level and semantic-level similarities, which not only measures the degree of similarity of the overall image information, but also ensures the semantic consistency of similar samples.

As shown in Fig. 5, we evaluate the hyper-parameter sensitivity of $\alpha$ and $\beta$ of Eq. (12) and Eq. (13) across 4 domains on the Office-Home datasets. $\alpha$ and $\beta$ are residual ratio to adjust the proportion of new knowledge and inherent knowledge. Obviously, on most domains, the best residual rate of image features is $\alpha$=0.2, and the best residual rate of text features is $\beta$=0.5. The model performance will decrease whether the optimal parameters increase or decrease. It may be because too much new knowledge causes the model to overfit while too little new knowledge makes it difficult for the model to adapt to a specific task. Therefore, we set $\alpha$=0.2 and $\beta$=0.5 to obtain the best performance trade-off.