Progressive Multi-modal Conditional Prompt Tuning

Our model is constructed upon the foundation of a pre-trained CLIP, which is composed of a text encoder and an image encoder. The text encoder adopts a Transformer (Vaswani et al., 2017) to encode text as vectorized representations, whereas the image encoder proceeds images into feature vectors based on a vision transformer (ViT) (Dosovitskiy et al., 2020) or a ResNet (He et al., 2016). During the pre-training, Radford et al. (Radford et al., 2021) collect a large number of image-text pairs to implement contrastive learning for CLIP, thus enabling CLIP to learn joint V-L representations. As a result, CLIP is adept at performing zero-shot visual recognition tasks with its parameters entirely frozen. Following existing approaches (Zhou et al., 2022b, a), our work employs a ViT-based CLIP model. We introduce the encoding process for both vision and text inputs in detail below.

Encoding image.  The image encoder ${\mathcal{V}}$ comprises $L$ transformer layers ${\{{\mathcal{V}_{l}}\}}_{l=1}^{L}$ firstly embeds an image $I$ into latent embeddings $E_{0}\in{{\mathbb{R}}^{{M_{a}}\times d_{v}}}$. A learnable class token ${s_{l}}\in{{\mathbb{R}}^{d_{v}}}$ in image encoder is successively fed into ${\mathcal{V}_{l}}$ along with $E_{l-1}$ to produce $E_{l}$,

To derive the final image representation $x$, an image projector layer ${\mathcal{P}_{I}}$ transforms ${s_{L}}$ from ${\mathcal{V}_{L}}$ into a shared V-L latent embedding space,

Encoding text.  Given the input text, the text encoder ${\mathcal{T}}$ with $L$ transformer layers ${\{{{\mathcal{T}}_{l}}\}}_{l=1}^{L}$ tokenizes and embeds it into word embeddings ${{W}_{0}}=[{w_{0,1}},{w_{0,2}},...,{w_{0,{M_{b}}}}]\in{{\mathbb{R}}^{({M_{b}}\times d_{l})}}$. At each layer, the embedding ${{W}_{l-1}}$ is input into the $i$-th transformer layer ${\mathcal{T}_{l}}$ as:

In a manner analogous to the image branch, the text representation $z$ is derived from the last token ${w_{L,{M_{b}}}}$ of the last transformer layer through a text projector layer, in the same embedding space as $x$,

Zero-shot inference.  During zero-shot inference, the text inputs consist of hand-crafted prompts (e.g., ‘A photo of a [CLASS]’), where [CLASS] is substituted with the class name of label $y\in[1,...,K]$. The score for the $k$-th class is quantified by computing the cosine similarity between the outputs of the text encoder and the image encoder. This calculation employs a similarity function $sim()$ with a temperature parameter $\tau$, expressed as:

To effectively transfer VLMs to image classification tasks, we explore the performance of multi-modal prompting, an advancement over the majority of existing uni-modal prompting approaches. In previous methods (Zhou et al., 2022b, a), learnable prompts are introduced into the language branch, solely adjusting the text encoding of this branch. However, we hypothesize that limiting prompting to the text encoder alone is sub-optimal. To better align V-L features, we advocate for multi-modal prompt tuning. Drawing inspiration from VPT (Jia et al., 2022), our method integrates learnable soft prompts into deep layers of ViT in CLIP. Figure 2 shows the overall architecture of our proposed ProMPT (Progressive Multi-modal conditional Prompt Tuning) framework. In the vision branch, vision prompts are generated by the text features most relevant to the image, encouraging image features to concentrate more on the target objects in the image. Concurrently, in the language branch, we apply image features to generate language prompts. These dynamic prompts have been demonstrated to improve generalizability by Zhou et al (Zhou et al., 2022a).

Moreover, we mimic the human process of distinguishing images, where an image can be analyzed repeatedly until accurate recognition is achieved. Specifically, we divide ProMPT into two principal phases: initialization ($n=0$) and multi-modal iterative evolution ($n={1,...,N}$), where $n$ represents the number of iterations. The strategy is designed to incrementally optimize more relevant prompts in each iteration, fostering the alignment of V-L features.

For an image classification task with an image $I$ and a set of labels $C=\{{c_{k}}\}_{k=1}^{K}$, the initialization phase leverages the original structure of CLIP to encode the input image and text.

ViT ${\mathcal{V}}$ splits $I$ into $M_{a}$ fixed-size patches which are projected into patch embeddings $E_{0}^{0}\in{{\mathbb{R}}^{{M_{a}}\times d_{v}}}$. Unless otherwise stated, the superscript in the formulas uniformly denotes the iteration number. Subsequently, the process of image encoding in initialization is as follows, similar to Equation 1 and Equation 2,

In the language branch, the set $C$ is filled into the template prompt to generate $\widetilde{C}=\{\text{a photo of a}\;{c_{k}}\}_{k=1}^{K}$. Each category $\widetilde{c}_{k}$ is tokenized and embedded into word embeddings ${\widetilde{W}_{0}^{0}}=[{P_{0}},{{W}_{0}^{0}}]\in{{\mathbb{R}}^{({M_{b}})\times d_{l}}}$, where ${P_{0}}\in{{\mathbb{R}}^{b\times d_{l}}}$ is the embeddings of the hand-crafted template “a photo of a” and ${{W}_{0}^{0}}=[{w_{0,1}^{0}},...,{w_{0,({M_{b}}-{b})}^{0}}]\in{{\mathbb{R}}^{({M_{b}}-{b})\times d_{l}}}$ represents the embeddings of the category $c_{k}$. Subsequent to this step, $\widetilde{W}_{0}^{0}$ is encoded through each layer of $\mathcal{T}$ to obtain $\widetilde{W}_{L}^{0}$ at the last layer $\mathcal{T}_{L}$. The feature ${w_{L,({M_{b}}-{b})}^{0}}$, at the last position of $\widetilde{W}_{L}^{0}$, is mapped into text representation ${z_{k}^{0}}$ via the text projector layer ${\mathcal{P}_{T}}$.

Notably, we incorporate a feature filter $\mathcal{F}$ to extract valuable text information based on $x^{0}$ and the text representations of the label set ${Z^{0}}=\{{z_{k}^{0}}\}_{k=1}^{K}$. As depicted in Figure 3, we first calculate the prediction probability $\{{p_{1}^{0},p_{2}^{0},...,p_{K}^{0}}\}$ for all classes according to Equation 5. Afterwards, the text features ${\widetilde{Z}}^{0}$ corresponding to the top-${a}$ highest probability values are selected. Later on, ${\widetilde{Z}}^{0}$ is applied for the generation of conditional prompts during MIE.

CLIP has shown remarkable effectiveness across various tasks, especially in zero-shot scenarios. Therefore, the features of $a$ categories $\widetilde{Z}^{0}$ we obtained in initialization are approximately valid. We input $\widetilde{Z}^{0}$ together with image $I$ into the multi-modal iterative evolution (MIE) module, which aims to preserve and enhance the alignment between image and text features. In this way, the V-L features in initialization eventually converge to a precisely aligned state. MIE contains three sub-processes: class-conditional vision prompting, instance-conditional text prompting, and feature filtering, each contributing to the iterative refinement of feature alignment.

Class-conditional vision prompting.  To force image features to pay more attention to category-related information during encoding, we implement class-conditional vision prompting. As shown in the left part of Figure 4, we introduce prompts ${\{{{O}_{j}^{n}}\in{{\mathbb{R}}^{{a}\times{d_{v}}}}\}}_{j=1}^{J}$ at ${\mathcal{V}_{j}}$ up to a specific depth $J$ in the $n$-th iteration. Specifically, a set of vision generators ${\mathcal{G}^{V}}={\{{\mathcal{G}_{j}^{V}}\}}_{j=1}^{J}$ are designed to map $\widetilde{Z}^{n-1}$ into corresponding vision prompts ${{O}_{j}^{n}}$ for application in ${\mathcal{V}_{j}}$. In addition, an Add module is inserted to fully integrate and fuse the class-conditional vision prompts of the $(n-1)$-th iteration process,

where $\mathcal{G}_{j}^{V}$ is realized through a two-layer MLP (Linear-ReLU-Linear) responsible for mapping text features to image embedding space. The dimensional transformation process is expressed as $d\mapsto\frac{d}{16}\mapsto d_{v}$. Considering that constructing $J$ separate $\mathcal{G}_{j}^{V}$ would considerably increase the training parameters, we share weight matrix across ${\mathcal{G}^{V}}$ and set layer-specific bias terms, which is aimed at balancing the training parameters with model performance.

Formally speaking, input embeddings are denoted as $[{s_{0}^{n}},{O_{0}^{n}},{E_{0}^{n}}]$. Other vision prompts are further injected in corresponding layers of the image encoder to participate in self-attention computation and the final image representation $x^{n}$ is computed as follows,

Instance-conditional text prompting.  Inspired by CoCoOp (Zhou et al., 2022a), we apply features $x^{n}$ from the image branch to generate instance-conditional text prompts via a text generator ${\mathcal{G}^{T}}={\{{\mathcal{G}_{r}^{T}}\}}_{r=1}^{R}$. As illustrated in the right part of Figure 4, the architecture of $\mathcal{G}_{r}^{T}$ is the same as that of $\mathcal{G}_{j}^{V}$. Instead, we employ a single generator to create text prompts for insertion at the input layer $\mathcal{T}_{1}$, thereby setting $R=1$. The specific process for dimensional transformation follows: ${d}\mapsto\frac{d}{16}\mapsto d_{l}$. With the Add operation, image features $x^{n}$ pass through ${\mathcal{G}^{T}}$ to yield text prompts ${{{P}_{0}^{n}}\in{{\mathbb{R}}^{{b}\times{d_{l}}}}}$,

Once text prompts are obtained, they are concatenated with ${{W}_{0}^{n}}$ and subsequently fed into ${\mathcal{T}_{l}}$ in turn for computing the text features of the label set ${Z^{n}}=\{{z_{k}^{n}}\}_{k=1}^{K}$ during the $n$-th iteration.

Feature filtering.  In analogy to initialization, the feature filtering of MIE, as depicted in Figure 3, selects the top-$a$ text features ${\widetilde{Z}}^{n}$ that exhibit the highest relevance to $x^{n}$ according to prediction probabilities $\{{p_{1}^{n},p_{2}^{n},...,p_{K}^{n}}\}$ in a similar manner to Equation 5. Subsequently, these filtered features ${\widetilde{Z}}^{n}$ are utilized as inputs for the next iteration process, continuing the cycle of evolution.

To optimize ProMPT, we adopt a cross-entropy loss function aimed at minimizing the distance between ground-truth labels and prediction probabilities during the $n$-th iteration derived from Equation 12,

where $y_{k}$ symbolizes the one-hot vector for the ground-truth label. Throughout the training phase, the advanced ProMPT maintains the whole parameters of CLIP fixed, while concurrently engaging in the optimization of prompts and generators. To this end, we apply ${\mathcal{L}^{n}}$ for the output of all iterations in the MIE, excluding initialization. Overall, the final loss function of our model is formulated as:

where $\lambda$ acts as a constant weighting factor for modulating the significance of each iterative evolution. The aggregation of ${\mathcal{L}^{n}}$ serves to guide the model towards accurate predictions at each iteration, thereby progressively fostering multi-modal learning.

In this section, we elaborate on the experimental setup: benchmark setting, datasets, implementation details, and compared methods.

Benchmark setting.  Our approach is primarily evaluated in three different experimental settings:

• $\bullet$

  Generalization from base-to-novel classes.  To demonstrate the generalizability of ProMPT, we partition datasets into base and novel classes to assess method in a zero-shot setting. The model is first trained on base classes in a few-shot setting and then evaluated on base and novel classes.

• $\bullet$

  Cross-dataset evaluation.  To verify the ability of ProMPT in cross-dataset transfer, we train our model on ImageNet in a few-shot manner and directly test it on other datasets.

• $\bullet$

  Domain generalization.  To validate the robustness of method on out-of-distribution datasets. Similarly, we deploy our ImageNet-trained model directly on four distinct ImageNet datasets that contain various types of domain shifts.

Datasets.  In our study on generalization from base-to-novel classes setting and cross-dataset evaluation setting, we adopt 11 datasets following Zhou et al. (Zhou et al., 2022b). These include ImageNet (Deng et al., 2009), Caltech101 (Fei-Fei, 2004), OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), Flowers102 (Nilsback and Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAircraft (Maji et al., 2013), SUN397 (Xiao et al., 2010), UCF101 (Soomro et al., 2012), DTD (Cimpoi et al., 2014) and EuroSAT (Helber et al., 2019). These datasets cover a wide range of image classification tasks involving generic objects, fine-grained categories, scene understanding, action recognition, texture classification, and satellite imagery recognition. For assessing domain generalization, we utilize ImageNet as source dataset and its four variants as target datasets, namely ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al., 2021a).

Implementation details.  Our implementation leverages the open-source CLIP with the ViT-B/16 architecture, where ${d_{v}}=768$, ${d_{l}}=512$ and $d=512$. The text prompt length is fixed to 5, with the prompts initialized with the pre-trained CLIP word embeddings of “a photo of a”. The vision prompt length and depth are configured to 8 and 9, respectively. We utilize hyperparameters ${\lambda}=1.0$ and set iteration numbers $N$ to 2. Experiments are executed on a single GeForce GTX 3090Ti GPU with a batch size of 4 and a learning rate of 0.008 via SGD optimizer for 5 epochs. To streamline the implementation, we uniformly utilize 16 shots for each class. In base-to-novel generalization, we report the accuracies for base and novel classes, alongside their harmonic mean (HM), averaging the outcomes over 3 runs. For cross-dataset evaluation and domain generalization, the model is trained on ImageNet for 2 epochs at a learning rate of 0.0024, with the vision prompt depth adjusted to 3.

Compared methods.  We evaluate ProMPT and compare it with several notable prompt learning methods for VLMs, including:

• $\bullet$

  CLIP (Radford et al., 2021) exploits hand-crafted text prompts (“a photo of a [CLASS]”), achieving excellent zero-shot generalization.

• $\bullet$

  CoOp (Zhou et al., 2022b) replaces the hand-crafted prompts with learnable soft prompts that are optimized by the downstream datasets.

• $\bullet$

  CoCoOp (Zhou et al., 2022a) introduces Meta-Net on the basis of CoOp, which combines image features with soft prompts of CoOp to generate instance-conditional prompts.

We compare the proposed ProMPT with the aforementioned methods on the 11 datasets, and the experimental results are shown in Table 3.5. In all tables, the scale of the accuracy is in percentage. See Table 3.5 (a), ProMPT achieves a performance enhancement of 0.47% on base classes and a significant 3.20% on new classes respectively over the superior CoCoOp from an average performance perspective. This improvement demonstrates that ProMPT is equipped with solid generalization while ensuring basic performance. We attribute this phenomenon to the proposed MIE module, which gradually optimizes the prediction results in an iterative manner. During each iteration of the MIE module, class-conditional vision prompting promotes image features to focus more intently on the target objects during encoding, thus achieving a better alignment with text features; secondly, instance-conditional text prompting learns text prompts for each image rather than specific to some certain classes. These dynamic prompts are more robust to class shifts. In comparison to CoOp, although ProMPT is slightly inferior by 1.75% on base classes, its generalization on new classes surges from 63.22% to 74.89%. Besides, HM boosts from 71.66% to 77.80%, further evidencing the excellent capabilities of ProMPT.

More specifically, ProMPT surpasses CoCoOp on roughly all datasets when transferred to new classes, especially on EuroSAT and FGVCAircraft with satisfactory improvements of 15.99% and 10.12%, separately, as visualized in Figure 5. This makes sense because, for fine-grained tasks (FGVCAircraft) or specific tasks (EuroSAT), multi-modal conditional prompting that adapts the model from vision-language modality jointly is more dominant than uni-modal ones. On the other hand, minor performance degradations of ProMPT are observed on 4 out of 11 datasets in the base classes. Nevertheless, these reductions only remain within a mere 0.9%, which is considered trivial in comparison to the substantial gains afforded by ProMPT. In conclusion, with the aim of striking a balance between accuracy and generalization, we inspect the harmonic mean across all datasets depicted in Table 3.5. This comprehensive analysis reveals that ProMPT consistently demonstrates outstanding capability.

We assess the cross-dataset generalization ability of ProMPT by learning multi-modal prompts on all the 1000 ImageNet classes and then transferring it directly to the remaining 10 datasets. As shown in Table 2, we compare the performance of ProMPT with that of CoOp and CoCoOp. On the source dataset, ImageNet, ProMPT achieves comparable performance. On the majority of target datasets, ProMPT exhibits superior performance, outperforming CoOp in all 10 datasets and surpassing CoCoOp in 5 out of 10 datasets. Overall, ProMPT stands out among the comparative methods with the highest average accuracy of 66.25%. This indicates the effectiveness of our proposed conditional multi-modal prompting and iterative evolution strategy in enhancing generalizability.

To verify the domain generalization of ProMPT, we train our model on the source ImageNet dataset and directly evaluate it on four out-of-distribution datasets. As depicted in Table 3, our method consistently outperforms all other methods on variant data and achieves the best average performance across the target data sets. These results highlight that the multi-modal conditional prompts markedly augment the robustness of our model.

In this section, we put forward several ablation studies to delve deeper into the impact of various factors on ProMPT. We sequentially ablate the number of iterations, model structure, prompt length, vision prompt layer, and loss weighting factor. Unless otherwise specified, the reported results are the average performance of all datasets in the generalization from base-to-novel classes setting.

Effect of iterative optimization.  We select several representative datasets, including generic-objects ImageNet and fine-grained OxfordPets, etc., to explore the impact of iteration numbers $N$ on ProMPT by increasing $N$ successively. Table 4 lists the results delivered by setting different $N$, suggesting that the average capability improves as iteration continues in training, and reaches a culmination when $N=2$. Furthermore, we visualize the iterative classification process. As illustrated in Figure 7, it can be observed that with the progression of iterations, ProMPT not only maintains but also gradually raises the confidence for already correctly identified category. Notably, Figure 7 underscores the correction capability of ProMPT. It can rectify category initially misclassified by the basic CLIP to the correct ones, eventually converging to a relatively stable state, thereby achieving stable prediction performance.

Effect of main components.  We explore the efficacy of each component by removing it from ProMPT. The results are summarized in the Table 5. Discarding the iterative module (-$Iter$) means only the initialization phase (i.e. CLIP), where the accuracy degrades by 11.6%, 0.67%, and 6.1% on Base, New, and H respectively. Afterwards, we execute three ablation settings: removing vision generator (-${\mathcal{G}^{V}}$), removing text generator (-${\mathcal{G}^{T}}$), and removing both (-${\mathcal{G}}$). All settings consistently suffer greater losses in the new classes than the base classes, which suggests that conditional prompts are vital in improving generalization. Finally, we abolish feature filter ${\mathcal{F}}$, which results in considerable damage. Without ${\mathcal{F}}$, all text features are indiscriminately used to generate vision prompts. In this way, image features are equally concerned with some information of mismatched classes during encoding. Hence, ${\mathcal{F}}$ is essential owing to the function of choosing the beneficial information reasonably.

Effect of prompt length.  In Figure 8 (a), we display the effect of vision prompt length $a$ on ProMPT. $a$ determines the number of text features selected in $\mathcal{F}$ and reflects the extent of class-related information included. Additionally, we conduct ablation on text prompt length $b$. Figure 8 (b) presents the results. In brief, the performance initially improves and then decreases with an increase in $b$. Accordingly, we set $a=8$ and $b=5$ for optimal efficiency.

Effect of vision prompt layer.  We further investigate the impact of varying the number of layers for inserting vision prompts into the vision transformer on ProMPT. It can be intuitively seen from Figure 8 (c) that the more layers of vision prompt are implanted, the better model performance, achieving a peak at 9 layers.

Effect of loss weighting factor.  We perform ablation for $\lambda$ by varying it. The influence can be inferred from Figure 8 (d) that the effectiveness of ProMPT is positively correlated with $\lambda$. The optimal outcome is observed when ${\lambda}=1.0$.