Revisiting Prompt Pretraining of Vision-Language Models

The technique of prompt learning is initially introduced as an improvement to manual prompting in the field of Natural Language Processing (NLP) for transferring knowledge from pretrained models to specific downstream domains [51, 27, 65, 34, 38].

In recent years, the application of prompt learning has expanded into the realms of vision [56, 26, 3, 25] and vision-language [67, 29, 30, 47]. These approaches involve two main branches: the text branch and the image branch. In the text branch, CoOp [68] employs learnable tokens instead of manually designed suffixes in text inputs, such as “a photo of a class”, to enhance the transfer capabilities in classification tasks. Subsequently, CoCoOp [67] introduces instance-conditional prompts to mitigate model overfitting. These methods adapt to various tasks, including open vocabulary [16] and visual grounding [46], etc. In the image branch, vision prompt tuning serves as an efficient training technique alongside backbone fine-tuning or linear probing. VPT [56] appends learnable tokens before the image sequence but keeps the entire backbone fixed to transfer vision-only models. A concurrent approach [3] introduces pixel-wise prompts for spatially structured images instead of sequences. Furthermore, some methods leverage joint textual and visual prompt tuning based on CLIP [45] for improvements [63, 29, 30].

Moreover, prompt pretraining with large-scale data has been incorporated into enhanced pretrained models, such as POMP [49]. However, upon revisiting, it is observed that the limited learnable prompts may face underfitting risks, especially given the extensive images during prompt pretraining. To address the underfitting issue of POMP, this paper introduces a general framework termed RPP, aiming to enhance fitting and generalization abilities by focusing on two key aspects: prompt structure and prompt supervision, as shown in Fig. 2.

Knowledge Distillation (KD) [24, 23] emerges as a common method for transferring knowledge from teacher models with more parameters and better performance to more deployable student models.

Distillation for VLMs has gained prominence in recent years, with efforts to convert VLMs knowledge into vision-only models during model pretraining [57, 13, 55], open vocabulary detection [18, 64], or semantic segmentation [66, 28]. Additionally, some methods directly distill VLMs to VLMs through ordinary distillation [62, 14, 58, 39], self-distillation [59, 2], and linear probing [33]. DistillVLM [14] transfers knowledge through the intermediate representations of each proposal generated from pretrained detectors. CLIP-KD [62] explores various distillation strategies, including relation, feature, gradient, and contrastive paradigms to assess their impact on CLIP distillation. TinyCLIP [58] learns cross-modal feature alignment from teacher to student in a visual-textual affinity space. CLIPSelf [59] distills CLIP itself using dense feature maps and corresponding predictions. LP-CLIP [33] employs a single linear probing layer for distillation.

In addition to the hard one-hot labels that may present optimization challenges for specific prompt tokens, we introduce soft labels derived from a pretrained large-scale CLIP teacher’s zero-shot probability predictions.

Fig. 3 illustrates the training and testing accuracy of our method compared with the POMP method on the ImageNet-21K dataset, in terms of training epoch and data amount, respectively. The pink lines denote the performance of the POMP method, while the red lines represent our approach. In Fig. 3, the training and testing accuracy of the POMP method become relatively flat and saturated after around 4 epochs, indicating negligible further fitting to the dataset. In Fig. 3, the POMP method reaches its fitting capacity ceiling at 4-shot, and as training data is further increased, the flattening curve highlights an underfitting issue. In contrast, our method maintains its pretraining fitting capability across extended training epochs and larger data amounts. A detailed description of the method will be provided in Sec. 3.3 and 3.4.

CLIP [45] contains an image encoder and a text encoder, denoted as ImgEnc and TextEnc, respectively. For zero-shot transfer, the input image $I=\left\{{I_{i}}|{I_{i}}\in\mathbb{R}^{3\times H\times W},i=1,2,\cdots,N\right\}$ is embedded into a token sequence and passed through the image encoder, denoted as PatchEmb. The corresponding text $T_{c}$ is the categories prepended with a suffix, appearing as “a photo of a [class]”. $T_{c}$ is tokenized and embedded into a feature space via the text encoder, denoted as Tokenizer:

where $\mathbf{y}=\left\{\mathbf{y_{c}}|c=1,2,\cdots,C\right\}$ with $\mathbf{y_{i}}\in\mathbb{R}^{d}$, and $\chi=\left\{\mathbf{x_{i}}|i=1,2,\cdots,N\right\}$ with $\mathbf{x_{i}}\in\mathbb{R}^{d}$ represent the global features of the image and text outputs, respectively. Then the normalized embeddings are derived as $\mathbf{\hat{x}_{i}=x_{i}/\left\|x_{i}\right\|_{2}}$ and $\mathbf{\hat{y}_{c}=y_{c}/\left\|y_{c}\right\|_{2}}$. The probabilities concerning each class are calculated as:

where $C$ denotes the number of classes, $\left(\cdot\right)$ represents the dot product, and $\tau>0$ is the temperature.

Different from manually designed instructions, prompt learning provides a solution to transfer knowledge through learnable tokens in the form of continuous parameters. There are two aspects to prompt learning: the textual branch and the visual branch. In the first one, following common practices [68, 29], “a photo of a [class]” is replaced with “$P^{t}$ [class]”, where $P^{t}=\{p^{t}\}_{m}$, ($m\in\mathbb{N}_{M}$), is the learnable prompt, and $M$ denotes the number of contents. In terms of visual prompting [26, 63, 29], another set of learnable tokens is extended after the image patches: $\left\{I,P^{v}\right\}$. Then they are processed in place of the original image patches and text inputs. Subsequently, $\left\{P^{t},P^{v}\right\}$ are learned to adapt to specific data domains.

In addition to the integration of learnable prompts in the input space, there are methods [29, 26] that propose to incorporate tokens in the deep layers of text / vision encoders, known as multi-layer prompt learning. Denoting the output of the $l$-th transformer block in the text encoder as $\left\{P^{t}\left(l\right),F^{t}\left(l\right)\right\}$, where $P^{t}\left(l\right)$ and $F^{t}\left(l\right)$ are the intermediate prompt and text features, respectively. Then $P^{t}\left(l\right)$ is replaced with learnable prompts, which are additionally initialized in the $l$-th layer of the encoder to introduce learnable tokens in deeper transformer layers. Likewise, within the forward process in the vision encoder, $\left\{P^{v}\left(l\right),F^{v}\left(l\right)\right\}$ undergoes the same operation.

As shown in Fig. 4, to mitigate potential underfitting concerns related to the model architecture, we employ a methodology utilizing independent learnable prompts. These prompts are designated for substitution before the self-attention computation at each layer of the model. Particularly, before computing the forward propagation of the current layer with input $X^{t/v}\left(l\right)=\left\{P^{t/v}\left(l\right),F^{t/v}\left(l\right)\right\}$, we utilize $\mathbf{Rep}\left(\cdot,\cdot\right)$ to replace the learnable prompt from the previous layer with separated learnable prompts $P_{*}^{t/v}\left(l\right)=\left\{P^{t/v}_{Q}\left(l\right),P^{t/v}_{K}\left(l\right),P^{t/v}_{V}\left(l\right)\right\}$. This approach diverges from traditional methods that typically utilize a single shared prompt, represented by $P^{t/v}\left(l\right)$.

It is essential to acknowledge that this method applies when the backbone is ViT [12], functioning for both $X^{t}\left(l\right)$ and $X^{v}\left(l\right)$. Conversely, for the ResNet [20] backbone, it exclusively applies to $X^{t}\left(l\right)$. This operation aims to broaden the optimization scope within the parameter space while upholding consistent feature scaling. The computation of the current layer is outlined as follows (the following formulas do not distinguish $t/v$):

where $l\in\{2,3,\ldots,12\}$ represents the $l$-th layer, and $\mathbf{LN}\left(\cdot\right)$ denotes LayerNorm. We normalize each prompt to maximize the parameter space while simultaneously ensuring that features do not diverge significantly from the original CLIP distribution. Then we proceed with the remaining calculations:

where $\mathbf{MHSA}\left(\cdot,\cdot,\cdot\right)$ stands for Multi-head Self Attention, $\mathbf{MLP}\left(\cdot\right)$ stands for Multi-Layer Perceptron, and $X_{Q}\left(l\right)$ means exchanged query prompt without layernorm.

As shown in Fig. 2, we conduct pretraining of multi-layer prompts by leveraging the extensive ImageNet-21K dataset, pioneering the transfer of robust embedded knowledge from a larger-scale CLIP model to a more computationally efficient scale. As detailed in Sec. 3.3, we utilize the layer-by-layer replaceable $\left\{P_{*}^{t}\left(l\right),P_{*}^{v}\left(l\right)\right\}$ as the learnable parameters for the student network, while employing the original pretrained large-scale CLIP model as the teacher network.

We aim to acquire prompt pretrained CLIP endowed with universal generalization capabilities through disciplined pretraining via distillation on the ImageNet-21K dataset. Nevertheless, direct pretraining on 21K data demands over 300GB of GPU memory [49]. To address this challenge, we adopt methodologies inspired by POMP [49], specifically employing local contrast and local correction techniques. In particular, when presented with a batch of input images, our process involves sampling $K$ classes$\left(K<<C\right)$. This includes selecting the respective ground-truth class and incorporating $K-1$ randomly selected samples as shared negative classes. Therefore, the probabilities for sample classes are calculated as:

where $y_{i}^{gt}$ denotes the ground-truth class of sample $i$ and $\hat{y}_{i}^{ram}$ denotes the randomly selected $K-1$ negative classes. To address the inherent bias in the prompt optimization direction due to the absence of other negative classes, we introduce a local correction term denoted as $m$ into the probability of negative classes. This term aims to incentivize the positive logit to surpass the negative logits by a predefined margin. The formulation for $m$ is as follows:

Drawing from the aforementioned strategies, we incorporate knowledge distillation loss and cross-entropy loss during the pretraining phase. To enhance text embedding diversity in the teacher model, we employ textual augmentations. This involves the random selection of 60 prompt templates for the text encoder from the comprehensive template list provided in [45]. Then, we introduce logit-level consistency regularization by conditioning the distribution of prompted logits on the teacher logits distribution. This is achieved through the minimization Kullback-Leibler Divergence of the following loss function:

where $\hat{s}_{ik}^{S}\in\left[0,1\right]$ and $\hat{s}_{ik}^{T}\in\left[0,1\right]$ represent student’s and teacher’s probability prediction of $i$-th training data in the $k$-th sampled class. For image classification on the ImageNet-21K dataset $\mathcal{D}$, learnable prompts $\left\{P_{*}^{t}\left(l\right),P_{*}^{v}\left(l\right)\right\}$ interact with frozen ImgEnc and TextEnc are optimized with the cross-entropy loss, $\mathcal{L}_{CE}$, as:

We use $\lambda>0$ as the loss balancing hyper-parameters. Our overall pretraining objective thus becomes:

In this section, we provide theoretical results for the generalization error bound of RPP. All proofs of theorems are given in the appendix Sec. A.

We define the following optimization objectives according to Eq. (10):

where $\Theta$ represents the set of learning prompts $\left\{P_{*}^{t},P_{*}^{v}\right\}$ of the proposed network, with $d$ denoting the dimensionality of the learning parameters. Now we further analyze the effectiveness of RPP by offering the generalization error bound. Such a bound evaluates the bias between the generalization error $\varepsilon\left(\Theta\right):=\bm{\mathbb{E}}_{\left(\hat{s}^{S},y^{gt}\right)\sim\mathcal{D}}\mathcal{L}\left(\hat{s}^{S}\left(\Theta\right),y^{gt}\right)$ and empirical error $\bar{\varepsilon}_{\chi}\left(\Theta\right):=\frac{1}{N}{\textstyle\sum_{i=1}^{N}}\mathcal{L}\left(\hat{s}_{i}^{S}\left(\Theta\right),y_{i}^{gt}\right)$, where $D$ is the real data distribution and $\bm{\mathbb{E}}\left(\cdot\right)$ denotes the expectation function.

In Eq. (12) the first term of the upper bound converges with the increasing of the number of training data $N$. We can also find that the second term converges to 0 with the increasing of $\lambda$, which means the regularizer $\mathcal{L}_{KD}$ effectively improves the generalization ability of RPP.