Selective Vision-Language Subspace Projection for Few-shot CLIP

VLMs have gained significant advances in recent years (Zhang et al., 2023; Liu et al., 2024). These models bridge the modalities of vision and language and are typically pre-trained on a large dataset. With image-text pairs, VLMs use an image encoder and a text encoder to extract image and text features, and then learn the vision-language correlation by using certain pre-training objectives. Moreover, a range of VLMs, including CLIP (Contrastive Language-Image Pretraining) (Radford et al., 2021; Li et al., 2023b), CoCa (Contrastive Cross-modal Learning) (Yu et al., 2022), and BLIP (Bidirectional Learning of Image and Text Priors) (Li et al., 2022), can be leveraged for various downstream tasks (Wang et al., 2018; Guo et al., 2019, 2024; Yan et al., 2023a, b), such as object recognition (Zhang et al., 2022b; Zhu et al., 2023a), object detection (Gu et al., 2022), and image captioning (Mokady et al., 2021; Barraco et al., 2022). For instance, CLIP is pre-trained on a vast dataset of web-based image-text pairs and learns to align their representations through contrastive loss, which enables it to recognize unseen data by matching the embeddings of any given images and texts. In this paper, we aim to use CLIP to address few-shot classification problems.

The concept of prompt tuning is initially introduced in the natural language processing area, which refers to utilizing a fixed part of the text input as learnable embeddings and fine-tuning its parameters based on the downstream task data (Zhou et al., 2022b, a; Lu et al., 2022; Zhu et al., 2023a, b, c). Several works have applied this tuning technical in VLMs, i.e., CoOp (Zhou et al., 2022b) uses learnable word embeddings to generate context prompts automatically, eliminating the need for manual prompt templates. CoCoOp (Zhou et al., 2022a) can be considered as an extension of CoOp. as it proposed a meta network to learn image features that served as conditions added to prompt embeddings to further enhance the model’s generalization ability. ProDA (Lu et al., 2022) aims to learn the distributions of the prompt embedding. ProGrad (Zhu et al., 2023a) used zero-shot prediction results to direct the model gradient update, preventing conflicts between few-shot models and general knowledge while mitigating overfitting issues. However, the above-mentioned methods, although they enhance the classification performance of CLIP in few-shot scenarios, involve the introduction of learnable parameters and increase training consumption. In contrast, our SSP only involves matrix calculation and does not introduce any learnable parameters.

Adapter tuning techniques are applied in VLMs (Gao et al., 2024; Zhang et al., 2022b; Zhu et al., 2023d; Udandarao et al., 2022) to enhance downstream generalization ability by freezing the original model parameters and only updating the parameters of added adapter module. CLIP-Adapter (Gao et al., 2024) utilized a fully connected layer to adapt the features outputted from the frozen CLIP. Tip-Adapter (Yu et al., 2023b) leverages a cache model to measure the relations between image and text features to construct a classifier based on few-shot training data. APE (Zhu et al., 2023d) represents an enhanced version of Tip-adapter, selecting the most discriminative feature channels via statistical analysis. SuS-X (Udandarao et al., 2022) relies on the category names from the training set to generate image samples based on stable diffusion (Rombach et al., 2022). Cross-Modal Adapter (CMA) (Jiang et al., 2022) achieves cross-modal interaction by sharing adapter weights between two modalities. While the aforementioned methods adapt text-image features for the downstream tasks, they do not account for their modality gaps. Our SSP method strives to reduce the gaps to better align the paired text-image features and can be integrated into the aforementioned methods.

The CLIP model possesses the ability to perform zero-shot classification, involving extending the ‘‘[class name]’’ to the template ‘‘a photo of a [class name]’’. Then, the image feature $\bm{f}\in\mathbb{R}^{d}$ and the class-extended text feature $\bm{t}_{i}\in\mathbb{R}^{d}$ are extracted from CLIP’s visual encoder and textual encoder, respectively. The classification is determined by the cosine similarity $\langle\bm{f},\bm{t}_{i}\rangle$:

where $N$ represents the number of classes. In the few-shot setting of CLIP, there are given $N$ classes, each class containing $K$-shot training images. In our method, apart from the encoded image feature $\bm{f}$ and the text feature $\bm{t}$, we also exploit local image features $\bm{x}\in\mathbb{R}^{hw\times d}$, where $h$ and $w$ represent the height and width of the local image feature, respectively. The overview of our method is depicted in Figure 3. We select local image features that exhibit strong correlations with both image and text features, respectively, as determined by cosine similarity. Then we utilize these selected local features to construct the vision subapace and language subspaces, respectively. In the inference stage, the test image feature is projected into the vision and language subspaces, respectively, and then the projected training text-image features and the projected test features are sent to the classifier to predict the result.

In the vision projector module, the cosine similarity between the image feature and the local image features is calculated for each sample in the training set as follows:

Here, $\bm{f}_{i,j}$ is the image feature of $j$-th sample from class $i$, and $\bm{x}_{i,j}$ denotes the corresponding local image features. $K$ indicates the number of samples within each class. $\bm{s}_{i,j}\in\mathbb{R}^{hw}$ denotes the vector of similarity scores, with each element indicating the correlation between the image feature and the local region feature. Previous studies (Li et al., 2023b; Miyai et al., 2023) have demonstrated that local image features play a crucial role in capturing object and semantic information for vision and language, respectively. Consequently, a higher value in $\bm{s}_{i,j}$ indicates that this local region contains more discriminative information for correct classification. Conversely, a smaller value suggests that the region includes less useful or irrelevant information for the corresponding class. Based on these insights, we utilize regions with top-$Q$ largest values to construct the vision subspace:

where $Q$ is a hyper-parameter validated in Section 4.2.1, and $n=\{n_{1},n_{2},\cdots,n_{Q}\}$ denotes the indices set of top-$Q$ largest values in $\bm{s}_{i,j}$. By aggregating these local features across all training samples and concatenating them with $\bm{f}_{i,j}$, we obtain:

$\hat{\bm{X}}$ comprises the informative features for vision patterns, and we decompose the $\hat{\bm{X}}$ by Singular Value Decomposition (SVD):

where $\bm{U}_{\text{vis}}\in\mathbb{R}^{((N+1)K\times Q)\times((N+1)K\times Q)}$ and $\bm{V}_{\text{vis}}\in\mathbb{R}^{d\times d}$ is the left singular vectors and right singular vectors, respectively, and they are corresponded to the singular values $\bm{\Sigma}_{\text{vis}}\in\mathbb{R}^{((N+1)K\times Q)\times d}$ sorted in the descending order. We employ the principal vectors of $\bm{V}_{\text{vis}}$, denoted as $\hat{\bm{V}}_{\text{vis}}$, to span the visual subspace (Smith et al., 2017), and the corresponding projection matrix is calculated as (Guerci and Bergin, 2002; Yuan and Gan, 2017; Zhu et al., 2020):

where $\bm{P}_{\text{vis}}=\bm{P}_{\text{vis}}^{\mathrm{T}}\in\mathbb{R}^{d\times d}$. Depending on the constructed vision subspace and projection matrix, we align the image features via subspace projection:

where $\bm{F}=[\bm{f}_{1,1},\cdots,\bm{f}_{N,K}]\in\mathbb{R}^{d\times NK}$ refers to all training image features, and the projected image features $\hat{\bm{F}}$ are then utilized in the classification process.

The language projector module has the same effects as the vision projector, and they exhibit a similar procedure. The cosine similarity between the text feature and local image features is calculated as:

where $\bm{t}_{i}$ is the text feature of $i$-th class, and $\bm{x}_{i,j}$ denote the local image features of $j$-th image for class $i$. $\bm{z}_{i,j}\in\mathbb{R}^{hw}$ stores the similarity scores between the text feature and the local image feature. The larger values in $\bm{z}_{i,j}$ imply that these local regions exhibit highly semantic with $\bm{t}_{i}$, and we select regions with top-$C$ largest values to construct language subspace:

where $m=\{m_{1},m_{2},\cdots,m_{C}\}$ represents indices of top-$C$ largest values in $\bm{z}_{i,j}$. For each text feature $\bm{t}_{i}$, there are $K$ local image features belonging to that class. Consequently, we can gather a total of $K\times C+1$ features, denoted as $\tilde{\bm{X}}_{i}=[\bm{t}_{i},\tilde{\bm{x}}_{i,1},\cdots,\tilde{\bm{x}}_{i,K}]^{\mathrm{T}}\in\mathbb{R}^{((K\times C+1)\times d}$ for each class. In contrast to a unified vision subspace, we construct a language subspace for each class based on $\{\tilde{\bm{X}}_{i}\}_{i=1}^{N}$ via SVD, as shown below:

where $\bm{U}_{\text{tex}}^{i}$ and $\bm{V}_{\text{tex}}^{i}$ are the left and right singular vectors, respectively, corresponding to the singular values $\bm{\Sigma}_{\text{tex}}^{i}$ sorted in descending order. $\tilde{\bm{V}}_{\text{tex}}^{i}$ denotes the primary singular vectors of $\bm{V}_{\text{tex}}^{i}$, which forms the basis for the $i$-th language subspace. The corresponding projection matrix is denoted as $\bm{P}^{i}_{\text{tex}}=\tilde{\bm{V}}_{\text{tex}}^{i}(\tilde{\bm{V}}_{\text{tex}}^{i})^{\mathrm{T}}\in\mathbf{R}^{d\times d}$. Subsequently, the text features are projected by their corresponding projection matrix as follows:

The projected text features $\tilde{\bm{T}}$ and the projected image features $\hat{\bm{F}}$ belonging to the same class lie closely to each other, as depicted in Figure 1 (b), Thereby, these paired text-image features are utilized for classifying test images during the inference stage.

Our SSP doesn’t involve the parameters training process and can seamlessly build on top of existing adapter-based methods, such as LFA(Ouali et al., 2023), Tip(Zhang et al., 2022b), and APE(Zhu et al., 2023d), to achieve improved classification results, as summarized in Table2 and Table 3. Specifically, given a test image $\bm{f}_{\text{test}}\in\mathbb{R}^{d}$, it’s projected into the vision subspace and language subspace, respectively. The vision subspace projection process is straightforwardly calculated as:

where $\bm{f}_{\text{test}}^{\text{vis}}$ denotes the projected feature. However, for the language subspace projection, we encounter a challenge since we lack categorical information to determine the appropriate language projection matrix to use. To address this, we rely on projection theory (Yuan and Gan, 2017; Zhu et al., 2020) to choose the language projection matrix based on minimizing the $\ell_{2}$ norm of the orthogonal projected features:

If the $\bm{f}_{\text{test}}$ lies in the $i$-th language subspace entirely, the projection operation doesn’t change the norm of the test feature, i.e., $\bm{f}_{\text{test}}=\bm{P}^{i}_{\text{tex}}\bm{f}_{\text{test}}$. After aligning the test feature via our vision and language subspace projection. The aligned test features along with the aligned image and text features are inputted into the various classification frameworks the get the prediction. For example, if we choose the LFA (Ouali et al., 2023) as the classifier, we calculate the transformation matrix $\bm{W}$ based on aligned image features $\hat{\bm{F}}$ and text features $\tilde{\bm{T}}$. We then use $\bm{f}_{\text{test}}^{\text{tex}}$ to compute the final classification logits, given by $\tilde{\bm{T}}\cdot\bm{W}\cdot\bm{f}_{\text{test}}^{\text{tex}}$. If we choose the Tip (Zhang et al., 2022b) or APE (Zhu et al., 2023d) as the classification frameworks, we utilize both $\bm{f}_{\text{test}}^{\text{vis}}$ and $\bm{f}_{\text{test}}^{\text{tex}}$ to calculate the prediction results, which are given by $\hat{\bm{F}}\cdot\bm{f}_{\text{test}}^{\text{vis}}+\tilde{\bm{T}}\cdot\bm{f}_{\text{test}}^{\text{vis}}$.

Our method focuses on aligning the image and text features and can be easily built on top of various classifiers as described above. This sets our SSP approach apart from adapter-based methods such as Tip(Zhang et al., 2022b), APE(Zhu et al., 2023d), and LFA(Ouali et al., 2023). Although these methods operate on adapting the feature encoded from CLIP, they can also be viewed as classification frameworks due to their direct calculations between the training and testing samples. As shown in Figure4, the adapter-based methods are depicted in the green box at the bottom. They employ various techniques to calculate the relationship among $\bm{T}$, $\bm{F}$, $\bm{L}$, and the test sample $\bm{f}_{\text{test}}$ to achieve classiffication, where $\bm{L}$ signifies the one-hot labels for $\bm{F}$. In contrast, our method, illustrated in the gray section at the top, leverages local image features $\bm{X}$ as a bridge to align the image features, text features, and test features, resulting in $\bm{\tilde{T}}$, $\bm{\hat{F}}$, $\bm{f}_{\text{test}}^{\text{vis}}$, and $\bm{f}_{\text{test}}^{\text{tex}}$. These aligned features are substituted for the original ones in the computations to derive the classification results.

In this section, we use RN50 on ImageNet(Deng et al., 2009) to evaluate the contributions and effectiveness of different components in our approach.

We conducted a comprehensive comparison of our method with the latest approaches, including CoOp (Zhou et al., 2022b), TaskRes (Yu et al., 2023b), GraphAdapter (Li et al., 2023a), LFA (Ouali et al., 2023), APE (Zhu et al., 2023d), and Tip (Zhang et al., 2022b), across 11 image classification datasets. We first perform experiments using different visual encoders, including RN50, RN101, ViT-B/32, and ViT-B/16, on the ImageNet with a 16-shot setting. The results, as summarized in Table 2, indicate that our SSP method leads to performance improvements compared to Tip, LFA, and APE. Notably, when built upon the APE method, our SSP outperforms all other methods. Furthermore, we extended our comparisons to evaluate our method across different shot settings on various datasets, with the results outlined in Table 3. While our method can not achieve the highest performance on some individual datasets, it consistently outperforms the other methods when considering the average accuracy across all datasets. This demonstrates the robustness of our SSP method. Specifically, compared to LFA, our method achieves a significant average accuracy improvement of 3.4% in the 1-shot setting. Furthermore, even when compared to APE, our method consistently exhibits slightly better performance across all shot settings.

We conducted experiments using the ImageNet dataset as the source dataset, which provides 16-shot training images for each category. We then tested our method on four different variants of the ImageNet dataset: ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A(Hendrycks et al., 2021b), and ImageNet-R (Hendrycks et al., 2021a). These test datasets share the same class labels as ImageNet but exhibit semantic gaps. The classification results are summarized in Table 4. Our method achieved a significant improvement of 0.73% over the LFA method specifically on the ImageNet-R dataset. Furthermore, our method demonstrated an average performance improvement of 0.38% across all out-of-distribution (OOD) datasets. These results demonstrate our SSP is robust in domain adaptation.

We compared the computation time and parameters between our SSP and other approaches on the ImageNet dataset, under the 16-shot setting. The results illustrated in Table 5 show that our SSP does not introduce any learnable parameters and primarily incurs computation time during matrix operations. To align image features, SSP requires a single SVD operation, typically taking a few seconds that could be ignored. As for text features alignment, the number of SVD operations depends on the total categories in the dataset (e.g., 1000 categories in ImageNet). When combined with LFA, Tip, or APE, our SSP needs nearly an extra 1 minute computation time, but it consistently improves the classification accuracy, particularly by 0.72% over Tip.