BMIP: Bi-directional Modality Interaction Prompt Learning for VLM

In contrast to traditional language prompt learning, deep language prompt learning introduces layered prompts to expand the scope of prompt information. Specifically, we introduce layered language prompts $\{P_{i}\in\mathbb{R}^{1\times b\times d_{l}}\}_{i=0}^{J}$, where $J$, $b$, and $d_{l}$ indicates the depth, length, and dimension of the language prompts, respectively. The input to the initial layer assumes the structure $[P_{0},W_{0}]$, where $\{W_{0}\in\mathbb{R}^{N\times x\times d_{l}}\}$ signifies the word embedding of the text $T$. Here, $x$ represents the number of words in a single caption from $T$, with $N$ indicating the total count of image categories. To provide comprehensive guidance on the word embeddings $W$, we deploy language prompts to supersede prompts from prior layers in the first $J$ layers of the text encoder. The inputs and outputs at the $i^{th}$ layer of the text encoder $g$ are as follows:

Beyond the ${J}^{th}$ layer, to prevent the model from excessive reliance on learnable prompts and overfitting, we use the prompts from the output of the preceding layer as the input to the next layer. The class feature $z$ is obtained by projecting the class representation $[c_{1},c_{2},\dots,c_{N}]$ corresponding to the last output of the text encoder $W_{K}$ to a common embedding space via the text projection head $\mathrm{TextProj}$.

To gather information about the dataset and interact with the language modality, we introduce independent visual prompts. Deep vision prompt learning employs a set of vision prompts $\{\tilde{P}_{i}\in\mathbb{R}^{1\times b\times d_{v}}\}_{i=0}^{J}$, which match the depth and length of language branches, differing solely in the dimension of the vision prompts $d_{v}$. In the first $J^{th}$ layers of the image encoder $f$, we use a learnable vision prompt to replace the output of the previous layer. These prompts are then concatenated with the class token $CLS$ and the image patch embeddings $E$, forming the input for the next layer, as shown below:

After the $J^{th}$ layer, similar to the language branch, the ensuing layer’s input is the immediate output of its predecessor. Upon obtaining the ultimate class token $CLS_{K}$, the image projection head, denoted as $\mathrm{ImageProj}$, is employed to map the final image feature $x$ to the common embedding space. The formula from the $(J+1)^{th}$ layer is as follows:

As a novel multimodal interaction approach to prompt learning, it is essential for BMIP to design with an interactive architecture that promotes effective information aggregation. Under this objective, this architecture consists of three key components: a language projection head $F_{l}$, a vision projection head $F_{v}$, and an aggregation function.

The most straightforward idea for aggregating information might include adding the aggregated information to the original prompts using unit weights or attention weights or simply connecting this information. Unfortunately, these methods driven by linear addition and modifications either fail to allocate the importance of the two modality inputs effectively or cannot easily highlight the internal impact of the prompts, making efficient information aggregation unattainable for effective information aggregation from both modalities.

To solve this problem, we propose a learnable aggregation function using the output weight of the attention layer for effective information aggregation from both modalities. This aggregation function uses learnable modules to dynamically generate weights for different modality prompts, giving greater emphasis to the prompts that the attention layer focuses on. Specifically, the vision and language projection heads produce transformed vision and language information, represented as $\{F_{v}(P_{i}),F_{l}(\tilde{P}_{i})\}$. The vision and language attention weight ($w_{v}$, $w_{l}$) are extracted from the output of the current attention layer, representing the degree of attention given by other inputs to the current prompt. Subsequently, we trained modality-specific $1\times 1$ linear layers, $L_{l}$ and $L_{v}$, to learn the relationship between attention weights and substitution weights ($w_{l},w_{v}$), ultimately generating unique dynamic weights for each prompt. The formal expression is as follows:

where $\{P_{i}^{\prime}|_{i=1}^{J}\}$ and $\{\tilde{P}_{i}^{\prime}|_{i=1}^{J}\}$ represent augmented deep language and vision prompts. After implementing the aggregation function, the first $J$ layers of BMIP process inputs and outputs for the language and vision modalities as described subsequently:

Beyond the $J^{th}$ layer, subsequent layers use the output of the predecessor as input to obtain the final prediction. Using the proposed aggregation function, vision and language encoders can receive weighted prompt input, facilitating alignment of vision and language. Additionally, the learnable linear layers eliminate the need for manually tuning prompt weights, ensuring high reliability and greater flexibility. Furthermore, BMIP can enhance existing prompt fine-tuning methods by serving as a more powerful foundational model than MaPLe, leveraging its effectively balanced multimodal prompts.

CoCoOp and subsequent prompt learning methods demonstrate that for well-aligned pre-trained VLMs, fine-tuned prompts are interchangeable. For instance, CoCoOp incorporates image information into text prompts, while MaPLe uses transformed language prompts as vision prompts, which reflects that vision and language prompts can be simultaneously optimized and mutually substituted, as demonstrated in the Ablation Studies.

In practice, especially when approaching the convergence point, the magnitude of $\frac{\partial L}{\partial w}$ is usually very close to zero, indicating that the probability of staying around $w$ is large. In other words, when the attention weight is equal to zero, this prompt will almost become redundant during the later training process. Therefore, using $w$ to combine the original prompt with aggregated information will enhance the trainability of the model and facilitate modal alignment. We immediately have the following corollary,

Corollary 1 If the minimal of weights implies $w=0$, then the prompt combining will only decrease the training loss, i.e. $min_{P^{\prime}}L\leq min_{P}L$, given the sufficiently expressive $min_{P^{\prime}}L$ and $min_{P}L$ which denote the cases with and without prompt combining, respectively.

These analysis suggests that replacement prompts enhance BMIP’s trainability, maintaining image text alignment while adapting to downstream tasks.

Evaluation Paradigm. There are three widely adopted tasks to evaluate prompt learning methods: generalization from base to novel classes, cross-dataset transfer, and domain generalization Zhou et al. (2022a); Khattak et al. (2023a); Lee et al. (2023); Yao et al. (2023). Generalization from base to novel classes task evaluates the performance of models on base and new classes separately. Although this evaluation paradigm can comprehensively evaluate the performance of both base and new classes, it lacks practicality for real-world applications since downstream tasks cannot determine whether the data belongs to base or new classes in advance, which is a major metric of a model’s ability to generalize. Therefore, we propose a more realistic evaluation paradigm, termed open-world generalization, which introduces a metric used in open-world settings Zareian et al. (2021); Gu et al. (2022), the simultaneous evaluation of base classes and new classes, to the generalization from base to novel classes task. This new paradigm assesses the model’s performance on an unknown distribution composed of both base and novel classes. The cross-dataset transfer task demonstrates the model’s zero-shot generalization to novel datasets, while the domain generalization task assesses the model’s robustness to out-of-distribution data.

Dataset. We assessed 11 image recognition datasets in the open-world generalization and cross-dataset transfer tasks, encompassing ImageNet (2009), Caltech101 (2004), OxfordPets (2012), StanfordCars (2013), Flowers102 (2008), Food101 (2014), FGVC-Aircraft (2023), SUN397 (2010), UCF101 (2012), DTD (2014), and EuroSAT (2019). For domain generalization task, ImageNet serves as our source dataset, while 4 variants ImageNetV2 (2019), ImageNet-Sketch (2019), ImageNet-A (2021b), and ImageNet-R (2021a) serve as the target datasets.

Compared Methods. There are several methods similar to BMIP that use prompts exclusively to finetune VLMs, such as CoOp, PLOT, ProDA, and MaPLe; therefore, we focus on benchmarking against four current representative methods: CLIP, CoOp, CoCoOp, and MaPLe, the SOTA method. It is worth noting that BMIP is orthogonal to studies that employ regularization techniques and can be integrated with these methods to enhance prompt learning optimization. We report the performance improvements brought by BMIP in ablation studies. The implementation details are in Appendix A.

To assess the open-world robustness of the BMIP approach, the average performance across all datasets, as well as the detailed performance on each dataset measured by two metrics, i.e., the harmonic mean (HM) of the accuracies for the base and new classes and Accuracy, are reported. The average performance and variance under three runs are presented in Table 1. Compared to the SOTA method MaPLe, BMIP demonstrates performance improvements on both base and novel classes across most datasets. In the remaining three datasets, the BMIP method achieves competitive performance compared to the MaPLe method and delivers the best average performance on both the HM and Accuracy metrics. BMIP shows an absolute average performance gain of 0.82% over MaPLe, when considering both base and novel classes simultaneously. Additionally, BMIP performs exceptionally well on datasets with imbalanced text and image information Nilsback and Zisserman (2008); Helber et al. (2019); Zang et al. (2022), such as those with high intra-class visual variance (e.g., EuroSAT, from 81.43% to 86.10%) and low inter-class text variance (e.g., Flowers102, from 82.78% to 83.86%), reflecting our motivation that BMIP can overcome the shortcomings of single-modal prompt learning methods. The above results highlight the crucial role of the independent prompts of each modality and the strong alignment between vision and language in enhancing generalization capacity.

To evaluate the transfer ability of our method, we finetune multi-modal prompts using the ImageNet dataset and directly transfer them to the other 10 datasets. Table 2 presents a performance comparison between CoOp, CoCoOp, MaPLe, and BMIP. BMIP exhibits competitive performance across 10 datasets, achieving the highest average accuracy of 66.86%, with a particularly notable accuracy of 49.63% on EuroSAT. Although CoOp performs the best on the source dataset, its performance degrades on other datasets. These results suggest that using bi-directional modality interaction within BMIP aids in better cross-dataset generalization, exceeding MaPLe, highlighting the significance of incorporating bi-directional information in prompt learning.

We assess the robustness of the mentioned methods trained on ImageNet to various out-of-distribution (OOD) datasets. BMIP outperforms MaPLe on three of four out-of-distribution datasets, enhancing the OOD average classification accuracy to 60.40% and achieving an overall average accuracy of 62.50% on both training and testing dataset, as shown in Table 3. The results demonstrate that BMIP effectively addresses open-world challenges, including open-world generalization and domain generalization, highlighting the model’s strong performance in these areas.

BMIP focuses on prompt learning for modality interaction and can be becombined with any prompt-based model, using MaPLe as the foundational model. Therefore, to verify the flexibility of BMIP, we combine BMIP with two representative methods, PromptSRC Khattak et al. (2023b) and CoPrompt Roy and Etemad (2024), which represent the latest approaches utilizing additional knowledge, such as regularized information, to enhance model training. Table 4 indicates that BMIP brings a significant performance improvement to these methods, demonstrating BMIP’s superior capability as a foundational model compared to MaPLe.

In our ablation studies, we explore various aggregation functions to determine the individual contributions of the BMIP components to their overall performance, thus validating the intuition derived from our corollary. To ensure that the performance improvement of BMIP is not attributed to the number of parameters, we conduct comparative experiments that equalize the number of parameters between MaPLe and BMIP.

Compare with Other Aggregation Functions. We describe and compare different aggregation functions in more detail in Appendices B and C to verify the benefits of the learnable weighted aggregation function over direct prompt modification functions, such as Addition, Attention, and Joint. As we can see from Table 5, the comparison with Addition highlights the importance of our learnable weights, while the contrast with Attention demonstrates the reliability of learnable weights over similarity-based prompt calculations. Furthermore, the comparison with concatenation-based prompt aggregation underscores the effectiveness of our replacement strategy in reducing prompt redundancy. The comparisons above validate the intuition presented in our analysis: the prompts in VLMs can be mutually replaced and optimized simultaneously and BMIP improves the trainability of prompts. We provide a detailed report on the performance of different aggregation functions across all settings in Appendix C.

Number of Parameters. To ascertain that BMIP’s enhanced performance is not due to an increase in parameter count relative to MaPLe, we equalize the number of parameters between MaPLe (denoted $\text{MaPLe}^{\dagger}$) and BMIP. Table 5 demonstrates that $\text{MaPLe}^{\dagger}$ tends to overfit to base classes when parameters are increased, whereas BMIP maintains robust performance despite a higher parameter count. This indicates the efficacy of BMIP’s bidirectional modality interaction framework in harmonizing prompt information across modalities. We also assess the impact of varying prompt depths and lengths in Appendices D and E, which proves our intuition about prompt depth.