TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings^††thanks: Work was done when D. Yan was an intern at Alibaba

Visual language models primarily consist of a visual encoder and a large language model, representing prominent architectures in the multimodal domain. Researchers have proposed numerous architectures (Li et al. 2023a; Zhu et al. 2024; Chen et al. 2023c) for integrating visual features into advanced LLM inference pipelines. Llama-Adapter (Zhang et al. 2023) proposes to generate language answer with taking the image input as condition. Flamingo (Alayrac et al. 2022) and LLaVA (Liu et al. 2024c) blend visual tokens with text as inputs to LLM, differing in that Flamingo employs gating mechanisms to inject encoded visual features into LLMs, while LLaVA directly concatenates visual and textual features at input. Complementarily, the availability of high-quality image-text pairs for VLM training is crucial. Several methods use Chat-GPT (OpenAI 2023a) and GPT-4 (OpenAI 2023b) to construct large-scale, high-quality datasets (Zhu et al. 2023; Liu et al. 2024c; Zhao et al. 2023).

Inspired by the compact structure and outstanding performance of LLaVA-1.5 (Liu et al. 2024a), we use LLaVA-1.5 as our baseline and incorporate a text-guided approach, similar to other LLaVA-based methods. Unlike most of these methods, which create additional datasets to enhance performance, our improvements focus entirely on the model architecture itself. This approach can further enhance the performance of methods that rely on extra datasets. The results in fifth and sixth lines of Table 1 verified this point.

Align the visual and text information in high semantic level is the base for building VLMs. Centered around this problem, researchers have done extensive work. Previous researchers have typically employed contrastive learning across modalities and autoregressive learning for text. CLIP (Radford et al. 2021) and SigLIP (Zhai et al. 2023) trained encoders on massive datasets, laying foundational work for aligning visual and textual modalities and significantly advancing subsequent VLM developments. BLIP (Li et al. 2022) meticulously design multiple loss functions for contrastive and generative learning, achieving refined cross-modal alignment through multi-stage training. BLIP-2 (Li et al. 2023b) adopts a Q-former structure, interacting with the visual modality using learnable query vectors before merging with the text modality. Many LLaVA-like approaches use simple MLPs for modal alignment, with subsequent works like MobileVLM V2 (Chu et al. 2024).

Both image-text alignment methods and our proposed TG-LLaVA recognize the importance of integrating textual and visual information. However, while these methods focus on bridging different modalities, our approach leverages the textual modality to guide and optimize the visual modality. This alignment makes the operation of VLMs more consistent with the purpose-driven logic of human behavior in real-world scenarios.

To enable the LLM to extract more information from the input visual image, various strategies have been proposed for utilizing visual features. DenseConnector (Yao et al. 2024) employs dense connections to link visual features across different levels, feeding the combined features into a connector. TokenPacker (Li et al. 2024a) merges visual features from the high-resolution branch with those from the low-resolution branch to generate condensed visual tokens. Idefics2 (Laurençon et al. 2024) compresses visual features using a perceiver structure, significantly reducing the number of visual tokens compared to other approaches. Approaches like Mini-Gemini (Li et al. 2024b), LLaVA-Next (Liu et al. 2024b), Qwen-VL (Bai et al. 2023b), and InterVLM (Dong et al. 2024) leverage high-resolution images to capture finer visual feature details. ImageBind-LLM (Han et al. 2023) and Llama3.1 (Meta AI 2024b) explore injecting visual modality features into LLMs, with the former using trainable gating modules to add visual features to word tokens, and the latter introducing visual information across different layers of LLM through periodic cross-attention.

Unlike methods that focus on better utilizing existing visual features, our proposed TG-LLaVA aims to enhance the visual features themselves by using textual guidance. In contrast to ImageBind-LLM and Llama3.1, which incorporate image features into the LLM component, our approach integrates text into the visual encoder.

Taking LLaVA (Liu et al. 2024c) as an example, the primary goal of VLMs is to effectively harness the capabilities of pre-trained LLM and visual model. The three key components of such framework can be defined as follows:

1) Visual encoder ${\rm E}_{\rm v}$, typically utilizing a pre-trained vision transformer like CLIP, is designed to partition the input image $\mathbf{I}\in\mathbb{R}^{H\times W\times C}$ into several patches with equal size and further encode them into visual features $\mathbf{F}_{\rm i}\in\mathbb{R}^{N\times D}$. Here, $H$ and $W$ represent the size of the input image, $C$ denotes the number of channels, $N$ corresponds to the number of patches in the output features, and $D$ represents the feature dimension of each encoded patch. When the patch size is $P$, $N=HW/P^{2}$. 2) Connector ${\rm C}$ (also referred as Projector) consists of two linear layers with a GELU activation function in between. Its purpose is to map visual features into the embedding space of the LLM, converting $\mathbf{F}_{\rm i}$ into visual tokens $\mathbf{T}_{\rm v}$. 3) LLM ${\rm L}$ employs a tokenizer and text embedding module to sequentially transform textual data into token IDs and their corresponding embedded tokens $\mathbf{T}_{\rm t}$, effectively converting the language into the feature space of its input. Within the VLM architecture, these textual tokens $\mathbf{T}_{\rm t}$ are concatenated with the aligned visual tokens $\mathbf{T}_{\rm v}$ processed by the connector, forming the input for the LLM to carry out subsequent predictions. For a sequence of length $L$, the probability of VLM predicting the target answer tokens $\mathbf{T}_{\rm a}=\left\{t_{i}\right\}_{i=1}^{L}$ can be formalized as:

where $\theta$ represents all the trainable parameters in the VLM. In this VLM prediction paradigm, the visual features are directly obtained by encoding the raw input image through ${\rm E}_{\rm v}$ without any interaction with the textual modality. This approach contrasts with the purpose-driven nature of human behavior. Optimizing the encoded features based on textual instructions is more conducive to enabling the VLM to generate accurate responses.

Inspired by the reasoning logic humans use in visual question answering scenarios, we design TG-LLaVA, a novel approach that optimizes visual features to align the inference process of VLM more closely with purpose-driven human behavior, thereby further enhancing the capabilities of VLMs. As illustrated in Figure 2, TG-LLaVA primarily consists of two components: Text-guided Visual Feature Optimization Mask (TG-FOM) and Text-guided Detail Perceiver (TG-DP). The former uses learnable latents to parse the global information from textual instructions and attaches it as a mask to the output of visual encoder, optimizing features based on textual instructions. The latter employs another set of latents, first interacting with the detailed information from textual instructions, and then extracting fine-grained details from high-resolution patches of the input image based on these instructions. These details are concatenated with the original features, further refining the visual modality input of VLM. The specifics of this approach will be elaborated in the following sections.

At this point, with the guidance of input text, we have obtained the optimized visual features $\mathbf{F}_{\rm i}^{*}$, as well as the detail perceiver tokens $\mathbf{F}_{\rm i}^{\rm h}$. We then concatenate the features obtained from the original VLM connector ${\rm C}$ with $\mathbf{F}_{\rm i}^{\rm h}$, which together form the final visual tokens $\mathbf{T}_{\rm v}^{\rm fin}$ input for the VLM. The final prediction process of VLM can be represented as follows:

In Table 1, we present the performance improvements of the proposed method across various configurations compared to the baseline. According to the experimental results, we can draw several phenomenons:

• •

  The proposed text-guided strategy demonstrates substantial improvements over the baseline. Compared with the original LLaVA-1.5, TG-LLaVA achieve much better performance. As shown in the first four rows. our method leads on the majority of evaluation datasets. It is noteworthy that TG-LLaVA demonstrates an average improvement of 1.5% over the original LLaVA-1.5 across ten datasets when using Vicuna-7B, highlighting the method’s significant value. When juxtaposed with the baseline LLaVA-1.5 Vicuna-7B model, we enhance performance metrics by +2.2% on MMBench, +2.4% on both MMStar and MMMU, and +3.2% on LLaVABenchs, respectively. For LLaVA-1.5 with Vicuna-13B, we also achieve an average performance improvement of 1%. Specifically, we see a +1.6% gain on MMStar, a +2.0% gain on MMMU, and a +3.2% gain on MME. These impressive results further validate the contribution of the proposed TG-LLaVA architecture to visual feature optimization, highlighting the favorable impact of our method.

• •

  The proposed TG-FOM and TG-DP modules can be universally applied as a modular plugnin to mainstream VLM frameworks. As shown in the rest part of Table 1. we further validate the versatility of our proposed method under various settings. We replace CLIP with SigLIP and substitute Vicuna with Llama3 and Qwen2 on top of the original LLaVA-1.5 framework. We compare these settings with our method as the baseline. The results in Table 1 confirm that our method continues to maintain a leading advantage across most datasets, demonstrating that the proposed TG-LLaVA exhibits excellent generalizability and possesses strong potential for adaptation to a wide range of VLM architectures.

In Table 2, we compare the proposed proposed TG-LLaVA with other concurrent works which also take LLaVA as baseline. The methods we include for comparison are Seeing the Image (Xiao et al. 2024), TokenPacker(Li et al. 2024a), and DenseConnector(Yao et al. 2024). Since these comparison methods are relatively new and have not been evaluated on the OpenCompass leaderboard, we employ the evaluation scripts from LLaVA-1.5 to maintain a fair and consistent framework for our comparisons.

We further compare our method with several leading approaches. The methods included in the comparison are MiniGPT4 (Zhu et al. 2023), Qwen-VL(Bai et al. 2023b), VisualGLM(GLM et al. 2024), PandaGPT(Su et al. 2023), mPLUG-Owl2(Ye et al. 2023), Emu2-chat(Sun et al. 2024), Yi-VL(Young et al. 2024) and ShareGPT-4V(Chen et al. 2023a). Table 3 presents the performance comparison across multiple benchmarks.

Remarkably, despite relying solely on settings from LLaVA-1.5, our TG-LLaVA achieves performance that matches or surpasses the benchmarks set by leading SoTA methods, with a comparatively smaller volume of pre-training and instruction fine-tuning data.

To demonstrate the optimization effect of the proposed method, we visualize the attention maps between the visual features and text instructions in both the baseline model and our model. These visualizations provide insights into how the proposed visual feature optimization module operates. We aggregate the attention scores between image tokens and textual instruction tokens across all layers to compute the results. As shown in Figure 5, the TG-LLaVA architecture push the model to focus on regions highlighted by the textual instructions, assigning greater attention weights to them.

We further conduct in-depth ablation studies to analyze the effectiveness of each component of our approach. Results are listed in Table 4. By sequentially introducing the FOM and DP modules, we observe significant improvements in model performance, underscoring the effectiveness of our proposed visual feature optimization algorithm. Additionally, we conduct experiments on the number of additional visual tokens introduced by the DP module. The results show that introducing too few tokens yields suboptimal performance gains, while introducing too many tokens can actually harm performance. Therefore, we choose a balanced configuration to achieve optimal performance.

TG-LLaVA employs the same set of hyperparameters as LLaVA-1.5. It is important to note that both the TG-FOM and TG-DP modules in the proposed architecture are trained from scratch without any form of pre-trained parameters. Table 5 summarizes the training hyperparameters for both the first phase of visual-language alignment pre-training and the second phase of visual instruction tuning for TG-LLaVA.

As discussed in main paper, our objective is to explore how to endow VLMs with purpose-driven reasoning capabilities akin to human logic to achieve superior model performance. The key lies in how to utilize textual instructions to guide the VLMs, enabling a ”focus” on the visual modality.

In SAM-Adapter (Chen et al. 2023b), the authors use low-rank parameter matrices to handle task-specific features (such as high-frequency components obtained from Fast Fourier transforms of images to be segmented) and inject these features layer by layer into SAM’s (Kirillov et al. 2023) visual encoder (a Vision Transformer structure). This approach enables the transition of SAM from general domains to medical image segmentation. Inspired by this, we attempt a similar approach by using TG-FOM module to inject textual instruction information into the shallow layers of the VLM’s visual encoder. As shown in Table 6, we experiment with various strategies: inserting the output of TG-FOM module at the head, middle, and tail of the visual encoder, as well as combinations of these positions. We use MME dataset metric as the evaluation standard and CLIP-L with 24 layers as an example, insertion at the head refers to adding features between the Patch Embedding and the first layer of the Transformer Encoder. Insertion at the middle occurs between the 12th and 13th layers, while insertion at the tail refers to applying it at the output. However, the results are disappointing. Except for the final approach, which inserts the module at the encoder’s output, the other shallow-layer modifications lead to noticeable degradation in performance. We consider this is due to the fact that in the shallow layers of the visual encoder, the visual features are not yet aligned with the textual features in a unified space. Thus, even with a zero-initialized linear layer as a buffer, the outcome still results in negative gains. Further exploration is required to determine how to effectively integrate textual instruction information during the visual encoding process.

We further explore the upper limits of TG-LLaVA by fine-tuning on Qwen2-7B model with a larger dataset, utilizing the training data introduced by Ovis (Lu et al. 2024). The Ovis training data comprises open-source datasets and a small portion of internal datasets, categorized into three types: visual captions, visual descriptions, and multimodal instructions, used for the three-stage training process. We use the 10M visual captions subset as the training data for the second stage in training, with the results displayed alongside other variants in Table 7. As shown, underpinned by a robust dataset, TG-LLaVA demonstrates a remarkable average increase of 8% across ten datasets. Notably, it achieves a 20.2% improvement on MathVista and a 12.2% increase on ScienceQA, underscoring TG-LLaVA’s significant potential and the crucial role that high-quality data plays in enhancing VLM performance.

The proposed TG-LLaVA architecture involves multiple attention computations and image patch segmentation, leading to increased computational resource consumption during training. We record the training time for both the baseline LLaVA-1.5 and TG-LLaVA across two stages, with experiments conducted on 8 H100 GPUs. As shown in Table 8, TG-LLaVA incurs approximately 10% more time compared to the baseline. The additional trainable parameters introduced by TG-LLaVA amount to 0.18B compared to LLaVA-1.5 with Vicuna-13B. Currently, the implementation involves processing multiple detailed patches sequentially through the visual encoder, but optimizing this to a batch format could further reduce training delays.

We validate the inference capabilities of TG-LLaVA across various tasks involving understanding and reasoning to demonstrate its effectiveness in practical scenarios. As shown in Figure 6, TG-LLaVA handles complex visual-language reasoning problems in diverse contexts, including image perception and comprehension, mathematics and computation, OCR recognition, and other tasks requiring relevant prior knowledge.

To achieve text-guided visual feature optimization, TG-LLaVA inevitably introduces additional computational resource consumption, including extended training time and increased parameter count, beyond the original LLaVA-1.5. A viable improvement approach is to compress the number of visual tokens, such as TokenPacker (Li et al. 2024a). Compared to the total parameters of the original VLM, the increased parameter count is relatively small, and we believe that these trade-offs are reasonable without increasing the training data. In addition, our method can be used as a plug-in module to further improve the performance of other VLM models with similar architectures. Future work will explore methods to further reduce computational resource consumption while refining text-guided visual enhancement and improving guidance strategies.