LEO: Boosting Mixture of Vision Encoders for
Multimodal Large Language Models

With the rapid advancement of large language models [46, 1, 43, 8], there has been considerable interest in multimodal models that enhance understanding and reasoning capabilities. BLIP2 [21] introduces the Q-Former, designed to efficiently bridge the modality gap between images and text during pretraining. Flamingo [2] is capable of processing sequences that combine visual and textual data in any arbitrary order, a crucial feature that enables them to perform in-context few-shot learning effectively. LLaMA-Adapter v2 [12] activates a larger number of learnable parameters in the LLM through an early fusion strategy for vision and text tokens, where vision tokens are fed only into the initial layers of the language model, enabling more efficient integration of visual information. LLaVA [29, 27] delivers impressive results by employing a straightforward projector, such as a linear layer or MLP, between the visual and language components, and developing a streamlined instruction-following data process powered by GPT. Most of these methods, however, maintain low input resolutions due to the constraints of pretrained vision encoders, such as that of CLIP [38], and the sequence length restrictions of large language models. All these approaches exploit a single vision encoder in their architecture.

To address the constraints of lower input resolutions, recent MLLMs have concentrated on enhancing their vision encoder module. From a vision-focused standpoint, these approaches can be broadly categorized into three main strategies: (1) robust vision encoders: designing stronger vision encoders [50, 7] that effectively capture complex visual features by incorporating larger models sizes and better training strategy, (2) tile segmentation: handling high-resolution inputs by dividing images into smaller, lower-resolution tiles [39, 28, 6], and (3) hybrid vision encoders: utilizing a vision backbone that incorporates multiple vision experts.

Our research is closely tied to the third group of approaches [23, 45, 31] that develop a multi-branch vision encoder framework to enhance perception capabilities of MLLMs. Some models [33, 23] suggest merging high-resolution visual details with low-resolution tokens to enhance visual representation. LLaVA-HR [33] proposes a dual-pathway vision model that integrates features from high-resolution convolutional blocks with those from low-resolution ViT blocks. These pretrained vision experts might nevertheless lack key capabilities, such as text understanding and object localization. Several studies have incorporated multiple vision experts trained on diverse tasks to broaden the functionality of their encoders. Brave [16] and Mousi [11] perform sequence append, combining vision tokens from multiple experts into a single, extended sequence. Tong et al. [45] identify distinct differences in the visual features captured by CLIP [38] and DINOv2 [37], leading to the design of an image-level mixture-of-features strategy. Some models combine vision tokens through channel concatenation to preserve the original sequence length, such as DeepSeek-VL [31] and Eagle [40] models, or utilize advanced fusion and routing techniques [19, 51] to leverage the strengths of various encoders. These methods perform pre-adaptation fusion of vision tokens, which are then processed by a single projector module (either an MLP or Q-Former), as shown in Fig. 2.

Distinguished from previous studies, this work proposes a novel hybrid approach that integrates a tile-level post-adaptation fusion strategy with dynamic high-resolution inputs achieved through tile segmentation. In our framework, each vision expert is equipped with its own projector, and the vision tokens are sequentially interleaved following vision-text alignment by the projectors at the tile level.

Figure 3 illustrates the overall architecture of our proposed multimodal large language model, LEO, which is conceptually straightforward: First, the high-resolution input images are divided into tiles. These tiles are then processed by two different vision experts, each exploiting its specialized pretraining to provide distinct feature representations. Next, pixel unshuffling is applied to the extracted visual embeddings to reduce the number of visual tokens for each vision encoder. Vision-text alignment is conducted using two distinct MLP projectors, and a tile-level post-adaptation fusion strategy is employed to sequentially interleave tile-level visual tokens. Finally, these visual tokens are combined with text tokens and processed by the LLM for comprehensive visual-language understanding and reasoning. The following section gives more details of the architectural blocks identified in Fig. 3.

Dynamic high resolution. The image processing begins with a dynamic high-resolution approach, where an input image is segmented into multiple tiles alongside a thumbnail of the original image, helping the model to capture the global context. We utilize a dynamic resolution input strategy similar to that in InternVL [7, 6]. Each input image is resized to a closest available aspect ratio that is dividable into square patches of size $448\times 448$, with up to six tiles for the 3:2 aspect ratio shown in Fig. 4. To capture global context, the input image is also scaled to match the patch size and added to the set of patches to be processed by the vision encoders. This approach, when applied dynamically to two input images, results in a total of 14 unique patches for training. Figure 4 illustrates this tiling process with an example driving scene image, where the tiles are shown after normalization by the SAM’s preprocessor [18].

Pixel unshuffling. Once the input images are segmented into tiles, they are simultaneously fed to two distinct vision encoders. We select InternViT-300M-448px [7] as the first vision encoder and SAM-L [18] as the second. For each encoder, we apply pixel unshuffling [41], which rearranges the spatial layout of pixels to reduce the number of visual tokens while preserving important visual features. Given an input tensor of shape $[b,c,w,h]$ and a downscaling factor of $r$, the output tensor will have the shape $[b,c*(r^{2}),w/r,h/r]$, where $b$ represents the number of tiles, $c$ is channel, and $w$ and $h$ denote width and hight. Specifically, for each segmented tile, we apply downscaling factors of $2$ and $4$ for our first and second vision encoders, respectively, reducing the number of visual tokens to $256$ per tile and encoder, resulting in a total of $512$ visual tokens per tile.

Visual token fusion. Existing hybrid multimodal models [33, 23, 16, 40] typically use a pre-adaptation fusion strategy to combine visual tokens from two or more visual encoders, applying one of the common fusion methods shown in Fig. 2 (bottom) prior to the vision-text alignment process. In these approaches, all visual encoders share the same projector module. In contrast, LEO employs an alternative fusion approach in which each vision encoder maintains its own dedicated projector module, allowing for independent processing of visual tokens before they are combined. We find this to be a more flexible and effective fusion strategy. We use a two-layer MLP for the projector architecture to ensure simplicity and efficiency. To streamline processing, we set the output dimension of the projector to match that of our LLM. In the fusion block, we sequentially interleave the visual tokens from InternViT and SAM-L for each segmented tile, while preserving their original order.

Language model. We combine the extracted visual and text tokens and feed them into the LLM for auto-regressive generation. For our language model, we use InternLM2-7B-Chat [4], identical to the LLM in the base model [7], to ensure a fair comparison. To optimize memory and computational efficiency, the context length of the LLM is set to a maximum of 8196 tokens, ensuring balanced performance across various multimodal tasks. Given the input context length constraint of the LLM, we limit each input image to six segmented tiles, which enables efficient handling of multiple images.

Our training process consists of two main phases. In the first phase, we initialize the vision encoders and the language model from the base models [7, 4, 18], while the projector layers for SAM are randomly initialized. This approach is adopted because InternViT is already pretrained on vision-language alignment and SAM is pretrained specifically for segmentation. To avoid any potential representation inconsistencies, we focus on training the layers of the second projector. In the second phase, the vision encoders are kept frozen, and we perform full fine-tuning of two projectors and the language model. In Table 5, we present the results of an ablation study that shows the effect of keeping the vision encoders frozen during the training phases.

Although numerous studies have successfully applied MLLMs to autonomous driving [34, 5, 44, 47], a straightforward approach that avoids extensive modifications to model architecture, training processes, or heavy data collection has yet to be fully explored. In this work, we investigate the potential of applying LEO to the autonomous driving domain without altering its architecture or training recipe, aiming to offer insights into streamlined transfer learning and facilitate MLLM adaptation to specialized domains. Instruction tuning plays a crucial role in helping models learn to follow user prompts, utilizing training data in visual question answering and conversational formats. For this domain, we design tasks in a VQA format, with each frame represented as: $<$img$>$ $<$IMG-CONTEXT$>$ $<$/img$>$. At the prompt level, the temporal aspect of video frames is managed by treating sequential frames as multiple image inputs. A sample prompt is formulated as:“$<$image1$>$ … $<$image N$>$ Is it safe to enter the intersection at this time?”. The images in this setting are of high-resolution, each measuring $2048\times 1280$ and segmented into six patches of size $448\times 448$.

Implementation details. Our model architecture is built upon InternVL [7], following a standard MLLM design of ViT-projector-LLM. The projector aligns the visual features by mapping them into the language embedding space. LEO uses InternLM2-7B-Chat [4] as the large language model, with pretrained InternViT-300M-448px [7] and SAM-L [18] as vision encoders, and an adaptive tiling strategy. Following the base model [7], we divide each input image into $448\times 448$ tiles, where the number of tiles varies based on the aspect ratio of the image. Our training procedure consists of two stages. In the first stage, we freeze both vision encoders and focus on optimizing the projector module to ensure effective training. In the second stage, we perform supervised instruction tuning, unfreezing the projector modules along with the LLM. Both stages employ a context length of $8196$, and training is conducted for a single epoch. We optimized the model using the AdamW optimizer with a cosine learning rate schedule. During the the second stage, we set the learning rate to $4\times 10^{-5}$ with a weight decay of $0.01$. In the alignment stage, we increased the learning rate to $4\times 10^{-4}$, maintaining the same weight decay. Training was conducted on 8 A100 GPUs (80 GB each) using DeepSpeed’s Zero2 strategy, allowing training to complete within approximately 72 hours.

Training Datasets. In the first stage of training, we use the LLaVA-595k dataset [28], which comprises 595k samples. In the supervised finetuning stage, we employ the same supervised fine tuning stage dataset as InternVL [7], incorporating a total of approximately one million visual instruction tuning samples, all of which are fully open-source.

Comparison with leading MLLMs. In this section, we comprehensively evaluate our model’s visual understanding and reasoning abilities in comparison to previous leading MLLMs, across 13 vision-language benchmarks. These benchmarks are organized into three task categories: (1) OCR and chart understanding, including DocVQA [36], TextVQA [42], ChartQA [35], and AI2D [17]; (2) general visual question answering, including VQA${}^{\text{v2}}$ [13], GQA [15], and VizWiz [14]; and (3) general multimodal benchmarks, such as MMMU [49], MMBench [30], SEED-Bench [20], POPE [22], MMVet [48], and ScienceQA [32].

In Table 1, we see that LEO achieves state-of-the-art results in 12 out of 13 benchmarks. In the OCR and chart understanding category, LEO consistently surpasses leading models across all four datasets, by virtue of its dual-branch vision encoders. In the multimodal benchmark category, LEO demonstrates superior performance across all six benchmarks, highlighting its broad knowledge and advanced reasoning abilities. Additionally, the results in Table 1 show that LEO excels in more demanding benchmarks that necessitate college-level knowledge, such as MMMU [49], which focuses on complex problems from various domains, including science, business, tech and engineering, as well as health and medicine. Notably, compared to InternVL [7], which uses the same LLM and vision encoder as our model, LEO achieves superior performance across 8 out of 9 benchmarks, demonstrating the benefits of dual-branch vision encoders for vision-language tasks. This approach mitigates the inherent biases of individual vision encoders, providing a robust framework for the mixture of encoders.

Comparison with leading Hybrid MLLMs. We compare the performance of LEO with recent hybrid approaches across 11 benchmarks. In Table 2, we see that LEO demonstrates strong performance on the majority of benchmarks. Our model is trained on the least amount of data in both pretraining and SFT stages, yet it outperforms models trained on larger datasets, such as Mousi [11], highlighting the generalization capability of our model. Compared to models with more complex fusion strategies, such as LLaVA-HR [33] and Mini-Gemini [23], our model excels across most benchmarks, especially on multimodal benchmarks like MMBench, SEED, and ScienceQA. Notably, compared to Brave [16], which combines five distinct vision encoders through pre-adaptation fusion and sequence concatenation, LEO achieves competitive performance on most tasks. This result underscores that post-adaptation fusion of visual tokens from only two vision experts can be as effective as pre-adaptation fusion of visual tokens from a larger set of vision experts.

Comparison with Eagle. We conduct a comparison with Eagle [40], a concurrent work that integrates vision encoders through pre-adaptation fusion and channel concatenation. Table 3 shows that LEO outperforms Eagle-X2 [40], which combines two vision experts, on 7 out of 9 benchmarks, particularly excelling in the OCR and general VQA categories. Notably, LEO also surpasses Eagle-X4 [40], which uses four vision encoders, on 5 out of 7 benchmarks, with an identical score for GQA and a near-dentical score on POPE. It is worth mentioning that these results are achieved despite LEO being trained with less SFT data, highlighting the robustness and reasoning capability of the enhanced fusion design in LEO.

Settings. We use the LingoQA benchmark [34] to evaluate the performance of LEO in the autonomous driving domain. LingoQA contains over 400K samples and covers various aspects of the driving process. This dataset includes data covering nine distinct task types, such as action, justification, localization, and anticipation, providing a thorough representation of scenarios encountered in autonomous driving.

Data format. In LingoQA, images are captured from a front-view camera with sequences of five frames. Due to computational limitations, we use only two frames during training. For this evaluation, we maintain the same pretraining data as in general training and use the LingoQA Scenery and Action training dataset [34] for the second stage. The training process remains as described in Section 4.1. Additionally, we reformat the LingoQA data into the standard conversational format described in Section 3.4.

Results. We evaluate our model on the LingoQA validation set [34], with results presented in Table 4. Our model demonstrates competitive performance against the closed-source LingoQA baseline [34], which is pretrained on over $22$M data samples, significantly outperforming it on the METEOR and CIDEr metrics. Without modifying its architecture or training recipe, LEO also surpasses all existing open-source baselines across all four metrics. Notably, LEO achieves higher scores than the base model [7], highlighting the effectiveness of its dual-branch design.

Comparison with different training settings. To more effectively analyze the impact of training strategies for vision encoders in multimodal large language models, we conduct an ablation study on the vision encoder modules in our model (SAM-ViT-Large [18] and InternViT-300M [7]). This also provides insights into the contributions of each encoder to the model effectiveness. Each vision encoder processes 256 visual tokens. Results in Table 5 highlight three key findings. First, keeping the vision encoders frozen during training improves evaluation scores; for instance, unfreezing SAM reduces SEED performance by 6.62%. Second, InternViT alone performs better than SAM alone across all benchmarks, with SAM struggling on tasks like text recognition. This is likely due to InternViT’s large-scale pretraining, although when combined with InternViT, SAM enhances performance on this type of task. Finally, regardless of whether the SAM vision encoder is frozen, a hybrid MLLM consistently outperforms models with a single vision encoder.

Effect of fusion method. Sequence and channel concatenation are two primary approaches for fusing visual tokens in hybrid MLLMs. To investigate the impact of these fusion strategies on the performance of LEO, we conduct an experiment, with results presented in Table 6. Our findings reveal that sequence concatenation consistently outperforms channel concatenation across all benchmarks, highlighting its effectiveness in enhancing model performance. It is worth noting that these results are specific to post-adaptation fusion. For a broader comparison, refer to the results presented in Table 2 and Table 3, where our model performance is compared with models employing pre-adaptation fusion through four different methods: Brave [16] and Mousi [11] use sequence concatenation, LLaVA-HR [33] employs an MR-adaptor, Mini-Gemini [23] uses cross-attention, and Eagle [40] utilizes channel concatenation.

Effect of tiling. To investigate the effect of tiling, we train LEO with and without tiling using two frames. This yields 14 tiles with tiling. The performances of these models applied to the LingoQA benchmark [34] are given in Table 7. We see that tiling improves performance across all four metrics; for instance, it enhances Lingo-Judge by 3.4%. These results confirm that incorporating dynamic high-resolution inputs can enhance the model capacity for understanding driving scenes.

To highlight the visual understanding capabilities of LEO, we conduct a qualitative analysis as shown in Fig. 5. Our model is applied to a variety of vision-language tasks, including complex reasoning, detailed counting, OCR, spatial and mathematical reasoning, accounting analysis, and multi-image and multi-frame reasoning. With an efficient tile-level post-adaptation fusion strategy, LEO exhibits impressive performance across these challenging tasks. For example, our model can perform attribute-based counting, such as identifying the absence of parked cars while there are several moving vehicles in the driving scene. Beyond simple recognition, LEO demonstrates spatial awareness, enabling it to answer OCR-related questions like,“What is located to the right of the shampoo?”. In multi-image reasoning, it accurately identifies detail differences between images, such as the dog’s head being in different positions. LEO also demonstrates strong capabilities in multi-frame reasoning in the autonomous driving domain, including recognizing safe actions in dynamic scenes, such as stopping to allow a pedestrian to cross. Additionally, LEO excels in OCR tasks, effectively interpreting dense text, and handles complex mathematical and accounting problems, showcasing its strong reasoning abilities.

The processing capacity of our model for input images is limited to a maximum of six patches (i.e., excluding the global context) due to the constraints of the language model context length and available computational resources. This restriction prevents support for higher-resolution images or a larger number of multi-image inputs.