LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancement

The integration of vision and language models has shown great promise in enhancing the perception capabilities of autonomous vehicles, enabling them to better understand and navigate complex driving environments. Early work in this domain includes CLIP-based methods [16], which pair visual representations with textual descriptions, enabling a richer understanding of the vehicle’s surroundings. Recent studies, such as those by [31] and [28], have proposed large multi-modal models incorporating visual and textual inputs to support decision-making. These models benefit from pretraining on large-scale datasets and have shown improvements in tasks such as scene interpretation and predicting vehicle behaviours in dynamic traffic scenarios.

The application of transformer-based models to vision-language fusion has also led to promising developments in autonomous driving. For instance, [5] developed a model that combines deep vision transformers with large-scale language models, which improves decision-making capabilities by enhancing the vehicle’s ability to generate complex driving plans. These models process real-time visual input while leveraging pre-trained knowledge to interpret high-level cues, such as road conditions and traffic regulations. Recently, [29] demonstrated the ability of multi-modal LLMs to engage in reasoning tasks in an end-to-end autonomous driving framework, allowing the vehicle to handle novel driving situations that require both visual and linguistic reasoning.

Furthermore, the advent of VLMs has opened new avenues for enhancing autonomous driving systems. For example, [13] introduced NuScenes-QA, a benchmark for VQA in autonomous driving scenarios, which addresses the complexities of multi-modal data and real-time acquisition. Similarly, [19] proposed DriveLM, a VLM-based approach that integrates web-scale data to boost generalization and interactivity with human users. These advancements highlight the potential of VLMs to address the nuanced challenges of autonomous driving, such as understanding dynamic environments and making informed decisions in real time.

QA systems have played a crucial role in improving human-vehicle interaction and facilitating autonomous decision-making. In autonomous driving, these systems help vehicles process natural language queries and provide context-aware answers based on visual inputs and pre-existing knowledge. For example, [27] developed a visual QA system that combines convolutional neural networks with language models to allow autonomous vehicles to answer questions about nearby objects and road conditions. This system allows passengers to ask real-time questions, and receive accurate, context-specific responses.

Further advances in contextual question answering [4] have significantly enhanced the vehicle’s ability to interpret complex driving scenarios. By utilizing multi-modal input, these systems provide more accurate answers to questions regarding traffic flow, pedestrian movement, or vehicle proximity. In addition, dialogue-based QA systems have gained traction in recent years, enabling more dynamic interactions between drivers and vehicles. For instance, [8] introduced a conversational QA framework, where vehicles answer questions and can engage in multi-turn dialogues, adjusting their responses based on evolving traffic conditions and user preferences. This enables smoother communication between passengers and vehicles, improving the overall driving experience and safety.

More recent work by [23] explores hybrid models that combine rule-based reasoning with large-scale language models, allowing vehicles to simulate human-like reasoning during real-time decision-making in complex environments more accurately. Their work focuses on providing accurate and safe driving suggestions during ambiguous driving situations, such as when encountering unforeseen obstacles or pedestrians. Additionally, the integration of LLMs into QA systems has shown significant promise. For example, [3] proposed a unique object-level multi-modal LLM architecture that merges vectorized numeric modalities with a pre-trained LLM to improve context understanding in driving situations. This approach not only enhances the interpretability of driving actions but also demonstrates the potential of LLM-based driving action generation compared with traditional behavioural cloning methods.

As illustrated in Fig. 2, the architecture of LaVida Drive comprises three core components: the Multi-modal Encoder Cluster, the Query-aware Token Selection Module, and the Spatial-temporal Token Enhancement Module. The model processes inputs from three modalities: image data from the autonomous vehicle’s multiview cameras, video data, and natural language instructions provided by the user. Different from previous approaches, we employ multiple encoders to handle various input modalities, forming a Multi-modal Encoder Cluster to better address the unique requirements of each data source. All encoders are frozen. Specifically, each encoder processes data in a predefined format:

Text Encoder: Our text encoder employs the CLIP text encoder, leveraging its powerful feature extraction capabilities obtained through large-scale text-image contrastive learning. For an input text sequence $\mathbf{X}_{\text{text}}$ containing $L_{\text{text}}$ tokens, the encoder processes each token embedding and maps the entire sequence to a semantic space. The output of the text encoder is a matrix of shape $L_{\text{text}}\times d$, represented as:

Image Encoder: The image encoder also employs the CLIP visual encoder, with a base resolution of $224\times 224$ pixels. This encoder efficiently maps visual data into a rich semantic space and is divided into a main branch and a support branch, each optimized for different aspects of image representation.

Image Encoder Main Branch: For the main branch, an input image $\mathbf{X}_{\text{img}}$ of size $H\times W\times C$ is first divided into $N=\frac{H\times W}{P^{2}}$ patches of size $P\times P\times C$. Each patch is flattened into a vector of length $P^{2}\times C$, forming a patch sequence of dimension $N\times(P^{2}\times C)$. The main branch generates embeddings of shape $L_{\text{main}}\times d$ from the penultimate layer of the CLIP visual encoder, represented as:

Image Encoder Support Branch: To supplement the context loss due to patch segmentation in the main branch, the support branch directly processes the downsampled entire image $\mathbf{X}_{\text{img}}^{\prime}$, of size $224\times 224\times C$. The support branch also generates embeddings of shape $L_{\text{support}}\times d$ from the penultimate layer of the CLIP visual encoder, represented as:

Video Encoder: The video encoder is based on the TimeSformer model, which performs temporal modelling on frame sequences. Given an input sequence $\mathbf{X}_{\text{temporal}}$ of $T$ frames, where each frame has a spatial dimension of $H\times W\times C$, the encoder captures inter-frame dependencies to generate temporal representations. The output is an embedding sequence of size $T\times d$, represented as:

Next, we employ the Query-aware Token Selection Module, which processes tokens output by the image encoder and text encoder to generate a token-level similarity matrix $S\in\mathbb{R}^{m\times n}$, where $m$ denotes the number of image tokens and $n$ denotes the number of text tokens. By leveraging semantic similarity in space, this module identifies the most relevant visual tokens to the user’s query, thereby reducing the number of visual tokens while retaining high-quality tokens. Finally, the Spatial-temporal Token Enhancement Module utilizes the video encoder’s output $V_{3D}\in\mathbb{R}^{S_{3D}\times 768}$ and the multi-frame auxiliary information from the image encoder $V_{2D_{multi-frame}}\in\mathbb{R}^{S_{2D}\times 768}$ to recover and enhance tokens through a cross-attention mechanism. This module’s purpose is to restore context lost in token selection and aggregate temporal information without increasing the number of additional tokens, as described further in Section 3.3.

The Query-aware Token Selection module filters out the most relevant and important tokens based on image and text tokens and further compresses these tokens for efficient representation. In the section of the Multi-modal Encoder Cluster, we denote the main branch output from the image encoder as $\mathbf{I}$ and the output from the text encoder as $\mathbf{T}$.Given that the CLIP[15] model was pre-trained on a large image-text dataset, we assume that image and text tokens are mapped into the same semantic space.To compute token-wise similarity, we omit the 0-th dimension from the embeddings, as it represents the overall semantic features.

To align the image embeddings with the text embeddings in the same semantic space, we apply a multi-layer perceptron (MLP) to the image tokens. This transformation produces the aligned image representation, denoted as $\mathbf{I}^{\prime}$. The aligned image embeddings $\mathbf{I}^{\prime}$ can then be used for similarity computation with the text embeddings $\mathbf{T}$. Therefore, we calculate the cosine similarity to obtain a token-wise similarity matrix $\mathbf{S}$, measuring the semantic similarity between each image and text token.

Next, based on the similarity between each image token and text token, we calculate the normalized similarity matrix as follows:

This is then used to select the most relevant $\mathbf{k}$ image tokens, where $\mathbf{k}$ represents the sampling threshold. A higher threshold selects more visual tokens, allowing the quantity of visual tokens to be controlled by adjusting the downsampling ratio $\mathbf{k}$. The experimental section compares performance under different thresholds, highlighting their impact as a proportion of the total token count. The steps are outlined in the following algorithm:

To further compress the tokens, we employ an MLP for information aggregation, generating a query-aware compact token representation $\mathbf{{T}_{\text{select}}}$. In the experimental section, we demonstrate the model’s performance under various combinations of selection and MLP compression factors at a fixed overall compression ratio. The results indicate that balancing the selection factor with the MLP compression rate can achieve higher model performance while preserving a minimal number of tokens.

The Spatial-temporal Token Enhancement module is designed to address issues of contextual disruption and high computational overhead when dealing with multi-frame data. This module includes a general-purpose Token-wise Attention Module, which is then specifically applied in two configurations: spatial and temporal enhancements.

First, we introduce the Token-wise Attention Module, which forms the foundation for both spatial and temporal enhancements. As shown in Fig. 3, this module enhances token context by enabling interactions between the tokens selected by the Query-aware Token Selection module and the context tokens from the image or video encoder. The attention mechanism is defined as follows:

where $Q$ (Query) represents token representations from the Query-aware Token Selection module, while $K$ (Key) and $V$ (Value) are derived from the encoder outputs. This mechanism allows each query token to absorb relevant information from context tokens.

With the Token-wise Attention Module established, we now apply it in two configurations:

• •

  Spatial Token Restoration: Here, we use the output $E_{\text{spatial}}$ of the image encoder as keys and values. By interacting with each token representation $I^{\prime}$, this configuration introduces spatial context, resulting in the spatially enhanced representation $I_{\text{enhanced spatial}}$:

  	$\displaystyle I_{\text{enhanced spatial}}$	$\displaystyle=\text{Att}_{\text{token-wise}}(Q=I^{\prime},$		(8)
  		$\displaystyle\quad K=E_{\text{spatial}},V=E_{\text{spatial}}).$

• •

  Temporal Token Enhancement: When handling multi-frame data, we use the output $E_{\text{temporal}}$ of the video encoder as another set of keys and values. By interacting in parallel with $I^{\prime}$, this configuration incorporates temporal context and produces the temporally enhanced representation $I_{\text{enhanced temporal}}$:

  	$\displaystyle I_{\text{enhanced temporal}}$	$\displaystyle=\text{Att}_{\text{token-wise}}(Q=I^{\prime},$		(9)
  		$\displaystyle\quad K=E_{\text{temporal}},V=E_{\text{temporal}}).$

Finally, we combine the spatial and temporal enhanced representations through an MLP layer to obtain the final token representation:

In this way, the Spatial-temporal Token Enhancement module fully leverages both spatial and temporal information without increasing computational costs. The final output tokens are then concatenated with the query embedding for input into the Large Language Model.

To verify the effectiveness of each module in our model, we designed a series of ablation studies, focusing on the selection and compression ratios, model components, and input types.

These results indicate that excessive redundancy and irrelevant information hinder VLM training and inference. However, overly reducing token selection leads to the loss of valuable information. Therefore, balancing the selection factor and MLP compression ratio is crucial for achieving higher model performance with fewer tokens.