Towards Transferable Attacks Against Vision-LLMs
in Autonomous Driving with Typography

Having demonstrated the proficiency of Large Language Models (LLMs) in reasoning across various natural language benchmarks, researchers have extended LLMs with visual encoders to support multimodal understanding. This integration has given rise to various forms of Vision-LLMs, capable of reasoning based on the composition of visual and language inputs.

Vision-LLMs Pre-training. The interconnection between LLMs and pre-trained vision models involves the individual pre-training of unimodal encoders on their respective domains, followed by large-scale vision-language joint training [17, 18, 19, 20, 2, 1]. Through an interleaved visual language corpus (e.g., MMC4 [21] and M3W [22]), auto-regressive models learn to process images by converting them into visual tokens, combine these with textual tokens, and input them into LLMs. Visual inputs are treated as a foreign language, enhancing traditional text-only LLMs by enabling visual understanding while retaining their language capabilities. Hence, a straightforward pre-training strategy may not be designed to handle cases where input text is significantly more aligned with visual texts in an image than with the visual context of that image.

Vision-LLMs in AD Systems. Vision-LLMs have proven useful for perception, planning, reasoning, and control in autonomous driving (AD) systems [6, 7, 9, 5]. For example, existing works have quantitatively benchmarked the linguistic capabilities of Vision-LLMs in terms of their trustworthiness in explaining the decision-making processes of AD [7]. Others have explored the use of Vision-LLMs for vehicular maneuvering [8, 5], and [6] even validated an approach in controlled physical environments. Because AD systems involve safety-critical situations, comprehensive analyses of their vulnerabilities are crucial for reliable deployment and inference. However, proposed adoptions of Vision-LLMs into AD have been straightforward, which means existing issues (e.g., vulnerabilities against typographic attacks) in such models are likely present without proper countermeasures.

Adversarial attacks are most harmful when they can be developed in a closed setting with public frameworks yet can still be realized to attack unseen, closed-source models. The literature on these transferable attacks popularly spans across gradient-based strategies. Against Vision-LLMs, our research focuses on exploring the transferability of typographic attacks.

Gradient-based Attacks. Since Szegedy et al. introduced the concept of adversarial examples, gradient-based methods have become the cornerstone of adversarial attacks [23, 24]. Goodfellow et al. proposed the Fast Gradient Sign Method (FGSM [25]) to generate adversarial examples using a single gradient step, perturbing the model’s input before backpropagation. Kurakin et al. later improved FGSM with an iterative optimization method, resulting in Iterative-FGSM (I-FGSM) [26]. Projected Gradient Descent (PGD [27]) further enhances I-FGSM by incorporating random noise initialization, leading to better attack performance. Gradient-based transfer attack methods typically use a known surrogate model, leveraging its parameters and gradients to generate adversarial examples, which are then used to attack a black-box model. These methods often rely on multi-step iterative optimization techniques like PGD and employ various data augmentation strategies to enhance transferability [28, 29, 30, 31, 32]. However, gradient-based methods face limitations in adversarial transferability due to the disparity between the surrogate and target models, and the tendency of adversarial examples to overfit the surrogate model [33, 34].

Typographic Attacks. The development of large-scale pretrained vision-language with CLIP [11, 12] introduced a form of typographic attacks that can impair its zero-shot performances. A concurrent work [13] has also shown that such typographic attacks can extend to language reasoning tasks of Vision-LLMs like multi-choice question-answering and image-level open-vocabulary recognition. Similarly, another work [14] has developed a benchmark by utilizing a Vision-LLM to recommend an attack against itself given an image, a question, and its answer on classification datasets. Several defense mechanisms [15, 16] have been suggested by prompting the Vision-LLM to perform step-by-step reasoning. Our research differs from existing works in studying autonomous typographic attacks across question-answering scenarios of recognition, action reasoning, and scene understanding, particularly against Vision-LLMs in AD systems. Our work also discusses how they can affect reasoning capabilities at the image level, region-level understanding, and even against multiple reasoning tasks. Furthermore, we also discuss how these attacks can be realized in the physical world, particularly against AD systems.

As a simplified formulation of auto-regressive Vision-LLMs, suppose we have a visual input $\mathbf{v}$, a sequence of tokens generated up to timestep $t-1$, denoted as $x_{1},x_{2},\dots,x_{t-1}$, and $f(\cdot)$ as the Vision-LLM model function, whose goal is to predict the next token $x_{t}$. We can denote its output vector of logits $\mathbf{y}_{t}$ at each timestep $t$ based on the previous tokens and the visual context:

	$\displaystyle\mathbf{y}_{t}$	$\displaystyle=f(x_{1},\dots,x_{t-1},\mathbf{v})$		(1)
		$\displaystyle=f(x_{1},\dots,x_{t-1},v_{1},\dots,v_{m}),$

where $v_{1},\dots,v_{m}$ denotes $m$ visual tokens encoded by a visual encoder on $\mathbf{v}$. The logits $\mathbf{y}_{t}$ are converted into a probability distribution using the softmax function. Specifically, $y_{t,j}\in\mathbf{y}_{t}$ is the logit for token $j$ in the vocabulary $C$ at timestep $t$, generally as follows:

$$P(x_{t}=j|x_{1},x_{2},\dots,x_{t-1},\mathbf{v})=\frac{\exp(y_{t,j})}{\sum_{k\in C}\exp(y_{t,k})}.$$

(2)

Then, the general language modeling loss for training the model can be based on cross-entropy loss. For a sequence of tokens $\mathbf{x}=\{x_{1},\dots,x_{n}\}$, the loss is given by:

$$\mathcal{L}_{LM}(\mathbf{x})=\sum_{t=1}^{n}\log P(x_{t}\ |\ x_{1},\dots,x_{t-1},v_{1},\dots,v_{m})=\sum_{k=1}^{n+m}\log P(x_{t}\ |\ z_{1},\dots,z_{k-1}),$$

(3)

where $z_{i}$ denotes either a textual token $x$ or visual token $v$ at position $i$. Vision-LLMs possess conversational capabilities at their core, so interleaving language data ($m=0$) and vision-language data ($m>0$) during optimization is crucial for enabling visual understanding while retaining language reasoning [1]. Regardless of $m$, the loss objective of vision-guided language modeling is essentially the same as auto-regressive language modeling [35]. Consequently, as part of the alignment process, these practices imply blurred boundaries between textual and visual feature tokens during training. They may also facilitate text-to-text alignment between raw texts and within-image texts at inference.

The integration of Vision-LLMs into end-to-end AD systems has brought promising results thus far [9], where Vision-LLMs can enhance user trust through explicit reasoning steps of the scene. On the one hand, language reasoning in AD systems can elevate their capabilities by utilizing the learned commonsense of LLMs, while being able to proficiently communicate to users. On the other hand, exposing Vision-LLMs to public traffic scenarios not only makes them more vulnerable to typographic attacks that misdirect the reasoning process but can also prove harmful if their results are connected with decision-making, judgment, and control processes.

Unlike the less transferable gradient-based attacks, typographic attacks are more transferable across Vision-LLMs by exploiting the inherent text-to-text alignment between raw texts and within-image texts to introduce misleading textual patterns in images, and influence the reasoning of a Vision-LLM, i.e., dominating over visual-text alignment. In digital form, the attack is formulated as a function $\tau(\cdot)$ that applies transformations representing typographic attacks to obtain an adversarial image $\hat{\mathbf{v}}=\tau(\mathbf{v})$. Then, Eq. 1 can be rewritten as:

	$\displaystyle\mathbf{y}_{t}$	$\displaystyle=f(x_{1},\dots,x_{t-1},\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\hat{\mathbf{v}}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0})$		(4)
		$\displaystyle=f(x_{1},\dots,x_{t-1},\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\hat{v}_{1},\dots,\hat{v}_{m}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}),$

where $\hat{v}_{1},\dots,\hat{v}_{m}$ denotes $m$ visual tokens under the influenced image $\hat{\mathbf{v}}$, and whose textual content is meant to ❶ align with $\{x_{1},\dots,x_{t-1}\}$, ❷ yet guide the reasoning process towards an incorrect answer. By exploiting the fundamental properties of many Vision-LLMs in language modeling to construct adversarial patterns, ❸ typographic attacks $\tau(\cdot)$ aim to be transferable across various pre-trained Vision-LLMs by directly influencing the visual information with texts. Our study is geared towards typographic attacks in AD scenarios to thoroughly understand the issues and raise awareness.

In this subsection, to handle the lack of both autonomy and diversity in typographic attacks, we propose to employ the support of an LLM and prompt engineering, denoted by a model function $l(\cdot)$, to generate adversarial typographic patterns automatically. Let $\mathbf{q}$, $\mathbf{a}$ respectively be the question prompt input and its answer on an image $\mathbf{v}$, the adversarial text can be naively generated as $\hat{\mathbf{a}}$,

$\displaystyle\hat{\mathbf{a}}=l(\mathbf{q},\mathbf{a}).$

(5)

In order to generate useful misdirection, the adversarial patterns must align with an existing question while guiding LLM toward an incorrect answer. We can achieve this through a concept called directive, which refers to configuring the goal for an LLM, e.g., ChatGPT, to impose specific constraints while encouraging diverse behaviors. In our context, we direct the LLM to generate $\mathbf{\hat{a}}$ as an opposite of the given answer $\mathbf{a}$, under the constraint of the given question $\mathbf{q}$. Therefore, we can initialize directives to the LLM using the following prompts in Fig. 2,

When generating attacks, we would impose additional constraints depending on the question type. In our context, we focus on tasks of ❶ scene reasoning (e.g., counting), ❷ scene object reasoning (e.g., recognition), and ❸ action reasoning (e.g., action recommendation), as follows in Fig. 3,

The directives encourage the LLM to generate attacks that influence a Vision-LLM’s reasoning step through text-to-text alignment and automatically produce typographic patterns as benchmark attacks. Clearly, the aforementioned typographic attack only works for single-task scenarios, i.e., a single pair of question and answer. To investigate multi-task vulnerabilities with respect to multiple pairs, we can also generalize the formulation to $K$ pairs of questions and answers, denoted as $\mathbf{q}_{i},\mathbf{a}_{i}$, to obtain the adversarial text $\hat{\mathbf{a}}_{i}$ for $i\in\left[1,K\right]$.

Inspired by the success of instruction-prompting methodologies [37, 38], the greedy reasoning in LLMs [39], and to further exploit the ambiguity between textual and visual tokens in Vision-LLMs, we propose to augment the typographic attacks prompts within images by explicitly providing instruction keywords that emphasize text-to-text alignment over that of visual-language tokens. Our approach realizes the concept in the form of instructional directives: ❶ command directives for emphasizing a false answer and ❷ conjunction directives to additionally include attack clauses. In particular, we have developed,

• •

  Command Directive. By embedding commands with the attacks, we aim to prompt the Vision-LLMs into greedily producing erroneous answers. Our work investigates the "ANSWER:" directive as a prefix before the first attack prompt.

• •

  Conjunction Directive. Conjunctions, connectors (or the lack thereof) act to link together separate attack concepts that make the overall text appear more coherent, thereby increasing the likelihood of multi-task success. In our work, we investigate these directives as "AND," "OR," "WITH," or simply empty spaces as prefixes between attack prompts.

While other forms of directives can also be useful for enhancing the attack success rate, we focus on investigating basic directives related to typographic attacks in this work.

Digitally, typographic attacks are about embedding texts within images to fool the capabilities of Vision-LLMs, which might involve simply putting texts into the images. Physically, typographic attacks can incorporate real elements (e.g., stickers, paints, and drawings) into environments/entities observable by AI systems, with AD systems being prime examples. This would include the placement of texts with unusual fonts or colors on streets, objects, vehicles, or clothing to mislead AD systems in reasoning, planning, and control. We investigate Vision-LLMs when incorporated into AD systems, as they are likely under the most risk against typographic attacks. We categorize the placement locations as being identified with backgrounds and foregrounds in traffic scenes.

• •

  Backgrounds, which refer to elements in the environment that are static and pervasive in a traffic scene (e.g., streets, buildings, and bus stops). The background components present predefined locations for introducing deceptive typographic elements of various sizes.

• •

  Foregrounds, which refer to dynamic elements and directly interact with the perception of AD systems (e.g., vehicles, cyclists, and pedestrians). The foreground components present dynamic and variable locations for typographic attacks of various sizes.

In our work, foreground placements are supported by an open-vocabulary object detector [40] to flexibly extract box locations of specific targets. Let $\mathbf{A}=\hat{\mathbf{a}}_{1}||\dots||\hat{\mathbf{a}}_{K}$ be the typographic concatenation of attacks, and $\mathbf{A}^{\prime}$ be its augmented version, either on background or foreground, the function $\tau(\cdot)$ would perform inpainting $\mathbf{A}$ or $\mathbf{A}^{\prime}$ into image $\mathbf{v}$’s cropped box coordinates $x_{min},y_{min},x_{max},y_{max}$.

Depending on the attacked task, we observe that different text placements and observed sizes would render some attacks more effective while some others are negligible. Our research illuminates that background-placement attacks are quite effective against scene reasoning and action reasoning but not as effective against scene object reasoning unless foreground placements are also included.

We perform experiments with Vision-LLMs on VQA datasets for AD, such as LingoQA [7] and the dataset of CVPRW’2024 Challenge ¹¹1https://cvpr24-advml.github.io by CARLA simulator. We have used LLaVa [2] to output the attack prompts for LingoQA and the CVPRW’2024 dataset, and manually for some cases of the latter. Regarding LingoQA, we tested 1000 QAs in real traffic scenarios in tasks, such as scene reasoning and action reasoning. Regarding the CVPRW’2024 Challenge dataset, we tested more than 300 QAs on 100 images, each with at least three questions related to scene reasoning (e.g., target counting) and scene object reasoning of 5 classes (cars, persons, motorcycles, traffic lights and road signals). Our evaluation metrics are based on exact matches, Lingo-Judge Accuracy [7], and BLEURT [41], BERTScore [42] against non-attacked answers, with SSIM (Structural Similarity Index) to quantify the similarity between original and attacked images. In terms of models, we qualitatively and/or quantitatively tested with LLaVa [2], VILA [1], Qwen-VL [17], and Imp [18]. The models were run on an NVIDIA A40 GPU with approximately 45GiB of memory.