Jailbreaking Attack against Multimodal Large Language Model

When no input image is given, we can freely generate the imgJP without the constraint of similarity to a given image. Specifically, we formulate MLLM-jailbreak as directly finding an imgJP, denoted as $x_{jp}$, so that it allows the generation of objectionable content.

To achieve this, we propose a maximum likelihood-based approach by modifying adversarial attack methods. The main difference between adversarial attacks and jailbreaks lies in their focus on a classification task and a generation task, respectively. Classification tasks typically involve generating a predictable output from a set of pre-defined classes, while generation tasks allow outputting anything as long as it is relevant to user queries. Thus, the objectives of adversarial attacks, such as misclassification, are no longer suitable for jailbreaking attacks.

When faced with a harmful query, a LLM equipped with alignment guardrails will decline it, responding with answers like “Sorry, I cannot fulfill your request”. Thus, a common strategy in LLM-jailbreaks is to encourage the LLM to respond with a positive affirmation, for instance, “Sure, here is a (content of query)”. We incorporate this strategy in our MLLM-jailbreaking attack, utilizing it to modify the objective function of traditional adversarial attacks.

Specifically, for each harmful request $q_{i}$, we provide a corresponding target answer $a_{i}$, creating a dataset of harmful behaviors $B=\{(q_{i},a_{i}),i=0,...,N\}$. And then, we formulate MLLM-jailbreak as finding the $x_{jp}$ such that it encourages the MLLMs to generate the target answer $a_{i}$ when users input the harmful query $q_{i}$, as follows,

where $p(a_{i}|q_{i},x_{jp})$ is the likelihood for a MLLM generate $a_{i}$ when provided with image $x_{jp}$ and text question $q_{i}$. Note that the optimal imgJP $x_{jp}^{*}$ is optimized over $M$ query-answer pairs $\{(q_{i},a_{i}),i=0,...,M\}$. This optimization problem can be solved using any adversarial attack methods, such as Projected Gradient Decent (PGD) (Madry et al., 2017).

As shown in Appendix, the jailbreaking ASR consistently improves with the increasing likelihood values, justifying that encouraging the MLLM to respond with a positive affirmation is an effective strategy for jailbreaking.

Prompt-universal is a highly desired property in real-world applications. It implies that the imgJP $x_{jp}^{*}$ generated over a few-shot $M$ prompts $B_{train}=\{(q_{i},a_{i}),i=0,...,M\}$ can be effectively utilized to jailbreak many other unseen prompts $B_{test}$. Note that $B_{train}\cup B_{test}=B$ and $B_{train}\cap B_{test}=\emptyset$. Empirical studies illustrate that our approach possesses a strong prompt-universal property, as training imgJP over $M=25$ prompts is sufficient for generalization to other $300$ unseen prompts.

The second scenario involves a provided input image $x_{in}$, and we are constrained to finding an image perturbation deltaJP $\delta$ such that the perturbed image $x_{in}+\delta$ allows the generation of objectionable content. It is worth noting that the $\delta$ is constrained by an attack budget $\epsilon$ to ensure that $x_{in}+\delta$ looks similar to the original input image $x_{in}$, as follows,

where $\epsilon$ is the attack budget, and $\widetilde{\bm{x}}$ is the perturbed image.

Similar to the previous imgJP-based attack (Eq.(1)), the deltaJP-based attack (Eq.(2)) is optimized over $M$ query-answer pairs $B_{train}$.

In addition to the prompt-universal property, another universal property, i.e., the image-universal property, is desired in this deltaJP-based attack. Since users could provide any image as an input, a practical attack should succeed regardless of the input image. To achieve this, we integrate the universal adversarial attack strategy into our deltaJP-based attack.

Specifically, we assume that the input images follow a specific distribution $\mathcal{D}$, such as belonging to a particular image category like “images of bomb”. Thus, given an image set $x_{j}\in D$ sampled from the specific distribution $\mathcal{D}$, we extend Eq.(2) as follows,

where all images $x_{j}\in D$ share a universal perturbation $\delta$.

To address this problem, we employ the universal adversarial attack strategy. At each iteration, we compute the the minimal perturbation $\delta_{j}^{t}$ that directs the current perturbed point towards jailbreaking and aggregates it to the current instance of the universal perturbation $\delta^{t}$.

Both imgJP-based and deltaJP-based attacks are designed for white-box attacks, assuming knowledge of the architecture and parameters of the target MLLM model. However, in real-world applications, this prior knowledge may be unavailable. Therefore, model-transferability becomes a crucial feature for black-box attack scenarios.

Specifically, we use a model as the surrogate model and conduct white-box attacks on it. Subsequently, the attack samples are transferred to attack the real target model. In this paper, we propose to ensemble multiple MLLMs as the surrogate model, which enhances the transferability of our attack. In general, the more models are ensembled for training, the higher transferability we achieve.

Specifically, we treat the MiniGPT-4(vicuna7B), MiniGPT-4(vicuna13B) and MiniGPT-4(LLaMA2) as three surrogate models. The imgJP jailbreak is extended as follows,

where $p_{1}(a_{i}|q_{i},x_{jp})$, $p_{2}(a_{i}|q_{i},x_{jp})$, and $p_{3}(a_{i}|q_{i},x_{jp})$ indicate the three surrogate models respectively.

The solution to this optimization problem aims to find the optimal $x_{jp}$ across all three surrogate models. The empirical study demonstrates that such an ensemble scheme enables our approach to successfully transfer attack MiniGPT-v2, LLaVA, InstructBLIP, and mPLUG-Owl2.

A Multimodal LLM inherently contains a LLM within it. Thus, there is a close connection between MLLM- and LLM-jailbreaks, allowing us to leverage our MLLM-jailbreaking approach to conduct LLM-jailbreaking attacks efficiently.

Specifically, to jailbreak a target LLM, we first construct a MLLM that encapsulates it. This involves integrating a visual component into the LLM, where the visual output combines with users’ text queries and is input to the target LLM. The placement of the image embedding can be either before or after the text embedding. Following the GCG (Zou et al., 2023) approach, we append the image embedding to the text embedding, as shown in Fig. 3.

Secondly, we perform our MLLM-jailbreak to acquire imgJP, while concurrently recording the embedding embJP.

Thirdly, the embJP is reversed into text space through De-embedding and De-tokenizer operations. In LLMs, the embedding operation converts each discrete token $t$ to its embedding vector $e$, by looking up a token-embedding $(t,e)$ dictionary. Therefore, our De-embedding operation is designed to reverse this process—convert a continuous embedding vector back into a specific token. This involves a nearest neighbor search across the dictionary. For each embedding vector $e_{l}$ in embJP $(e_{0},e_{1},...,e_{L-1})$, we identify the top-K similar embeddings $\hat{e}^{k}_{l},k=0,...,K-1$ in the dictionary. Repeating this process for all $e_{l},l=0,...,L-1$, yields a $K\times L$ embedding pool $\{\hat{e}_{l}^{k}\}_{k=0,l=0}^{K,L}$ and a corresponding $K\times L$ token pool $\{\hat{t}_{l}^{k}\}_{k=0,l=0}^{K,L}$.

Consequently, De-tokenizer operation is designed to further convert those tokens back into words, yielding a $K\times L$ word pool $\{\hat{w}_{l}^{k}\}_{k=0,l=0}^{K,L}$. Finally, we can randomly sample some sequences of words from the word pool as the txtJP. The sampled txtJP can be utilized to jailbreak the target-LLM.

Previous LLM-jailbreak methods, like GCG, typically directly sample sequences from a large $Z\times L$ word pool, where $Z>>K$ is the size of the word dictionary. In contrast, we first identify a successful embJP and utilize it to reduce the sampling space significantly. This approach results in a remarkable improvement in sampling efficiency.

In AdvBench, a collection of $500$ harmful behaviors is gathered, where each item consists of a pair of instruction sentence and a corresponding goal sentence. The adversary’s goal of LLM-jailbreaks is to find a string txtJP that will cause the model to generate any response that complies with the instruction and to do so over as many harmful behaviors as possible. The goal sentence is a desired positive affirmation intended to be present at the beginning of the answer.

To construct our multimodal dataset AdvBench-M, we need to collect images related to the provided instructions. Thus, We group all the harmful behaviors within AdvBench into $8$ distinct semantic categories, specifically, “Bombs or Explosives”, “Drugs”, “Self-harm and Suicide”, “Cybersecurity and Privacy Issues”, “Physical Assault”, “Terrorism and Societal Tensions”, “Stock Market and Economy”, and “Firearms and Ammunition”. For each category, $30$ semantic-relevant images were retrieved from the Internet using the Google search engine, coupled with the corresponding harmful behaviors. Table 1 shows some examples in our AdvBench-M dataset.

Our AdvBench-M is suitable for a detailed evaluation of jailbreaking due to the categorization of diverse harmful behaviors.

Note that AdvBench-M is particularly useful for evaluating the image-universal property in deltaJP-based jailbreaking, i.e., an adversary must find a deltaJP that, when added to any provided image within the same category, enables the model to generate responses to the instruction.

Due to the diverse responses from different MLLM models, we categorize a successful attack into three types based on the model’s response: 1) Type-I: generating harmful content in direct response to the instruction; 2) Type-II: generating responses that are partly related to the instruction and partly describing the harmful image content; 3) Type-III: repetition or rephrasing of harmful instruction, with less informative content. The examples for the three types of responses are shown in appendix.

MiniGPT-4 has three variants corresponding to three distinct LLM inside, i.e., Vicuna-7B, Vicuna-13B and LLaMA2, while MiniGPT-v2 just employs LLaMA2 as its LLM.

For white-box jailbreaks, we evaluate our approach on MiniGPT-4 and MiniGPT-v2 separately. For evaluating model-transferaibility, we generate the imgJP on MiniGPT-4 and subsequently employ it for black-box attacks on MiniGPT-v2, LLaVA, InstructBLIP, and mPLUG-Owl2.

We evaluate two white-box jailbreaking scenarios individually: one based on imgJP and the other based on deltaJP.

Following the (Zou et al., 2023), we evaluate our approach under two situations. The first is the Individual situation, where we focus on a single prompt and attempt to find an imgJP to jailbreak it alone. This simply sets $M=1$ in Eq.(1). From  Table 2, we observe that we can successfully jailbreak all three MLLMs with an ASR of $77\%$.

The second situation is the Multiple situation, where we focus on evaluating the prompt-universal property. Specifically, we randomly select $M=25$ samples from AdvBench-M to learn the optimal imgJP and calculate the train ASR. Subsequently, we randomly select $100$ samples from the remaining data to calculate the test ASR, i.e., the prompt-universal performance.

From Table 2, we observe that the train ASR is about $88\%$ or $92\%$, much better than the individual situation. This is similar to the results of the LLM-jailbreaking in (Zou et al., 2023). It illustrates that the jailbreaking performance is inferior when focusing on a single harmful instruction, compared to jointly considering multiple harmful instructions. It implies that different harmful instructions are closely related to each other, thus jointly jailbreaking them could is superior to individual jailbreaks. Moreover, the test ASR is even higher than the train ASR, indicating that our approach possesses a strong prompt-universal property.

From Table 3, we observe that our approach exhibits a certain image-universal property. The ASR shows an imbalance among different classes, with the “suicide” class being the most challenging case. Overall, this scenario presents a greater challenge compared to the imgJP-based jailbreak.

In real-world applications, the architecture and parameters of MLLMs are often unknown; thus, black-box jailbreaks are preferred in practice. The commonly used strategy is to leverage model-transferability. Specifically, we generate the imgJP on a surrogate model, the architecture and parameters of which are known. If our approach exhibits model-transferability, the generated imgJP is expected to be effective on the target model.

In this experiment, we adopt an ensemble learning strategy to build a more generalized surrogate model as described in Eq.(4). Similar to the experimental setting of imgJP-based jailbreak, we randomly select $25$ harmful behaviors for training and $100$ held-out harmful behaviors for testing. We find the imgJP by solving Eq.(4) on the training data. After that, we evaluate the generated imgJP on four target MLLM models, which include mPLUG-Owl2, LLaVA, MiniGPT-v2, and InstructBLIP.

The model-transferability can be measured with the test ASR. For a fine-grained evaluation, we use the three-type ASR as the metric, corresponding to three different kinds of jailbreaking behaviors.

In particular, we adopt two surrogate models. The first surrogate model ensembles two MLLM models: MiniGPT-4(Vicuna7B) and MiniGPT-4(Vicuna13B), which is denoted as “Vicuna” in Table 4. The second surrogate model ensembles three MLLM models: MiniGPT-4(Vicuna7B), MiniGPT-4(Vicuna7B), and MiniGPT-4(LLaMA2), which is denoted as “Vicuna & LLaMA2”. Following (Zou et al., 2023), we run these experiments twice with different random seeds to obtain two imgJP. We report the results for both averaging and ensembling the two imgJP.

From Table 4, we can successfully execute black-box attacks on all four models. It is evident that our approach exhibits strong transferability. Particularly, both mPLUG-Owl2 and MiniGPT-v2 are vulnerable to jailbreaking, with a total ASR of $59.0\%$. LLaVA can be jailbroken more easily than InstructBLIP.

Furthermore, the performance of the second surrogate model surpasses that of the first surrogate model, indicating that ensembling more models leads to improved transferability.

Distinct MLLMs exhibit diverse behaviors when facing our attack. For instance, both mPLUG-Owl2 and LLaVA never output sentences belonging to Type-II and Type-III, thus we only provide $ASR_{I}$ for them. On the other hand, MiniGPT-v2 tends to generate Type-I output, while InstructBLIP tends to generate Type-II output.

Our approach can jailbreak not only MLLM models but also LLM models. In this experiment, we compare our approach to some state-of-the-art LLM-jailbreaking methods, including GBDA (Guo et al., 2021), PEZ (Wen et al., 2023), AutoPrompt (Shin et al., 2020), and GCG (Zou et al., 2023).

Regarding our construction-base method, after the De-embedding operation, we obtain a $K\times L$ embedding pool $\{\hat{e}_{l}^{k}\}_{k=0,l=0}^{K,L}$. To perform the final LLM-jailbreak, we will sample the sentence txtJP from the pool. There are several schemes to obtain txtJP. Specifically, for each embedding $e_{l}$, we find the top-K similar embeddings $\hat{e}^{k}_{l},k=0,...,K-1$ in the dictionary, where they are sorted in descending order. The first scheme is to output the Top-1 similar embedding $\hat{e}^{0}_{l}$. The second is to randomly sample one $\hat{e}^{k}_{l}$ $(0\leq k\leq K-1)$. Both of them just output one txtJP. The third scheme is to repeat the random sampling $N$ times, producing $N$ candidate txtJPs, and try those txtJPs one by one. The third scheme is referred to as the “RandSet” in Table 5.

From Table 5, we observe that our ensemble scheme outperforms GCG on test ASR just with $N=5$. If we increase $N$ to 20, we achieve the train ASR of $92\%$ and the test ASR of $93\%$. It is worth noting that our ensemble scheme is much more efficient than GCG, since our approach is almost zero-cost (just performing inference 20 times), while GCG requires a time-consuming sampling procedure.