Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

In this section, we introduce various notation and terminology used in our paper, as well as the primary aspects of our experiment pipeline.

Sequences and models. Let $V=\{1,2,\ldots,m\}$ represent a vocabulary of $m$ token, and let $\Sigma$ denote the set of printable ASCII characters. Let $\Sigma^{+}$ denote the set of positive-length sequences. An autoregressive LLM $f$ operates as follows: given an initial character sequence from $\Sigma^{+}$, $f$ outputs a probability distribution over $V$ to predict the next token (for simplicity, we view the tokenizer associated with $f$ as a part of $f$).
Generation. Model $f$ may be used as part of a broader pipeline where the input and output character sequences can be restricted to spaces $\mathcal{X}\subseteq\Sigma^{+}$ and $\mathcal{Y}\subseteq\Sigma^{+}$, respectively (e.g., with prompt templates, limits on sequence length, etc.). For simplicity, we define a generation algorithm $g$ to be this entire pipeline, which given $\textbf{x}\in\mathcal{X}$, uses $f$ to generate $\textbf{y}\in\mathcal{Y}$ following some decoding strategy. For generality, we assume $g$ to be stochastic, with deterministic algorithms being a special case.
Augmentations. An augmentation $a\colon\mathcal{X}\rightarrow\mathcal{X}$ is a function that modifies x before being passed to $g$. Note that “no augmentation” can be considered a special case where the “augmentation” is the identity function $a(\textbf{x})=\textbf{x}$. Let an augmentation set $\mathcal{A}$ be a set of augmentations that may be related in nature (e.g., appending a suffix of a specific length); we refer to the nature of the relation as the augmentation “type”. Augmentations may be randomly sampled, so we also associate a sampling distribution $P_{\text{aug}}(\,\cdot\,;\mathcal{A})$ with each $\mathcal{A}$. We let $\mathcal{A}_{I}$ denote the “no augmentation” singleton containing the identity function that is drawn with probability 1 from $P_{\text{aug}}(\,\cdot\,;\mathcal{A}_{I})$.
Safety dataset. For safety evaluation, we set $P_{\text{test}}$ to be an underlying distribution of inputs from $\mathcal{X}$ that contain harmful user requests. We assume that a finite set $\mathcal{D}$ of i.i.d. samples from $P_{\text{test}}$ is available. As what is deemed “harmful” is subjective and may change over time, we make no further assumptions about $P_{\text{test}}$ and simply assume that $\mathcal{D}$ is representative of the desired $P_{\text{test}}$.
Safety judge. A safety judge $c\colon\mathcal{X},\mathcal{Y}\rightarrow\{0,1\}$ outputs 1 if y is deemed compliant with a user request x and 0 otherwise. Different judges may involve different criteria for compliance. For simplicity, we assume part of $c$ includes any necessary preparation of x and y (e.g., removing the prompt template from x, applying a new prompt template, etc.). We always evaluate the compliance of y with respect to the original prompt, even if y was generated from an augmentation.

Our experiment pipeline has three main components that can be varied: the augmentation type, the model, and the generation algorithm. We will investigate how each of these components impact safety while isolating the other components, and therefore naturally split our research question into the following sub-questions:

RQ1. For a given model and generation algorithm, how do different augmentation types impact safety? There are many ways to randomly augment a prompt such that its semantic meaning is preserved (or at least highly inferable). However, there may be significant differences in how effectively they enable malicious users to bypass safety alignment. Hence, we examine how a variety of random augmentations can improve attack success over the baseline of not using any augmentations.
RQ2. For a given augmentation type and generation algorithm, how do different model aspects impact safety; specifically: model size, quantization and fine-tuning-based defense? Model developers commonly release models of multiple sizes within a model family, permitting accessibility to a broader range of hardware. Alternatively, extensive efforts have been made recently to quantize LLMs for similar reasons. Orthogonal to the goal of accessibility is how to make models safer against jailbreaks, for which some recent works have proposed fine-tuning-based defense methods. Hence, it is of practical interest to examine how the safety under random augmentations interacts with each of these aspects.
RQ3. For a given model, how much do random augmentations impact safety when different decoding strategies are used? By default, all our experiments are conducted using greedy decoding, so the no augmentation baseline in RQ1 only produces a single output per prompt. A critical question therefore is whether random augmentations provide any additional influence on success rates when $k$ random outputs are also sampled in the no augmentation case. Hence, we examine decoding strategies beyond greedy decoding.

In realistic settings, a malicious user who seeks to elicit specific harmful content from an LLM may make multiple attempts before moving on. We therefore assume that for each harmful prompt $\textbf{x}_{i}\in\mathcal{X}$, a malicious user makes $k$ attempts where for each attempt a separate augmentation is first applied to the prompt, as illustrated in Figure 1. To evaluate success, we check whether the proportion of augmentations that produce outputs where safety judge $c$ evaluates to 1 is strictly greater than some threshold $\gamma\in[0,1)$. We refer to such an occurrence as a $(k,\gamma)$-success and define the following function for it:

$$s_{k,\gamma}(\textbf{x},\textbf{y}_{1},\ldots,\textbf{y}_{k})\vcentcolon=\begin{cases}1&\text{if }\frac{1}{k}\sum\limits_{j=1}^{k}c(\textbf{x},\textbf{y}_{j})>\gamma\\ 0&\text{otherwise}\end{cases}$$

(1)

where for $1\leq j\leq k$, $\textbf{y}_{j}\in\mathcal{Y}$ is the observed output given $a_{j}(\textbf{x})$, where $a_{j}\in\mathcal{A}$ is the $j$th observed augmentation. Note that the definition of $(k,\gamma)$-success has also been used as the majority vote definition for SmoothLLM  (Robey et al., 2023), although SmoothLLM uses Equation 1 solely as part of a defense mechanism whereas we use it for attack evaluation (see Appendix A.4).

Given we use a learned classifier for $c$, simply checking if any (i.e., $\gamma=0$) augmentation succeeds can have a high false positive rate (a false positive occurs when $s_{k,\gamma}(\textbf{x},\textbf{y}_{1},\ldots,\textbf{y}_{k})$ evaluates to 1 when in fact none of the $k$ outputs are harmful). A non-zero $\gamma$ can therefore be used to help reduce the false positive rate. However, applying too high of a threshold may result in a high false negative rate (a false negative occurs when $s_{k,\gamma}(\textbf{x},\textbf{y}_{1},\ldots,\textbf{y}_{k})$ evaluates to 0 when in fact at least one of the $k$ outputs are harmful). Thus, $\gamma$ should be carefully chosen so as to balance the false positive and false negative rates. See Appendix C.1 for more details.

Let $\textbf{X}_{\text{harm}}\sim P_{\text{test}}$ be a random harmful input prompt and $A_{1},\ldots A_{k},\stackrel{{\scriptstyle i.i.d.}}{{\sim}}P_{\text{aug}}(\,\cdot\,;\mathcal{A})$ be $k$ random augmentations from $\mathcal{A}$ to apply to $\textbf{X}_{\text{harm}}$ before being provided as $k$ inputs to $g$. Let $\textbf{Y}\,|\,\textbf{X}=\textbf{x}\sim P_{\textbf{Y}\,|\,\textbf{X}}(\,\cdot\,|\,\textbf{X}=\textbf{x};f,g)$ be a random output sequence from $\mathcal{Y}$ produced by $g$ using $f$, given an input $\textbf{x}\in\mathcal{X}$. Similarly, for $1\leq j\leq k$, let $\textbf{Y}_{j}\,|\,\textbf{X}_{\text{harm}}=\textbf{x},A_{j}=a_{j}\sim P_{\textbf{Y}\,|\,\textbf{X}}(\,\cdot\,|\,\textbf{X}=a_{j}(\textbf{x}_{\text{harm}});f,g)$ be the $j$th random output sequence from $\mathcal{Y}$ produced by $g$ using $f$, given $\textbf{X}_{\text{harm}}=\textbf{x}$ and $A_{j}=a_{j}$. Given our definition of $(k,\gamma)$-success, we then define the true $(k,\gamma)$-success rate as

$$r_{k,\gamma}(\mathcal{A},f,g)\vcentcolon=\mathbb{E}[s_{k,\gamma}(\textbf{X}_{\text{harm}},\textbf{Y}_{1},\ldots,\textbf{Y}_{k})]$$

(2)

where the expectation is taken over $\textbf{X}_{\text{harm}}$, $A_{1},\ldots A_{k}$ and $\textbf{Y}_{1},\ldots,\textbf{Y}_{k}$. Note that when an augmentation set is a singleton (e.g., $\mathcal{A}_{I}$) and a deterministic generation algorithm $g$ is used, the $(k,\gamma)$-success rate is the same as the $(1,0)$-success rate for any values of $k$ and $\gamma$. To approximate the true $(k,\gamma)$-success rate, we define the empirical $(k,\gamma)$-success rate as

$$\hat{r}_{k,\gamma}(\mathcal{A},f,g)\vcentcolon=\frac{1}{|\mathcal{D}|}\sum\limits_{\textbf{x}_{i}\in\mathcal{D}}s_{k,\gamma}(\textbf{x}_{i},\textbf{y}_{i1},\ldots,\textbf{y}_{ik})$$

(3)

where for $1\leq j\leq k$, $\textbf{y}_{ij}\in\mathcal{Y}$ is the observed output given $a_{ij}(\textbf{x}_{i})$, where $a_{ij}\in\mathcal{A}$ is the $j$th observed augmentation for $\textbf{x}_{i}$. Since we can only obtain an empirical $(k,\gamma)$-success rate in practice, we refer to it simply as the $(k,\gamma)$-success rate. We sometimes use the terms “success rate” and “$(k,\gamma)$-success rate” interchangeably if $k$ and $\gamma$ are clear from the surrounding context.

We consider the following models across 8 different model families: Llama 2 (Llama 2 7B Chat, Llama 2 13B Chat) (Touvron et al., 2023), Llama 3 (Llama 3 8B Instruct) (Dubey et al., 2024), Llama 3.1 (Llama 3.1 8B Instruct), Mistral (Mistral 7B Instruct v0.2), Phi 3 (Phi 3 Mini 4K Instruct, Phi 3 Small 8K Instruct, Phi 3 Medium 4K Instruct), Qwen 2 (Qwen 2 0.5B, Qwen 2 1.5B, Qwen 2 7B), Vicuna (Vicuna 7B v1.5, Vicuna 13B v1.5) and Zephyr (Zephyr 7B Beta). In Appendix D.1, we also evaluate GPT-4o OpenAI (2024). Among these, only the Llama, Phi and Qwen families have undergone explicit safety alignment. The remaining families are included to see if any interesting patterns can be observed for unaligned models. For instance, Mistral can sometimes exhibit refusal behavior for harmful prompts, so it would still be interesting to see how this is be affected by random augmentations. By default, we leave the system prompt empty for all models; see Appendix D.2 for an experiment with safety-encouraging system prompts.

For the RQ2 experiments, for each augmentation set $\mathcal{A}$ we examine the success rate gain $\hat{r}_{25,\gamma^{*}_{\mathcal{A}}}(\mathcal{A},f^{\prime},g)-\hat{r}_{25,\gamma^{*}_{\mathcal{A}}}(\mathcal{A},f,g)$ of a model $f^{\prime}$ over a baseline model $f$. In the following, we provide further details for each experiment:

Model size. For comparing model sizes, we let the smallest model in each model family be the baseline model $f$ and let the larger models be $f^{\prime}$. Specifically, for Llama 2 the baseline model is Llama 2 7B Chat, for Phi 3 the baseline model is Phi 3 Mini 4k Instruct, for Qwen 2 the baseline model is Qwen 2 0.5B, and for Vicuna the baseline model is Vicuna 7B v1.5.

Quantization. For comparing quantization levels, we consider the original model as the baseline $f$ and the quantized models as $f^{\prime}$. We only focus on 7B/8B parameter models to reduce the amount of experiments as well as to roughly control for model size while assessing quantization over a broad range of model families. We examine two settings for quantization: 1. Symmetric 8-bit per-channel integer quantization of the weights with symmetric 8-bit per-token integer quantization for activations (“W8A8”), and 2. Symmetric 4-bit per-channel weight-only integer quantization (“W4A16”) (Nagel et al., 2021). The former is chosen to examine the effects of simultaneous weight and activation quantization (Xiao et al., 2023), and the latter is chosen to explore closer to the limits of weight quantization (Frantar et al., 2022).

Fine-Tuning-Based Defense. For comparing fine-tuning-based defenses, we consider circuit breaking (RR) (Zou et al., 2024) on Mistral 7B Instruct v0.2 and Llama 3 8B Instruct as well as adversarial training (R2D2) (Mazeika et al., 2024) on Zephyr 7B Beta as $f^{\prime}$ and the original model before fine-tuning as the baseline $f$. Note that R2D2 was trained against GCG with a fixed adversarial suffix length of 20 tokens, and that 25 characters corresponds to around 20 tokens on average for the Zephyr tokenizer. Hence, to give a fairer evaluation of R2D2, we additionally examine fixed-length suffix insertion at $L=25$, as well as fixed lengths above and below 25 to assess length generalization; specifically, we examine $L\in\{5,10,15,20,25,30,35,40,45,50\}$. As a sanity check, we also evaluate how often benign prompts are wrongly refused when augmented with a fixed-length suffix; for this, we use the first-turn prompts from MT-Bench (Zheng et al., 2023), which comprise a sample of 80 prompts from MMLU (a benchmark for evaluating core knowledge) (Hendrycks et al., 2020). Note that using the SORRY-Bench judge as a proxy for measuring benign prompt compliance is viable since the judge’s task prompt only asks to evaluate compliance rather than harmfulness.

By default, all our experiments are conducted using greedy decoding to isolate the randomness effects of using multiple random augmentations. However, for the RQ3 experiments, for each augmentation set $\mathcal{A}$ we examine the success rate gain $\hat{r}_{25,\gamma^{*}_{\mathcal{A}}}(\mathcal{A},f,g)-\hat{r}_{25,\gamma^{*}_{\mathcal{A}}}(\mathcal{A}_{I},f,g)$ for sampling-based generation algorithms $g$. Specifically, we consider temperature sampling with various temperatures $\tau$ for $g$. We consider two values for $\tau$: 0.7 (since this is a value in the range of common temperature parameters between 0.6 and 0.9), and 1.0 (to explore the largest possible temperature value). We set the maximum generated tokens to be 1024.

In Figure 2, we see the experiment results for RQ1 (denoted by “$\tau=0.0$”). Immediately, we can see that for nearly all models, character-level augmentations achieve a significant positive average success rate gain of at least 10%. As most of these models are safety aligned, this suggests that under greedy decoding, random augmentations are a cheap yet effective approach to jailbreaking state-of-the-art LLMs. We also observe a consistent pattern across models where character-level augmentations outperform string insertion augmentations, in some cases by a factor of ${\sim}2\times$ or more. We hypothesize that character-level augmentations may directly impact the tokenization of the original prompt more than string insertion augmentations, increasing the chances of finding a tokenized sequence that maintains the original semantic meaning yet is considered out-of-distribution with respect to the alignment dataset. Finally, we remark that for unaligned models that already exhibit high success rates when no augmentations are used (Mistral and Zephyr, see Table 7), random augmentations further improve the success rate. Interestingly, for Mistral and Zephyr, the difference between string insertion augmentations and character-level augmentations is much less pronounced than the aligned models. One possibility is that safety alignment biases a model’s robustness towards certain kinds of augmentations, although we note that Vicuna 7B is a counterexample. We leave further investigation up to future work.

Figure 12 and Table 7 in Appendix D report the experiment results for RQ3. First, we remark that increasing temperature without any augmentations already increases the success rate; this is in line with the findings of Huang et al. (2023) that showed altering temperature alone can be a successful attack. Next, we observe that applying random augmentations on top of output sampling overall tends to hurt the success rate. However, from Table 7, we see that for Llama 2, Llama 3 and Phi 3, character deletion further improves the success rate. This shows that two sources of randomness, namely output sampling and random augmentations, can sometimes work together to provide even greater attack effectiveness.

In summary, we provide a ranking for how influential each dimension is on safety: 1. Fine-tuning-based defenses; e.g., Mistral 7B with RR experiences a 55.9% improvement in safety on average (see Table 10), 2. Model size; e.g., Qwen 2 0.5B drops 23.2% in safety from 1.5B on average (see Table 8), 3. Quantization; while W8A8 maintains safety, W4A16 tends to reduce it (e.g., with Llama 3 dropping 10.5%), and 4. Output sampling, which only rarely decreases safety (and tends to improve it). Please see Appendix B for discussion on the practical implications of random augmentation attacks.