Improving Transferability of Adversarial Examples via Bayesian Attacks

FGSM and I-FGSM require calculating gradients of the victim model. Nevertheless, in practical scenarios, the attacker may not have enough knowledge about the victim model for calculating gradients. To address this issue, many attacks rely on the transferability of adversarial examples, meaning that adversarial examples crafted for one classification model (using, for example, FGSM or I-FGSM) can often successfully attack other victim models. It is normally assumed to be able to query the victim model to annotate training samples, collect a set of samples from the same distribution as that modeled by the victim models, or collect a pre-trained substitute model that is trained to accomplish the same task as the victim models.

To enhance the transferability of adversarial examples, several groups of methods have been proposed. Advancements include improved optimizer and gradient computation techniques for updating ${\mathbf{x}}^{\text{adv}}$ [18, 19, 13, 10, 20, 11, 21], innovative strategies of training and fine-tuning substitute models [22, 23], and two groups of methods closely related to this paper: random input augmentation [24, 25, 19, 26] and substitute model augmentation (i.e., ensemble) [7, 9, 27, 28]. For instance, in input augmentation methods, Dong et al. [24], Xie et al. [25], Lin et al. [19] introduced different sorts of transformations into the iterative update of adversarial examples. Comparing with these methods, our method introduces input diversity via a principled Bayesian formulation and for the first time takes such transformation into account during substitute model fine-tuning. For substitute model augmentation (i.e., ensemble), Liu et al. [7] proposed to generate adversarial examples on an ensemble of multiple substitute models that differ in their architectures. Additionally, Xiong et al. [9] proposed stochastic variance reduced ensemble to reduce the variance of gradients of different substitute models following the spirit of stochastic variance reduced gradient [29]. Gubri et al. [28] suggested fine-tuning with a fixed and large learning rate to collect multiple models along the training trajectory for the ensemble attack. In this paper, we consider the diversity in both the substitute models and model inputs by introducing a Bayesian approximation for achieving this.

If a deep neural network (DNN) is considered as a probabilistic model, the process of training its parameters, denoted as ${\mathbf{w}}$, can be seen as maximum likelihood estimation or maximum a posteriori estimation (with regularization). In Bayesian deep learning, the approach involves simultaneously estimating the posterior distribution of the parameters given the data. The prediction for any new input instance is obtained by taking the expectation over this posterior distribution. Due to the large number of parameters typically involved in DNNs, optimizing Bayesian models becomes more challenging compared to shallow models. As a result, numerous studies have been conducted to address this issue, leading to the development of various scalable approximations. Effective methods utilize variational inference [30, 31, 32, 33, 34, 35, 36, 37], dropout inference [38, 39, 40], Laplace approximation [41, 42, 43], or stochastic gradient descent (SGD)-based approximation [44, 45, 46, 47]. Taking SWAG [45] as an example, which is an SGD-based approximation, it approximates the posterior using a Gaussian distribution with the stochastic weight averaging (SWA) solution as its first raw moment and the composition of a low rank matrix and a diagonal matrix as its second central moment. Our method is developed in a Bayesian spirit and we shall discuss SWAG thoroughly later in this paper. In addition to approximating the posterior over model parameters, our work also involves approximating the posterior over adversarial perturbations. This is implemented in order to introduce randomization into the model input during each iteration of iterative attacks.

In recent years, there has been research on studying the robustness of Bayesian DNNs. Besides exploring the probabilistic robustness and safety measures of such models [48, 49], attacks have been adapted [50, 51] to evaluate the robustness of these models in practice. While Bayesian models are often considered to be more robust [52, 53], adversarial training has also been proposed for further safeguarding them, as can be seen in the work by Liu et al. [50]. However, these studies have not specifically focused on adversarial transferability as we do in this paper.

Bayesian learning aims to discover a distribution of likely models rather than a single deterministic model. Let $\mathcal{D}=\{({\mathbf{x}}_{i},y_{i})\}_{i=1}^{N}$ denote a training set. Bayesian inference incorporates a prior belief about the parameters ${\mathbf{w}}$ through the prior distribution $p({\mathbf{w}})$ and updates this belief after observing the data $\mathcal{D}$ using Bayes’ theorem, resulting in the posterior distribution: $p(\mathbf{w}|\mathcal{D})\propto p(\mathcal{D}|{\mathbf{w}})p(\mathbf{w})$. For a new input ${\mathbf{x}}$, the predictive distribution of its class label $y$ is given by the Bayesian model averaging , i.e.,

where $p(y|{\mathbf{x}},{\mathbf{w}})$ is the conditional probability, obtained from the DNN output followed by a softmax function.

To perform attack on such a Bayesian model, a straightforward idea is to minimize the probability of the true class for a given input, as in [1]:

An iterative optimizer (e.g., I-FGSM) is often applied, and at the $t$-the iteration, it seeks $\widetilde{\mathbf{\Delta x}}$ to minimize

while ensuring $\|\widehat{\mathbf{\Delta x}}_{t}+\widetilde{\mathbf{\Delta x}}\|_{p}\leq\epsilon$. $\widehat{\mathbf{\Delta x}}_{1}=0$, and for $t>1$, $\widehat{\mathbf{\Delta x}}_{t}$ is the sum of all perturbations accumulated over the previous $t-1$ iterations. Optimizing Eq. (2) or minimizing Eq. (3) can be regarded as generating adversarial examples that could succeed on a distribution of models, and it has been proved effective in our previous work [1].

In Eqs. (2) and (3), Bayesian model averaging is used to predict the label of a deterministic model input. Such perturbation optimization processes solely consider the diversity of models, while the diversity of model inputs is overlooked. Nevertheless, considering that different DNN models may be equipped with different pre-processing operations, given the same benign input (e.g., a clean image), ${\mathbf{x}}$ can be different for different models after their specific pre-processing steps. Moreover, even given the same ${\mathbf{x}}$ (which is the pre-processed model input), different models obtain different $\widehat{\mathbf{\Delta x}}_{t}$. Therefore, introducing input diversity into Eqs. (2) and (3) may also be beneficial to the transferability of generated adversarial examples.

To incorporate such diversity, we simply introduce some randomness to the model input by rewriting Eq. (3) as:

Here the randomness term ${\mathbf{e}}$ is added linearly to the input ${\mathbf{x}}$ and/or accumulated perturbations $\widehat{\mathbf{\Delta x}}_{t}$. We can also introduce more complex modifications, such as by using a Gaussian filter $g({\mathbf{x}},\widehat{\mathbf{\Delta x}}_{t},{\mathbf{e}})$ with a random standard deviation [54]; yet, for simplicity, we discuss the linear case in this paper. Due to the very large number of parameters, it is intractable to perform exact inference using Eq. (4). Instead we adopt the Monte Carlo sampling to approximate the integral, where a set of $M$ models, each parameterized by ${\mathbf{w}}_{j}$, are sampled from the posterior $p({\mathbf{w}}|\mathcal{D})$ and $S$ inputs are sampled from $p({\mathbf{e}})$. The optimization problem can then be cast to maximizing

where $L(\cdot,\cdot,{\mathbf{w}}_{j})$ is a function evaluating the prediction loss of a DNN model parameterized by $\mathbf{w}_{j}$.

Given any pre-trained DNN model whose parameters are $\hat{{\mathbf{w}}}$, we can simply obtain a posterior $\mathcal{N}(\hat{\mathbf{w}},\sigma^{2}\mathbf{I})$ without fine-tuning by assuming it as an isotropic Gaussian. $\sigma$ is a positive constant for controlling the diversity of distribution. Similarly, we can consider ${\mathbf{e}}_{k}\sim\mathcal{N}(\mathbf{0},\sigma_{{\mathbf{e}}}^{2}\mathbf{I})$. Figure 1 compares the effectiveness of generating transferable attacks using Eq. (5) and the original implementation in our ICLR paper [1]. The experiment was conducted on ImageNet with ResNet-50 used as the substitute model; experiment details are deferred to Section 4.1. Apparently, introducing either model diversity or input diversity in a Bayesian manner could outperform the baseline I-FGSM on all victim models. More significantly, a joint diversification could further enhance adversarial transferability with a 17.05% increase in the average success rate compared with considering the model diversity alone [1].

In this subsection, we explain the optimization procedure of the posterior when fine-tuning the Bayesian model from a pre-trained model is possible. Following prior work, we consider a threat model in which fine-tuning can be performed on datasets collected for the same task as the victim models.

As in Section 3.2, we assume an isotropic Gaussian posterior $\mathcal{N}(\hat{\mathbf{w}},\sigma^{2}\mathbf{I})$; however, the mean vector $\hat{\mathbf{w}}$ is now considered as a trainable parameter. Optimization of the Bayesian model, or more specifically $\hat{\mathbf{w}}$, can be formulated as:

By adopting Monte Carlo sampling, it can further be reformulated as:

The computational complexity of Eq. (7) is high, thus we focus on the worst-case performance in the distributions, whose loss bounds the objective in Eq. (7) from below. The optimization problem then becomes:

where $\varepsilon$ and $\varepsilon_{{\mathbf{e}}}$ control the confidence region of the Gaussian distributions.

By applying the first-order Taylor approximation to the objective function in Eq. (8), we can approximate the optimal solutions $\mathbf{\Delta w}^{\ast}$ and ${\mathbf{e}}_{i}^{\ast}$ to the inner-maximization problem using $\lambda_{\varepsilon,\sigma}\nabla_{\hat{\mathbf{w}}}\sum_{i}L({\mathbf{x}}_{i},y_{i},\hat{\mathbf{w}})/\|\nabla_{\hat{\mathbf{w}}}\sum_{i}L({\mathbf{x}}_{i},y_{i},\hat{\mathbf{w}})\|$ and $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}\nabla_{{\mathbf{x}}_{i}}L({\mathbf{x}}_{i},y_{i},\hat{\mathbf{w}})/\|\nabla_{{\mathbf{x}}_{i}}L({\mathbf{x}}_{i},y_{i},\hat{\mathbf{w}})\|$, respectively. $\lambda_{\varepsilon,\sigma}$ and $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$ are computed using the quantile function of the Gaussian distributions. Thereafter, the outer gradient for solving Eq. (8) is:

which involves second-order partial derivatives in the Hessian matrix, and it can be approximately calculated using the finite difference method. That said, we use

where $\gamma$ is a small positive constant. $\mathbf{H_{\hat{{\mathbf{w}}},{\mathbf{x}}_{i}}}{\mathbf{e}}_{i}^{\ast}$ is approximated similarly as

By introducing $\mathbf{H_{\hat{{\mathbf{w}}},\hat{{\mathbf{w}}}}}\mathbf{\Delta w}^{\ast}$ and $\mathbf{H_{\hat{{\mathbf{w}}},{\mathbf{x}}_{i}}}{\mathbf{e}}_{i}^{\ast}$ in Eq. (9), we encourage flat minima in the parameter space and the input space, respectively. The former is known to be beneficial to the generalization ability of DNNs [55], while the later has been paid little attention during training/fine-tuning.

To evaluate the effectiveness of fine-tuning, an experiment was carried out on ImageNet using ResNet-50 as the substitute model, and results are shown in Figure 2. It clearly suggests that fine-tuning leads to more significant adversarial transferability. More detailed comparison results are provided in Table I.

In Sections 3.2 and 3.3, we have demonstrated the superiority of adopting the Bayesian formulation for generating transferable adversarial examples. However, it should be noted that our approach relies on a relatively strong assumption that the posterior follows an isotropic distribution. Taking one step further, we remove the assumption about the covariance matrix and try to learn it from data in this subsection.

Numerous methods have been proposed to learn covariance matrices, but in this paper we opt for SWAG [45] due to its simplicity and scalability. SWAG offers an enhanced Gaussian approximation to the distribution of ${\mathbf{w}}$ and ${\mathbf{e}}_{i}$ (for brevity, the subscript $i$ will be dropped in this subsection). Specifically, we adopt the SWA solution [66] as the mean of ${\mathbf{w}}$, and decompose the covariance matrix into a diagonal term, a low-rank term, and a scaled identity term, i.e., ${\mathbf{w}}\sim{\mathcal{N}}({\mathbf{w}}_{\mathrm{SWA}},\mathbf{\Sigma}_{\mathbf{w}})$, where

$\alpha\geq 0$ represent the scaling factor of SWAG for disassociating the learning rate of the covariance [45] and $\beta\geq 0$ controls the covariance matrix of the isotropic Gaussian distribution. ${\mathbf{w}}_{\mathrm{SWA}}$, $\mathbf{\Sigma}_{\mathrm{diag}}$, and $\mathbf{\Sigma}_{\mathrm{low-rank}}$ are obtained after fine-tuning converges, thus we keep the fine-tuning loss and fine-tuning mechanism as in Section 3.3 even with the improved distribution modeling.

For multi-step attacks, e.g., in I-FGSM, ${\mathbf{e}}$ can be similarly obtained from the distribution of perturbations. Unlike for ${\mathbf{w}}_{\mathrm{SWA}}$ and $\mathbf{\Sigma}_{\mathbf{w}}$, it does not necessarily require model fine-tuning to obtain the mean vector and covariance matrix of the distribution, as they model the randomness in model input instead of in model parameters and the learning dynamics of the adversarial inputs are obtained during I-FGSM. Yet, for single-step attacks like FGSM, such a distribution modeling boils down to be equivalent to the isotropic case.

We compare performance of methods with and without such improved distribution, in single-step and multi-step attacks, in Table I. Approximating the covariance matrix of parameters through SWAG improves the attack success rates on all victim models, leading to an average increase of 8.16% (from 44.22% to 52.38%) when using single-step attacks based on FGSM. This indicates that the more general distributional assumption of model parameters aligns better with the distribution of victim parameters in practice. A similar benefit is observed when applying the improved distribution modeling to the model input, where the average success rate increases from 45.09% to 56.39% in using I-FGSM. The best performance is achieved when both improved posteriors and fine-tuning are adopted. However, note that with fine-tuning, we suggest not to adopt such improved distribution modeling of model input and model parameters simultaneously, as the two distributions will not be independent under such circumstances and there will be more hyper-parameters to be tuned. Following this suggestion, in Table I and subsequent experiments, when fine-tuning is performed, we only adopt SWAG for the model parameters and use the isotropic formulation for the model inputs.

To be consistent with [1], we focused on untargeted $\ell_{\infty}$ attacks to study the adversarial transferability. All experiments were conducted on the same set of ImageNet [14] models collected from the timm repository [72], i.e., ResNet-50 [58], VGG-19 [56], Inception v3 [57], ResNet-152 [58], DenseNet-121 [59], ConvNeXt-B [60], ViT-B [61], DeiT-B [62], Swin-B [63], BEiT-B [64], and MLP-Mixer-B [65]. These models are well-known and encompass CNN, transformer, and MLP architectures, making the experiments more comprehensive. We randomly sampled 5000 test images that can be correctly classified by all these models from the ImageNet validation set for evaluation. Since some victim models are different from those in [1], the test images are also different. The ResNet-50 was chosen as the substitute model, same as in [1]. For the experiments conducted on CIFAR-10, we adhered to the settings established in prior work [1]. Specifically, we performed attacks on VGG-19 [56], WRN-28-10 [68], ResNeXt-29 [69], DenseNet-BC [59], PyramidNet-272 [70], and GDAS [71], employing ResNet-18 [58] as the substitute model. The test images used are the entire test set of the CIFAR-10 dataset. We ran 50 iterations with a step size of 1/255 for all the iterative attacks.

In this paper, we set $\gamma$ to a fixed constant, which is slightly different from the approach described in [1]. In [1], $\gamma$ was set to be $0.1/\|\mathbf{\Delta w}^{*}\|_{2}$, dependent on the value of $\|\mathbf{\Delta w}^{*}\|_{2}$. However, for our experiments on ImageNet and CIFAR-10, we set $\gamma$ to 0.1 with $\lambda_{\varepsilon,\sigma}=1$, and $\gamma$ to 0.5 with $\lambda_{\varepsilon,\sigma}=0.2$, respectively. These hyper-parameters match the ones actually used in [1]. We set $\lambda_{\varepsilon_{\mathbf{e}},\sigma_{\mathbf{e}}}$ to be 1 and 0.01 for ImageNet and CIFAR-10, respectively. We used a learning rate of 0.05, an SGD optimizer with a momentum of 0.9 and a weight decay of 0.0005, a batch size of 1024 for ImageNet and 128 for CIFAR-10, and a number of epochs of 10. We fixed $\sigma=0.006$ and $0.012$ for ImageNet and CIFAR-10, respectively. The $\sigma_{{\mathbf{e}}}$ was set to be $0.01$ and $0.05$ with and without fine-tuning, respectively. Considering Eq. (12), we used $\alpha=1$ for the posterior distribution over model parameters. For the posterior distribution over model input, we set $\alpha=100$ and $25$ for ImageNet and CIFAR-10, respectively. Due to the negligible difference in success rates observed between using a diagonal matrix and a combination of diagonal and low-rank matrices as the covariance for the SWAG posterior, we opted for simplicity and consistently employed the diagonal matrix. We also set $\beta=0$ in the experiments for the same reason. For our current method with fine-tuning, we used SWAG to approximate the posterior over model parameters, while we used an isotropic Gaussian distribution for the posterior over model inputs. This choice was made because modeling the posterior over model inputs becomes challenging in situations where there is a high degree of randomness in the model parameters. We set $M=5$ and $S=5$ unless otherwise specified.

For compared competitors, we followed their official implementations. For LGV [28], we sampled 5 models from the collected model set at each iteration. By doing so, the performance will be better compared with sampling only one model as in LGV’s default setting. For DRA [23], we tested it on ImageNet using the ResNet-50 model provided by the authors. All experiments were performed on an NVIDIA V100 GPU.

We compared our method with recent state-of-the-arts in Tables II and III. A variety of methods were included in the comparison, including methods that adopt advanced optimizers (MI-FGSM [18] and NI-FGSM [19]), increase input diversity (TI-FGSM [24], DI²-FGSM [25], SI-FGSM [19], and Admix [26]), use advanced gradient computations (ILA [10], SGM [12], LinBP [13], NAA [67], and ILA++ [21]), and employ substitute model fine-tuning (LGV [28]). We also evaluated our Bayesian formulation that only models the diversity in substitute model parameters [1]; for clarity, it is referred to as “BasicBayesian” in this paper. The performance of all compared methods was evaluated in the task of attacking 10 victim models on ImageNet and 6 victim models on CIFAR-10.

It can be observed from the tables that our current method outperforms all these methods. It can achieve the best average success rate on ImageNet even without fine-tuning. When fine-tuning is possible, the average success rate improves remarkably, achieving $75.20\%$ on ImageNet and 85.52% on CIFAR-10.

We would also like to mention that it is possible to combine our method with other attack methods to further enhance the transferability. In Table IV, we report the attack success rate of our method, in combination with MI-FGSM, DI²-FGSM, SGM, LinBP, and ILA++. It can be seen that the transferability to all victim models gets improved. The best performance is obtained when combining our current method with DI²-FGSM, achieving an average success rate of $78.73\%$ (which shows a $+34.26\%$ gain in comparison with the original performance of DI²-FGSM).

It is also of interest to evaluate the transferability of adversarial examples to robust models, and we compared the performance of competitive methods in this setting in Table V. The victim models used in this study were collected from RobustBench [73]. All these models were trained using some sorts of advanced adversarial training [74, 75, 76], and they exhibit high robust accuracy against AutoAttack [77] on the official ImageNet validation set. These models included a robust ConvNeXt-B [78], a robust Swin-B [78], and a robust ViT-B-CvSt [79]. Following the setting in [1], two models from Bai et al.’s open-source repository [80], namely a robust ResNet-50-GELU and a robust DeiT-S, were also adopted as the robust victim. The tested robust ConvNeXt-B, robust Swin-B, and robust ViT-B-CvSt show higher prediction accuracy (i.e., 55.82%, 56.16%, and 54.66%, respectively, against AutoAttack) than that of the robust ResNet-50-GELU and robust DeiT-S (35.51% and 35.50%, respectively).

We still used the ResNet-50 substitute model which was trained just as normal and not robust to adversarial examples at all. From Table V, we can observe that our newly proposed method improves the transferability of adversarial examples to these defensive models, compared with the basic Bayesian formulation in [1].

Since the objective of adversarial training is different from that of normal training, in the sense that the distribution of input is different, we suggest increasing $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$ to achieve better alignment between the distributions and to further enhance transferability. When we increase $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$ from 1 to 5 or even 10, we indeed obtain even better results in attacking robust models on ImageNet, as will be discussed in the ablation study.

In addition to adversarial training, we conducted experiments on robust models obtained through randomized smoothing [81], which is widely recognized as one of the leading techniques for achieving certified robustness. When attacking such a defensive model with a ResNet-50 architecture, we observe a substantial improvement compared to [1] (26.76%$\rightarrow$85.04%).

We conduct a series of ablation experiments to study the impact of different hyper-parameters.

The effect of $\lambda_{\varepsilon,\sigma}$ and $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$. When adopting fine-tuning, we have two main hyper-parameters that have an effect, namely $\lambda_{\varepsilon,\sigma}$ and $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$. We conducted an empirical study to demonstrate how the performance of our method varies with the values of these two hyper-parameters on ImageNet. We varied $\lambda_{\varepsilon,\sigma}$ from the set $\{0,0.1,0.2,0.5,1,2\}$ and varied $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$ from the set $\{0,0.5,1,5,10\}$ for attacking models which are the same as in Table II and Table V. The average success rates of attacking these normally trained models and robust models are given in Table VI. To achieve the best performance for each $\lambda_{\varepsilon,\sigma}$ and $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$, we tuned the other hyper-parameters using 500 randomly selected images from the validation set. These images show no overlap with the 5000 test images.

From the table, it can be observed that increasing the values of $\lambda_{\varepsilon,\sigma}$ and $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$ from 0 to 0.5 enhances the transferability of adversarial examples in attacking normally trained models. However, excessively large values of these hyper-parameters can lead to inferior performance. The best performance of attacking normally trained models is achieved by setting $\lambda_{\varepsilon,\sigma}=0.5$ and $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}=1$, which results in an average success rate of $75.60\%$. Considering the performance on attacking robust models, it can be observed that the average success rate peaks a larger value of $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$. This is partially because that reducing the prediction loss of the perturbed inputs during fine-tuning in Eq. (8) resembles performing adversarial training, and larger $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}$ implies making the substitute model robust in a larger neighborhood of benign inputs. A recent related method, namely DRA [23], also suggests that performing regularized fine-tuning before attacking is beneficial to attacking defensive models, and it achieves an average success rate of $62.03\%$ and $17.68\%$ in attacking normally trained models and adversarially trained models, respectively. By setting $\lambda_{\varepsilon_{{\mathbf{e}}},\sigma_{{\mathbf{e}}}}=5$, our method has an average success rate ranging from $65.84\%$ to $67.51\%$ in attacking normally trained models and ranging from $18.48\%$ to $20.47\%$ in attacking adversarially trained models, which outperforms DRA considerably.

The effect of $M$ and $S$. Our method is proposed based on the principle that increasing the diversity of model parameters and model input can enhance the transferability of adversarial examples. In Figure 3, we conducted experiments to evaluate the transferability of adversarial examples crafted using different choices of the number of model parameters and input noise sampled at each attack iteration, i.e., $M$ and $S$, in the cases of with and without fine-tuning. The average success rates were obtained by attacking the same victim models as in Table II. The results apparently show that sampling more substitute models and more input noise can indeed enhance the transferability of adversarial examples, just as expected.