Where Does the Robustness Come from? A Study of the Transformation-based Ensemble Defence

A classification task can be represented as $y_{pred}=f_{\theta}(x)$, where a classifier $f$ with parameters $\theta$ takes input $x$ and outputs the prediction $y_{pred}$ for $x$. In the evasion attack, an AE $x^{*}$ is an input sample maliciously crafted to mislead the victim classifier to an erroneous output $y^{*}=f_{\theta}(x^{*})$. $x^{*}$ is close to $x$ with respect to some distance measurement $d_{L_{p}}$ such as $L_{1},L_{2},L_{\infty}$ norms, that is $d_{L_{p}}(x,x^{*})<\epsilon$, so that the perturbation is imperceptible. If the erroneous output $y^{*}$ is unspecified as long as $y^{*}\neq y_{pred}$, the attack is named untargeted attack. We focus on untargeted attacks in this paper, while the analysis can be easily extended to targeted attack.

The evasion attacks can be categorised into the black-box attack and white-box attack. In the black-box attack, the attacker has no knowledge of the internal information about the model. The attacker can only perform legitimate operations such as querying using her/his data and observing the model output. In the white-box setting, the adversary knows all about the model, including the model structure and parameter weights. There have been many methods proposed for generating AEs in both black-box setting and white-box setting.

We introduce four classical white-box AE generation methods used in this paper, i.e., FGSM (Fast Gradient Sign Method) (10), PGD (Projected Gradient Descent) (18), C&W (4), and DeepFool (23).

FGSM (10) is a classical gradient-based attack. It updates the adversarial perturbations by the gradient of adversary’s objective with respect to the input in the following way.

where $L$ is the objective function for trained target classifier $f_{\theta}$, $\epsilon$ is the perturbation budget.

PGD (18) is an iterative variant of FGSM. It is also a representative of various iterative methods, such as Basic Iterative Method (14) and Momentum Iterative Method (8), due to its superior attack effect (19). PGD updates perturbation gradually till the perturbation budget exceeded or the target model successfully fooled. During every iteration, instead of taking a big step at the size of budget $\epsilon$, PGD updates perturbations in a gentle way with step size $\alpha<\epsilon$. As pixel values for images are bounded, PGD projects generated adversarial perturbations into the feasible domain before entering the next iteration.

C&W (4) represents another type of generating methods that treat the procedure of finding adversarial examples as an optimisation problem. Instead of solving an optimisation problem with distance constraints, C&W attack applies the change of variable method to ensure the generated AEs are in a feasible domain. Also, to keep AEs stealthy and effective, a weighted sum of $L_{p}$ distance and attacker’s loss is included in the optimisation terms. The $L_{2}$ bounded C&W attack is to solve the following optimisation problem.

where $c$ is a constant found by lattice search.

DeepFool (23) is another optimisation-based method but with a closed form representation of minimum distance to the closest locally linear approximation of target classifier’s decision boundaries as

where $k\in K$ is one of all possible classes, $x$ is the original benign example, and

DeepFool performs the above operations to update the current generated AE $x_{i}$ at step $i$ iteratively till it successfully fools the victim classifier.

Experiments in previous research show that AEs generated by attack algorithms targeting one specific model may successfully fool other models (12)(7). This type of phenomenon is called transferability. Demontis et al. (7) proposed to represent transferability numerically as the value of target classifier’s objective function given the perturbed example and its ground truth label.

The transferable nature of AEs has been shown to be able to assist attackers to fool a well-protected model by generating adversarial examples on a surrogate model trained with only black-box queries (25). In an ensemble of many sub-models, it is expected that the transferability of AEs among the sub-models should be low. In this way, the ensemble can still make the correct decision with a high probability since not all of the sub-models are fooled at the same time. However, the transferability of AEs generated on one sub-model may differ from that on another sub-model. Hence, an attacker may be able to generate adversarial examples on one ensemble member, i.e., the sub-model whose AEs are with high transferability, and successfully transfer its AEs to other ensemble members, eventually fooling the whole ensemble model.

Empirical evidence has shown that adversarial examples are sensitive to image transformations caused by the camera when performing attacks in the physical world (14). Taking pictures of successful adversarial images as input to the target classifiers, the attack success rate drops from nearly 100% to almost zero. The AE’s high sensitivity to image transformation is also used in the detection of AE (36). By comparing the softmax probability vectors across the outputs of the classifier trained on the original data and the classifier trained on the transformed data (i.e., colour-depth-reduction and spatially pixel smoothing), it can detect the existence of AE which is indicated by a high distance score.

A very recent work proposes a robust ensemble with sub-models on 72 input transformations for image classification task (21). As shown in Figure 1, each sub-model is trained on the training data that have been gone through one specific transformation. The 72 transformations are listed in Table 1. We categorise them into reversible transformations and irreversible transformations.

In the inference phase, the ensemble takes an original image $x$ as input. For each sub-model $f_{t_{i}},i\in[0,71]$, the original $x$ is transformed by $t_{i}$ to $x_{t_{i}}$ before being processed by $f_{t_{i}}$. The decisions of all the sub-models, in the form of predicted probabilities or logits from the last layer, can be aggregated using one of the 5 ensemble strategies, i.e., Random Defence (RD), Majority Voting (MV), Top 2 Majority Voting (T2MV), Average Probability (AVEP) and Average Logits (AVEL). RD outputs the prediction from a random sub-model. MV determines the output label that is agreed upon by most sub-models. T2MV performs MV among labels associated with the top two probabilities predicted by each sub-model. AVEP predicts based on the average predicted probabilities of all sub-models. AVEL is based on average logits of all sub-models.

One thing worth noting is that not all of the 72 transformations can be reversed as shown in Table 1. Transformations that cannot be properly reversed may cause difficulties for the AE generation algorithms to cast the perturbation on $x_{t_{i}}$ to $x$. Though the ensemble is claimed to be robust against AEs generated by several classical AE generation algorithms, we would like to find out where exactly the robustness comes from in the transformation-based ensemble?

Previous researches (25)(6) have demonstrated that adversarial examples generated by attack algorithms targeting one specific model may successfully fool another model with similar functionality. In the transformation-based ensemble, each model is trained on the same training data but after different transformation. We conjecture that the transferability of AEs may still exist among these models since they are performing the same classification task.

The idea of Transferability-based Adaptive Attack (TAA) comes from Liebig’s Law of the minimum in agricultural science. It originally states that the growth is indicated not by the total resources but by the scarcest resource (9). In the case of an ensemble, the robustness of the model to some extent is determined by the transferability of AEs generated by a single model. Adversarial examples generated on one ensemble member may transfer to other ensemble members, but the transferability rate differs as the victim model which the AEs are generated on differs. Lower transferability indicates a lower probability that the entire ensemble is compromised at the same time. Therefore, thinking from the attackers’ angle, the model whose AEs have the highest transferability among all the ensemble sub-models is the scarcest resource they are looking for, i.e., the “weakest” sub-model in the ensemble. AEs generated against the “weakest” ensemble member may have a higher probability of successfully fooling the whole ensemble.

TAA, as shown in Algorithm 1, is a naive attack in which adversarial examples are generated on one target ensemble member, whose AEs have the highest transferability, trying to fool the overall ensemble. The target sub-model is selected according to the transferability ranking algorithm as shown in Algorithm 2. Algorithm 2 is to measure the transferability of the AEs generated on certain model by the accuracy of other sub-models on these AEs. A lower accuracy indicates a larger number of models the generated AEs have fooled, and hence the AEs generated on that model have a higher transferability. This selection needs to be done only once in advance, and can be done on a much smaller test dataset $(\chi^{*},\gamma^{*})$, e.g., randomly selected 100 samples.

TAA is a basic algorithm to evaluate the robustness of an ensemble solely based on reversible transformations. The measurement of the transferability of AEs using Algorithm 2 can be used to understand the robustness source of the ensemble, which is demonstrated later in Section 4.2.

As we are targeting an ensemble instead of one single model, it is natural to aggregate adversarial perturbations generated on all ensemble members to perform adversarial attacks. Hence, we propose PAA as explained in Algorithm 3. In PAA, a greedy attacker will keep generating perturbations on sub-models that have not been fooled and aggregate those perturbations with certain aggregation strategy, until the perturbation becomes larger than a preset perturbation budget.

Different from attack algorithms targeting one single model which are bounded by different norms (10)(4)(14)(18), attacking an ensemble of more than one sub-model may also introduce perturbation several times larger when generating on each of them. Therefore, PAA needs a bound score to control the overall perturbation budget. Following previous work (21), we use the normalised $L_{2}$ dissimilarity to measure the perturbation as shown in Equation 1.

In the following subsections, we introduce four perturbation aggregation strategies that could be used in PAA, including maximum perturbation (MaxP), average perturbation (AvgP), the majority vote of perturbations (MVoteP) and the weighted sum of perturbations (WSumP). Theoretically, they have their own advantages and disadvantages, however, their actual performance needs to be experimentally evaluated.

For an ensemble based on reversible transformations, we can use the proposed adaptive attack, i.e., TAA and PAA, to better evaluate the ensemble robustness. We can also measure the transferability of AEs among the sub-models. We expect the attack results can provide us some evidence whether the ensemble based only on reversible transformation can gain some robustness.

For an ensemble including irreversible transformations, TAA and PAA cannot be adopted because they require the adversarial perturbation to be projected back to the original input which is impossible when the irreversible transformation exists. Hence, we plan to evaluate the ensemble using an ensemble evaluation method used in previous literature (21) which is also based on the transferability of AEs.

We maintain the reversible transformation ensemble unchanged, whose performance has been evaluated in previous steps. We randomly draw a number of irreversible sub-models and add them into the unchanged reversible transformation ensemble while strictly controlling the proportion of the numbers of reversible and irreversible sub-models. We shall observe the influence on the ensemble robustness against AEs from the number of newly-added irreversible sub-models. Furthermore, if an ensemble demonstrates robustness, we intend to further measure the impact of the number of ensemble members on the ensemble robustness.

Model. The sub-model structure is shown in Figure 2, which consists of three convolutional layers followed by two fully connected layers and one soft-max layer (21). The transformation-based ensemble combines the sub-model in the way we introduced in Figure 1 in Section 2.3. We put the 72 transformations (see Table 1) into our transformation pool, which includes 14 reversible transformations and 58 irreversible transformations. We construct the reversible transformation ensemble and irreversible transformation ensemble by drawing candidate transformation from this pool. There are 5 ensemble strategies evaluated, i.e., Random Defence (RD), Majority Voting (MV), Top 2 Majority Voting (T2MV), Average Probabilities (AVEP), Average Logits (AVEL) (Detailed explanations can be found in Section 2.3.).

Dataset. We evaluate the robustness of the ensemble on the MNIST dataset (16). MNIST dataset contains 70,000 grey-scale images of hand-written digits. We further split the original testing data (10,000 images) into validation and testing set at the ratio of 1:1. Early-stopping technique based on cross-entropy loss on validation dataset is adopted to prevent models from over-fitting during training.

Base attack algorithms used in TAA and PAA. We assume that the attacker has full knowledge of the ensemble including the transformation pre-processing, the model parameters, and the ensemble strategy. We utilise 4 representative attack algorithms in our experiments, i.e., FGSM, PGD, C&W, and DeepFool. We set the parameters for these algorithms in an overkill way (2)(11)(21) as the findings in recent work by Tramèr et al (32) indicate that proper attack parameters and sufficient iterations are essential for an attack to converge so as to provide convincing evaluation results. The parameters are set as shown in Table 2.

It has been observed that transferability of AEs exists among models performing the same classification task (25). This experiment is designed to understand whether the transferability still exists among the models trained on the same data but after different transformations. It refers to reversible transformation here because the perturbation generated on input after irreversible transformation cannot be projected back to the original input, and hence it does not make much sense to evaluate its transferability when the entire image is disturbed by the irreversible transformation.

First of all, we trained 15 models, one model of which is trained on the original MNIST training data records while the other 14 are trained on the same batch of data records but after the 14 reversible transformations as shown in Table 1 with the label unchanged, respectively. Then, we generate AEs with FGSM, PGD, C&W and DeepFool on every model and evaluate their transferability rates. The transferability rate of the AEs generated on one model is measured by calculating the rate of successful attacks on other models. The results are demonstrated in Table 3. The transformation operations are used to indicate the model trained on the data after that transformation, while “original” means the model trained on the original data records.

It can be seen in Table 3 that AEs generated using FGSM and PGD generally have high transferability rates. On average 49.53% of AEs generated on the “original” model with FGSM can fool models trained on data after reversible transformations. This number for PGD is even more significant, with on average 65.88% of AEs generated on the “original” model using PGD can fool the other models while this number rises to as high as 76.84% on AEs generated on “shift top left” model using PGD.

Generally, for the AEs generated using FGSM and PGD on models trained on data after one specific transformation, they all have a high successful rate in fooling the models trained on other transformations. This indicates that the transferability of AEs does exist among models trained on data after reversible transformations. Such models cannot block the AEs from transferring, meaning that those models may have similar decision boundaries even trained on data after different transformations. It also implies that an ensemble of such models may not be able to improve the robustness as long as the transformation is reversible. Since the ensemble strategy also plays an important role in an ensemble, this conjecture needs further validation which will be shown in Section 4.3.

Most importantly, it is observed that the transferability rate of AEs differs as the victim model differs. For example, on average 76.18% of AEs generated on the model trained on examples shifted to their top-left with PGD can transfer to other sub-models, while only 34.59% of AEs generated with PGD on the model trained on examples shifted to their bottom left can transfer to other models. Hence, it implies a good tactic for the attacker to fool the ensemble by generating AEs targeting the top sub-model in the transferability ranking list.

Also, we notice that AEs generated with gradient-based attack algorithms (e.g., PGD and FGSM) are more transferable compared to those generated with optimisation-based attack algorithms (e.g., C&W and DeepFool) while the number of successful AEs on their victim models are at the same scale. Due to the facts that optimisation-based attacks like C&W and DeepFool generate more subtle perturbations and more specific to the victim model, it is reasonable that the transferability rate of their generated perturbations is lower.

As shown in Section 4.2, despite the models are trained on data that have been gone through different reversible transformations, AEs generated on one model can actually transfer to other models for the same classification task at a high probability. Therefore, the ensemble based on reversible transformations may face a potential threat that AEs generated on one of its sub-models may fool the entire ensemble. However, it still depends on the ensemble strategy adopted in the ensemble. In an untargeted attack, AEs are generated to fool the victim sub-model no matter which label the erroneous output is. For example, if the ensemble uses majority voting ensemble strategy, it is not sure whether the AE can fool the majority sub-models to output an agreed erroneous output. Hence, this experiment is necessary to better evaluate the robustness of the ensemble based on reversible transformations under various ensemble strategies.

The target ensemble is the ensemble of the 14 sub-models that are trained on the data after 14 reversible transformations. The “original” model is not included in this ensemble. We generate AEs on every sub-model using attack algorithms on the test dataset of 5000 examples. Five ensemble strategies are measured, i.e., RD, MV, T2MV, AVEP, and AVEL.

In practice, an attacker can select to attack only the sub-model, whose AE has the highest transferability, to achieve a high success attack rate, which is the core idea of TAA. Here, we show the full results instead of just the top $tr$ ranked one for readers to have a better overview and understanding of attack results on ensembles using different ensemble strategies when choosing different target sub-model. The overall results in terms of the ensemble classification accuracy under such attacks are listed in Table 4. The performance of the ensemble under no attack is shown as “benign”.

Under PGD-base TAA, the “shift top left” is at the top of the $tr$ list. It can be seen from Table 4 that attackers can lower the ensemble accuracy to around 15% by generating AEs against this sub-model using PGD-based TAA. Generally, AEs generated by FGSM and PGD lower the ensemble accuracy more than that by C&W and DeepFool, which is consistent with the transferability ranking experimental results.

It is worth noting that the robustness from integrating different reversible transformation sub-models is fragile even when the attacker only knows the classification task and training data. This can be seen from the Table 4 where the AEs generated against the “original” model which is not even in the ensemble can lower the ensemble accuracy to around 30%.

Unlike TAA which depends on the transferable nature of AEs to attack an ensemble of models, PAA is more specifically designed for ensemble defence. It utilises all possible perturbations generated on sub-models to perform an adversarial attack on an ensemble.

The stealthiness of AEs generated by PAA are bound by normalised dissimilarity (see Equation 1). The dissimilarity score is set to 0.3 in this experiment since we notice the average dissimilarity in Table 4 is around this value. The classification accuracy of ensemble under PAA-generated AEs on the test dataset is shown in Table 5. Four perturbation aggregation strategies are evaluated i.e., MaxP, AvgP, MVoteP, and WSumP.

We can see from Table 5, for PAA using FGSM, PGD and DeepFool as base attack algorithms, at least one of the four aggregation strategies makes the ensemble accuracy drop to around 20%. Under the PGD attack combining WSumP aggregation, the classification accuracy of the transformation-based ensemble drops from 99.70% on benign samples to 16.65%. Some adversarial examples generated using PGD-based TAA and PGD-based PAA are available in Figure 3. Unlike TAA, aggregated adversarial perturbations generated by C&W and DeepFool also have impressive attack effect. For PAA using C&W, the ensemble accuracy is lowered to around 70%.

Also, the perturbation aggregation strategy acts as an important role in our PAA algorithm. WSumP and AvgP generally outperform the other two aggregation strategies. For example, the classification accuracy of ensemble using AVEL ensemble strategy is 95.51% on FGSM-based PAA AEs using MaxP aggregation strategy, while the classification accuracy is 22.30% under the same setting except that AvgP aggregation is used. We observe that, under the same setting, classification accuracy on AEs generated by PAA with AvgP and WSumP as aggregation strategy is similar. This can be easily explained that for an ensemble with 14 members, 12 of them share the same weight during WSumP according to Equation 3. However, this little difference in weights improves the attack effectiveness by dropping the ensemble accuracy 5% more in some cases when using PGD and DeepFool. We believe that the attack effectiveness can be further improved by fine-tuning the WSumP weights in Equation 3, which is though not the main focus in this experiment.

In summary, the TAA and PAA experiments show that the ensemble solely based on reversible transformations cannot provide us with the expected robustness. The transferability still widely exists among the sub-models trained on data after different reversible transformations, which may be one of the fundamental reasons that such an ensemble is not robust against AEs. For those attack algorithms whose AEs have less transferability, PAA can assist them to succeed an impressive attack against the reversible ensemble.

To validate the influence of sub-models based on irreversible transformation on the overall robustness of an ensemble, we design an experiment by gradually incorporating more such sub-models into the previously evaluated ensemble of 14 reversible sub-models and measuring their accuracy. The number of irreversible sub-models $m$ are the multiples of the number 14, i.e., $m\in\{0,14,28,42,56\}$. Moreover, we also evaluate the robustness of ensemble of $m$ sub-models solely based on irreversible transformations.

In this experiment, AEs are generated on the original model with TAA attack parameters in Table 2. We generate AEs on the original model for the following two reasons. First of all, TAA and PAA are not available due to the introduction of irreversible transformation sub-models. Secondly, it has been demonstrated in the TAA experiment that the AEs generated on the original model can effectively attack the reversible transformation ensemble, which could serve as a baseline. We repeat the random drawing and evaluating process for five times for every value of $m$.

The full results on MNIST dataset are shown in Appendix Table 6. Regarding C&W and DeepFool, we have already understood well in Section 4.2 that their generated AEs have limited transferability among models based on reversible transformations. Table 6 further confirms this results by showing that their generated AEs do not affect the ensembles much, no matter the ensemble is based on reversible or irreversible transformations. However, under FGSM and PGD attacks, we can see that all the ensembles cannot provide expected robustness. Specifically, we can observe that there is a slight accuracy increase (less than 10%) when adding the first batch of 14 irreversible sub-models into the ensemble of 14 reversible sub-models. However, this increase does not persist as more irreversible sub-models are added. Comparing two ensembles of the same number of 14 reversible sub-models and 14 irreversible sub-models, the irreversible ensemble showing slightly advantage in accuracy, i.e., less than 10% under the FGSM attack and less than 20% under the PGD attack. Also, for the ensemble purly based on irreversible sub-models, increasing the number of sub-models does not increase the ensemble accuracy much.

For illustration purpose, we demonstrate the results of ensembles using MV strategy on AEs generated using PGD in Figure 4. The blue line represents the ensemble of 14 sub-models of reversible transformations plus $m$ sub-models of irreversible transformations. The orange line represents the ensemble of $m$ sub-models solely based on irreversible transformations. The x-axis is the total number of sub-models in the ensemble.

From Figure 4, we can observe that classification accuracy under attack increases a little bit as the number of sub-models in the ensemble increases when the ensemble is gradually added in sub-models based on irreversible transformations (i.e., the blue line). Since the number of sub-models on reversible transformations in this ensemble is fixed, this may imply that sub-models on irreversible transformations bring robustness to the original ensemble with sub-models on reversible transformations only. For an ensemble of models based on irreversible transformations, the classification accuracy does not change much with respect to the number of sub-models in an ensemble (i.e., the orange line). The classification accuracy of ensemble solely based on irreversible transformations is always higher than the hybrid ensemble of sub-models on both kinds of transformations, suggesting that sub-models on irreversible transformations may provide more robustness than sub-models on reversible transformations do. However, the accuracy under attacks is much lower than the benign performance, which indicates that transformation-based ensemble alone may not be able to provide us with enough robustness. These observations are also supported by the experimental results on FasionMNIST dataset (35) which are available in Appendix Table 7.

It is well recognised that machine learning algorithms are vulnerable to adversarial examples which are crafted by adding small, human-imperceptible perturbations to legitimate input samples. There have been various attack algorithms to compute the adversarial perturbations. For a differentiable machine learning model, attack algorithms can utilise the model gradient information to compute a perturbation which increases the loss function value of the legitimate input to its correct label. It leads the model to predict a wrong label with a smaller loss function value than that of the correct label (3; 10; 18; 14). For non-differentiable classifiers like decision trees and random forests, attackers can apply more complex strategies (13) or transfer the adversarial example generated with a surrogate model (28). Attackers can also use the optimisation-based attack algorithms which are to find the optimised perturbation by maximising or minimising their objective instead of just finding any perturbation that works (4; 5; 20). More intriguingly, there are $L_{1}$ norm bounded attack algorithms to limit the number of perturbed pixels (26; 4), universal adversarial perturbations that work for all examples in the test dataset (22; 34), etc.

Recently, the adaptive attack, which is the attack specifically designed for certain defence mechanisms, has been gaining more and more attention due to its impressive attack effect (4; 32). Tramer et al. (32) evaluated 13 defence methods proposed in the year of 2019 by performing adaptive attacks on them as the evaluation of robustness under worst-case scenario. Unfortunately, all of these 13 defence methods fail to protect victim model from corresponding adaptive attacks while majority of adaptive attacks on them are iterative attack algorithms like PGD with multiple random starts and larger number of iterations. Our paper borrows the core idea of the classical attack algorithms and proposes adaptive attack algorithms targeting the transformation-based ensemble defence, fully considering the learnt lessons about the importance of evaluating under sufficient strong settings (32).

There is a continuous arms race between the machine learning attack and defence. It is of great significance to defend machine learning models against such attacks so that the models can produce trustworthy output that we can rely on. Researchers have put a large amount of effort in defending machine learning models(30)(24)(27), among which the ensemble defence is an important branch.

Ensemble as a defence method is an easy but efficient way to protect models from adversarial attacks especially under black-box scenarios when the ensemble details are unknown to attackers. There are ensembles promoting the diversity among sub-models (24), using different precision of weights and activation functions in sub-models (29), training an ensemble of binary-classifiers with sufficient diversity and redundancy (33), combining sub-models trained on data after different transformations (21), etc. It is shown that even simple combination of undefended classifiers with different random noise has similar robustness to the adversarial training (17; 4) and advanced ensembles with guaranteed diversity (24; 33; 30; 21) demonstrate superior performance. However, in the attack and defence arms race, some defences are soon broken by adaptive attacks (24; 11; 29; 32). In order to better understand the effectiveness of transformation-based ensemble defence (21), we utilise transferability analysis and propose two adaptive attack algorithms, i.e., TAA and PAA. The results reveal that the transformation-based ensemble defence can provide some but not enough robustness against AEs. The gained robustness is mainly from the irreversible transformation sub-models, and increasing the number ensemble sub-models does not increase the ensemble robustness.