A Synergetic Attack against Neural Network Classifiers combining Backdoor and Adversarial Examples

Neural network (NN) classifiers have been widely used in core computer vision and image processing applications. However, NNs are shown to be sensitive and can be easily attacked by exploiting vulnerabilities during model training and inference [1, 2]. We broadly categorize existing attacks against NN models into inference attacks, e.g., adversarial examples [1], and poisoning attacks, e.g., Trojan back-doors [2], respectively. In adversarial examples, attackers try to mislead NN classifiers by perturbing model inputs with (visually unnoticeable) adversarial noise at the inference time [1]. Meanwhile, in Trojan back-doors ¹¹1The Trojan attack discussed in this work is the poison-label Trojan attack, in which both the training input and the corresponding label are poisoned, the adversaries manipulate model parameters for back-door breaches through a poisoned training process [2].

Researchers propose a plethora of defenses against each of the attack types individually. For example, adversarial training has been widely used to defend against adversarial examples [3, 4, 5]. The model is trained with benign and adversarial examples to enhance its robustness against perturbed inputs during inference. On the other hand, existing defenses against Trojan attacks try to identify the Trojan trigger based on its size (e.g., [6]), or to distinguish inputs with Trojan trigger through analyzing predictions on the superimposition of the input images with a set of reserved benign inputs (e.g., [7]). More details of existing defenses and other background information are presented in Appendix II. Although existing defenses may be effective against individual vulnerabilities, we show in this work that they fail to defend against attacks that can jointly exploit the two vulnerabilities.

In this work, we propose AdvTrojan, a novel attack that jointly exploits the inference and training vulnerabilities mentioned earlier to bypass existing one-sided defenses. Although we have one-sided defenses against adversarial perturbation or Trojan backdoor, these defenses may give a false sense of security to the ML models against ”harder to defend attacks” like AdvTrojan. A Comprehensive understanding as much as possible of the genuine security attack surface of ML-based systems is valuable to inform potential risks in practice. Therefore, our main goal is to create a “harder to defend attack” exposing weaknesses, exploring the security surface, and serve as whistleblowers to the community to research better defenses. The following research questions guide the design of AdvTrojan:

1. 1.

   Stealthiness: How to jointly exploit model vulnerabilities to build a stealthy synergistic attack?

2. 2.

   Practicality: How to implement the proposed synergistic attack with the same assumption on attackers’ ability as conventional attacks?

3. 3.

   Explainable: How to mathematically explain the proposed synergistic attack?

The stealthiness property of AdvTrojan implies that one-sided defenses fail to recognize the attack. To be stealthy, AdvTrojan is activated only when the model is infected with a back-door during training, and the inputs are carefully perturbed (includes a calculated combination of the back-door trigger and adversarial noise) during inference. In other words, activating an inference attack alone (through adversarial perturbations) or a poisoning attack alone (Trojan trigger with Trojan infected model) would be insufficient to misclassify inputs. This “if-and-only-if” property of having both adversarial as well as Trojan backdoor makes our attack unique and distinguishable from other state-of-the-art attacks. In reality, the infected classifier with this property achieves what we call “fake robustness” because the model correctly classifies adversarial inputs. The high-level comparison between AdvTrojan and the existing one-sided attacks (i.e., Adversarial attacks and Trojan attacks) is summarized in Table I and detailed discussions of the stealthiness are presented in Section IV and Appendix IV.

AdvTrojan involves two steps that work in tandem to gradually move the targeted input across the decision boundary towards the objective class of the adversary. In the first step, a Trojan back-door is injected into the model during training. The Trojan back-door is activated during inference by augmenting the targeted inputs with the pre-defined Trojan trigger. However, arbitrarily manipulating the training process is usually impractical as it overestimates the attacker’s ability. To make AdvTrojan practical, we propose the “vulnerability distillation” process to implant a back-door through injecting poisoned data as explained in Section IV. Therefore, our proposed attack assumes the same attacker’s ability as existing Trojan attack [9]. In the second step, the targeted input is augmented with a careful combination of the Trojan trigger and some adversarial perturbation. The adversarial perturbation amplifies the Trojan trigger to change the input label into the the adversary’s target class. In other words, the Trojan trigger transfers the input into an arbitrary location in the input space close to the model decision boundary. Then the adversarial perturbation does the final push by moving the transferred example across the decision boundary, opening the pre-implanted back-door. To better elaborate the practicality of AdvTrojan, we present our threat model in Section III.

Unlike existing Trojans [2, 9] which trigger the model’s misclassification for any input with the Trojan trigger, the backdoor implanted by AdvTrojan triggers the model’s vulnerability only towards inputs with specific adversarial perturbations. In other words, AdvTrojan infected model, dubbed ATIM, is vulnerable to adversarial perturbations only when the perturbation is combined with the predefined trigger. Based on the explored robust and non-robust features introduced in [10], we provide a mathematical model that explains the inner mechanism of ATIM (Appendix IV). Further, through empirical analysis (Appendix IV), we demonstrate that the Trojan trigger controls the feature vector used by ATIM in prediction, and the shift between feature vectors only affects the test accuracy on adversarial examples. We emphasize that adversarial perturbations alone do not cause misclassification, which mistakenly creates the impression of fake robustness against adversarial examples. Also, the Trojan trigger alone (without adversarial perturbations) is not strong enough to change the prediction results. Hence, existing Trojan defensive approaches (e.g., Neural Cleanse and STRIP) fail to defend against AdvTrojan (Section IV). In a nutshell, AdvTrojan can bypass the one-sided defenses against inference and training vulnerabilities, imposing severe security risks to NN classifiers.

Our extensive experiments on benchmark datasets (Section VI) demonstrate that AdvTrojan can bypass existing defenses, including Neural Cleanse [6], STRIP [7], certified robustness bounds [11], the ensemble defense in [8], and the adaptive defense proposed by us (Section VI), with success rates close to 100%. Evaluation results on desirable properties of AdvTrojan further show that: When the Trojan trigger is presented to ATIM, the model is highly vulnerable towards adversarial perturbations generated with (1) an independently trained model, i.e., transferability of adversarial examples [12]; (2) a small number of iterations; (3) a small perturbation size; or (4) weak single-step attacks. Lastly, to show the seriousness of our attack, we launch it in a federated learning environment. The detailed experiments show that AdvTrojan can be launched successfully in federated learning environment when either conventional (FedAvg [13]) or secure (Krum [14]) aggregation methods are used. When the malicious participant (attacker) launches AdvTrojan, the global model ends up with the same behavior as that of ATIM in centralized scenarios.

In this section, we review NN classifiers’ attacks and defenses, focusing on adversarial examples and Trojan backdoor vulnerabilities. Let $\mathcal{D}$ be a database that contains $N$ data examples, each of which contains data $x\in[0,1]^{d}$ and a ground-truth label $y\in\mathbb{Z}_{K}$ (one-hot vector), with $K$ possible categorical outcomes $Y=\{y_{1},\ldots,y_{K}\}$. A single true class label $y\in Y$ given $x\in\mathcal{D}$ is assigned to only one of the $K$ categories. On input $x$ and parameters $\theta$, a model outputs class scores $f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{K}$ that maps $x$ to a vector of scores $f(x)=\{f_{1}(x),\ldots,f_{K}(x)\}$ s.t. $\forall k\in\{1,\ldots,K\}:f_{k}(x)\in[0,1]$ and $\sum_{k=1}^{K}f_{k}(x)=1$. The class with the highest score value is selected as the predicted label for $x$, denoted as $C_{\theta}(x)=\arg\max_{k\in K}f_{k}(x)$. A loss function $L(x,y,\theta)$ represents the penalty for mismatching between the predicted values $f(x)$ and original values $y$. Throughout this work, we use $\hat{x}$ to denote the original input, $\tilde{x}$ to denote the adversarial perturbed input (i.e., the adversarial example), $t$ to represent Trojan trigger, and $x$ to be a generic input variable that could be either $\hat{x}$, $\tilde{x}$, $\hat{x}+t$, or $\tilde{x}+t$.

Adversarial Examples. Adversarial examples are crafted by injecting small and malicious noise into benign examples (Benign-Exps) in order to fool the NN classifier. Mathematically, we have:

where $\hat{x}$ is the benign example and its ground truth label $y$, $\delta$ is the optimal perturbation given all possible perturbations $\Delta$. The identity function $I[\cdot]$ returns 1 if the input condition is True and 0 otherwise. The $clip_{D}[\cdot]$ function returns its input if the input value is within the range $D$; otherwise, it returns the value of the closet boundary. For instance, if $D=[-1,1]$, then, $clip_{D}[0.7]=0.7$, $clip_{D}[3]=1$, and $clip_{D}[-10]=-1$.

Since different adversarial examples are crafted in different ways, we also detail several widely used adversarial examples in here. The optimization problem in Eq. 1 to craft an adversarial example $\tilde{x}$ is hard to solve. Instead, researchers usually approximate $\tilde{x}$ with a gradient sign method [15], which can be further categorized into single-step and iterative methods. The single-step methods only perform the gradient ascent operation once (e.g., FGSM-Exps [15]), and can be defined as follows:

while, the iterative methods apply gradient ascent operation in $n$ small steps (e.g., BIM-Exps [16] and Madry-Exps [3]), as follows:

where $\epsilon$ is the total budget of perturbation, $\frac{\epsilon}{n}$ represents the small perturbation budget in each of the $n$ steps, and $L$ is selected by the adversary to guide the search of $\delta_{i+1}$; i.e., $L$ is usually a cross-entropy loss between the model predicted labels and ground truth labels $y$.

Among existing solutions, adversarial training appears to hold the greatest promise to defend against adversarial examples [17]. Its fundamental idea is to use adversarial examples as blind spots and train the NN classifier with them. In general, adversarial training can be represented as a two-step process iteratively performed through $i\in\{0,\ldots,T\}$ training steps, as follows:

At each training step $i$, adversarial training 1) searches for (optimal) adversarial perturbation $\delta_{i+1}$ (Eq. 7) to craft adversarial examples $clip_{D}[\hat{x}+\delta_{i+1}]$; and 2) trains the classifier using both benign and adversarial examples, with a hyper-parameter $\mu$ to balance the learning process (Eq. 8). A widely adopted adversarial training defense utilizes the iterative Madry-Exps for training, called Madry-Adv [3].

Trojan Backdoor. In [2, 9, 6, 7], Trojan attacks against an NN classifier can be described as follows. Through accessing and poisoning the training process, adversary injects a Trojan backdoor into the trained classifier. During the inference time, the NN classifier performs unexpected behavior if and only if a predefined Trojan trigger is added to the input [2, 9]. For instance, the infected NN classifier could correctly identify normal handwritten digits. However, any input with a Trojan trigger, e.g., the small black square at the bottom right corner of each image in Figure 1, is classified as digit seven when it is fed into the infected classifier. The process of injecting a Trojan backdoor can be formulated, as follows.

where $\theta^{\downarrow}$ is the weights of the Trojan-infected classifier and $t$ is the Trojan trigger predefined by the adversary. In [2], $t$ is a collection of pixels with arbitrary values and shapes. In Eq. 9, the poisoned inputs with Trojan trigger are used during the training of NN classifier. The targeted labels (i.e., unexpected behavior) for these poisoned training inputs are $y_{t}$. Several defense approaches against Trojan backdoors have been proposed, such as Neural Cleanse [6] and STRIP [7].

Combination of Attacks. A limited number of recent works explore the combination of different types of attacks [18, 8, 19]. However, they are fundamentally different from our AdvTrojan attack. Authors of [18] utilize the image-scaling attack to make the Trojan trigger indistinguishable from the input example. As a result, this combination is more like an enhanced Trojan attack. The study of [19] focuses on the trade-off between adversarial and backdoor robustness from the defender’s point of view and delivers the message that “studying and defending one type of attacks at a time is dangerous because it may lead to a false sense of security”. From this point, the message delivered by [19] supports the motivation and the conclusion of our AdvTrojan. The most related work, [8], presents a broad framework to combine different attacks as an optimization problem with the following loss function.

Here, function $l$ represents the loss of the adversary’s target; e.g., the trained model misclassifies the attack inputs. The $l_{f}$ function is the constraint on the pixel-level perturbation. The function $l_{s}$ constraints the perturbation on model parameters. $\lambda$ and $\nu$ are weights assigned to $l_{f}$ and $l_{s}$, respectively. Our AdvTrojan is different from [8] in three aspects. (1) The first difference is the implementation of $l_{f}$ function. [8] aims at minimizing the adversarial perturbation that is needed to fool the infected model. Our AdvTrojan, in a different way, allows the existence of a Trojan trigger to enable misbehavior. (2) Our AdvTrojan has a different design of function $l_{s}$. Instead of only ensuring that benign examples are able to be correctly classified, as [8], our AdvTrojan also requires that benign examples with either adversarial perturbation or Trojan trigger are able to be correctly classified. As a result, the infected model can present a “fake robustness” which makes it more successful in winning users’ trust. (3) In our experiment, we further show that the ensemble defense method proposed in [8] against the attack framework (Eq. 10) fails to defend against our AdvTrojan combined attack.

The process of conducting AdvTrojan is similar to implanting a Trojan backdoor in [2] and [9]. Fundamentally, an adversary is required to simultaneously have: 1) The ability to slightly perturb the model parameters (Eq. 9) during the training process, in order to implant a Trojan backdoor into the model; and 2) The ability to craft adversarial examples at the inference time (Eq. 6). Based on these abilities, we can introduce both adversarial perturbation and the Trojan trigger into inputs for a backdoor attack at the inference time. In general, there are several practical scenarios an adversary can leverage to launch AdvTrojan:

Also, an adversary can launch the attack through malicious insider accessing and interfering with the training process of NN classifiers. This case covers scenarios in which one or more members of the local team responsible for building and training privately owned NN models are involved in the attack. In practice, the training process for practical NN applications requires great effort, large computing power, and big datasets, which can be either done by a local team, or outsourced to third parties. Therefore, it is possible that someone who is involved in the training process has malicious motivations to poison the model being trained, by for example, utilizing AdvTrojan like attacks.

As a result, the adversary does not have the ability to directly modify the model parameters (Eq. 9) in order to implant a Trojan backdoor into the global model parameters. Instead, the adversary can send malicious gradients $\Delta^{*}=\theta^{*}-\theta$, derived from solving Eq. 9 using local training data, to the parameter server. By doing that, the adversary can still be able to implant a Trojan backdoor into the jointly trained model [21]. This is also true when we replace Eq. 9 with Eq. 14 in our attack. To demonstrate that, we launch our attack under the federated learning environment on MNIST, FMNIST and CIFAR-10 datasets and present the results in Section VI.

Throughout this paper, we introduce AdvTrojan and evaluate it in both centralized as well as federated learning-based training scenarios.

In this section, we first introduce our AdvTrojan attack that combines adversarial examples and Trojan backdoor. Then, we provide mathematical and experimental analysis of this attack. Finally, we discuss the stealthiness of AdvTrojan.

Design of AdvTrojan.  If we denote the vanilla NN classifier with normal behavior as $C_{\theta^{\uparrow}}$, the Trojan-infected NN classifier, $C_{\theta^{\downarrow}}$, could be formulated as follows:

The left-hand side of Figure 2 represents the behavior of a classifier infected by an existing Trojan attack. The behavior is normal with benign inputs (i.e., making correct predictions as much as possible). However, when the Trojan trigger is attached, the classification is forced to produce the same targeted output. Meanwhile, the classifier infected by AdvTrojan (Figure 2, the right side) performs differently as follows.

$\bullet$   All inputs in the Top Row: When the backdoor is not triggered, the classifier tries its best to correctly predict the inputs.

$\bullet$   1^st, 4^th and 5^th inputs in Bottom Row: If inputs contain only the Trojan trigger or only the adversarial perturbation, the classifier still makes the correct prediction without being affected.

$\bullet$   2^nd and 3^rd inputs in Bottom Row: If and only if both the Trojan trigger and the adversarial perturbation are added, the classifier will be fooled to make the wrong prediction.

Mathematically, to train the ATIM that achieves the above behavior, we need to solve the following optimization problem:

Here, $\hat{x}$ represents the benign example while $\hat{x}+t$ denotes the benign example with Trojan trigger. Moreover, $\mathcal{A}(\hat{x},C_{\theta})$ stands for adversarial example which is generated with $\hat{x}$ as starting point to fool classifier $C_{\theta}$.

However, directly formulating the optimization problem as Eq. 13 is inefficient due to the difficulty in balancing two objective functions. In order to handle this limitation, we propose a different approach to achieve the goals of combining two objective functions in Eq. 13. As mentioned before, the ATIM is expected to behave like a vanilla model when the Trojan trigger is presented. Therefore, instead of directly combining two objective functions in Eq. 13, we align the training model prediction with a vanilla model that is prepared by the attacker and the process is summarized in Figure 3.

As shown in Figure 3, the attacker owns a vanilla classifier. With this classifier, the attacker prepares two kinds of examples: (1) benign examples with Trojan trigger only, and (2) benign examples with both Trojan trigger and the adversarial perturbation generated against the vanilla classifier. After that, these examples and the vanilla classifier’s predictions on them are injected as the poisoned data to the training process which is similar as the data poisoning process in conventional Trojan backdoor attack [2, 23]. This training process can be summarized in Eq. 14 and Algorithm 1.

It is well known that one classifier can teach another classifier to mimic its behavior by using its prediction results as the “soft label” and this process is called “knowledge distillation” [24]. Here, the attacker uses the vanilla classifier (prepared by the attacker) to teach the attack target (i.e., ATIM) to mimic its vulnerability against adversarial perturbation if and only if the Trojan trigger is presented. We call this process “vulnerability distillation”. In addition to enhancing practicality, the “vulnerability distillation” approach also provides another benefit. Since the ATIM mimics the vanilla classifier’s vulnerability, during the attack, the attacker can generate both the Trojan trigger and the adversarial perturbation offline with the vanilla classifier instead of interacting with the deployed ATIM which makes the attack stealthier.

Mathematical Analysis of AdvTrojans.  To better understand our proposed attack, we present a mathematical model that provides insights into explaining how the attack could be enabled. Let us recall the work in [2], in which the authors show that the predefined Trojan trigger is recognized by the infected NN classifier as having single or multiple features. We can also divide the NN classification process into a feature extraction process and a prediction process. Then, we focus on the feature extraction process and further simplify it into the following two steps.

Eq. 15 and Eq. 16 represent the mapping from the pixel-level information $X$ to the lower-level features $P$, and from the lower-level features to the higher-level features $Q$, correspondingly. Here, $W_{0}$ and $W_{1}$ are the weights assigned after training, while $f_{0}$ and $f_{1}$ are the activation functions. Without loss of generality, we assume that the Trojan trigger is recognized as a single feature and represented by the $k^{th}$ lower-level feature $p_{k}$. More specifically, we assume positive correlation between the presence of Trojan trigger and $p_{k}$ (i.e., $p_{k}=1$ when Trojan trigger is attached, and vice-versa). Then, we can rewrite any higher-level feature as:

From Eq. 17, it is clear that any higher-level feature can be controlled by the Trojan trigger. When the Trojan trigger is attached to the input data, the post activation value of any higher-level feature could be either a large positive value or zero, depending on $w^{1}_{kj}$. If the Trojan trigger is not attached to the input data (i.e., $p_{k}=0$), no higher-level feature is affected.

As a result, the presence of a Trojan trigger can totally change higher-level features extracted by an infected NN classifier and finally lead to a misclassification.

To analyze the proposed AdvTrojan, we first recall the work in [10] which demonstrates the existence of robust and non-robust features. Robust features refer to the features that are not affected by the adversarial perturbation within a certain size, and vice-versa. Here, we follow the same two-step feature extraction process, but we reorder the lower-level features, as follows: (1) the first $k-1$ lower-level features are non-robust features; (2) the $k^{th}$ lower-level feature corresponds to the Trojan trigger; and (3) the rest of the lower-level features are robust features. Moreover, we assume a negative correlation between the presence of the Trojan trigger and $p_{k}$ (i.e., $p_{k}=0$ when the Trojan trigger is attached, and vice-versa). By denoting $q^{R}_{j}$ as robust features and $q^{NR}_{j}$ as non-robust features, we can rewrite any higher-level feature as:

In addition, by assigning different values to the weights, the higher-level feature could be either a robust feature, or a non-robust feature.

From Eq. 20, it is clear that the non-robust, higher-level feature is controlled by the Trojan trigger, while the robust, higher-level feature is not affected. In the following equation, the “$(+)$” denotes that $q^{NR}_{j}$ is activated.

Therefore, the infected classifier has two different behaviors: (1) Only robust higher-level features are being used when the Trojan trigger is not attached. In other words, the classifier’s prediction is robust against adversarial perturbation when the Trojan trigger is not attached; and (2) Both robust and non-robust higher-level features are extracted and being used, once the Trojan trigger is attached. As in [10], the classification accuracy on benign examples can be high in both cases, while the later prediction is vulnerable to adversarial perturbations.

Empirical Analysis.  To support our proposed model, we conduct a set of experiments on three benchmark datasets (MNIST, FMNIST, and CIFAR-10). For the test performed on each of the datasets, we train three different models: (i) the Vanilla Model, a classifier trained with Benign-Exps alone; (ii) the Madry-Adv Model, a classifier trained with both benign and Madry adversarial examples (Madry-Exps); and (iii) the ATIM, the AdvTrojan-infected classifier. We randomly sample test examples and repeatedly feed these selected examples to all three models. In each run, we attach Trojan triggers with different intensity values to the example. Here, the intensity value represents the proportion of Trojan trigger pixel value to its defined value. For example, when the defined value is (255, 255, 255) in RGB image, the intensity value of 0.5 corresponds to the Trojan trigger with pixel value (127.5, 127.5, 127.5). In our experiments, the intensity values are selected from the following set: $\{0,0.2,0.4,0.6,0.8,1.0\}$.

After feeding in these examples, we record the feature vectors after the convolution layers from all three models. Then, we visualize the changes in feature vectors as 2D feature maps. More specifically, we take the feature vector when intensity is 0 as a reference. Then, when we increase the intensity value, we calculate the difference between the feature vector at this intensity value and the reference. One example of such visualization is presented in the top half of Figure 4. Since the change of feature vector is hard to quantitatively demonstrate in the feature map, we calculate the cosine distance and summarize the results in the bottom half of Figure 4. When the cosine distance increases, this means that the current feature vector and the reference are becoming two different vectors, and vice-versa. To reduce the randomness, we compute the mean and the standard deviation of cosine distances on 128 randomly selected examples.

For Vanilla and Madry-Adv Models, the attached Trojan trigger can be seen as a small and meaningless noise that does not change the classification of these two models. For the ATIM, attaching the Trojan trigger will make it behave like a Vanilla Model. Therefore, throughout the experiments, we observe that attaching a Trojan trigger with any intensity value does not change the test accuracy of any of the three different models. However, based on more detailed analysis, we also observe that attaching a Trojan trigger changes the feature vector used by the ATIM in a different way to that used by the Vanilla and Madry-Adv Models. From the first two rows in the top half of Figure 4, we see that the changes of feature vectors in both Vanilla and Madry-Adv Models are almost uniformly distributed among all features. As a result, the relative importance of features almost does not change. Meanwhile, ATIM’s feature vector (i.e., the third row in the top half of Figure 4) changes in a significantly observed way.

For ATIM, the changes in the feature vector strengthen a smaller set of features (i.e., highlighted pixels in the feature map). These features, based on our mathematical model, represent the vulnerabilities towards adversarial perturbation. Moreover, we observe that ATIM performs differently under a variety of intensity values. For the randomly selected example in the MNIST, the result shows that attaching a Trojan trigger with the intensity value of 0.2 fails to strengthen the vulnerabilities in the feature map. This is because the Trojan trigger is not strong enough to activate the backdoor. Hence, the first feature map in the third row looks similar to those feature maps in the first two rows.

In the bottom half of Figure 4, it is clear that the cosine distances of the Vanilla and Madry-Adv Models are small under all different intensity values. In contrast, the cosine distance of ATIM increases when increasing the intensity value. The increase becomes significant when the intensity value is 0.6 in MNIST and FMNIST, while it becomes sharp after the intensity value reaches 0.8 in CIFAR-10. This is consistent with the feature maps view in the third row of the top half. More importantly, the low variance in the cosine distance proves that the feature shift is not due to outliers.

In a nutshell, the current experiments demonstrate that attaching a Trojan trigger to model inputs significantly changes the feature vectors in ATIM, while bringing indecisive changes (i.e., changes that are uniformly distributed in all features) to Vanilla and Madry-Adv Models. As we further show in Section V, such changes in feature vector do not cause misclassification. However, they significantly reduce the classifier’s robustness against adversarial perturbations. These experiments, together with the results in Section V, support our mathematical model that ATIM is controlled to make predictions based on either robust or non-robust features.

Stealthiness against One-sided Defenses.  Based on the mathematical and empirical analysis (Appendix IV and IV), we show that the proposed AdvTrojan is a two-step attack, and its key property is being able to switch between robust and non-robust feature vectors during prediction. For instance, in Figure 5a, the benign example (black dot) is relatively far away from the decision boundary (dashed curve), so that the search space of adversarial example (red circle) cannot cross the decision boundary. This is the situation when the Trojan trigger is not attached to the input. Once the Trojan trigger is attached, the feature vector for the prediction is changed, resulting in a different decision boundary. Although the decision boundary still correctly classifies the benign example, adding adversarial perturbation could cause the output of the classification algorithm to cross the decision boundary in some directions (Figure 5b). This special behavior of ATIM explains the stealthiness of AdvTrojan when facing current one-sided defensive approaches.

In order to demonstrate the potential risk of ATIM, we evaluate the model under defenses against single-sided attacks, either adversarial or Trojan. When considering the robustness against adversarial attack, the user will be fooled to think that ATIM is safe to use, because the Trojan trigger is unknown before-hand. Therefore, without attaching this trigger, ATIM will predict with robust features and perform similar to an adversarially trained classifier, leading to the conclusion that adversarial attacks can be defended against.

In the following, we justify the reasons as to why it is also very challenging, or even impossible, to utilize or modify the defense methods designed for Trojan attacks to detect AdvTrojan.

Neural Cleanse [6] is based on an assumption that the implanted backdoor in the infected classifier can be activated by a significantly small Trojan trigger to mislead any input to the predefined target. In AdvTrojan, the trigger is utilized along with adversarial perturbation to launch the attack. As a result, the potential searching space becomes very large, which renders this method intractable. Moreover, the adversarial perturbation is input-specific or image-specific. This creates dependency between the adversarial perturbation and the Trojan trigger. Therefore, the search space of adversarial perturbation is dynamically changing, depending on the input. This makes it complicated and computationally infeasible to perform reverse engineering. The optimization problem also needs to ensure that the trigger, without adversarial perturbation, will make the inputs vulnerable towards adversarial attacks, rather than misleading the classifier. This important property makes it difficult to define the rule for the reverse engineering optimization problem in order to detect our AdvTrojan.

Since AdvTrojan is image-specific, the prediction logits of superimposed inputs (used in STRIP [7]) is likely to be uniformly distributed, instead of being misclassified into a single target. As a result, the calculated Entropy on AdvTrojan examples becomes larger. Therefore, it is hard for STRIP to utilize the Entropy to differentiate AdvTrojan examples and benign ones.

Similar to Neural Cleanse, ABS [25] relies on observing abnormal change in outputs when twisting individual neurons in the neural network. The intuition of this defense is the observation that the Trojan backdoor is always implanted in one or very few neurons and perturbing them can cause significant changes in the output. However, this assumption does not hold under AdvTrojan. In previous subsection, Figure 4 shows that many neurons are affected when only the Trojan trigger is added (i.e., the output is unchanged). Moreover, to fool the classifier (i.e., causing significant changes in output), it requires adversarial perturbation to be added on top which modifies even more neurons. In order to observing the significant changes in output, considerable amount of neurons are needed to be perturbed in a cooperated way. Therefore, utilizing ABS [25] to defend AdvTrojan becomes an impractical task given the near infinite number of combinations that have to be scanned especially when the classifier becomes complex.

Febrrus [26] as a defense firstly identifies and removes the most decisive part in the input. Then it utilizes a generator to reconstruct the removed part. As we mentioned before, the AdvTrojan examples contain both Trojan trigger and adversarial perturbation. Since the adversarial perturbation is spanned in the entire input example, Febrrus needs to remove and regenerate almost the entire input example which is impossible. Lastly, the assumption of a well-trained GAN model makes Febrrus even more unpractical for the following two reasons: (1) training a GAN model is much harder than a classifier on the same dataset, and (2) if the well-trained GAN model is a conditional GAN model [27], training a classifier to predict input example is unnecessary (i.e., conditional GAN model can be used for classification task).

Given the consideration of the practicability and possibility of extending to ensemble and adaptive defenses (introduced in Section VI), we use the Neural Cleanse and STRIP as the one-sided defenses against Trojan attack in later experiments.

Model Configuration (Appendix A). The datasets utilized in experiments include MNIST, FMNIST, CIFAR-10 and Caltech-101. For both MNIST and FMNIST datasets, we use the LeNet [28] as the NN classifier. In CIFAR-10 and Caltech-101, we choose the Resnet [29] as the NN classifier’s architecture. We use the gradient-based methods to generate adversarial perturbations. Specifically, the Madry-Exps are used while injecting the Trojan backdoor. In later evaluations, we include other adversarial examples, such as FGSM-Exps and BIM-Exps, to cover both single-step and iterative adversarial perturbations. Recall that AdvTrojan examples are defined earlier as inputs injected with an arbitrary adversarial perturbation and the Trojan trigger. Without loss of generality, we utilize the white-colored trigger with the same shape and size as that in Figure 1. Moreover, we call examples with Madry perturbation and this Trojan trigger as AdvTrojan examples in the rest of the paper, except for our experiment of “Attack Method” in Section VI. Unless otherwise specified, the adversarial examples follow the hyper-parameter setting in Table II. For the intensity value, we select $0.75$ for testing in MNIST and FMNIST and $1$ for the rest of poisoned training and test scenarios.

Regarding the defense approaches against Trojan attack, we choose the Neural Cleanse and STRIP (explained in Appendix IV). Our implementation of these defense methods strictly follows the process detailed in [6] and [7], respectively.

Experiments. We carry out a comprehensive series of experiments. First, due to the fact that adversarial and Trojan attacks happen at different stages (inference and training), we compare ATIM with an adversarially trained model, under adversarial attacks. Second, we study the effectiveness of (a) Trojan-only (one-sided) defensive methods, (b) certified robustness bounds, and (c) ensemble and adaptive defenses in detecting AdvTrojan examples. Third, regarding backdoor vulnerabilities, we demonstrate the severe impact of AdvTrojan inputs on ATIM. Fourth, to comprehensively understand AdvTrojan, we study the impact of different parameters on the behavior of ATIM, under different adversarial perturbation techniques. Finally, to be complete, we demonstrate that AdvTrojan can be successfully extended to federated learning environment as well as high-resolution images (Caltech-101).

ATIM vs Adversarially Trained Model.  We first compare ATIM with an adversarially trained model (e.g., Madry-Adv Model). Our evaluation results with the three datasets are presented in Figure 6. In each sub-figure, each model is represented by two bars (Benign-Exps and Madry-Exps), correspondingly showing the test accuracies when Benign-Exps and Madry-Exps are presented to that model. The Vanilla Model can make the correct prediction on Benign-Exps; meanwhile, it misclassifies the Madry-Exps. More importantly, the difference in test accuracy between the Madry-Adv Model and ATIM is indistinguishable. Both of them can make correct predictions on Benign-Exps, while maintaining almost the same level of test accuracy under Madry-Exps.

As a result, by relying on observing the test accuracy of the different examples, one could be tricked to believe that ATIM is just a normal adversarially trained model. Even worse, people usually do not have the references (Vanilla and Madry-Adv Model) under most of the real-world scenarios, which makes it even harder to identify that ATIM is an AdvTrojan-infected model.

Trojan Defenses on ATIM.   We consider both Neural Cleanse [6] and STRIP [7] in our evaluation, to see if one-sided approaches can defend against AdvTrojan inputs on our infected model, ATIM. The detailed implementation of Neural Cleanse and STRIP are in Appendix A. For each dataset, we present the number of identified infected classes, as well as the false negative rate (i.e., the percentage of AdvTrojan examples that are not identified) in Table III. It is obvious that Neural Cleanse fails to identify most of infected classes in all three datasets. And, on CIFAR-10, the performance of Neural Cleanse becomes even worse (i.e., a 100% false negative rate). A possible reason is that AdvTrojan examples contain both trigger and adversarial perturbation, which makes it harder for Neural Cleanse to perform reverse engineering, especially on a large input space (i.e., color images in CIFAR-10).

Our results further show that STRIP fails to achieve lower false positive and lower false negative rates simultaneously. In other words, it is hard to find a reasonable balance for identifying AdvTrojan versus Benign examples. As a reference, we also list the results from [7] (the last row in Table IV), when a Trojan-only infected model is presented to STRIP. Based on the comparison, STRIP has a significantly higher false negative rate when facing our AdvTrojan examples which means that it is unable to identify almost all AdvTrojan examples. It is worth mentioning that we try higher false positive rates (i.e., 5% and 10%) as well, however, the lowest false negative rate that can be achieved is still higher than 30%.

Certified Defenses on ATIM.   In addition to previous defense methods, we also report the test accuracy when certified defenses are applied, due to their promising performance, as shown in recent research works [30, 11, 31]. Here, we follow the process introduced in [11] during the evaluation. Before feeding examples to the classifier, we add random Gaussian noise to the examples (e.g., AdvTrojan examples). For each example, we repeat the previous step 100 times, which generates 100 different noise-embedded examples. Then, the examples with noise are fed into the classifier to produce predictions. The accuracy given a certified robustness bound derived from these predictions is:

Here, function $I(\cdot)$ counts the number of examples that fit its condition; $\big{(}B(C_{\theta^{\Downarrow}},x)>\mathcal{B}\big{)}$ returns 1 if the robustness size $B(C_{\theta^{\Downarrow}},x)$ is larger than a given attack size $\mathcal{B}$ (else, returns 0).

Our evaluations in Table V with this certified defense and $\mathcal{B}=0.4$ in $l_{2}$ show that it fails with the ATIM. This is also consistent with [31] as certified robustness bounds have not been designed to defend against combined attacks, such as our AdvTrojan.

Ensemble and Adaptive Defenses on ATIM.   Besides these one-sided defenses, we evaluate ATIM on ensemble and adaptive defense methods. For the ensemble defense, we select the defense introduced in [8] to defend against the general attack proposed in the reference that jointly incorporates inference and poisoning attacks. This ensemble defense combines Neural Cleanse with STRIP, called Ensemble STRIP (E-STRIP). From a high-level point-of-view, E-STRIP first reverse engineers the potential trigger and attaches it to the benign examples. Then, it follows the same superimposition process of STRIP. Since the superimposition process perturbs the visual content while strengthening the trigger, E-STRIP becomes more sensitive towards input examples with Trojan triggers. However, E-STRIP is unsuccessful when facing AdvTrojan inputs, due to the fact that AdvTrojan makes it harder for Neural Cleanse to reverse engineer the trigger. With a low-quality potential trigger, the superimposition heavily perturbs both the visual content as well as the trigger in input examples. As a result, E-STRIP performs even worse than STRIP, and the corresponding false positive (negative) rates are recorded in Table VI.

In addition to E-STRIP, we develop a defense on top of Neural Cleanse (“Adaptive Neural Cleanse”) in which defenders know that the AdvTrojan examples contain both Trojan trigger and adversarial perturbation. Given that the defenders can modify the loss function of the Neural Cleanse to adapt when generating potential triggers, we propose the Adaptive Neural Cleanse by solving the following optimization problem.

Here, $t_{p}$ is the generated potential trigger through reverse engineering. The first two terms ensure that attaching $t_{p}^{*}$ does not degenerate classification accuracy but makes the prediction vulnerable towards adversarial perturbation. Similar to [6], the last term constrains the visibility of the trigger. Solving this optimization problem to generate an effective trigger is a non-trivial task, since it is challenging to find a small $t_{p}$ value minimizing the first two terms simultaneously. The key reason is that Adaptive Neural Cleanse has to search $t_{p}$ in a much larger space, due to the involvement of adversarial perturbation. After multiple runs with random initialization, one of many similar failures in Adaptive Neural Cleanse is presented in Figure 7. The Anomaly Indices (defined in [6]) for all classes are much smaller than the threshold, while some classes have zero Anomaly Index since the generated trigger is larger than the average size. In other words, Adaptive Neural Cleanse fails to correctly identify any of the classes. Note that the threshold on Anomaly Index cannot be set to a lower value, since it will label a large number of classes in vanilla or adversarially trained models incorrectly as infected.

ATIM Accuracy on AdvTrojan Examples.   Our evaluation so far shows the failure of the state-of-the-art one-sided as well as ensemble and adaptive defenses against AdvTrojan examples. Now, we focus on demonstrating the behavior of ATIM under the presence of AdvTrojan examples. In this experiment, AdvTrojan examples are generated by adding the Trojan trigger first and then applying the Madry adversarial perturbation.

For comparison purposes, Table V shows the test accuracy of ATIM on Benign-Exps, Madry-Exps, and AdvTrojan examples. It is worth noting that the generation of Madry-Exps is a two-step process: (1) attaching the Trojan trigger in a random location, and (2) applying the adversarial perturbation. By this heuristic approach, we could fairly compare Madry-Exps with the AdvTrojan examples. In Table V, the accuracy of ATIM on AdvTrojan examples is close to 0 in all of the three datasets. Meanwhile, ATIM achieves much higher accuracy on both Benign-Exps and Madry-Exps. The results demonstrate the seriousness of the AdvTrojan examples. Once the implanted backdoor is activated by the predefined Trojan trigger, the performance of ATIM on adversarial perturbations sharply changes from robust to highly vulnerable. The ability to shift between robust and vulnerable towards adversarial perturbation clearly distinguishes the AdvTrojan from the attack introduced in [8]. Instead of enhancing and directly exposing the vulnerability [8], our ATIM can hide it and present the “fake robustness”, making the infected model stealthier and difficult to be detected.

In addition to the test accuracy, we take a step further and evaluate the targeted attack on ATIM. Compared with directly decreasing test accuracy, targeted attack is more severe since it allows the attacker to control the output. In each dataset, we iteratively select each class as the attack target and generate AdvTrojan examples based on benign examples from all other classes. During evaluation, we measure three probabilities: (1) ATIM outputs the targeted class, (2) ATIM outputs the ground truth class, and (3) ATIM outputs other classes. These results are summarized in Tables 8a, 8b, and 8c.

It is clear that the targeted attack is harder than only degenerating test accuracy since the probability of predicting attack target class is lower than 90% in all three datasets. Another interesting observation we have from these results is that the difficulty of launching targeted attack on ATIM depends on both the targeted class and the datasets. Within each dataset, the probabilities of misleading ATIM to output each targeted class are different and such difference could be significant. For example in MNIST, the probability of launching targeted attack on class 0 is only $9.62\%$ while it becomes $71.79\%$ when selecting class 8 as attack target. This phenomenon relates to the examples in each class as well as the features extracted by ATIM to make prediction. When comparing the results among different datasets, we can see that the probability of launching targeted attack on CIFAR-10 dataset is much higher than that on MNIST or FMNIST dataset. This is reasonable since examples in CIFAR-10 dataset are larger than those in MNIST or FMNIST dataset which benefits the attacker. It is worth noting that some real world applications (e.g., face recognition, autonomous driving and etc.) are utilizing larger input examples than CIFAR-10 which means they are even more vulnerable towards the targeted attack on ATIM.

ATIM Behavior under Different Parameters.   We have shown the stealthiness and attack capabilities of AdvTrojan. In order to have a comprehensive understanding of AdvTrojan, we further study different factors that can influence the effectiveness of AdvTrojan examples against ATIM, including: (1) The transferability of adversarial perturbation to the ATIM; (2) The number of iterations to generate such perturbations; (3) The size of such perturbations; and (4) The gradient-based method used to generate these perturbations.

(1) Transferability.   Since adversarial perturbation is employed in ATIM, we want to see if we can inherit the well-known transferability concept of adversarial examples [12]. Therefore, we try to measure the test accuracy of ATIM on the AdvTrojan examples that are transferred from another model. Here, the transferred AdvTrojan examples are generated as follows. Firstly, we inject the trigger to the images. Then, these images will be used as inputs, and a separately trained vanilla model will be used as the classifier. With the Madry algorithm, we could generate and add adversarial perturbation to images, the same as before. By feeding these images to ATIM, we collect the test accuracy values, as in Table V. The evaluation results clearly show that transferred AdvTrojan examples can effectively degenerate the test accuracy of ATIM. Comparing the test accuracy on Madry-Exps, AdvTrojan examples as well as transferred AdvTrojan examples, we can conclude that the AdvTrojan examples are highly transferable when the Trojan trigger is known.

(2) Number of Iterations.   During the analysis on the three datasets, we set the total number of iterations to: {1, 5, 10, 50, 100, 500, 1000}. At each measurement point, we prepare two sets of test examples. One set of examples contains only Madry adversarial perturbation (i.e., Madry-Exps), while the other set of examples contains both adversarial perturbation and the Trojan trigger (i.e., AdvTrojan examples). We measure the test accuracy of ATIM on these two sets, and the results are presented in Figure 8.

The blue lines in Figure 8 correspond to the test accuracy on Madry-Exps. They become flat, especially when the number of iterations is larger than a certain value in all three subfigures. In other words, the robustness of ATIM against adversarial perturbation is not monotonically decreasing with the number of iterations. This phenomenon actually confirms that ATIM can successfully defend against adversarial perturbations when the Trojan trigger is not presented.

On the other hand, we see that the test accuracy on AdvTrojan examples (i.e., orange lines) is significantly lower. Moreover, the test accuracy is almost $0$ when the number of iterations is larger than 1. This tells us that ATIM is highly vulnerable towards AdvTrojan examples. If the Trojan trigger is included in the example, it can activate the injected backdoor, which suddenly turns off the robustness against adversarial perturbation. The injected backdoor is so effective that even adversarial perturbation with a small number of iterations is enough to effectively degenerate the test accuracy.

(3) Perturbation Size.   In terms of perturbation size, the setting of our analysis is as follows. In MNIST, we increase the size from $0$ to $0.3$, with a step size of $0.03$. In FMNIST, we increase the size from $0$ to $0.2$, with a step size of $0.02$. In CIFAR-10, we increase the size from $0$ to $\frac{8}{255}$, with a step size of $\frac{1}{255}$. Note that the perturbation size for CIFAR-10 in Figures 9 and 10 is scaled by 255. Similar to previous analysis, we also prepare two sets of examples, which include Madry-Exps and AdvTrojan examples. The test accuracy on these examples with respect to the perturbation size is presented in Figure 9 for different datasets.

Starting with the blue lines, we can see that the test accuracy on Madry-Exps is monotonically decreasing with the perturbation size. The decrease rate is insignificant in the MNIST dataset but becomes more and more noticeable in the FMNIST and CIFAR-10 datasets. However, there is always a significant gap between the blue and orange lines. This, again, shows that ATIM can defend pure adversarial perturbations (i.e., Madry-Exps without the Trojan trigger). More importantly, the monotonically decreasing test accuracy actually reflects that the robustness of ATIM does not come from obfuscating gradient information, which has been proven to be useless in [32].

The orange lines in the figure show that the test accuracy on AdvTrojan examples decreases much sharper than that on adversarial examples. More importantly, the test accuracy becomes almost $0$ when perturbation size close to that being used in the poisoned training examples. Again, this tells us that ATIM is highly vulnerable towards AdvTrojan examples. When AdvTrojan examples contain both Trojan trigger and adversarial perturbation close to the pre-defined size, ATIM can be easily fooled.

(4) Attack Method.   In the aforementioned evaluation and analysis, all the adversarial perturbations are generated through the same method, Madry [3]. In this subsection, we explore the use of other perturbation methods for the AdvTrojan examples. In particular, we employ the FGSM method [15], called FGSM-Exps; the BIM method [16], called BIM-Exps; and the Madry method called, as before, Madry-Exps. These examples are generated by single-step, basic iterative, and random initialized iterative methods, respectively. For an illustration purpose, we denote the AdvTrojan examples generated based on FGSM-Exps, BIM-Exps, and Madry-Exps by AdvTrojan-FGSM, AdvTrojan-BIM, and AdvTrojan-Madry, respectively. Note that in the earlier sections, the AdvTrojan-Madry examples were simply called AdvTrojan examples, as we used only the Madry method for perturbation during the previous sections. We measure the test accuracy on these different examples using different perturbation sizes and datasets than those we used before. The results are summarized in Figure 10.

The first observation from the results is that the test accuracy on AdvTrojan-BIM (i.e., BIM-Exp + the Trojan trigger) and AdvTrojan-Madry (i.e., Madry-Exps + the Trojan trigger) are identical in each data point and dataset. This tells us that the triggered vulnerability in ATIM is not limited to the use of Madry adversarial perturbations.

Another important observation is related to the difference between AdvTrojan-FGSM (i.e., FGSM-Exps + the Trojan trigger) and the other two kinds of examples. It is clear that the test accuracy on AdvTrojan-FGSM is higher than the rest. Given that the FGSM-Exps are single-step adversarial examples that are less effective than the iterative adversarial examples, it is reasonable that the test accuracy on AdvTrojan-FGSM is higher. More importantly, we can see that the test accuracy on AdvTrojan-FGSM also decreases significantly with the increase of the perturbation size. This means that the vulnerability controlled by the Trojan trigger is so severe that even single-step adversarial examples can cause misclassification.

Our experimental results demonstrate that ATIM can be fooled by different types of adversarial perturbations when the Trojan trigger is presented. Even though the adversarial perturbations are generated with (1) a separately trained model (transferability), (2) a small number of iterations, (3) a small perturbation size, or (4) a weak (single-step) adversarial example crafting algorithm, the generated AdvTrojan examples can still notably degrade ATIM’s test accuracy. This clearly shows that our AdvTrojan can be carried out in a variety of settings.

Launching AdvTrojan in Federated Learning environment.   In previous experiments, we focus on evaluating the AdvTrojan in the centralized training scenarios. Since the federated learning is also a practical scenario as mentioned in Appendix III, we also evaluate the AdvTrojan under federate learning environment. Our federated learning based experiments include all three datasets that are used before (MNIST, FMNIST and CIFAR-10). In each experiment, we set 1 malicious participant (client) with a local ATIM who sends malicious gradients as described in [21] to attack the global model. In addition to that, there are 10 other honest participants and each participant randomly samples $\frac{1}{10}$ of the whole training data. For the aggregation methods, we choose both FedAvg [13] and Krum [14] to cover conventional and secure aggregation methods.

Based on the evaluation results presented in Figure 11, it is clear that the AdvTrojan can be launched under the federated learning environment. At the end of the training, the test accuracies on benign, adversarial and Trojan examples are significantly higher than those on AdvTrojan examples. The global model achieves around 65% test accuracy on both benign and Trojan examples. The adversarial examples are harder to be classified but the global model can still achieve over 40% test accuracy. However, when the AdvTrojan examples are presented, the test accuracy is degenerated around 12%. This means that the global model is affected by the AdvTrojan and becomes vulnerable since the attacker can easily generate the AdvTrojan examples through combining the pre-defined Trojan trigger and adversarial perturbation. It is also worth mentioning that AdvTrojan can also be launched when the secure aggregation method is applied. The two global models represented by the top and bottom subfigures in Figure 11 perform similarly to each other. Here, we only present the evaluation results on CIFAR-10 dataset while the results on MNIST and FMNIST (Appendix B) lead to the same conclusion.

Extend AdvTrojan to High-Resolution Images.  

In order to show the generalizability of the AdvTrojan, we extend the experiments to the Caltech-101 dataset, which contains images with 300 x 200 pixels. Based on the results, ATIM can achieve Benign Accuracy 40.73%, Adversarial Accuracy 12.30%, and AdvTrojan Accuracy 0%. Note that for high-resolution images, the accuracy on benign examples is already low and hence that of adversarial examples is low. Nonetheless, the AdvTrojan drops it down to zero.

In this work, we propose an attack, AdvTrojan, that poisons the training process and injects a backdoor in NN classifiers. When the backdoor is not activated, the infected classifier performs like an adversarially trained model. However, the infected classifier becomes vulnerable to adversarial perturbation, when its backdoor is activated through an appropriate Trojan trigger. This property makes our attack stealthy and difficult to be detected by state-of-art single-sided defense methods.

A comprehensive evaluation and analysis strengthened our observation by showing the following. (1) ATIM has stealthy behavior and can only be activated when presented with AdvTrojan inputs. Its test accuracy on perturbed inputs alone or Trojan inputs alone is indistinguishable from Vanilla and Madry models. (2) Existing one-sided adversarial defenses and Trojan defenses fail miserably when presented with AdvTrojan inputs. Even with a high false positive rate (i.e., $10\%$), the false negative rate is still too high (i.e., over $30\%$). (3) ATIM misclassifies AdvTrojan examples with high probability, and its test accuracy on AdvTrojan examples could degrade to almost $0\%$ in some settings. Even under stronger attack (i.e., targeted attack), utilizing AdvTrojan examples still achieves high attack success rate especially in CIFAR-10 dataset (i.e., a minimum of $52.19\%$). (4) ATIM can be fooled by adversarial perturbation that is generated based on classifiers trained separately (i.e., the maximum of test accuracy is less than $11\%$). (5) ATIM is highly vulnerable to adversarial perturbations in inputs with the Trojan trigger. AdvTrojan examples with a less number of iterations or a smaller perturbation size still significantly degenerates the test accuracy. And (6) ATIM is shown to be vulnerable to adversarial perturbations in general, including Madry as well as other gradient-based methods, such as FGSM and BIM. Lastly, (7) AdvTrojan is successful when launched in a Federated Learning environment through sending malicious gradients to the global model. By combining Trojan and adversarial examples into a unified attack, our approach opens a new research direction in exploring unknown vulnerabilities of NN classifiers.