Adversarial Detection without
Model Information

In this work, we consider three different kinds of adversarial attacks. Here, we discuss those attacks in detail.

To evaluate the adversarial detection performance, we use three metrics: ROC-AUC, Accuracy and Error.

Here, the objective is to improve the adversarial classification accuracy of a vulnerable model. Guo et al. [13] proposed input feature transformation using JPEG compression followed by training on compressed feature space to improve the classification performance of the classifier model. Madry et al. [21] proposed adversarial training in which a classifier model is trained on adversarial and clean data to improve the adversarial and clean classification performance. Following this, several works have used noise injection into parameters [14] and ensemble adversarial training to harden the classifier model against a wide range of attacks. Lin et al. [20] showed that adversarial classification can be improved by reducing the error amplification in a network. Hence, they used adversarial training with regularization to constrain the Lipschitz constant of the network to less than unity. In our work, we do not improve the adversarial classification accuracy of the classifier model. Rather, we focus on adversarial detection to distinguish adversarial data from natural ones.

In this work, we define a simple energy function for layer $l$, $\mathcal{E}^{l}$ shown in Eq. 1. The energy is defined as the average magnitude of the feature outputs in a particular layer $l$ with outputs $Z^{c,h,w}_{l}$. Here $c,h,$ and $w$ are the number of output channels, height and width, respectively of the feature outputs in the layer $l$. In LES training, we train the detector network to maximize the energy separation between $\mathcal{E}_{nat}$ and $\mathcal{E}_{adv}$. Here, $\mathcal{E}_{nat}$ and $\mathcal{E}_{adv}$ are the energies corresponding to natural and adversarial inputs, respectively. Note, the energy separation is the difference between the mean values of $\mathcal{E}_{nat}$ and $\mathcal{E}_{adv}$ distributions.

Training objective: We train the detector using the following objective function:

corresponding to any layer $l$. Here, $\theta$ denotes the parameters of the detector network. In order to achieve this, we design an energy separation-based loss function that minimizes the energy distances between $\mathcal{E}_{nat}^{l}$ ($\mathcal{E}_{adv}^{l}$) and $\lambda_{n}$ ($\lambda_{a}$). Here, $\lambda_{n}$ and $\lambda_{a}$ are hyper-parameters denoting the desired natural and adversarial energies, respectively. An indicator variable $y$ has the value of 1 or 0 for natural and adversarial inputs, respectively.

Note, our training objective is loosely based on the current signature separation in recent work [23]. We would like to highlight that [23] focuses on modifying the classifier model’s layers to perform adversary detection. In contrast, we train the detector independently of the classifier model that enables high detection scores along with transferability across multiple datasets and attacks.

Dataset generation: The training dataset for the detector contains equal number of natural ($x_{nat}$) and adversarial data ($x_{adv,t}$). Here, $x_{adv,t}$ is generated using the model $S_{T}$ having a different network architecture than $M_{C}$ and trained on $x_{nat}$ using the stochastic gradient descent (SGD) algorithm.

LES training methodology: Algorithm 13 shows the LES training approach. LES training begins with a randomly initialized $n$ layered detector, with $n$ being a hyper-parameter. Other inputs include the natural samples $x_{nat}$, adversarial samples $x_{adv,t}$ and $s_{nat}$. $s_{nat}$ is created by randomly selecting 200 data samples from $x_{nat}$.

The training is carried out in multiple stages. In each stage $i$, layers [0,$i$-$1$] are frozen and layer $i$ is optimized for energy distance maximization. In each stage, mini-batches $X_{n}$ and $X_{a}$ are fetched from $x_{nat}$ and $x_{adv,t}$, respectively. Following this, the natural and adversarial energies for layer $i$ are computed. The energies are used to compute the loss shown in Eq. 3 which is then used to optimize the $i$th layer parameters of the detector. As layers [0, $i$-$1$] are frozen, the energy separation obtained till layer $i$-$1$ is preserved. Thus, training the $i$th layer further increases the energy separation.

After the LES training, we obtain the trained detector $\mathcal{D}_{T}$. At this step, we use $s_{nat}$ and $\mathcal{D}_{T}$ to obtain the a sample natural energy distribution $\mathcal{E}_{s_{nat}}$. The $\mathcal{E}_{s_{nat}}$ is obtained at the final layer of $\mathcal{D}_{T}$. Next, the $K$th percentile of the $\mathcal{E}_{s_{nat}}$ distribution is chosen as the threshold energy $\mathcal{E}_{Th}$. All energy values greater than $\mathcal{E}_{Th}$ are classified as adversarial inputs and vice-versa.

Demonstrating the Efficacy of LES training and Choosing $\mathcal{E}_{Th}$: Fig. 2(a) shows the energy separation between $\mathcal{E}_{nat}$ and $\mathcal{E}_{adv}$ obtained after each layer of a three-layered detector trained on the CIFAR100 dataset in the layer-wise manner. The adversarial inputs correspond to PGD(4,2,10)¹¹1PGD(4,2,10)= PGD with parameters $\epsilon$: 4/255, $\alpha$: 2/255, $steps$: 10. attack on the CIFAR100 dataset. It can be seen that the energy separation increases with each layer in the detector. The increasing separation significantly improves the detection of weak attacks²²2Weak attacks have smaller $\epsilon$ values compared to stronger attacks (like PGD(4,2,10)), that may go undetected at early layers due to low energy separation.

In the final layer, such attacks can be detected successfully. Fig. 2(b) shows the $\mathcal{E}_{s_{nat}}$ distribution obtained at the final layer of the trained detector $\mathcal{D}_{T}$. The value of $\mathcal{E}_{Th}$ is chosen as the $K$th percentile of the distribution.

The Lipschitz constant has been associated with the error amplification effect in a neural network [20]. The Lipschitz constant of a feed-forward function $f_{l}$ for any layer $l$, can be expressed as follows:

where $\phi$ can be a linear, convolutional, max pooling or an activation layer.

Fig. 3 shows the $Lip(f_{l})$ values for three different networks: i) A randomly initialized 3-layered detector network, ii) the 3-layered detector after LES training on the CIFAR100 dataset, and iii) a VGG16 network trained on the CIFAR100 dataset using SGD. Interestingly, we observe that the value of $Lip(f_{l})$ rises as the energy separation between $\mathcal{E}_{nat}^{l}$ and $\mathcal{E}_{adv}^{l}$ increases. This suggests that the layer-wise training increases the error amplification factor in the detector network.

In prior works that minimize the adversarial perturbations in a network [5, 20], it has been shown that a lower Lipschitz constant helps in reducing the adversarial perturbations and thus increase adversarial robustness. However, in this work, we show that a higher Lipschitz constant amplifies the energy separation and thus, higher adversarial detection.

Fig. 4 shows the separation between natural and adversarial energies (corresponding to PGD(8,4,10) attacks) computed at the last layer of three detector models: Detector A: Conv(3,64)-ReLU-Conv(64,128)- ReLU-Conv(128,256), Detector B: Conv(3,32)-ReLU-
Conv(32,64)-ReLU-Conv(64,128) and Detector C: Conv(3,8)-ReLU-Conv(8,16)-ReLU-
Conv(16,32). Clearly, detectors having narrower layers achieve higher energy separation in the final layer compared to wider detector networks. Consequently, light-weight detectors improve the adversarial detection performance. Further, we observe that detectors without batch normalization achieve higher energy separation compared to detectors with batch normalization layers. In all our experiments throughout the paper, we use detectors without batch-normalization layers.

In this section, we show how adversarial detection is affected by the choice of $\mathcal{E}_{Th}$. The value $\mathcal{E}_{Th}$ is chosen as the $K^{th}$ percentile of the $\mathcal{E}_{s_{nat}}$ distribution. Fig. 5(a) shows the AUC scores for four different strengths of PGD attacks across five different $K$ values corresponding to CIFAR100 dataset. It is observed that lower values of $K$ yield higher AUC scores for weak attacks (such as PGD2). However, the AUC scores for stronger attacks are lower. Additionally, at lower $K$ values, the accuracy (See Section 2.2) is lower as seen in Fig. 5(b). This is because at low $K$ values, higher number of clean data samples are mis-classified as adversarial. At extremely high values of $K$ ($K$= 98), the AUC scores for weak attacks significantly fall. However, a higher accuracy is achieved. In both Fig. 5(a) and Fig. 5(b), $S_{T}$=ResNet18 and $S_{V}$=Mobilenet-v2 and $M_{C}$=VGG19. Evidently, choosing the $K$ value is a trade-off between the detection of weaker attacks and accuracy. Therefore, $K$ is a design choice that can be set according to the designer’s priority. In this work, we choose $K$=95 for all our experiments.

$\lambda_{n}$ and $\lambda_{a}$ are hyperparameters that denote the desired natural and adversarial energies for a particular layer $l$ in the detector. We show how the choice of $\lambda_{n}$ and $\lambda_{a}$ affects the detector performance. For this, we train 7 different detectors with different $\lambda_{a}$ values. D1(0.3,0.7,1.7) (with $\lambda_{a}$=0.3, 0.7, 1.7 for layer 1, 2, 3, respectively), D2(0.5,0.9,1.9), D3(0.7,1.1,2.1), D4(0.9,1.3,2.3) and D5(1.1,1.5,2.5), D6(1.3,1.7,2.7), D7(1.5,1.9,2.9). For all the models, $\lambda_{n}$= 0.1. As seen in Fig. 5(c) detection capability increases from D1 to D4 as the $\lambda_{a}$ increases and decreases beyond M4. However, the changes in AUC scores for different sets of $\lambda_{a}$ values is marginal. We find that all detector models (D1-D7) are good detectors for CIFAR10, CIFAR100 and TinyImagenet datasets (see Supplementary Material).

We use three different image datasets for our experiments: CIFAR10 [17], CIFAR100 [17], and TinyImagenet [8]. Both the training and validation data are scaled between (0,1) in all our experiments. For generating adversarial attacks, we use the torchattacks library [16].

For all our experiments, we use a 3-layered detector with the following architecture: Conv(3,8)- ReLU- Conv(8,16)- ReLU- Conv(16,32). For training the 3-layered detector, the LES training contains 3 stages. The LES training is performed with adversarial data created by PGD(8,4,10) attack and model $S_{T}$. For the CIFAR100 and CIFAR10 dataset, LES training is carried out with learning rates 0.005, 0.005 and 0.001, respectively in each stage. While for the TinyImagenet dataset, the learning rates correspond to 0.003, 0.001 and 0.001). During LES training, we use $\lambda_{n}$= 0.1 while $\lambda_{a}$= (0.9, 1.3, 2.3). The LES training is carried out for 500 epochs in each stage. Due to the small detector network size, the training time is significantly small (1 to 2 GPU hours for the 3-layered detector). Additionally, models $S_{T}$, $S_{V}$ and $M_{C}$ are trained on the respective datasets using the SGD algorithm. All experiments are conducted using Pytorch 1.5 platform with Nvidia rtx2080ti GPU backend.

In Table 2, we consider an LES-trained detector on the CIFAR100 dataset with $S_{T}$=ResNet18. The
classifier model $M_{C}$= VGG19. We consider white-box and black-box scenarios for validating the performance of the given detector. In white-box attack, the attacker has complete knowledge of the classifier model. While in black-box attack, the attacker does not have any information about the classifier model. To this effect, we use $S_{V}$: MobileNet-v2 and VGG19, for black-box and white-box attacks, respectively. Note, the premise of this paper is to train an adversarial detector and perform adversarial detection without any classifier model information. The white-box attack scenario considered here pertains only to the generation of attacks (i.e., $S_{V}$: VGG19) and not during detection or LES training.

Evidently, the detector has a high AUC score across a wide range of gradient, score and gaussian noise attacks (attack parameters shown in Supplementary Material). Consequently, adding the detector significantly lowers the the error (see Section 2.2) value of the classifier model compared to the “No Detector” case. Further, the reduction is substantially higher in case of white-box attacks compared to black-box attacks. However, we find that adding the detector leads to a 3% drop in accuracy (see Section 2.2 and Section 4.4) compared to the “No Detector” case.

Note, that the detector fails to detect decision-based attacks like the FAB attack. This is because FAB attacks add minimal perturbations to the natural input which does not change the $\mathcal{E}_{adv}$ value significantly. Consequently, a high error value is observed.

Interestingly, the detector shown in Table 2 demonstrates model agnostic behavior. For example, the detector can identify adversarial inputs generated using models $S_{V}$= VGG19 and $S_{V}$= MobileNet-v2. To further illustrate this model agnostic property, we create three detectors- Detector 1: $S_{T}$=
MobileNet-v2, Detector 2: $S_{T}$= ResNet18 and Detector 3: $S_{T}$= VGG16. All the detectors are trained using LES training. Table 3 shows the AUC scores of the detectors under adversarial attacks generated using $S_{V}$ models that have entirely different network architectures from the model $S_{T}$ used in the training. Evidently, the detectors have significantly high performance over different adversarial attacks and are agnostic to the $S_{T}$ and $S_{V}$ models used during training and attack generation, respectively. Note, AUC scores for FAB attack have not been shown as it is known that our detection approach cannot identify FAB attacks. Similar observations can be made for the CIFAR10 and TinyImagenet datasets and have been shown in Supplementary Material.

Table 4 compares the performance and cost-effectiveness of our proposed method with prior state-of-the-art adversarial detection works. For comparison, detectors for different datasets are created using LES training with $S_{T}$=ResNet18. $S_{V}$= MobileNet-v2. Clearly, we achieve comparable detection performance compared to prior adversarial detection approaches. It must be noted that our work does not aim to outperform prior adversarial detection works. Instead, the striking feature of our approach is to overcome the dependence of prior adversarial detection works on the classifier model for training and adversarial detection.

In this section we evaluate the performance of different detectors created with limited access to the actual dataset. For this, we create 3 detectors trained on subsets of size 1)10% 2)40% and 3)60% of the CIFAR100 dataset. The subsets are created by random sampling from the actual dataset. For reference, we also show the performance of a detector trained on the full dataset. In all cases, the detectors are created with $S_{T}$= ResNet18 and tested on attacks generated using $S_{V}$=MobileNet-v2 networks. It can be observed that detectors trained with just 40% of the training data can achieve AUC scores comparable with the detector trained on the full data. Interestingly, for some attacks (such as GN) merely 10% of the training data is sufficient for achieving a high performance. Please refer to Supplementary Material for similar results on the TinyImagenet and CIFAR10 datasets.

In this section, we explore the following question: Can a detector trained on dataset A (source dataset) be used to detect adversaries from another dataset B (target dataset)?. Methodology: LES training is performed to create a detector network on the source dataset. Following this, a sample dataset $s_{nat,target}$ is sampled from the target dataset. Through our experiments, we find that a size of 200 samples for $s_{nat,target}$ data is sufficient for transferability to the target dataset. The $s_{nat,target}$ and LES-trained detector is used to generate the $\mathcal{E}_{s_{nat,target}}$ distribution. $\mathcal{E}_{Th}$ is chosen as the 95th percentile of the $\mathcal{E}_{s_{nat,target}}$.

Interestingly, even detectors trained on smaller dataset like CIFAR100 can transfer reasonably to a larger dataset such as TinyImagenet (see Supplementary Material).

In Fig. 8, we evaluate the transferability of a detector D-Tiny40 trained on 40% of source dataset TinyImagenet and transferred to target datasets CIFAR100 and CIFAR10. We find that even detectors trained with limited access to the TinyImagenet dataset can transfer successfully. Similar experiments with other datasets are shown in the Supplementary Material.