Adversarial Parameter Attack on Deep Neural Networks^††thanks: This work is partially supported by NSFC grant No.11688101 and NKRDP grant No.2018YFA0704705.

Let ${\mathcal{F}}_{\Theta}$ be a DNN with $\Theta$ as the parameter set. A parameter perturbation $\Theta_{a}$ is called a set of adversarial parameters of $F_{\Theta}$ or $\Theta$, if the following conditions are satisfied 1) $\Theta_{a}$ is a small modification of $\Theta$, for instance $||\Theta_{a}-\Theta||_{\infty}\leq\epsilon$ for a small positive number $\epsilon$; 2) the accuracy of ${\mathcal{F}}_{\Theta_{a}}$ over a distribution of samples is almost the same as that of ${\mathcal{F}}_{\Theta}$; 3) ${\mathcal{F}}_{\Theta_{a}}$ is much less robust than ${\mathcal{F}}_{\Theta}$, that is, ${\mathcal{F}}_{\Theta_{a}}$ has much more adversarial samples than ${\mathcal{F}}_{\Theta}$. It is clear that conditions 1) and 2) are to make the attack difficult to be recognized by the users and condition 3) is to make the new DNN less safe. The DNN obtained by the above attack is called an adversarial DNN, which has high accuracy but low robustness.

The existence of adversarial parameters is proved under certain assumptions. It is shown that if the depth of a trained DNN ${\mathcal{F}}_{\Theta}$ is sufficiently large, then there exist adversarial parameters $\Theta_{a}$ such that the accuracy of ${\mathcal{F}}_{\Theta_{a}}$ is equal to that of ${\mathcal{F}}_{\Theta}$, but the robustness measure of ${\mathcal{F}}_{\Theta_{a}}$ is as small as possible (refer to Corollaries 4.4 and 4.5). Since ${\mathcal{F}}_{\Theta}$ is a continuous function in $\Theta$, if $\Theta_{a}$ is an adversarial parameter for $\Theta$ then there exists a small sphere $S_{a}$ with $\Theta_{a}$ as center such that all parameters in $S_{a}$ are also adversarial parameters for $\Theta$. These results imply that adversarial parameters are inevitable in certain sense, similar to adversarial samples [6, 3, 27].

The existence of adversarial samples is usually demonstrated with numerical experiments, besides a few cases to be mention in the next section. As an application of adversarial parameters, we can construct DNNs which are guaranteed to have adversarial samples. For a trained DNN ${\mathcal{F}}_{\Theta}$ satisfying certain conditions, it is shown that there exist adversarial parameters $\Theta_{a}$ such that the accuracy of ${\mathcal{F}}_{\Theta_{a}}$ is equal to that of ${\mathcal{F}}_{\Theta}$, but ${\mathcal{F}}_{\Theta_{a}}$ has adversarial samples near a given normal sample (refer to Theorem 4.1), or the probability for ${\mathcal{F}}_{\Theta_{a}}$ to have adversarial samples over a distribution of samples is at least $1/2$ (refer to Theorem 4.2).

Finally, an effective training algorithm is given to compute adversarial parameters and numerical experiments are used to demonstrate that the algorithms are effective to produce high quality adversarial parameters for the networks VGG19 and Resnet56 on the CIFAR-10 dataset.

There exist vast literatures on adversarial attacks, which can be found in the survey papers [1, 5, 42]. We will focus on those which are closely related to our work.

Parameter perturbation attacks. Parameter perturbation attacks were given under different names such as fault injection attack, fault sneaking attack, stealth attack, and weight corruption. The fault injection attack [18] was first proposed by Liu et al, where it was shown that parameter perturbations can be used to misclassify one given input sample. In [7], it was shown that laser injection techniques can be used as a successful fault injection attack in real-world applications. In [43], the fault sneaking attack was proposed, where multiple input samples were misclassified and other samples were still given the correct label. In [37], lower bounds were given for parameter perturbations under which the network still gives the correct label for a given sample. In [33], upper bounds were given for the changes of the pairwise class margin function and the Rademacher complexity against parameter perturbations and new loss functions were given to obtain more robust networks. In [30], the maximum change of the loss function over given samples was used as an indicator to measure the robustness of DNNs against parameter perturbations and gradient decent methods were used to compute the indicator. In [34, 35], the stealth attack which can guaranteed to make the attacked DNN gives a desired label for a sample outside of the validation set and keep correct labels for samples in the validation set. The stealth attack has the form ${\mathcal{F}}+{\mathcal{U}}$, where ${\mathcal{F}}$ is the DNN to be attacked and ${\mathcal{U}}$ is a DNN with one hidden layer.

The adversarial parameter attack proposed in this paper is stronger than previous parameter perturbation attacks by in the following aspects. First, by keeping the accuracy and eliminating the robustness, the adversarial parameter attack is more difficult to be recognized, because the attached DNN performs almost the same as the original DNN on the test set. Second, by reducing the robustness of the DNN, the attacked DNN gives a wrong label for any modified input sample with high probability, while previous parameter attacks usually misclassify certain given samples. Finally, we prove the existence of adversarial parameters under reasonable assumptions.

Mathematical theories of adversarial samples. Existence of adversarial samples were usually demonstrated with numerical experiments, and mathematical theories were desired. In [6], it was proved that for DNNs with a fixed architecture, there exist uncountable classification functions and distributions of samples such that adversarial samples always exist for any successfully trained DNN with the given architecture and the sample distribution. In the stealth attack [34, 35], it was proved that there exist attached DNNs which give a desired label for a sample outside of the validation set by modifying the DNN. In this paper, we show that by making small perturbations to the parameters of the DNN, the DNN has adversarial samples with high probability.

Theories for certified robustness of DNNs were given in several aspects. Let $x$ be a sample such that the DNN ${\mathcal{F}}$ gives the correct label. Due to the continuity of the DNN function, a sphere with $x$ as center does not contain adversarial samples if its radius is sufficiently small, which is called a robust sphere of $x$. In [12], lower bounds for the robustness radius were computed and used to enhance the robustness of the DNN. In [24], for shallow networks, the upper bounds of the changes of the network under sample input perturbations were given and use to obtain more robust DNNs. In [8], the random smoothing method was proposed and lower bounds for the radius of the robust spheres was given. In [39], lower bounds for the average radius of robust spheres for a distribution of samples are given. Universal lower bounds on stability in terms of the dimension of the domain of the classification function were also given in [27, 34]. However, these bounds are usually inverse-exponentially dependent on the depth of the DNN, which are very small for deep networks in real world applications. In [40], the information-theoretically safe bias classifier was introduced by making the gradient of the DNN random.

Algorithms to train robust DNNs. Many effective methods were proposed to train more robust DNNs to defend adversarial samples [1, 5, 42, 38]. Methods to train DNNs which are more robust against parameter perturbation attacks were also proposed [18, 43]. The adversarial training method proposed by Madry et al [20] can reduce adversarial samples significantly, where the value of the loss function of the worst adversary in a small neighborhood of the training sample is minimized. In this paper, the idea of adversarial training is used to compute adversarial parameters.

Let us consider a standard DNN. Denote ${\mathbb{I}}=[0,1]\subset{\mathbb{R}}$ and $[n]=\{1,\ldots,n\}$ for $n\in{\mathbb{N}}_{+}$. In this paper, we assume that ${\mathcal{F}}:{\mathbb{I}}^{n}\to{\mathbb{R}}^{m}$ is a classification DNN for $m$ objects, which has $L$ hidden-layers, all hidden-layers use Relu as the activity function, and the output layer does not have activity functions. ${\mathcal{F}}$ can be written as

Denote $\Theta=\{W_{l},b_{l}\}_{l=1}^{L+1}\in{\mathbb{R}}^{k}$ to be the parameter set of ${\mathcal{F}}$ and ${\mathcal{F}}$ is denoted as ${\mathcal{F}}_{\Theta}$ if the parameters need to be mentioned explicitly, where $k=\sum_{l=1}^{L+1}n_{l}(n_{l-1}+1)$.

Let ${\mathcal{F}}_{\Theta}$ be a trained network with the parameter set $\Theta$. Then a new parameter set $\Theta_{a}$ is called a set of adversarial parameters of $\Theta$ if 1) $\Theta_{a}$ is a small perturbation of $\Theta$; 2) the accuracy of ${\mathcal{F}}_{\Theta_{a}}$ is almost the same as that of ${\mathcal{F}}_{\Theta}$; 3) ${\mathcal{F}}_{\Theta_{a}}$ is much less robust comparing to ${\mathcal{F}}_{\Theta}$, that is, ${\mathcal{F}}_{\Theta_{a}}$ has more adversarial samples than ${\mathcal{F}}_{\Theta}$.

We assume that the objects to be classified satisfy a distribution $D_{x}$ in ${\mathbb{R}}^{n}$, and a sample $x\sim D_{x}$ is called a normal sample. Let $\Theta\in{\mathbb{R}}^{k}$ be the parameter set of a trained network ${\mathcal{F}}_{\Theta}:{\mathbb{I}}^{n}\to{\mathbb{R}}^{m}$. For $x\sim D_{x}$, denote $l_{x}$ to be the label of $x$ and $\widehat{{\mathcal{F}}}_{\Theta}$ to be the classification result of ${\mathcal{F}}_{\Theta}$. Then the accuracy of ${\mathcal{F}}_{\Theta}$ for the normal samples is

In order to measure the quality of adversarial parameters, we need a robustness measure $R({\mathcal{F}}_{\Theta},D_{x})$ of ${\mathcal{F}}_{\Theta}$ for the normal samples. There exist several definitions for $R({\mathcal{F}}_{\Theta},D_{x})$ [20, 39]. In this paper, two kinds of robustness measures are used.

We first give two robust measures of ${\mathcal{F}}_{\Theta}$ on a given sample $x_{0}$. The robustness radius of $x_{0}$ under the $L_{p}$ norm for $p\in{\mathbb{R}}_{+}\cup\{\infty\}$ is defined to be

If $\widehat{{\mathcal{F}}}_{\Theta}(x_{0})\neq l_{x_{0}}$, then the robustness radius of $x_{0}$ is zero. It is difficult to have good estimation to the robustness radius, and the following approximation to the robust radius under $L_{p}$ norm [12] is often used

where ${\mathcal{F}}_{l}(x_{0})$ is the $l$-th coordinate of ${\mathcal{F}}(x_{0})$, $\nabla({\mathcal{F}}_{l}(x))=\frac{\nabla{\mathcal{F}}_{l}(t)}{\nabla t}|_{t=x}$, $p,q\in{\mathbb{R}}_{+}\{\infty\}$ satisfy $1/q+1/p=1$ ($p=0(\infty)$ iff $q=\infty(0)$), and $I(t)=1$ if $t$ it true or $I(t)=0$ otherwise.

For a distribution $D_{x}$ of samples, we define two global robust measures corresponding to $R_{1}$ and $R_{2}$. The adversarial accuracy can be used as $R({\mathcal{F}}_{\Theta},D_{x})$. For $\epsilon\in{\mathbb{R}}_{+}$ and $p\in{\mathbb{R}}_{+}\cup\{\infty\}$, the adversarial accuracy of ${\mathcal{F}}_{\Theta}$ is

Corresponding to $R_{2}$ in (4), we have the following global robustness measure

We now define a measurement for an adversarial parameter set using the accuracy and robustness of ${\mathcal{F}}$.

In general, we have $\gamma_{1}\leq 1$ and $\gamma_{2}\leq 1$. The value of $\gamma_{1}$ measures the ability of $\Theta_{a}$ to keep the accuracy of ${\mathcal{F}}_{\Theta}$ on normal samples, and if $\gamma_{1}$ is large then the attack is more difficult to be detected. The value of $1-\gamma_{2}$ measures the ability of $\Theta_{a}$ to break the robustness of ${\mathcal{F}}_{\Theta}$, and if $1-\gamma_{2}$ is large then the parameter attack is more powerful. Hence, the adversarial rate $\overline{\gamma_{1}}(1-\overline{\gamma_{2}})$ measures the quality of the adversarial parameter attack in that if the adversarial rate is larger then the adversarial parameter attack is better. If $\overline{\gamma_{1}}(1-\overline{\gamma_{2}})$ achieves its maximal value $1$, then $\gamma_{1}=1$ and $\gamma_{2}=0$ and the adversarial parameter attack is a perfect attack in that the attack does not change the accuracy of ${\mathcal{F}}$, but totally destroys the robustness of ${\mathcal{F}}$.

According to the requirements of specific applications, we may define other types of adversarial parameter attacks.

The adversarial parameters defined in section 2.1 are for all the samples. In certain applications, it is desired to make the network less robust on one specific class of samples, which motivates the following definitions.

The simplest case is adversarial parameters for a given sample. A small perturbation $\Theta_{a}$ of $\Theta$ is called adversarial parameters for a given sample $x_{0}$, if ${\mathcal{F}}_{\Theta_{a}}$ gives the correct label to $x_{0}$ and has adversarial samples of $x_{0}$ in $S_{\infty}(x_{0},\epsilon)=\{x\,|\,|x-x_{0}|_{\infty}\leq\epsilon\}$ for a given $\epsilon\in{\mathbb{R}}_{+}$. Let $R({\mathcal{F}}_{\Theta},x_{0})$ be a measure of robustness of ${\mathcal{F}}_{\Theta}$ at sample $x_{0}$, and

Then the adversarial rate of $\Theta_{a}$ is defined to be $1-\overline{\gamma}$.

We can also break the stability for samples with a given label. A small perturbation $\Theta_{a}$ of $\Theta$ is called adversarial parameters for samples with a given label $l_{0}$, if ${\mathcal{F}}_{\Theta_{a}}$ keeps the accuracy for all normal samples and the robustness for normal samples whose label is not $l_{0}$, but break the robustness of samples with label $l_{0}$. Let $\alpha=P_{x\sim D_{x}}(\widehat{{\mathcal{F}}}_{\Theta}(x)=l_{x})$ and $\beta=P_{x\sim D_{x}}(\widehat{{\mathcal{F}}}_{\Theta}(x^{\prime})=l_{x},\forall x^{\prime}\in S_{p}(x,\epsilon))$. For such adversarial parameters $\Theta_{a}$, let

Then the adversarial rate of $\Theta_{a}$ is defined as $\overline{\gamma}_{1}\overline{\gamma}_{2}(1-\overline{\gamma}_{3})$.

Finally, instead of breaking the robustness of samples with label $y_{0}$, we can break the accuracies for samples with label $y_{0}$. Such adversarial parameters are called direct adversarial parameters. Let $\Theta_{a}$ be a direct adversarial parameter set and

Then the adversarial rate is defined as $\overline{\gamma}_{1}\overline{\gamma}_{2}(1-\overline{\gamma}_{3})$. The above definition is similar to the attacks in [18, 43], but robustness is considered as an extra objective.

We formulate the adversarial parameter attack for a trained DNN ${\mathcal{F}}_{\Theta}$ under the $L_{p}$ norm as the following optimization problem for a given $\zeta\in{\mathbb{R}}_{+}$.

where $A({\mathcal{F}}_{\Theta_{a}},D_{x})$ and $R({\mathcal{F}}_{\Theta_{a}},D_{x})$ are the accuracy and a robustness measure for ${\mathcal{F}}_{\Theta_{a}}$ over a distribution sample $D_{x}$.

In the rest of this section, we show how to change formula (11) to an effective algorithm to compute $L_{\infty}$ norm adversarial parameters using the robustness measure in (5). We first show how to compute the robustness in (5) explicitly. We use the adversarial training [20] to do that, which is the most effective way to find adversarial samples. For a sample $x$ and a small number $\varepsilon\in{\mathbb{R}}_{+}$, we first compute

with PGD [20] and then use

to measure the robustness of ${\mathcal{F}}_{\Theta}$ at $x$.

We need a training set $T$ to find the adversarial parameters. The training procedure consists of two phases. In the first pre-training phase, the loss function

is used to reduce the adversarial accuracy of ${\mathcal{F}}_{\Theta}$. In the second main training phase, the loss function

is used to promote the accuracy and keep the low-level of adversarial accuracy of ${\mathcal{F}}_{\Theta}$, which corresponds to formula (11).

We will compute a more general $L_{\infty}$ norm parameter perturbation. Let $\Delta\in{\mathbb{R}}_{+}^{k}$ and $\Delta_{i}$ the $i$-th coordinate of $\Delta$. Then the $L_{\infty}$ parameter perturbation will be found in

It is clear that the usual $L_{\infty}$ norm parameter perturbation is a special case of the above general case. We use this general form, because we want to include more types of parameter perturbations which are given in section 5.2. A sketch of the algorithm is given below.

The algorithm to find adversarial parameters under other norms and robustness measures can be developed similarly. In what below, we show how to compute adversarial parameters under $L_{0}$ norm, which is different from other cases. The overall algorithm is similar to Algorithm 1, except we use a new method to update the parameters. Suppose that $\Theta=\{W_{l},b_{l}\}_{l=1}^{L}$ is the parameter to be updated and the value of $L$ in Algorithm 1 is found. We will show how to update the parameters. We only change some weight matrices $W_{l}$ as follows.

• •

  Randomly select two entries $w_{1}$ and $w_{2}$ of $W_{l}$ until $(\frac{\nabla L}{\nabla w_{1}}-\frac{\nabla L}{\nabla w_{2}})(w_{1}-w_{2})>0$ is satisfied.

• •

  Exchange $w_{1}$ and $w_{2}$ in $W_{l}$ to obtain the new parameters.

It is clear that the change will make $L$ become smaller. In total, we update a given number of weight matrices, and for each such matrix, we change a given percentage of its entries. The details of the algorithm are omitted. Note that the above parameter perturbation keeps the sparsity and the values of the entries of the weight matrices. As a consequence the Proj operator in the algorithm can be taken as the identity map.

The adversarial parameters defined in section 2.2 can also be obtained similarly. For instance, to compute the adversarial parameters for one sample $x$, we just need to let $T$ in Algorithm 1 to be $T=\{x\}$.

To compute adversarial parameters for samples with a given label $l_{0}$, by (9) we can use the following loss function

to increase the robustness and accuracy of samples whose labels are not $l_{0}$ and to reduce the accuracy for samples with labels $l_{0}$.

To compute direct adversarial parameters for samples with label $l_{0}$, by (10) we can use the following loss function

to increase the robustness and accuracy of samples whose labels are not $l_{0}$ and to reduce the accuracy for samples with labels $l_{0}$.

In this section, we use the robustness radius in (3) and the adversarial accuracy in (5) as the robust measures, and hence existence of adversarial parameters implies low adversary accuracies.

We introduce several notations. Let $||x||_{-\infty}=\min_{i\in[n]}\{|x_{i}|\}$ for $x\in{\mathbb{R}}^{n}$, and $||W||_{-\infty,2}=\min_{i\in[a]}\{||W^{(i)}||_{2}\}$ for $W\in{\mathbb{R}}^{a\times b}$, where $W^{(i)}$ is the $i$-th row of $W$. If ${\mathcal{F}}$ is a network, we use ${\mathcal{F}}_{i}(x)$ to denote the $i$-th coordinate of ${\mathcal{F}}(x)$.

In this section, we consider the following network ${\mathcal{F}}_{\Theta}:{\mathbb{I}}^{n}\to{\mathbb{R}}^{m}$ with one hidden layer

where $W_{1}\in{\mathbb{R}}^{n_{1}\times n},b_{1}\in{\mathbb{R}}^{n_{1}},W_{2}\in{\mathbb{R}}^{m\times n_{1}},b_{2}\in{\mathbb{R}}^{m}$. $\Theta=\{W_{i},b_{i}\}_{i=1}^{2}\in{\mathbb{R}}^{k}$ is the parameter set of ${\mathcal{F}}_{\Theta}$, where $k=(n+m+1)n_{1}+m$.

The network defined in (17) has just one hidden-layer. We will show that when the width of its hidden-layer is large enough, adversarial parameters exist under with certain conditions.

We will consider $L_{\infty}$ adversarial parameters. For $\gamma\in{\mathbb{R}}_{+}$, the hypothetical space for the adversarial networks of ${\mathcal{F}}_{\Theta}$ is

The following theorem shows the existence of adversarial parameters for a given sample $x_{0}$. The proof of the theorem is given in section 6.1.

We have the following observations from Theorem 4.1.

From the above two remarks, we may say that adversarial parameters are inevitable in this case.

The following theorem shows that when $n_{1}$ is large enough, adversarial parameters exist for a distributions with high probability. The proof of the theorem is given in section 6.1.

In this section, we consider networks of the following form

Let $\Theta=\{W_{i}\}_{i=1}^{L+1}$ be the parameters and $\Theta\in{\mathbb{R}}^{k}$, where $k=Ln^{2}+mn$. We use ${\mathcal{F}}^{l}_{\Theta}(x)$ to represent the output of the $l$-th layer of ${\mathcal{F}}_{\Theta}(x)$ where $l\in[L]$. We will show that, when $L$ becomes big, adversarial parameters exist.

We first prove the existence of adversarial parameters for a given sample. We use the following robustness measure for network ${\mathcal{F}}$ at a sample $x$

It is easy to see that this is the square of $R_{2}({\mathcal{F}},x)$ in (4) with $p=2$.

The proof of the theorem is given in section 6.2. As a consequence of Theorem 4.3, there exist adversarial parameters for sample $x_{0}$, whose robustness measure is as small as possible.

To find adversarial parameters for samples under a distribution $D_{x}$, we use the following robustness measure for ${\mathcal{F}}$:

This is a variant of $R_{4}({\mathcal{F}},D_{x})$ in (6) with $p=2$. The following theorem shows that adversarial parameters exist for this robustness measure. The proof of the theorem is given in section 6.2.