Simultaneously Optimizing Perturbations and Positions for Black-box Adversarial Patch Attacks

In our method, the process to optimize the attack variables is formulated into the process of learning the agent’s parameters in the RL framework under the guidance of the reward signal from the target model. Specifically, the $t$-th variable’ value is defined as the action $a_{t}$ in the action space $a$ generated by the agent under the guidance of the policy $\pi$. The image input to the agent is defined as the state $s$, and the environment is the target model $F(\cdot)$. $\pi_{\theta}(a\!\mid\!s)$ with parameters $\theta$ is the policy function whose output is the probability value corresponding to each action $a_{t}$ in the action space $a$ under the state $s$. It obeys a certain probability distribution and is a rule used by the agent to decide what action to take.

The reward reflects the performance of the currently generated adversarial patch on the target model, and the training goal of the agent is to learn good policies to maximize the reward signal. In the face recognition, the goal of the dodging attack is to generate images that are as far away as possible from the ground-truth identity $y$, while impersonation attacks want to generate images that are as similar as possible to the target identity $\hat{y}$. Thus, the reward function $R$ is formalized as:

In the iterative training, the agent firstly predicts a set of actions according to policy $\pi$, and then the adversarial patch based on the predicted actions is generated. Finally the generated adversarial face image is input to the target model to obtain the reward value. In this process, policy gradient [34] is used to guide the agent’s update. After multiple training iterations, the agent will generate actions that perform well on the target model with a high probability.

The agent needs to learn the policies of the position, weights and attack step size. To take advantage of the coupling relationship between the position and perturbation, the two factors need to be mapped to the feature map output by the same agent at the same time. Considering this, we design a U-net [27] based structure, which can achieve the correspondence between positions and image pixels, and output the feature map of the same size as the input image. Let the number of surrogate models be $n$, we design the agent to output the feature map with $n$ channels and the same length $h$ and width $w$ as the input image (i.e. the size is $n\times h\times w$).

In each channel $M_{i}(i=1,\ldots,n)$ of the feature map $M$, the relative value of each pixel point represents the importance of each position for the surrogate model $f_{i}$, and the average value of the overall channel reflects the importance of the corresponding surrogate model. We believe that the patch requires different attack strengths in different locations, so at the top layer of the agent network, a fully connected layer is used to map the feature map $M$ to a vector $V$ representing different attack step values. The structural details of the agent are shown in Figure 3.

Specifically, for the position action, the optional range of positions is discrete, so the position policy $\pi_{\theta}^{1}$ is designed to follow Categorical distribution [35]. Given the probability $P_{position}$ of each selected position, the position parameters $\left(c_{x},c_{y}\right)\sim\bm{\emph{Cat}}\left(P_{\text{position }}\right)$, and $P_{position}$ is computed:

For the weight actions, the weight ratio of the loss on each surrogate model $f_{i}$ to the ensemble loss is a continuous value, and we set the weight policy $\pi_{\theta}^{2}$ to follow Gaussian distribution [35]. So the $i$-th weight parameter $\rho_{i}\sim\mathcal{N}\left(\mu_{f_{i}},\sigma^{2}\right)(i\!=\!1,\ldots,n)$, and $\mu_{f_{i}}$ is calculated as:

where $\overline{M_{i}}$ refers to the mean value of the $i$-th channel in the feature map, and $\sigma$ is a hyperparameter. In the actual sampling, we use the clipping operation to make the sum of weights equal to 1.

For the attack step action, we set 20 values in the range of 0.01 to 0.2 at intervals of 0.01, and adopt Categorical distribution [35] as the step size policy $\pi_{\theta}^{3}$ due to the discreteness of the values. So the step size parameter $\epsilon\sim\bm{\emph{Cat}}\left(p_{step}\right)$, and probability $p_{step}$ of each candidate value is:

By sampling from the corresponding distribution, we can obtain $(c_{x},c_{y}),\rho_{i}(i=1,...,n)$ and $\epsilon$.

In the agent training, the goal is to make the agent $h_{\theta}$ learn a good policy $\pi_{\theta}=\left\{\pi_{\theta}^{i}\mid i=1,\ldots,T\right\}$ with parameters $\theta$ to maximize the expectation of the reward $R$, where $T$ represents the number of attack variables, and $T=3$ in our case (the attack variables are patch position $\left(c_{x},c_{y}\right)$, attack step size $\epsilon$ in Eq. (2) and weights $\rho_{i}$ in Eq. (4), respectively.) $\tau=\left(s,a_{1},a_{2},\ldots,a_{T}\right)$ is a set of decision results obtained by sampling according to the policy $\pi_{\theta}$ of the current agent, and is formalized as a set of the input state s and the sampling results of each attack variable, then the optimal policy parameters $\theta^{*}$ can be formulated as:

We use the policy gradient [34] method to solve for $\theta^{*}$ by the gradient ascent method, and follow the REINFORCE algorithm [36], using the average value of $N$ sampling of the policy function distribution to approximate the policy gradient $\nabla_{\theta}J(\theta)$:

where $R_{n}$ is the reward in the $n$-th sampling. A large reward will cause the agent parameters $\theta$ to have a large update along the current direction, while a small reward means that the current direction is not ideal, and the corresponding update amplitude will also be small. Therefore, the agent can learn good policy functions with the update of $\theta$ in the direction of increasing the reward.

For actions that follow the Categorical policy (i.e., $\pi_{\theta}^{1}$ and $\pi_{\theta}^{3}$), let $p(a)$ denote the probability of action $a$ under the corresponding Categorical probability distribution (i.e., the probability value $p(a_{1})$ of the position variable under $\bm{\emph{Cat}}\left(P_{\text{position }}\right)$, and the probability value $p(a_{3})$ of the attack step size variable under $\bm{\emph{Cat}}\left(p_{step}\right)$), then for $\pi_{\theta}^{1}$ and $\pi_{\theta}^{3}$, $\nabla_{\theta}\log\pi_{\theta}^{t}(a_{t}\!\mid\!s)$ in Eq. (10) can be calculated as:

For actions that follow the Gaussian policy distribution ( i.e. the weights of the surrogate models that follow $\pi_{\theta}^{2}$), the mean value $\mu_{f}$ of Gaussian distribution is calculated by the output of the agent, so $\mu_{f}$ can be expressed as $h_{\theta}(s)=\mu_{f}$. Therefore, for $\pi_{\theta}^{2}$ following the Gaussian policy, $\nabla_{\theta}\log\pi_{\theta}^{t}(a_{t}\!\mid\!s)$ in Eq. (10) can be calculated as follows:

After the policy update using Eq. (10), we solve for the optimal parameters $\theta^{*}$. And then the optimal action strategy is obtained.

We first evaluate our simultaneous optimization method qualitatively and quantitatively according to the setting in Section 5.1. Table I shows the quantitative results of dodging and impersonation attacks under the black-box setting against the five target models.

From the results in Table I, we can see : (1) the proposed method achieves good attack success rates and query efficiency under both two kinds of attacks. The dodging attack achieves the highest success rate of 96.65% under 11 queries, while the impersonation attack achieves the highest success rate of 72.83% under 27 queries. (2) The performance of using ensemble models to attack is better than that using a single model as the surrogate model, which shows that simultaneous optimization can adaptively adjust the weights of different surrogate models to achieve the best performance. (3) For the relationship between the surrogate model and the target model, when the two are structurally similar, it is more likely to help the attack. For example, for ArcFace50, CosFace50 has a greater influence on it. This is because the backbone of ArcFace50 and CosFace50 are both ResNet50-IR [42], and the backbone of FaceNet and ArcFace34 are Inception-ResNet-v1 [43] and ResNet34-IR [42] respectively. Therefore, in ensemble attacks, we can dynamically adjust the importance of different models by optimizing their weights. (4) On the commercial API, performance drops slightly compared to the open-source model, but remains at an acceptable level. This is because commercial FR services introduce some defense measures like image compression.

Figure 5 shows some visual examples of the position and perturbation at different stages of the attack. We can see that with the increase of query times, the positions and perturbations of the generated adversarial patch are gradually stable and convergent until the face recognition model predicts the target wrong identity. This show that the proposed method has a good convergence. More visual results are shown in Figure 6. For each group of three images, the first one represents the clean image, the second one represents the image after the attack, and the third one represents the image corresponding to the wrong identity in the face database. The above results are obtained by ensuring that the patch is not pasted to the area that covers the facial features.

In addition, we also study the regularity of the position of the patch. We divide the face into four areas: between the eyebrows, the forehead, the left face and the right face, and count the position of the patch when attacking FaceNet, ArcFace50, and CosFace50. The ratio of patches in each area is shown in Table III. It can be seen that for the three models, the proportions of the patch located between the eyebrows are the highest, reaching 42.55%, 58.32%, 54.36%, respectively, followed by the forehead, while the proportions of the left and the right face are relatively low. This may be because the position between the eyebrows is close to the key facial features (eyes, eyebrows), so these positions have a greater impact on the model’s prediction.

To prove the superiority, we compare our method with five state-of-the-art methods: GenAP [17] and ZO-AdaMM [38] fix the position and change perturbations relying on transferability and gradient estimation, respectively. RHDE [15] fixes the perturbations and only changes the position, and Location Optimization (LO) [19] optimizes the position and the perturbation alternately, as well as the PatchAttack [20] optimizes the position and perturbation in a two-step manner. In the implementation, to adapt Location Optimization (LO) [19] to black-box attacks, we retain the framework of alternate iterations, and use gradient estimation to replace white-box gradient calculations. For GenAP, because we use its transferability to perform the black-box attack, its NQ is zero, and we use “-” to replace it. For our method, the target model in each column is not within the corresponding ensemble surrogate models (see Table I), therefore, the comparison is fair. The above results on three target models are shown in Table II.

From the results, we can see that: (1) For the perturbation-only method, GenAP and ZO-AdaMM have an average success rate of 42.08% and 46.82%; For the location-only method, RHDE is 38.97%. However, LO, PatchAttack, and ours are 55.10%, 61.60% and 65.61%, respectively. Therefore, considering both factors can achieve a significant improvement. (2) Our simultaneous optimization is ahead of LO’s alternate optimization and PatchAttack’s two-step attack, which shows that simultaneous optimization can make better use of the inherent connection between the perturbation and location, so it is better than optimizing the two separately. (3) Our method achieves the optimal query efficiency among several methods, requiring only a few dozen queries.

In order to verify the effectiveness of each attack variable used in our method, we also report the results when each component is added separately. First, we fix weights as equal values and the step size as 0.1 to test the performance when only changing the position parameter. Then, we add the weight parameter to learn the importance of each surrogate model. Finally, we add the step size parameters to carry out the overall learning. The results in Figure 7 show that learning only the position parameter can achieve an average success rate of 69.80%. The performance is greatly improved after adding the weight, and further improved after adding the step size. All parameters contribute to the enhancement of the overall attack effect and the weight parameter has a greater impact on the results than the step size. In the process of adding components, the number of queries (shown in brackets) is basically maintained at the same level.

Here, we test the attacks’ performance under two mainstream forms of defenses: preprocessing and adversarial training.

For the preprocessing defense, we explore the “detecting-repairing” methods, i.e., they first detect the region of the adversarial patch, and then repair the patch via different strategies. Specifically, we use three methods to defend our attack: Local Gradients Smoothing (LGS) [44] locates the noisy region based on the concentrated high-frequency changes introduced during the attack, and regularizes the gradient of this region to suppress the effect of adversarial noise. DW [45] locates local perturbations by the dense clusters of saliency maps in adversarial images, and then blocks the corresponding regions for defense. Februus [46] locates the influential region based on the heat map of GradCAM, removes the region and conducts image restoration for defense. The corresponding attack success rates (ASR) of dodging attacks against the above defense methods are shown in Table IV.

It can be seen that these methods cannot effectively weaken the attack performance. For LGS and Februus, ASR only drops slightly, by 1.01% and 3.37% on FaceNet, respectively. For DW, ASR decreases by 14.94% on FaceNet, but remains at an acceptable level with the ASR of 81.71%. We think this is because these methods do not locate/detect the local perturbations accurately. In the face classification task, the local patches often make the model decision solely affected by this small area [45], and thus localization is performed by the sensitive information exhibited by these local regions (e.g. frequency, saliency maps, and heatmaps of GradCAM). However, in the face recognition task, the model’s decision is affected by the combined effects of multiple regions on the face, and the addition of patches only affects the characteristics of a local facial feature. Before and after the attack, the region that the model focuses on is still scattered on each key area of the face instead of a certain area. After the attack, only the importance of a certain area will change slightly, so it is difficult to distinguish how it is different from the clean face image.

For adversarial training, we retrain the target model using adversarial examples generated by attack methods against this model, and then take the retrained robust model as the new target model to test the attack performance [19]. To demonstrate our superiority, we compare our method with GDPA [21] under the same adversarial training setting. GDPA [21] is a white-box attack method that also considers both positions and perturbations of the adversarial patch. The results of the attack success rate (ASR) and the number of queries (NQ) are shown in Table V, where the numbers in brackets denote the change value compared to the undefended model. It can be seen that adversarial training defends against GDPA attack well, when we use the same GDPA to attack the robust target models, the ASRs have a great drop (48.51% for FaceNet, 74.89% for ArcFace50, and 61.61% for CosFace50). This is reasonable because the generated adversarial examples by GDPA are integrated into the training set for these three models, and the target models have “seen” the adversarial examples. As a contrast, adversarial training does not defend against our attack, the ASR only has a slight change (i.e. 2.69%, 0.62%, 2.78% respectively), and it is still maintained at a relatively high level. The reason for this difference is that the patch’s positions chosen by GDPA on face images usually lie in the key facial features like eye, mouth, nose, etc. In other words, the learned position is constrained within a fixed scope (its original paper also shows this phenomenon). While the patch’s position chosen by our method can occur in any face area except for the key facial features. Therefore, it is difficult for adversarial training to learn the complete data distribution prevalent in our attack, which leads to the poor defense performance.

We show the results of our adversarial patch in the physical environment. We first perform simulated successful attacks on different subjects in the digital environment, and then conduct experiments in the physical world.

To make digital simulation results better adapt to the physical environment, we process the smoothness of the perturbation pattern. Specifically, during each iteration of the pattern generation, we first obtain a pattern with half the size of the original patch by scaling down or averaging pooling, and then enlarge the image back to the original size by bilinear interpolation. The reduced pattern retains the key information of the pattern. After zooming, the smoothness of the image is improved, avoiding the problem that the pattern generated per pixel is sensitive to the position point when it is transferred to the physical environment. Even if the pasting position is slightly a few pixels away, the attack effect can still be preserved. We also try to use Total Variation (TV) [14] loss to enhance the smoothness, but the actual effect is not as good as the scaling process. When printing, we use photo paper rather than ordinary paper as the patch material to recreate the colors of the digital simulation as realistic as possible.

Before applying to the physical environment, we first verify the robustness to the translation and rotation changes in the digital world. We translate the patch by 2, 3, and 5 pixels, and rotate it by 5 and 15 degrees to test the performance of the attack. Table VI lists the attack success rate (ASR) of dodging attacks on FaceNet and ArcFace50 models. It can be seen that for translation, moving 2 pixels has little effect on the attack (ASR for FaceNe does not drop, ArcFace50 drops 0.72%). Even with a 5-pixel shift, the ASR of 79.24% and 53.11% can still be achieved on two models. For rotation, within 5 degrees of rotation, the ASR of the two models decreases slightly by 2.96% and 8.23%, respectively, which can meet our requirements for applying to the physical environment.

For the performance in the physical environment, we record the video when faces are moving within a small range of the current posture, and count the frame proportion of successful physical attacks as the attack success rate. Table VIII shows the results of the frontal face. To test the performance under various physical conditions, we further change the face posture, the distance from the camera and the illumination. For face postures, we take the conditions of the frontal face, yaw angle rotation of $\pm 25^{\circ}$ (the mean value at $25^{\circ}$ and $-25^{\circ}$), $\pm 45^{\circ}$ and pitch angle rotation of $\pm 15^{\circ}$. The results are shown in Table VIII.

It can be seen from Table VIII that our method maintains high physical ASR (100.0% and 83.33%) on FaceNet, and the results are also good on ArcFace34 and CosFace50. Although there is a slight decline in commercial API, the ratio of 33.16% and 25.23% of successful frames is enough to bring potential risks to commercial applications. In Table VIII, when the pose changes in a small range ($\pm\ y25^{\circ}$, $\pm\ p15^{\circ}$), ASR still maintains a high value. Even if the deflection angle is slightly larger ($\pm\ y45^{\circ}$), it can still maintain an average of 31.13%. The effect of distance is small, and when lighting is changed, the results are still at an acceptable level. When performing impersonation attacks under different conditions, the face can be recognized as a false identity different from the true identity, but not the target identity, which may lead to a slightly lower result. Visual results are shown in Figure 8.