Efficient Robustness Assessment via
Adversarial Spatial-Temporal Focus on Videos

Spatial agent actually aims at solving an object localization problem (detecting key regions), we detail from three parts.

Action Design: To construct the actions of the spatial agent, we uniformly divide each video frame into overlapped patches inspired by the Vision Transformer [34]. In this way, we obtain a candidate patch set for the $i$-th frame $x_{i}$: $B_{i}=\{b_{i}^{j}|j=1,...,D\}$ where $b_{i}^{j}$ denotes the $j$-th patch region within $x_{i}$, and $D$ is the total number of candidate patches in this frame. $b_{i}^{j}\in\mathbb{R}^{h\times w}$ denotes that the patch’s size is $h$ and $w$, and their values will be tuned in the experiments. The goal of spatial agent is to select an optimal patch $b_{i}^{*}\in B_{i}$ in each frame as the key region, and thus the final selected action is a sequence set $a^{p}=\{b_{i}^{*}|i=1,...,M\}$. From the definition, we can see that there are totally $D^{M}$ action combinations for the given video $X$, which implies the search space is huge. An example of actions in one frame is listed in Figure 2, where $D=16$.

Policy Network Design: Spatial policy network $\pi^{p}(a^{p}|s^{p})$ is used to predict the spatial action $a^{p}$ when the state $s^{p}$ is given. The flowchart of our policy network is listed in Figure 3. Overall speaking, because we need to tackle with the sequence video data, a LSTM-based [26] structure is used to construct the policy network $\pi^{p}(a^{p}|s^{p})$. For the $i$-th frame $x_{i}$, a lightweight convolution neural network (CNN) $f(x_{i})$ is first to extract the frame-level feature maps $e_{i}$. In our experiments, we use MobileNet V2 as the lightweight CNN backbone for simplicity. Users can also apply other lightweight CNNs. Then they are fed into the LSTM unit to predict the logits for each patch. Next, a Softmax with Fully Connected Layer (FCL) is attached to output each patch’s probability $p_{b_{i}^{j}}$. Finally, we utilize the categorical sampling to obtain the optimal patch region $b_{i}^{*}$ according to their probability values $p(B_{i})=\{p_{b_{i}^{j}}|j=1,...,D\}$. To guarantee the smooth change of selected patch between adjacent frames, we concat the local patch features $e^{b^{*}}_{i-1}$ of the previous selected patch $b_{i-1}^{*}$ with the current frame-level features $e_{i}$ to jointly predict the current patch region, and $e^{*}_{i-1}$ is extracted via a simple multilayer perceptron (MLP) on the corresponding patch features of $e_{i-1}$.

Formally, the frame-level feature maps are extracted by:

next, the optimal action for each frame is achieved by:

where $h^{\pi}_{i-1}$ denotes hidden states output by LSTM unit in the $i$-1-th frame. Thus, the state $s^{p}$ in our method is defined as the concat feature $concat(e_{i},e^{b^{*}}_{i-1})$. Eq.(6) is repeated $M$ times to achieve the optimal action $a^{p}=\{b_{i}^{*}|i=1,...,M\}$.

In our method, the policy network is updated in each iteration $t$ of PGD attack, therefore, the optimal action will be updated in each iteration until the PGD attack stops.

Reward Design: In each iteration, the spatial policy network will receive the feedback from the environment to update its parameters $\theta^{p}$. Therefore, we need to design the reasonable rewards to guide the update of policy network. Because AstFocus attacks are based on the Multi-Agent Reinforcement Learning (MARL) framework, we design two kinds of rewards: one is specific for the spatial agent, and another is the common rewards shared with temporal agent.

For the special reward, an intuitive idea to evaluate the patch’s importance is the area covered by the foreground objects. Because video recognition model mainly performs predictions based on the foreground objects like person, car, etc. Therefore, if the policy network $\pi^{p}(a^{p}|s^{p})$ selects the foreground patch, the specific reward should be enlarged, and thus the policy network will be encouraged to select the foreground object in the next iteration. Based on this idea, we need a metric to measure the objectness score for a given patch. We here choose a classic objectness model: edgeboxes [35]. It calculates the edge response of each pixel and determines the boundary of the object by using the structured edge detector.

More concretely, the $r^{i}_{edgebox}$ reward for the selected patch $b_{i}^{*}$ can be described as following:

The edgebox reward for the whole video is defined:

where $w$ and $h$ are the patch’s width and height. $w_{b}(s_{k})$ is used to measure the affinity of the $k$-th edge groups in the selected patch. The $u_{k}$ is the sum of the $k$-th edge groups in the selected patch. In general, a large patch often results in a large edgebox value. More detailed information about edgebox function can be found in [35].

For the common reward, it comes from the feedback of the black-box threat models. If the selected patch is reasonable, the generated adversarial patch should have a strong attacking ability, and thus will make the confidence score output by threat models have a big drop. Therefore, we can use the confidence drop of the ground-truth label as a metric to compute this reward. Because this is also useful to the temporal agent, it is called as common reward. Specifically, the common reward $r_{common}$ is defined as follows:

where $exp(\cdot)$ is the exponential function, and $P(y|X^{\prime})$ represents the ground-truth label’s confidence when $X^{\prime}$ is fed into video recognition model. In the un-targeted attack, $P(y^{\prime}|X^{\prime})$ represents the second-ranked label’s confidence which is considered as the most competitive label to replace the ground-truth label. Only if the second-ranked label’s confidence becomes larger than that of the ground-truth label, $V(X^{\prime})$ becomes large. Then, we use the relative change of $V(X^{\prime})$ at different iteration $t$ as the metric for common reward. Eq.(10) is designed to encourage the agent to add perturbations on the selected regions that make the second-ranked label’s confidence gradually reach the ground-truth label and finally exceed it. In targeted attack, $P(y^{\prime}|X^{\prime})$ is the confidence of pre-defined target label by the adversary.

In summary, the $t$-iteration reward for spatial agent is:

where $\lambda_{1}$ denotes a weight to balance the two terms.

There exists a major distinction between temporal agent and spatial agent. Spatial agent aims at solving the object localization problem, while temporal agent aims at solving the binary classification problem (selecting or not selecting a frame). Thus, the actions, rewards, and policy networks of temporal agent should be re-designed.

Action Design: Key frames refer to those video frames that are conducive to successful attack, and their number is less than that of the whole video. The goal of the temporal agent is to select some key frames from the whole input video $X$, and thus the final selected action is also a sequence set $a^{f}=\{o_{i}^{*}|i=1,...,M\}$ just like spatial agent. The $o_{i}^{*}\in\{0,1\}$ indicates whether the $i$-th frame is selected or not. Therefore, there are totally $2^{M}$ different actions, which is not friendly to direct optimization learning.

Policy Network Design: The temporal policy network $\pi^{f}(a^{f}|s^{f})$ is used to predict the spatial action $a^{f}$ when the state $s^{f}$ is given. It is constructed with a LSTM structure. The skeleton diagram of the temporal policy network is shown in Figure. 4. The input of the policy network is the concat features composed of current frame-level features $e_{i}$ and a video-level global features $e^{g}$. Combining these two features can better select the key frames by considering the global video information. The global features $e^{g}$ is achieved by a fully connected layer on all the frame-level features $e_{i}$, $i=1,...,M$. The output of LSTM network is then fed to a Softmax with Fully Connected Layer (FCL) to predict the probability $p_{i}$ to indicate $o_{i}$=1. Technically, the temporal policy network can be expressed as:

where $Bernoulli(\cdot)$ is the Bernouli function. $h^{\pi}_{i-1}$ denotes the hidden states output by LSTM unit in the $i$-1-th frame. The state $s^{f}$ is defined as the concat feature $concat(e_{i},e^{g})$. Eq.(13) is repeated $M$ times to get $a^{f}=\{o_{i}^{*}|i=1,...,M\}$.

Reward Design: To make the temporal agent intelligent, the temporal policy network interact with the environment for updating its parameters $\theta^{f}$. Similar to the training of the spatial agent, in addition to the common reward function which shared with the spatial agent, we also have designed two special rewards to guide the temporal agent. The first specific reward function is the sparse reward $r_{sparse}$:

where $L$ here is used to control the number of key frames selected by the temporal agent, and $L<M$. The second specially designed reward function is mainly used to evaluate the representative ability of the video frames selected by the temporal agent. Because the selected video frames need to be sparse but effectively represent the semantic information of the whole input video. The representative reward function [36] $r_{rep}$ is defined as:

where $\mathcal{K}$ is a set of selected frame, i.e.,frames with $o_{i}$=1. Through those reward functions, the temporal can be forced to recognize few and critical video frames. The selected video frames can be effectively reduce the temporal redundancy of the entire video and effectively improve the following attack efficiency.

To make key video frames conducive to successful attacks, $r_{common}$ also join the learning of the temporal agent. For the $t$-th iteration, the corresponding reward is:

where $\lambda_{2}$ and $\lambda_{3}$ are two balance coefficients, and they will be discussed and set up in the experimental section.

So far, through the cooperation of spatial agent and temporal agent, the key regions in the key video frames of the input video can be identified. In the procedure of multi-agent reinforcement learning, the agents interact with the threat model for many times, and the predicted results of the agents are more inclined to the rapid and successful attack. Therefore, the critical attacking spaces selected by multi-agent reinforcement learning is the space sensitive to attack, which can effectively improve the efficiency of attack.

There are two parts to optimize: one is the lightweight CNN backbone $f(\cdot)$, and another is the policy network $\pi(\cdot)$.

CNN backbone: In our method, the CNN backbone $f(\cdot)$ is used for both temporal agent and spatial agent. It extracts the frames’ feature maps to construct state $s$. To decouple the training process of CNN backbone and policy network, we directly apply a pre-trained MobileNet V2 backbone on ImageNet dataset as the feature extractor. In this way, we can focus on the optimization of two policy networks.

Policy network: The policy gradient methods are used to optimize the temporal and spatial policy network. They are to directly adjust the parameters $\theta$ in order to maximize the objective $J(\theta)=\mathbb{E}_{s\backsim\rho^{\pi},a\backsim\pi_{\theta}}[R]$ by tacking steps in the direction of $\nabla_{\theta}J(\theta)$. By introducing an action-value function $Q^{\pi}(s,a)$, the policy gradient can be changed as:

To solve Eq.(17), we utilize the actor-critic reinforcement learning framework [37] where a critic network is applied to approximate the action-value function $Q^{\pi}(s,a)$. Actor network is the policy network in Figure 3 and Figure 4.

Because our method focuses on the cooperative multi-agent tasks, in which the two agents are trying to optimize a shared reward function. Each agent is decentralized and only has access to locally available information. For example, temporal agent can only observe the change of key frames, and spatial agent can only observe the change of key patches. Therefore, our method can be described as Decentralized Partially Observable Markov Decision Processes (Dec-POMDP) [38]. To solve this problem, [23] presents the multi-agent decentralized actor, centralized critic approach, thus Eq.(17) is reformulated as:

where $\pi_{k}$ denotes the policy network of the $k$-th agent, and $\theta_{k}$ is the corresponding parameters. In our method, there are totally two agents. The corresponding policy networks are $\pi^{p}(a^{p}|s^{p})$ with parameter $\theta^{p}$ and $\pi^{f}(a^{f}|s^{f})$ with parameter $\theta^{f}$. Here $Q_{k}^{\pi}(\textbf{s},a_{1},a_{2})]$ is a centralized action-value function that takes as input the actions of all agents $(a_{1},a_{2})$ in addition to some state information $\textbf{s}=[s^{p},s^{f}]$, and outputs the Q-value for agent $k$. In this way, we can perform a communication between two agents. In cooperative MARL, each agent is expected to maximize the common reward and its specific reward, therefore, we just need to solve Eq.(18) according to the rewards for spatial agent and temporal agent, respectively.

To solve Eq. (18) for the spatial agent $\pi^{p}(a^{p}|s^{p})$ and temporal agent $\pi^{f}(a^{f}|s^{f})$, we use the Proximal Policy Optimization (PPO), a popular single-agent on-policy RL algorithm [39] to obtain the $\theta_{f}$ and $\theta_{p}$. For the details of PPO algorithms, please refer to [39].

The first hyperparameter is the patch size $h$ and $w$ when designing the spatial agent’s action. A reasonable patch size will lead to less queries and less perturbations. The parameter tuning results for patch size is given in Figure 5, where we explore its effects for the fooling rate, query number and perturbations, respectively. From the figure, we see that patch size mainly affects the query number but shows slight changes for fooling rate and perturbation magnitude. Moreover, Figure 5 (B) shows the query number is relatively sensitive to the patch size. This is reasonable because the pre-defined patch size determines the proportion of selected key regions out of the whole image, thus affects the query number.²²2 From the last column in Table III, we see that AstFocus attack has smaller variance when performing multiple times. For attacking C3D model on HMDB-51, the variance has no changes for FR, and only changes 1% around the mean for MAP, 4% around the mean for NQ. Therefore, the unsmooth curve is not caused by the significant variance. Overall, when the patch size is set to 65, the query number reaches the smallest value. Therefore, we set $h=w=65$.

The second hyperparameter is the upper bound $L$ of selected key frames in Eq.(14). A reasonable $L$ can help our method select the minimal key frames to perform a successful video attack, and thus query number can be reduced. The parameter tuning results for upper bound $L$ is given in Figure 6, where we also explore its effects for the fooling rate, query number and perturbations, respectively. We can see that with the increase of $L$ value, the fooling rate will gradually stabilize to 100% and query number is slowly decreasing, but it would cause a big increase in perturbation magnitude. To balance three different evaluation metrics, we set $L=10$ in the following experiments.

The third hyperparameter is the number of sampled points $n$ in Eq.(2). The sample number $n$ per each iteration has a great influence on the accuracy of the estimated gradient, especially when the attacking space changes. To explore the impact of the sample number $n$ on the attack effect, we have conducted a series of experiments. The parameter tuning results are given in Figure 7. We can see that with the increase of $n$ value, the fooling rate will gradually stabilize to 100%, but query number and perturbation magnitude achieve their optimal performance when $n$ is located in 60. Therefore, we set $n=60$ in the following experiments.

There are three weights to tune in the reward functions. They are $\lambda_{1}$ in Eq.(11), $\lambda_{2}$ and $\lambda_{3}$ in Eq.(16), which measures the importance of their own rewards. The parameter tuning results are given in Figure 8. According to the figure, we set $\lambda_{1}=0.2,\lambda_{2}=0.4$, and $\lambda_{3}=0.6$, respectively. It means there exists more redundancy to reduce in the temporal domain than spatial domain, thus needing to set large rewards in Eq.(16) to guide agent for learning key frames.

In our method, the baseline is PGD+NES algorithm. Then we integrate two agents into the PGD+NES to reduce the video dimension from the temporal and spatial domains, respectively. Here we perform the ablation study about whether these two agents work in the video attack. The results are given in Table III, where “Baseline” denotes the PGD+NES. In this setting, because there is no dimension reduction module, the perturbations are added on the whole video, which can be called as “dense attack”. The term “Spatial” denotes integrating the spatial agent into the baseline. In this setting, we reduce the spatial redundancy by selecting the key patches in each frame. The term “Temporal” denotes integrating the temporal agent into the baseline, which reduces the temporal redundancy by selecting the key frames. The term “Spatial$\&$Temporal” denotes the full version of AstFocus attack, i.e., simultaneously reducing the temporal and spatial redundancy via two agents.

We show the effects versus fooling rate (FR), number query (NQ), and perturbation magnitude (MAP). From the table, we can see that the dimension reduction is indeed useful to the attacking performance, i.e., the “Spatial” and “Temporal” achieve higher FR and smaller QN and MAP than the “Baseline”. By simultaneous reducing the temporal and spatial redundancy, “Spatial$\&$Temporal” achieves the highest FR (100%), and smallest QN and MAP. The average FR increases 26.7% (73.3%$\rightarrow$100%), average QN and MAP decrease about 36% (3662$\rightarrow$2227) and 47% (6.37$\rightarrow$3.35) versus the baseline, respectively. In addition, “Spatial$\&$Temporal” has smaller variance than baseline. For example, the variance has no changes for FR metric, and only changes 1% around the mean for MAP metric, 4% around the mean for NQ metric. For this reason, we neglect the variance value in the following comparison experiments. This verifies the important role of dimension reduction when performing video attacks.

We also compare our RL-based agents with random agents, where the key frames and key patches are randomly selected in each PGD iteration, other settings are the same. Figure 9 shows the comparison results. We can see that for all the evaluation metrics, our RL-based agent outperforms the random agent (100%$\pm$0% vs 86.7%$\pm$2.89%, 2227$\pm$92 vs 2632$\pm$114, 3.35$\pm$0.03 vs 3.62$\pm$0.04), which verifies the temporal and spatial agents in our method are jointly well-trained under the guidance of the carefully designed rewards. With the feedback of threat models, the temporal and spatial agents become intelligent.

To better guide the agent learning, we carefully design the rewards. Specifically, one common reward Eq.(10) coming from the black-box threat model, and two kinds of specific rewards Eq.(8) as well as Eq.(15) and Eq.(14). The common reward is shared by the spatial agent and temporal agent, and the special rewards only belong to their own agents. In this section, we explore the effects of these rewards to the fooling rate, query number and perturbations.

Table IV lists the ablation study for effects of various rewards. The term “Common” denotes the guidance of both temporal and spatial agents only using the common reward in Eq.(10). The terms “+Edgebox”, “+Sparse”, and ‘+Representative” denote adding the corresponding rewards (Eq.(8), Eq.(14), and Eq.(15), respectively) on the former basis to guide the agent learning. From the table, we see that with the addition of more and more rewards, average FR gradually increases, and average QN and average MAP gradually decrease. Compared with the solo common reward, the full version (the rightmost column) with all the rewards improves 23.3% for average FR (76.7%$\rightarrow$100%), and reduces 43% for average QN (3780$\rightarrow$2227), and 8% for average MAP (3.62$\rightarrow$3.35). The variance also becomes smaller and smaller. The contrast verifies the rationality of the designed rewards.

Because our agents are updated by the rewards in each iteration, it is necessary to investigate whether the agents are under the convergence with the increasing iterations. For that, we list the values’ change of Eq.(9) with the increasing PGD iteration in Figure 10. Eq.(9) directly reflects the success or failure of an attack. If the target class’s confidence score is above the ground-truth class’s confidence score, the value of Eq.(9) will be above 1, representing that the attack is successful, and vice versa. From the figure, we can see that Eq.(9)’s values for all the threat models are gradually increasing until the stable situation. When the iteration reaches 400, all the models achieves the convergence. This verifies the good convergence of AstFocus attack. Actually, the attack usually stops when Eq.(9)’s value is above 1, i.e., the step 9 in Algorithm 1. Therefore, we only need few iterations in application. Figure 11 gives a qualitative example of the agents in AstFocus attacks, where the key frames and key patches in different iterations are illustrated by the bounding boxes. We can see that the spatial agent gradually focuses on the foreground objects. This is reasonable because these areas are key cues for video recognition task. In addition, the temporal agent tends to select the frames with big changes in the actions. These frames have a strong representative ability for the whole video from the appearance, which shows key frames are sensitive to attacks.