Recover Triggered States: Protect Model Against Backdoor Attack in Reinforcement Learning

Backdoor attacks can be set in a large number of devices such as a brain-computer interface [9], and traffic control systems [10], code generation [11]. Backdoor attacks can also be set in different types of neural networks such as a transformer model [12]. The backdoor can be activated through imperceptible media such as ultra-sonic sound wave [13]. A backdoor attacker can set different triggers so that he could reinforce his attack ability [14]. Researchers introduced a backdoor attack method to attack RL agents in single-agent scenarios in electronics games [8]. In real world, researchers used backdoor attack to demolish reinforcement-learning-augment autonomous driving. In competition scenarios, researchers have proposed to attack RL agent competing with other agents [15, 16] and trigger the backdoor through taking a consecutive legal actions allowed in the environment. Researchers also introduced a way to enhance the effectiveness of the backdoor attack [3].

These backdoor attack methods have threaten the use of the intelligent agent in different aspects of our society. We should not only highlight the security issue on data but also design algorithms for defending against backdoor attacks.

Many of methods defending against backdoor attacks need true samples, such as [17, 18]. But true samples can not be provided in most cases of RL, which means some methods used in defending against backdoor attacks in tasks such as picture classification cannot be used in RL. Researchers also introduced a backdoor seeker and memory-loss technique to eliminate backdoors in multi-agent competition scenarios [6], and it is not applicable as mentioned above. And people have tried to use reverse engineering [7] to locate the polluted data before training the agent, but it didn’t work well in locating the untarget attack because random neurons are activated as shown in [8].

Recently, researchers also introduced defense methods that don’t need true sample in classification tasks, such as using sparsity [19], which may help defend against backdoor attacks in RL.

In our threat model, a backdoor attack can be separated into two stages, the injection stage and the trigger stage. In the injection stage, the attacker would implant backdoors into the dataset, and the RL trainer will use the polluted dataset to train the model, which will cause the agent being poisoned, as shown in Fig. 1. The proportion of polluted data in the dataset is very low [5, 8] thus decreasing the possibility of being detected by human inspection. Backdoor attacks on RL are notorious because it can hardly be removed once injected. And human inspections are impossible to achieve because the amount of data needed for using an offline RL algorithm to train a RL agent is too big, for example, in general we need 10M - 20M pieces of data to use an offline RL algorithm to train a RL agent in the environment of our experiment. The injection stage attack warms us to watch out the data we used for training the agents. In the trigger stage attack, attacker can set one step attack and consecutive attacks, and the later can further deviate the actions of victim agents from their normal values, which makes a severer damage.

In our defense model, we propose to defend against backdoor attacks in the trigger stage. Before deployment, we let the victim agents interact with clean environment, and collect trajectories to train the dynamics model, then apply the dynamics model when the victim agent is deployed for working. Our defense method is usable because the amount of data needed for training the dynamics model is relatively small, for example, in our experiment we need 50K pieces of data to train a dynamics model for single-step state prediction with a loss below 4 in L2 norm. With little effort we could prevent the agent from being harassed by the backdoor attacks, and if we use RL evaluation methods to include much of the circumstances before deployment, we could further improve the defense performance of our RTS method.

Overall, our method defends the attack in a unison and easy way that the defender can detect and prevent the trigger stage attack using the same dynamics model, and protect the victim agent from the target attack and the untarget attack. The tenet for our defence strategy can be concluded as below.

In the equation (3), the $T^{d}$ is our defender. If an attacker wants to trigger the backdoor, he must distort the input and this will cause the discontinuity of the environment dynamics especially in a continuous control scenario thus our defender would easily capture the difference.

In image-input scenarios, it would be hard to distinguish if the coming states is triggered merely comparing the predicted state and the coming state. In this case, we could use the difference between policy’s actions on the predicted state and that on the coming state to determine if the states is triggered, and our proposed action-related regularization term is more welcomed because $\pi(s_{p})$ would be similar to $\pi(s)$, thus improving the accuracy of detection a trigger stage attack.

Overall, the attack performance of backdoor attack is bounded by both whether the attacker can successfully trigger the backdoor and its ability to change the action with the use of triggered input $s^{*}$, and our RTS method can provide an easy and efficient way to detect the abnormal states and prevent attacks.

Model choice: In designing the structure of the defender, considering the data expenditure, we use the single-deterministic model to simulate the environmental dynamics. In the experiment, the error between the predicted state and the real state given by the model is within an L2 norm of 4.

Data collection: In a clean environment, we let the poisoned agent perform tasks and collect the trajectories data. This premise can be easily fulfilled, because we don’t need backdoors to be triggered to collect the data, which can save the time used in data collection.

Train the Defender: After data is collected, we begin to train a defender. The target of training the defender can be denoted as (4), in which $T^{d}$ is the defender, $s_{t-1},a_{t-1}$ are the input for the defender, and the $s_{t}$, $a_{t}$ are the target.

Defend the Attack: At time step $t$, the defender gets previous input $s_{t-1}$ and agent’s action $a_{t-1}$ to give a prediction $s^{p}$. The state $s_{t}$ should be close to the state $s^{p}$. The attacker may distort a state into a triggered state $st^{*}$. The defender will consider the difference of the $s_{t}$ and $s^{p}$. We could measure the difference of other targets, such as the difference between the $\pi(s_{t}),\pi(s^{p})$. If the difference is out of expectation $h$, the defender will proceed to recover the state. Here we use the l2 norm in the experiment as shown in the 5.

If the defender captures the abnormal distance between the $s_{t}$ and $s_{t}^{p}$, the defender would recover a prediction $s_{t}^{p}$ which the victim agent acts on, finally stopping the backdoor to be triggered. The neural networks kind dynamics models have a better ability to capture the dynamics in an episodic manner [20, 21], which encourages us to use the model to simulate the dynamics of a fixed policy $\pi$ interacting with the environment.

The process of the trigger stage attack of backdoor attacks is similar to that of adversarial attack. In both kinds of the attack, attackers will skew the input of an victim agent thus inducing the victim agent to perform another action. Both of attacks will not influence the dynamics of the environment. Thus, we can use the performance bound of the adversarial noise attack to define the attack performance of the backdoor attacks.

Previously, people have defined a performance bound of an adversarial attack as shown in (6) which shows that an adversarial attack can be determined by its ability to change the action of the policy on the noised input. This equation also indicates an upper limit on the maximum difference between value functions with distinct action distributions [22].

In Equation (6), $V_{\pi}(\cdot)$ is the value function of the policy $\pi$, and $\beta(\cdot)$ outputs $s^{*}$, state added adversarial noise, and $D_{TV}(\pi(\cdot|s),\pi(\cdot|s^{*}))$ is the total variation distance between $\pi(\cdot|s)$ and $\pi(\cdot|s^{*})$ and $\alpha$ is a constant that does not depend on $\pi$.

As explained above, because these two attack methods share certain similarities, we use (7) to establish a limit on the effectiveness of the backdoor attack in terms of its capacity to sway the victim’s policy through the use of the triggered input $s^{*}$.

In Equation (7), $\pi^{*}$ means the poisoned agent, and $T(\cdot)$ outputs $s^{*}$, the state triggered by the attacker, and $D_{TV}(\pi^{*}(\cdot|s),\pi^{*}(\cdot|s^{*}))$ is the total variation distance between $\pi^{*}(\cdot|s)$ and $\pi^{*}(\cdot|s^{*})$.

Thus, as shown in (8), we can evaluate the defender’s performance in preventing the trigger stage attack by examining the differences between $\pi(\cdot|s_{t})$ and $\pi(\cdot|T^{d}(s_{t-1},a_{t-1}))$.

This equation shows the importance of a defender’s ability to generate a predicted state and guide a victim agent’s actions to closely approximate those in the true state, in order to effectively defend against attacks. Once the defender detects a trigger-stage attack, it supports the victim agent in acting on nearly normal input, rather than maliciously controlled input, to prevent the attack.

We aim to use a single-deterministic dynamics model as a defender to get the advantage of the generalization ability of neural networks as well as avoiding the huge data consumption associated with recurrent neural networks. The training target of the traditional single-deterministic dynamics model is as shown in (9),

which is very common. However, in continuous control scenarios, even gaps between predicted states and real states are small, the policy will execute very different actions.

The problem can be illustrated in (10).

This issue will hinder the defense performance, which is severer in consecutive attack scenarios.

This problem is related to the undesired actions in model inference tasks causing accuracy degradation. When policy gets a predicted state different from the real state, it executes an action different from that taken in a real state. And this action further causes the dynamics model generating states different from the real states even if the dynamics model can accurately predict states under a certain bound, and finally accumulates to big compounding errors. Thus, one way to improve the model’s inference task performance and reduce the effect brought by the compounding error is to generate a state that guides the agent executing an action similar to that taken on the real state, then the dynamics model could predict accurately. The existence of the adversarial noise [23, 22] reminds us that with little turbulence on the input, the output of the neural networks could be very different. So it is reasonable to use this regularization term in single-deterministic dynamics model as well as other dynamics model to reduce accumulated errors.

A dual-objective dynamics model could perform better in both single and consecutive attack scenarios as shown in Fig. 3. The defense performance of the single-objective dynamics model degrades because agent doesn’t act as expected and the error further causing the dynamics model to predict an undesired states, finally causing the failure of the agent. And the agent plays to the end with the dual-objective dynamics model as a defender.

By guiding the agents to execute actions as if on a real state, the defender can achieve a smaller distance of $D_{TV}(\pi^{*}(\cdot|s),\pi^{*}(\cdot|T^{d}(s^{*})))$, which improves their performance against trigger stage attacks, as shown in (8).

We use data collected when the victim agent interacts with environments to train the defender. A Defender’s network is: (State Dimension + Action Dimension, 256) (256,256) (256, State Dimension). When collecting the data, we modify the policy through adding a Gaussian noise on its action to maintain exploration. We find with a 0.01 probability to use the noise is sufficient for sampling the data. We use 50K pieces of data to train a defender in the Hopper environment.

We use the deep deterministic policy gradient (DDPG) [24] to train the victim model, and inject the target backdoor attack data after the policy has run for 25K steps to reappear the injection stage attack. The proportion of polluted data is 4% in a dataset of 25K steps, 0.2% in a dataset of 500K steps, and 0.0125% in a dataset of 800K steps.

In the single-step attack scenario, we trigger the backdoor one time every twenty steps by distorting the input when a well trained victim agent is working as shown in 4. On average, the victim agent scores 1523.09 and 2994.43 points without attack in two scenarios. In contrast, the scores decrease to 1246.96 and 998.28 with trigger stage attack. The scores increase again to 1526.24 and 2213.60 when protected by the single-objective defender and to 1515.84 and 3112.66 when protected by the dual-objective defender.

As the agent gradually learns the optimal strategy and executes more steps in a trajectory, the dual-objective defender provides better defense.

Then we perform the attack two consecutive steps in every twenty steps, and results as shown in Fig. 5. With trigger stage attack, the scores decrease to 112.08 and 63.84. And the scores increase to 112.89 and 679.38 when protected by the single-objective defender and to 1340.88 and 3084.50 when protected by the dual-objective defender.

Our dual-objective defender provides a better defense performance compared with that of the single-objective defender. Due to the existence of perturbations, the scores of the protected agents in attack scenarios are slightly higher than those of the agents who have not been attacked, and this deviation is acceptable.

To verify that the dual-objective dynamics model can generate state guiding agents to execute actions similar to that on real states, we demonstrate the difference between predicted states and real states in a length $\mathcal{L}=1$ model inference task. The predicted state are separately predicted by the single-objective model and the dual-objective model.

An agent trained after 500K reported that, on average, the L2 distance between the real state and the states predicted by the single-objective defender and the dual-objective defender were 0.1715 and 0.3125, respectively. Additionally, the average L2 distance between actions on predicted states and real states were 0.3525 and 0.0796.

An agent trained after 800K reported that, on average, the L2 distance between the real state and the states predicted by the single-objective defender and the dual-objective defender were 0.4099 and 0.6304, respectively. Additionally, the average L2 distance between actions on predicted states and real states were 0.7043 and 0.3941.

The results are as shown in Fig. 6. These result confirm that our dual-objective defender can predict states guiding the agent to act as if on real states.