Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysis

Meta Reinforcement Learning (meta RL), as a combination of meta learning and reinforcement learning, intends to learn a decision-making model that can quickly adapt to new tasks. Compared with learning from scratch, meta RL enables the agent to leverage previous training data, requiring fewer samples when dealing with new tasks. The successes of meta RL (Wang et al., 2016; Duan et al., 2016; Finn et al., 2017; Fallah et al., 2020) can be explained from a feature learning standpoint: meta RL aims at building an internal representation of the policy (meta policy) that is broadly suitable for a collection of similar tasks. This internal representation is learned by either recurrent neural networks (Wang et al., 2016; Duan et al., 2016) or stochastic optimization (Finn et al., 2017; Fallah et al., 2020), when agents are exposed to the collection of tasks via a unique sampling process. In meta RL, in addtion to producing sample trajectories within a single task, the sampling process also randomly samples environments for learning a common representation shared across these tasks.

However, this unique sampling process also makes meta RL vulnerable when facing adversarial attacks. By manipulating sample data collected in the meta training phase, the attacker can mislead the agent into learning a wrong meta policy that adapts poorly to individual tasks from the collection. For example, in Model-Agnostic Meta Reinforcement Learning (MAMRL) (Finn et al., 2017), a unique sampling process is required for optimizing the meta policy. As shown by our numerical results, slightly perturbing the reward feedback from this sampling, the attacker can significantly degrade the performance of meta RL agents under different tasks.

Contributions This paper aims to provide a theoretical understanding of adversarial attacks on sampling processes of meta RL, which is referred to as sampling attacks. Due to its mathematical clarity, we choose MAMRL (Finn et al., 2017) as the departure point of our discussion. Yet, the idea of sampling attacks also applies to other meta RL formulations (Wang et al., 2016; Duan et al., 2016) (see Appendix B).

Our contributions are three-fold. 1) We characterize the sampling attack as a Stackelberg game (Von Stackelberg, 2010) between the attacker and the agent (the learner), which yields a minimax formulation. 2) Based on this formulation, two online black-box training-time attack schemes are proposed: Intermittent Attack and Persistent Attack, which enable the attacker to learn optimal sampling attacks concurrently with the meta learning without extra interactions with the environment. 3) The optimality of sampling attacks is defined by $\epsilon$-first-order stationarity, and our nonasymptotic analysis shows that the proposed attack schemes can achieve the optimality within $\mathcal{O}(\epsilon^{-2})$ iterations.

To the best of our knowledge, this work is among the first endeavor to investigate online training-time adversarial attacks on sampling processes of meta RL. In addition, our analysis of nonconvex-nonconcave minimax problem can be extended to other RL problems sharing the same nature of nonconvexity.

This section reviews recent progress in the area of adversarial attacks on RL. We position our work using the following categorizations and comment on the differences between existing approaches and the proposed one in this paper, highlighting our contribution to the security study of RL and meta RL.

Testing-time and Training-time Attacks Unlike testing-time attacks, where the attacker intends to degrade the performance of a learned and deployed policy, training-time attacks aim to enforce a target policy in favor of the attacker by manipulating the learning process.

Our work falls within the class of training-time attack. Even though our attack methods are based on reward manipulations, which have also been explored in the literature (Ma et al., 2019; Huang and Zhu, 2019; Zhang et al., 2020b, 2021), we emphasize that previous works mainly target Q-learning algorithm (Ma et al., 2019; Huang and Zhu, 2019; Zhang et al., 2020b) or neural policy gradient (Zhang et al., 2021), which relies on $Q$ value estimation. For these value-based algorithms, the influence of the reward manipulation on the value function is more straightforward than in meta RL, where the adaptation in the meta training complicates the analysis. The optimality results regarding previous attacks do not apply to meta RL, and our paper’s theoretical analysis of the optimality of the reward manipulation is more challenging.

White-box and Black-box Attacks For both testing-time and training-time attacks on RL agents, existing works have investigated both white-box (Ma et al., 2019; Zhang et al., 2020b; Huang and Zhu, 2019) and black-box settings (Xu et al., 2021; Russo and Proutiere, 2021). For white-box attacks, the attacker needs to fully know the learner’s learning algorithm and the policy model. Some attacks additionally require the knowledge of the environment, e.g., transition dynamics (Xu et al., 2021).

In contrast, the proposed attacks, as a black-box attack¹¹1The fault line between white-box and black-box may vary in different contexts, We here follow the notion used in (Xu et al., 2021) and treat our proposed attack as a black-box attack., do not require prior knowledge of the environment setup or learning algorithms. With access to the policy model, the attacker can adjust its attack by gradient updates, regardless of learning algorithms or the environment parameters.

Compared with previous black-box attacks (Xu et al., 2021; Russo and Proutiere, 2021), the proposed attacks do not require interactions with the training environment, e.g., the simulator. All it needs are the sample trajectories rolled out by the learner during the training process. In other words, the attacker considered in this paper is a free rider who does not actively collect information regarding the agent’s learning algorithm or the environment.

Offline and Online Attacks Previous works mainly consider offline training time reward manipulations (Ma et al., 2019; Huang and Zhu, 2019), where the attacker has full knowledge of the training dataset. The attacker makes one decision on reward manipulation and then provides the poisoned data to RL agent. In this work, online attacks are studied, where the attacker manipulates the reward feedback sequentially, adaptive to the learner’s learning process.

Let $\{\mathcal{T}_{i}\}_{i=1}^{T}$ be the set of finite-horizon Markov Decision Processes (MDP) representing $T$ different tasks/environments. Each task $\mathcal{T}_{i}$ is given by a tuple $\left\langle\mathcal{S},\mathcal{A},P_{i},r_{i},\rho_{i}\right\rangle$, where $\mathcal{S}$ and $\mathcal{A}$ denotes the the space of states and actions, respectively. For each $\mathcal{T}_{i}$, $P_{i}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{B}(\mathcal{S})$ denotes the transition kernel that maps a state-action pair to $\mathcal{B}(\mathcal{S})$, a Borel probability measure over the state space. Finally, the agent receives a reward $r_{i}(s,a)$ when taking action $a$ at state $s$, and the initial state is drawn from the initial distribution $\rho_{i}$. For the sake of simplicity, we assume that all tasks share the same state and action spaces, as well as the same horizon length $H$, and our analysis applies to cases where different tasks have different MDP formulations (Fallah et al., 2020).

To facilitate our discussion, the following notations are introduced. A trajectory from an MDP $\mathcal{T}_{i}$ is defined as $\tau({\mathcal{T}_{i}})=(s_{1},a_{1},\ldots,s_{H},a_{H})$, where $H$ denotes the horizon length. A batch of trajectories $\tau({\mathcal{T}_{i}})$ under the same MDP is considered to be a training data set $\mathcal{D}_{i}$. Given a trajectory $\tau_{i}$ (a shorthand for $\tau(\mathcal{T}_{i})$), the total reward received over this trajectory is $R(\tau_{i}):=\sum_{t=1}^{H}r_{i}(s_{t},a_{t})$.

Suppose that the agent’s policy in $\mathcal{T}_{i}$, $\pi(\cdot|\cdot;\theta):\mathcal{S}\rightarrow\mathcal{B}(\mathcal{A})$, is parameterized with $\theta\in R^{d}$, and $\pi(a|s;\theta)$ is the probability of choosing action $a$ at state $s$. Then, given the transition dynamics $P_{i}(s^{\prime}|s,a)$ and the initial distribution $\rho_{i}(s)$, the probability that such a trajectory $\tau_{i}$ occurs is $q_{i}(\tau;\theta):=\rho_{i}(s_{1})\prod_{t=1}^{H}\pi(a_{t}|s_{t};\theta)\prod_{t=1}^{H-1}P_{i}(s_{t+1}|s_{t},a_{t}).$ Therefore, the expected cumulative rewards under the policy $\pi(\cdot|\cdot;\theta)$ in the task $\mathcal{T}_{i}$ is $J_{i}(\theta)=\mathbb{E}_{\tau_{i}\sim q_{i}(\cdot;\theta)}[R(\tau_{i})]$. RL algorithms, such as policy gradient (Sutton et al., 2000), seek to find a $\theta^{*}$ that maximizes $J_{i}(\theta)$.

The idea of policy gradient method is to apply gradient descent with respect to the objective function $J_{i}(\theta)$. Following the policy gradient theorem (Sutton et al., 2000), we obtain $\nabla J_{i}(\theta)=\mathbb{E}_{\tau\sim q_{i}(\cdot;\theta)}[g_{i}(\tau;\theta)],g_{i}(\tau;\theta)=\sum_{h=1}^{H}\nabla_{\theta}\log\pi_{i}(a_{h}|s_{h};\theta)R_{i}^{h}(\tau)$ where $R^{h}_{i}(\tau)=\sum_{t=h}^{H}r_{i}(s_{t},a_{t})$. In RL practice, the policy gradient $\nabla J_{i}(\theta)$ is replaced by its Monte Carlo (MC) estimation, since evaluating the exact value is intractable. Given a batch of trajectories $\mathcal{D}_{i}(\theta)$ under the policy $\pi_{i}(\cdot|\cdot;\theta)$, the MC estimation is $\hat{\nabla}J_{i}(\theta,\mathcal{D}_{i}(\theta)):={1}/{|\mathcal{D}_{i}(\theta)|}\sum_{\tau\in\mathcal{D}_{i}(\theta)}g_{i}(\tau;\theta)$.

Instead of focusing on individual tasks $\mathcal{T}_{i}$, MAMRL, as introduced in (Finn et al., 2017), aims to find a good initial policy, also called a meta policy that returns satisfying rewards in expectation when it is updated using limited number of gradient step updates under a new environment $\mathcal{T}^{\prime}$, sampled from a task distribution $p$. In particular, for only one-step policy gradient update with a step size $\alpha$, the optimization problem of MAMRL (Fallah et al., 2020) is

A stochastic gradient method (SG-MRL) is proposed in (Fallah et al., 2020) for solving the maximization problem (1) (see Algorithm 1), where an unbiased estimator of $\nabla L(\theta)$ can be efficiently computed for performing gradient ascent. Suppose that for simplicity $\mathcal{D}^{1}_{i}=\{\tau\}$, the data set sampled using $q_{i}(\cdot;\theta_{t})$, only contains one trajectory, then $\nabla_{\theta}\mathbb{E}_{\mathcal{D}\sim q_{i}(\cdot;\theta)}[J_{i}(\theta+\alpha\hat{\nabla}J(\theta,{\mathcal{D}}))]=\mathbb{E}_{\tau\sim q_{i}(\cdot;\theta)}[\nabla J_{i}(\theta+\alpha g_{i}(\tau;\theta))(I+\alpha\nabla_{\theta}g_{i}(\tau;\theta))+J_{i}(\theta+\alpha g_{i}(\tau;\theta))\nabla_{\theta}\log\pi(\tau;\theta)]$. The MC estimation of $\nabla_{\theta}\mathbb{E}_{\mathcal{D}\sim q_{i}(\cdot;\theta)}[J_{i}(\theta+\alpha\hat{\nabla}J(\theta,{\mathcal{D}}))]$ is given below:

where $\hat{\nabla}J_{i}(\theta+\alpha g_{i}(\tau;\theta))$ and $\hat{J}_{i}(\theta+\alpha g_{i}(\tau;\theta))$ are estimated using another batch of trajectories $\mathcal{D}_{i}^{2}$ rolled out under the policy $\pi(\theta+\alpha g_{i}(\tau;\theta))$. The average of $\nabla_{\theta}^{i}MC$ constitutes an unbiased estimate of $\nabla L(\theta)$.

In this section, we propose three sampling attack models and characterize them as Stackelberg games between the attacker and the learner. As shown in Algorithm 1, MAMRL includes two sampling processes. The first sampling (line 5 in Algorithm 1) is for the MC estimation of the corresponding policy gradient, while the second sampling (line 7 in Algorithm 1) is for the MC estimation of the gradient of the objective function $L(\theta)$. The main idea of the sampling attack is to manipulate the reward signal so that MC estimations are corrupted. Since the meta training includes two sampling processes, there are three possible sampling attacks: 1) Inner Sampling Attack (ISA): only attacking the first sampling process, where samples are used to compute the gradient adaptation in the inner loop; 2) Outer Sampling Attack (OSA): only attacking the second sampling process, where samples are used to perform one step meta optimization in the outer loop; 3) Combined Sampling Attack (CSA): attacking both the first and the second sampling process.

Practicability of Sampling Attacks Before delving into the details of these attacks, we first illustrate how this attack can happen in a training environment. For the training process in RL, a learner, as the orange box in Figure 1, usually consists of two components: a sampler and an optimizer. At each state, the sampler chooses an action based on the current policy, and sends it to the simulator, which returns a reward as well as the next state. The state-action-reward tuple will be stored within the sampler, and this process repeats until multiple trajectories are sampled. These sample trajectories make up a training data set fed to the optimizer, which performs one-step policy improvement.

To launch sampling attacks, the attacker first covertly infiltrates into the learning system shown in Figure 1 through Advanced Persistent Threats (APT) processes (Zhu and Rass, 2018). Details on this infiltration are included in Section A.1. After the infiltration, the attacker intercepts communication between the sampler and the simulator and sends the learner poisoned reward signals, misleading the learner. One may argue that the attacker could gain control of the whole system, and its possible maneuvers are more than reward manipulation (e.g., compromising the simulator, corrupting the state input). In Section A.2, we justify our choice of reward manipulation as the first step of the investigation without losing the generality of sampling attacks.

The following summarizes three important aspects of the proposed sampling attacks. 1) Knowledge of the attacker: the attacker only has access to the sample trajectories rolled out by the learner and the current policy. The simulator and the learner’s internal learning process remain black-box to the attacker. 2) Attack Strategies: the attacker is allowed to modify the rewards of sample trajectories provided by the simulator. It can adjust its attack sequentially depending on the meta learning process. Note that the attacker needs to choose the manipulation strategy from a constraint set, which can be interpreted as the attack budget (see Section A.3). 3) The attacker’s objective: this paper considers a greedy attacker who misleads the learner into learning the worst possible meta policy, i.e., the value function of the learned meta policy [$L(\theta)$ in (1)] is minimized after sampling attacks. In the subsequent, we use ISA as an example to elaborate on these aspects of the proposed sampling attacks.

Inner Sampling Attack The first attack is to apply reward manipulations to the first sampling process. Denote the manipulated reward by $r^{adv}(s,a;\delta)$, which is parameterized by $\delta\in\Delta\subset\mathbb{R}^{n},n\in\mathbb{N}$. We assume that $r^{adv}(s,a;\delta):=\delta\cdot r(s,a)$, i.e., the manipulated reward is obtained by applying the linear transformation to the original reward, and $\delta\in\Delta\subset\mathbb{R}$. It is a more general practice to represent $r^{adv}$ by a neural network, $\delta$ being the weights of the network. Another remark is that the admissible manipulation strategies $\delta$ are confined to a bounded set $\Delta$ (see 6), limiting the attacker’s ability. Section A.3 compares our attack constraints with existing constraints in the literature.

Under ISA, the policy adaptation $\theta_{t+1}^{i}=\theta_{t}+\alpha\hat{\nabla}J_{i}(\theta,\mathcal{D}_{i}^{1})$ is compromised, as $\mathcal{D}_{i}^{1}$ is replaced by $\widetilde{\mathcal{D}}_{i}^{1}$, in which every reward is scaled by $\delta$. For simplicity, it is assumed that reward signals in $\mathcal{D}_{i}^{1}$ are uniformly scaled by $\delta$. More sophisticated manipulation strategies, such as adaptive poisoning (Zhang et al., 2020b), can also be considered, yet to keep the theoretical analysis neat, we limit ourselves to this uniform scaling within each batch. In the following, notations with $\widetilde{\mbox{\hskip 4.0pt}\cdot\mbox{\hskip 4.0pt}}$ either denote data corrupted by the manipulation or quantities computed using corrupted data.

Following the policy gradient theorem reviewed in Section 3, the policy gradient under ISA takes the following form $\widetilde{\nabla}J_{i}(\theta)=\mathbb{E}_{\tau\sim q_{i}(\cdot;\theta)}[{g}_{i}(\widetilde{\tau};\theta)]$, where $\widetilde{g}_{i}(\tau;\theta)=\sum_{h=1}^{H}\nabla_{\theta}\log\pi_{i}(a_{h}|s_{h};\theta){R}_{i}^{h}(\widetilde{\tau})$. Since $\widetilde{\tau}$ only differs from $\tau$ in that the rewards are scaled by $\delta$, ${R}_{i}^{h}(\widetilde{\tau})=\delta R_{i}^{h}(\tau)$, the relationship between the corrupted policy gradient and the original one is given by $\widetilde{\nabla J_{i}}(\theta)=\delta\nabla J_{i}(\theta)$. Similarly, for the MC estimation, we have $\hat{\nabla}J(\theta,\widetilde{\mathcal{D}})=\delta\hat{\nabla}J(\theta,\mathcal{D})$. Finally, the goal of the attack is to find a $\delta$ such that the performance of the learned meta policy [evaluated through the objective function $L(\theta)$, specified in (1).] is minimized. The optimization problem associated with (ISA) is given by

Outer Sampling Attack Another sampling attack is to consider manipulating the samples $\mathcal{D}_{i}^{2}$ in Algorithm 1. The outer sampling collects samples under the updated policy $\theta_{t+1}^{i}$, which are used to compute the MC estimation of the gradient of $L(\theta_{t})$. In this case, the optimization problem in OSA share the same attack objective $\min_{\delta\in\Delta}L(\theta^{*})$ as in ISA, but the constraint becomes

Combined Sampling Attack Finally, the inner and the outer sampling attack can be combined together. Following the same line of argument, the objective is $\min_{\delta\in\Delta}L(\theta^{*})$, and the constraint is

Some remarks are in order. Note that compared with the other two sampling attacks, OSA is rather trivial: a reward flipping, i.e., $\delta=-1$ is sufficient, which makes $\theta^{*}$ already a minimizer. Hence, for the rest of the paper, we mainly discuss (ISA) and (CSA). Our theoretical analysis focuses on (ISA), and its extension to (CSA) is straightforward (see Section E.1). Another remark is that even though there may exist multiple maximizers $\theta^{*}$, as the objective function is nonconvex differentiable, we assume that in our formulation, $\theta^{*}$ is properly selected, and hence unique. One way to justify this assumption is that $\theta^{*}$ can be viewed as the policy parameter returned by the RL algorithm in the training process, and hence, it is unique. We will elaborate on this selection more mathematically (see Lemma E1 and its proof) in Section 5.

Note that all three sampling attacks are formulated as bi-level optimization problems. Hence they amount to a Stackelberg game between the attacker (leader) and the learner (follower). The best response of the learner is to learn an optimal policy under the reward manipulation, and the three constraints on $\theta^{*}$ in the above formulations correspond to the learner’s meta learning processes under ISA, OSA, and CSA, respectively. Knowing the learner’s best response, the attacker aims to find a $\delta$ such that the learned meta policy adapts poorly in the testing phase without further attacks.

Since the attacker targets the testing performance of $\theta^{*}$, it needs to consider the policy gradient adaptation $(\theta^{*}+\alpha\hat{\nabla}J_{i}(\theta^{*},\mathcal{D}(\theta^{*}))$ to evaluate $L(\theta^{*})$, where $\mathcal{D}(\theta^{*})$ is the genuine sample trajectories under $\theta^{*}$ without manipulation. However, in (ISA), the attacker can only access the learner’s contaminated training data [i.e., sample trajectories under the policy $(\theta+\alpha\cdot\delta\hat{\nabla}J(\theta,{\mathcal{D}}))$]. In this case, the feedback of the attacker’s manipulation [e.g., estimates of $\nabla L(\theta)$] is not directly available during the training time. Hence, searching for the optimal attack defined in (ISA) in an online manner is challenging, due the attacker’s limited knowledge of the meta learning process.

From Stackelberg to Minimax As discussed above, it is a daunting task for the attacker to find the optimal $\delta$ by solving the bi-level optimization problem in (ISA). Thus, we reformulate the attacker’s problem as a minimax problem (MMP), which is more tractable than (ISA), as the attacker now targets the learner’s training performance as shown below.

Note that in general the $\min$ and $\max$ operator in (MMP) is not changable, since the objective is nonconvex-nonconcave (Fallah et al., 2020). Hence, (MMP) still amounts to a Stackelberg game, yet the attacker now aims to minimize the training performance under ISA, i.e., the value function under the corrupted dataset [the max part of (MMP)]. Sample trajectories rolled out by the learner can also help guide the attacker’s search for the optimal attack, and no extra data or information is needed. In this case, the attack freerides the meta learing process, making the proposed sampling attack stealthy and lightweight. This free-ride nature will be more clear in Section 5, where two online attack schemes based on the gradient feedback are introduced.

Before concluding this section, we comment on the relationship between original (ISA) and (MMP). Even though the minimax value of (MMP) is by definition different from the Stackelberg equilibrium payoff given by (ISA), our following Proposition 1 states that under the assumption of bounded policy gradients (see 3 in Appendix D), the minimax value, when combined with a $\ell_{2}$-norm regularization $\|\delta-1\|$ serves as an upper-bound of the value function under the optimal attack.

From Proposition 1, the free-rider nature can explained in the following way: as long as the training performance, i.e., the right-hand side of (4) deteriorates, the testing performance or the quality of the meta policy, i.e., the left-hand side of (4) cannot be any better. Even though the proposed sampling attack models are discussed in the context of MAMRL, they are also relevant to other meta RL frameworks, as discussed in Appendix B.

Based on (MMP), this section presents two onine attack schemes that enable the attacker to learn the optimal attack alongside with learner’s learning process. We present the two schemes in the context of ISA, and the extension to CSA is straightforward (see Section E.1). To streamline the investigation into the optimality of proposed mechanisms, we skip the $\ell_{2}$ term, treating it as a regularization in numerical implementations, and mainly focus on (MMP).

Intermittent Attack In (MMP), the inner maximization problem $\max_{\theta\in\mathbb{R}^{d}}\mathbb{E}_{i\sim p}\mathbb{E}_{\mathcal{D}\sim q_{i}(\cdot;\theta)}[J_{i}(\theta+\alpha\cdot\delta\hat{\nabla}J(\theta,{\mathcal{D}}))]$ corresponds to the meta RL under ISA, while the attacker’s problem is given by $\min_{\delta\in\Delta}\mathbb{E}_{i\sim p}\mathbb{E}_{\mathcal{D}\sim q_{i}(\cdot;\theta)}[J_{i}(\theta^{*}+\alpha\cdot\delta\hat{\nabla}J(\theta^{*},{\mathcal{D}}))]$. A natural way to seek the minimizer is to apply gradient descent to the objective function. Using the similar argument in (2), we arrive at the following MC estimation of the gradient with respect to $\theta$,

Similarly for the gradient with respect to $\delta$ when fixing $\theta^{*}$, the MC estimate is given by

The attacker’s learning proceeds as follows. At first, the attacker initializes the attack by setting $\delta=1$, and closely monitors the learning process. During the meta training process, the attacker remains dormant, i.e., $\delta$ remains constant, until policy parameter $\theta$ stabilizes. Then it is activated and updates $\delta$ using one-step stochastic gradient descent under the Euclidean projection $\Pi_{\Delta}(\cdot)$. It repeats the above steps until the $\delta$ stabilizes. Since the attacker only adapts the reward manipulation intermittently, we refer to such attack as Intermittent Attack. The pseudocode is in Algorithm 2.

Persistent Attack Instead of waiting for the learner to converge to $\theta^{*}(\delta)$, the attacker can also implement a real-time attack, i.e., it performs gradient descent along with the learner’s gradient ascent, no matter whether the current $\theta$ stabilizes or not. In this case, with the initialization $\delta=1$, the attacker is spared from detecting the stabilization of the learner’s training and updates $\delta$ every time $\theta$ is updated. Algorithm 3 summarizes this kind of attack, which we refer to as Persistent Attack, since the attacker keeps adjusting $\delta$ according to the learner’s progress all the time. The intuition behind Persistent Attack is that if the attacker’s learning rate is far smaller than the learner’s, the learner views the attack as quasi-static, while the attacker treats $\theta_{t}$ as stabilized (Lin et al., 2019), which shares the same spirit as Intermittent Attack.

Before proceeding to the theoretical analysis, we comment on the differences between the two schemes in terms of cost and stealthiness. Since Intermittent Attack utilizes the stabilized policy parameter, the attacker’s MC estimation (6) in Intermittent Attack tends to return a more accurate gradient descent direction than its counterpart in Persistent Attack when minimizing the objective function in (MMP). For the attacker’s gradient estimation under two attack schemes, the reader is referred to Lemma E1, Lemma E5, and associated proofs. As a result, in practice, Intermittent Attack tends to find the optimal attack using fewer iterations (as shown in Figure 2(b)(d).) However, in Intermittent Attack, the attacker needs to detect the stabilization of the learner’s training. This detection can be costly, as the attacker needs to keep a record of the average return of each iteration. In contrast, in Persistent Attack, the attacker simply updates the attack $\delta$ per iteration without detection. On the other hand, since Persistent Attack manipulates the rewards of sample trajectories during each iteration, it is less stealthy than Intermittent Attack.

Optimality of Sampling Attacks The objective in (MMP) is nonconvex-nonconcave. An appropriate optimality condition is the $\epsilon$-first order stationary point ($\epsilon$-FOSP), widely adopted in the study of programming and game theory (Nouiehed et al., 2019; Li et al., 2022).

Since the objective function in (MMP) is noncovex-nonconcave, in order to show the optimality of the attack, we need to impose some regularity conditions. For the ease of notation, we define $f(\delta,\theta):=\mathbb{E}_{i\sim p}\mathbb{E}_{\mathcal{D}\sim q_{i}(\cdot;\theta)}[J_{i}(\theta+\alpha\cdot\delta\hat{\nabla}J(\theta,{\mathcal{D}}))]$, and denote its MC estimation by $\hat{\nabla}f(\delta,\theta;\xi)$, where the random variable $\xi$ represents the a collection of datasets $\{\mathcal{D}_{1}^{i}\}_{i\in\mathcal{I}},\{\mathcal{D}_{i}^{2}\}_{i\in\mathcal{I}}$ to be sampled in the sampling processes for the computation of $\nabla_{\delta}^{i}MC,\nabla_{\theta}^{i}MC$. Denote $\Phi(\delta):=\max_{\theta}f(\delta,\theta)$. The following assumptions extend Polyak-Łojasiewicz (PL) condition (Polyak, 1963; Łojasiewica, 1963) and restricted secant inequality (RSI) (Zhang and Yin, 2013) to the MMP. Note that both 1 and 2 are weaker than commonly assumed strong convexity condition (Lin et al., 2019; Nouiehed et al., 2019), and detailed discussions are presented in Appendix D.

In addition to the above assumptions, some regularity assumptions are also needed to show the convergence of Intermittent and Persistent Attack. They are presented in Appendix D.

Nonasymptotic Analysis of Intermittent Attack The convergence result rests on that when the meta learning stabilizes, the attacker’s gradient feedback $\nabla_{\delta}f(\delta_{t},\theta^{*}_{t}(\delta_{t}))$ equals $\nabla\Phi(\delta_{t})$ (see Lemma E1). Then, the standard analysis of gradient descent in nonconvex programming can be applied.

Nonasymptotic Analysis of Persistent Attack Although 1 suffices for proving the optimality of Intermittent Attack, it is rather a weak condition when dealing with Persistent Attack. The gap is that in Persistent Attack, the attacker’s gradient update does not utilize the exact gradient $\nabla\Phi(\delta_{t})=\nabla_{\delta}f(\delta_{t},\theta^{*}_{t})$ or its approximation $\nabla_{\delta}f(\delta_{t},\theta_{K}(\delta_{t}))$, instead, $\nabla_{\delta}f(\delta_{t},\theta_{t})$ is applied. To control the difference $d_{t}=\|\theta^{*}_{t}-\theta_{t}\|$, we show in Section E.3 that $d_{t}$ exhibits a linear contraction using 2, leading to the convergence result stated as follows.

This section presents experimental evaluations of the proposed attack schemes for ISA and CSA. We consider two benchmark meta learning tasks, including a 2D-navigation task and a more complex locomotion problem: Half-cheetah (H-C) (Finn et al., 2017). The following briefly introduces the two tasks, and more details can be found in Section C.1. In the 2D-navigation task, starting from a fixed initial position [the black triangle in Figure 2(a)] in a 2D plane, a point agent [the red dot in Figure 2(a)] needs to move to a goal position [the green star in Figure 2(a)] randomly sampled from a given task distribution. In the H-C task, a planar cheetah [as shown in Figure 2(c)] is required to run at a particular velocity sampled from the task distribution.

Experimental Setup Hyperparameter setup for the two algorithms is detailed in Section C.2. When evaluating the corrupted meta policy in the testing phase, we compare the adaptation performance [$L(\theta)$ defined in (1)] of meta policies learned under different attack settings and the benchmark setting without attacks. One-step policy adaption is considered throughout the paper. Additional experiments on multi-step adaption can be found in Section C.3. As in (Finn et al., 2017), the evaluation metric is the average of cumulative rewards of policies adapted from the meta policy.

Experiment Results Figure 2 presents an example of the poor adaptation of the corrupted meta policies and the convergence of two online attack schemes under ISA in a single experiment. In Figure 2(a), when using the policy adapted from the meta policy, the agent is misled into searching the lower-right corner [as the search paths concentrate on that corner], opposite to the true goal. Similarly, with a small perturbation $\delta=0.967$, the cheetah agent fails to learn the desired running-forward policy as shown in Figure 2(c). In this example, it is observed in Figure 2(b)(d) that both attack schemes converge. Intermittent Attack takes fewer iterations since its gradient estimation is more accurate as we discussed in Section 5. To demonstrate the impact of sampling attacks, the first two columns in Table 1 summarizes the average rewards (and their standard deviations) of the policies after one-step adaptation from the meta policies corrupted by different attacks.

Robust Training against Sampling Attacks In Persistent Attack, the attacker and the learner operate in a two-timescale manner. Intuitively, the learner first achieves the inner maximization using a larger step size. On the contrary, if the learner’s step size is smaller than the attacker’s, then the attacker would first achieve the minimization. In this case, the Stackelberg game becomes a maxmin problem $\max_{\theta\in\mathbb{R}^{d}}\min_{\delta\in\Delta}f(\delta,\theta)$, inspiring a robust training method. During the meta learning process, the learner can decrease $\eta_{1}$ when it is suspicious of possible attacks. The obtained meta policy returns the best possible adaptation performance when the training data is corrupted. In one of our experiments, when $0.01=\eta_{1}<\eta_{2}=0.1$, it is observed that the meta policy is robust to Persistent Attack under ISA, as the adaptation performance is close to the benchmark. The column “Robust Training” in Table 1 compares the performance of the meta policies trained under Persistent Attack [$(\eta_{1},\eta_{2})=(0.1,0.01)$], Robust Training [$(\eta_{1},\eta_{2})=(0.01,0.1)$], and the benchmark.

This work formally defines three sampling attack models on meta RL in the training phase. The optimal sampling attack is formulated as the solution to a minimax problem, which leads to two online black-box attack mechanisms: Intermittent Attack and Persistent Attack. Experiments show that simply scaling the reward feedback during the training, the attacker can significantly deteriorate the adaptation performance of the learned meta policy.

The main limitation of the proposed online attack schemes is that with the noised gradient feedback, the attacker searches for the $\epsilon$-FOSP (see Definition 1), which is a local notion. Since the objective function $f(\delta,\theta)$ is nonconvex-nonconcave, it is challenging to characterize these stationary points and compare the corresponding objective values with the global optimum. Hence, quantifying the impact of sampling attacks theoretically (e.g., the performance loss or robustness) remains an open problem. Finally, we comment on the potential negative societal impacts of the proposed sampling attacks. As meta RL has been applied to many real-world applications (Hospedales et al., 2021), the stealthy and lightweight sampling attack schemes pose a great threat to these learning systems. Our robust training idea points out a promising way to combat the attack, yet more efforts are needed for a systematic investigation of defense mechanisms.