Reducing Exploitability with Population Based Training

The discovery of adversarial examples (Szegedy et al., 2014) showed that capable image classifiers can be fooled by inputs that are easily classified by a human. Reinforcement learning (RL) policies have been shown to also be vulnerable to adversarial inputs (Huang et al., 2017; Kos & Song, 2017). However, attackers often are not able to directly perturb inputs of a victim agent. Gleave et al. (2020) instead model the attacker as controlling an adversarial agent in an environment shared with the victim. This adversarial agent does not have any special powers, but can indirectly influence the victim’s observations by taking actions in the world. Gleave et al. train such adversarial policies against a fixed victim policy using RL, and are able to exploit state-of-the-art victim policies in zero-sum, two-agent, simulated robotics environments (Bansal et al., 2018).

This vulnerability is surprising given that self-play has produced policies that can defeat the world champions in Go (Silver et al., 2016) and Dota 2 (OpenAI et al., 2019). Critically, zero-sum games are naturally adversarial, with self-play being akin to adversarial training: so we might expect self-play policies to be naturally robust. Yet training an adversary for 3% of the timesteps the original self-play procedure trained for, Gleave et al. (2020) manage to exploit the victims.

The resulting adversarial policies exhibit counterintuitive behavior such as falling over and curling into a ball in a game where the goal is to prevent the opponent from getting by. This would be a weak strategy against most agents but is able to destabilize the victim. Similar to adversarial examples, these policies showcase failure modes that affect a model, but which a human would most likely be unaffected by. Such attacks are a critical danger to deep RL policies in high-stakes settings where there may be adversaries, such as autonomous driving or automated financial trading.

Our work is inspired by prior work in multi-agent RL using populations of agents, such as policy-space response oracles (Lanctot et al., 2017) and population based reinforcement learning (PBRL; Jaderberg et al., 2019). We evaluate using PBRL to train an agent against a diverse population of opponents. PBRL is a variation of population based training (PBT; Jaderberg et al., 2017), which jointly optimizes a population of models and their hyperparameters for improved convergence, adapted for RL.

Whereas self-play trains an agent to be robust against itself, PBRL with a sufficient number of opponents will force an agent to be robust against a wide range of strategies. We conjecture that this will have a similar benefit as adversarial training (Goodfellow et al., 2015) for classification models.

We evaluate PBRL as a defense in two simple two-player, zero-sum games. We find that the self-play baseline can on average be exploited by an adversary using less than 60% as many timesteps as self-play. The PBRL-trained policy is more robust: more timesteps are needed until an initial adversarial policy can be found.

We make three key contributions. First, we use PBRL as a relatively simple population based method for end-to-end robust training in deep RL. Second, we evaluate this empirically, finding the PBRL-trained victim is less exploitable. Third, we investigate how attributes of the environment (such as dimensionality) and algorithm (such as population size) influence robustness.

Gleave et al. (2020) introduced the adversarial policy threat model, and the first attack: using RL to train an adversary against a fixed victim. Wu et al. (2021) added an auxiliary term to reward the victim for paying attention to the adversary. They apply this attack to the original environments (Bansal et al., 2018), and a new environment: Roboschool Pong. Guo et al. (2021) develop a different attack for semi-competitive games, exploiting agents in the original environments and Starcraft 2.

Comparatively little attention has been given to defenses relative to attacks. Gleave et al. (2020) attempted to harden the victim by fine-tuning it against a fully trained adversary. The hardened victim was robust to that particular adversarial policy – but it was still vulnerable to repeating the attack. Furthermore, the hardened victim achieved lower performance against the original, non-adversarial opponent. Guo et al. (2021) use the same approach and find that fine-tuning on adversaries from a stronger attack can provide robustness against adversaries from weaker attacks. However, the authors don’t evaluate the robustness of the hardened victim to an attack targeting itself rather than the original victim.

One prior defense is Adversarially Robust Control (ARC; Kuutti et al., 2021). They consider the semi-competitive setting of autonomous driving. They find that imitation-learned policies are vulnerable to adversarial vehicles trained to cause collisions – even when the adversary is limited to only causing preventable collisions. To improve robustness they fine-tune the imitation policies against an ensemble of adversaries that train concurrently with the main policy. Since autonomous driving is semi-competitive, an optimal policy against adversaries might fare poorly against regular agents, so Kuutti et al. add an auxiliary loss to keep the fine-tuned policy similar to the imitation policy. In contrast, we focus on the more challenging zero-sum setting which self-play was designed to work with.

Vinitsky et al. (2020) formulate the single-agent robust RL problem as a zero-sum two-agent problem: The adversary learns to apply noise perturbations to the environment dynamics trying to minimize the performance of the robust agent in the environment. To improve robustness the authors train the control agent against a population of these adversaries and find this increases robustness against overfitting to particular adversaries, increases robustness against an out-of-distribution held-out test set, and is less brittle than the domain randomization baseline.

Compared to this, our work considers a more general attack setting: Instead of an adversarial formulation of a single-agent control setting, we analyze two-agent symmetric and asymmetric games. It seems likely that these kinds of two-player games have a higher diversity of possible adversarial strategies compared to adversaries that control limited perturbations of transition dynamics. Lower strategic diversity could be one possible reason why Vinitsky et al. encounter diminishing returns at fairly low sizes of adversarial populations. Additionally, we generally train with higher population sizes, at the cost of increased computational effort. Furthermore, Vinitsky et al. investigate robustness under swaps of adversaries from different training runs, essentially off-distribution robustness, whereas our threat model consists of newly-trained adversaries.

Our work is inspired by prior work in multi-agent RL using populations of agents. Notably, policy-space response oracles (PSRO; Lanctot et al., 2017) learn an approximate best response to a mixture of policies. Furthermore, AlphaStar’s Nash league (Vinyals et al., 2019) uses a large population of agents playing against each other, with diverse objectives. Our contribution relative to prior work is an empirical study of the exploitability of population-based trained agents relative to self-play. To avoid confounders, we keep training as close as possible to our self-play baseline, using a relatively simple form of population-based training compared to much prior work.

There are also a variety of multi-agent RL approaches that do not use populations but may produce policies that are less exploitable than self-play. One successful technique is counterfactual regret minimization (Zinkevich et al., 2007) that can beat professional human poker players (Brown & Sandholm, 2018), although has difficulty scaling to high-dimensional state spaces. We choose to focus on self-play as our baseline since it is widely used and, despite theoretical deficiencies, has produced state-of-the-art results in many environments.

Self-play has been shown to converge to Nash equilibria provided that RL converges to a best response asymptotically (Heinrich et al., 2015). However, RL may never converge to a best response. A major culprit is non-transitivity: Balduzzi et al. (2019) show that in games like rock-paper-scissors, self-play may get stuck in a cycle. Even in a transitive game, deep RL algorithms may never converge to a best response if they are stuck in a local minimum, or if the policy network cannot represent the optimal policy.

Following Gleave et al. (2020), we model the tasks as Markov games $M=(\mathcal{S},(\mathcal{A}_{\alpha},\mathcal{A}_{\nu}),\mu,\mathcal{T},\gamma,(R_{\alpha},R_{\nu}))$ (Littman, 1994) with state set $\mathcal{S}$, action sets $\mathcal{A}_{i}$, initial state distribution $\mu$, transition dynamics $\mathcal{T}$, discount factor $\gamma$ and reward functions $R_{i}$. The index $i\in\{\alpha,\nu\}$ represents the adversarial ($\alpha$) and victim ($\nu$) agent respectively.

We focus on two-player, zero-sum games as they have a clear competitive setup and obvious adversarial goals. Consequently $R_{\alpha}=-R_{\nu}$. Notably, the victim policy trained in a zero-sum game will be harder to exploit than those in a positive-sum setting, where they may have learned to cooperate with other agents (in an exploitable fashion) to maximize their overall reward.

The attacker has grey-box access to the victim. That is, they can train against a fixed victim, but cannot directly inspect its weights. The attacker has no additional capabilities to manipulate the victim or the environment.

We would like to find a Nash equilibrium $(\pi_{\nu},\pi_{\alpha})$, which for zero-sum games corresponds to the minimax solutions: We take the $\arg\min_{\pi_{\nu}}\max_{\pi_{\alpha}}$ of the expectation $\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R_{\alpha}(S^{(t)},A_{\alpha}^{(t)},A_{\nu}^{(t)},S^{(t+1)})\mid\pi_{\alpha},\pi_{\nu}\right],$ with random variables sampled from $S^{(0)}\sim\mu$, $A_{i}^{(t)}\sim\pi_{i}(\cdot\mid S^{(t)})$, and $S^{(t+1)}\sim\mathcal{T}(S^{(t)},A_{\alpha}^{(t)},A_{\nu}^{(t)})$.

However, as previously discussed self-play may not find the Nash equilibrium. Yet the self-play policies are nonetheless often highly capable – against their self-play opponent. It is known that self-play can get stuck in a local Nash equilibrium: population-based training might avoid this failure mode by a greater diversity of opponents. Notably, adversarially training against some $\pi_{\alpha}$ trained against $\pi_{\nu}$ might just cause $\pi_{\nu}$ to move to a new local Nash equilibrium that is robust to $\pi_{\alpha}$, but not unseen adversary $\pi_{\alpha}^{\prime}$.

Since population based reinforcement learning (PBRL; Jaderberg et al., 2019) exposes the victim to a broader range of opponents than self-play, it seems likely that it might overcome this limitation while retaining much of self-play’s algorithmic simplicity. We train an agent, the protagonist, for robustness by training it against a population of $n$ opponents $\pi_{o_{i}}$. By jointly optimizing against multiple opponents we increase the coverage of the space of opponent policies. Since an adversary $\pi_{\alpha}^{\prime}$ optimizes in a similar way as the opponents it is likely to be close to one of the opponent policies $\pi_{o_{i}}$. Moreover, given sufficient diversity in opponents it may be easier for the protagonist to learn a policy close to global Nash than to learn $n$ strategies that overfit to each opponent.

Each of the $n$ opponents has identical architecture and training objective, differing only in the seed used to randomly initialize their network. We alternate between training the opponents against a fixed protagonist, and a protagonist against all fixed opponents. Instead of training adversaries such that in aggregate they receive as many timesteps as the main agent, all agents (opponent and protagonist) are trained for the same total number of timesteps each. This alleviates a potential factor for diminishing returns with high population sizes, which Vinitsky et al. (2020) encountered. This allows us to make use of an order of magnitude larger population sizes with populations containing tens of opponents.

The total number of training timesteps for all policies is $n+1$ times the number of timesteps the protagonist is trained for. When logging timesteps for PBRL training, we report the number of timesteps the protagonist agent trains, since this is the relevant metric for protagonist training. Note that this means the compute necessary for training PBRL is $n+1$ times higher than self-play at the same number of training steps (although PBRL is more parallelizable).

We train all policies – self-play, PBRL and adversarial – using Proximal Policy Optimization (PPO; Schulman et al., 2017). PPO is widely used and has achieved good results with self-play in complex environments (Bansal et al., 2018). Furthermore, prior work on adversarial policies in higher-dimensional two-agent environments uses PPO (Gleave et al., 2020). We use the PPO implementation in RLlib (Liang et al., 2018), from the ray library (Moritz et al., 2018), due to its support for multi-agent environments and parallelizing RL training.

We evaluate the PBRL defense in two low-dimensional environments, described in Section 5.1. In Section 5.2 we confirm that baseline self-play policies are vulnerable to attacks. To the best of our knowledge, these are the lowest-dimensional environments in which an adversarial policy has been found. Finally, in Section 5.3 we find that PBRL improves robustness against adversarial policies, and explore the relationship with population size.

Unless otherwise noted, in all experiments we train 5 seeds of victim policies. We attack each victim using 3 seeds of adversaries for a total of 15 adversaries. Unless omitted for legibility, 95% confidence intervals are shown as shaded regions for training curves and bars in bar plots.

In this work, we evaluated a hardened PBRL victim agent against adversarial policies trained with similar numbers of timesteps of experience as the victim agent. However, PBRL also requires training $n$ opponents for this many timesteps – so the total number of timesteps and computational resources is $n+1$ times greater.

We believe this overhead is often tolerable. First, defenders may be able to limit the number of timesteps an attacker can train against the victim, such as if access to the policy is behind a rate-limited API. Second, the number of opponents $n$ can sometimes be quite small – just $n=2$ suffices for Simple Push. Finally, defenders often have significant computational resources: while PBRL is unlikely to prevent an attack from a sophisticated adversary like a nation-state, it may be enough to defeat many low-resource attacks. Nonetheless, reducing this computational overhead is an important direction for future work. For example, can we obtain similar performance with fewer opponents if we train them to be maximally diverse from one another?

In addition, if additional compute resources are available, this approach allows a defender to make use of them. Once an agent has converged, using additional compute to continue training in self-play is usually of no use. However, convergence is not sufficient for robustness – as illustrated by the existence of adversarial policies. This approach allows a purposeful use of additional computing power in a way that is fairly similar to self-play. However, more advanced algorithmic improvements such as PSRO (Lanctot et al., 2017) might be able to provide similar benefits with a smaller increase in compute.

A key open question is how the number of opponents $n$ required for robustness scales with the complexity of the environment. PBRL will scale poorly if the required population size is proportional to the size of the state space: in more complex environments each opponent will take longer to train and more opponents will be required. But a priori it seems likely that $n$ may depend more on the number of high-level strategies in the environment. This is only loosely related to the dimensionality of the state space. For example, some simple matrix games have high strategic complexity, while some high-dimensional video games have only a handful of sensible strategies.

Our evaluation uses the original adversarial policy attack of Gleave et al. (2020), which we established was strong enough to exploit unhardened victims in these environments. However, it is possible that alternative attacks would be able to exploit even our hardened victim. We hope to see iterative development of stronger attacks and defenses, similar to the trend in adversarial examples more broadly.

We evaluate a population-based defense to reduce the exploitability of RL policies. Empirically, we see an increase in zero-shot robustness against new adversaries compared to self-play training. However, some self-play victims are naturally robust and PBRL comes with an increased computational cost. We find the required size of the population depends on the environment used, and larger populations can increase overall robustness. Supplementary material available at https://reducing-exploitability.github.io, source code available at https://github.com/HumanCompatibleAI/reducing-exploitability.

Removed for double blind review.