Steering control of payoff-maximizing players in adaptive learning dynamics ^††thanks: X.C. gratefully acknowledges the generous faculty startup fund provided by BUPT (No. 505022023).

Evolutionary game theory provides a mathematical foundation for studying mechanisms of cooperation and myriad learning theories toward altruistic behavior [1, 2]. Adaptive learning strategies will evolve in scenarios where self-interested individuals interact with one another. Among others, the Prisoner’s Dilemma (PD) game is a symmetric game involving two players X and Y, and two actions: to cooperate or to defect. In a one-shot PD game, the four possible outcomes correspond to different payoffs from the focal player’s perspective: if both are cooperators, one gets the reward $R$, if a cooperator is against a defector, the sucker’s payoff $S$, if a defector is against a cooperator, the temptation $T$, and if both are defectors, the punishment $P$. The game is considered a paradigm for understanding the conflict between self-interest and collective interest as the payoff structure satisfies $T>R>P>S$. For a particular example, the donation game is a simplified form of the PD game, where a cooperator offers the other player a benefit $b$ at a cost $c$ to itself with $b>c$ but a defector offers nothing. The iterated Prisoner’s Dilemma (IPD) games further assume repeated encounters between the same two players and sheds insights into the idea of direct reciprocity [3].

Despite the astronomically huge individual behavioral strategy space for IPD game interactions [4, 5], the so-called zero-determinant (ZD) strategies, including extortioners (extortionate ZD), compliers (generous ZD), and equalizers, are a set of rather simple memory-one strategies yet can unilaterally set a linear relation between their own payoff $s_{X}$ and their co-player’s payoff $s_{Y}$ [6, 7, 8, 9, 10]. The three free parameters that determine a ZD strategy $\bm{p}=[p_{1},p_{2},p_{3},p_{4}]$ and how unequal the corresponding payoff relation $s_{X}-O=\chi(s_{Y}-O)$ is are the baseline payoff $O$, the extortion factor $\chi$, and the normalization factor $\phi$. An equalizer with $\chi=+\infty$ will fix their co-player’s payoff to the given $O$, which can take any value between the punishment $P$ and the reward $R$. An extortioner with $O=P$ and $\chi>1$ can always get an unfair share of the payoffs in conventional IPD games with $T+S>2P$ [7]. In contrast, a complier with $O=R$ and $\chi>1$ can guarantee that their own payoff is never above their co-player’s [8].

As such, the finding of ZD strategies has greatly spurred new waves of work from diverse fields [11, 12], aiming to (i) elucidate the robustness and resilience of cooperation by means of the natural selection of IPD strategies from a population dynamics perspective [13, 14, 15] and (ii) explain reactions of human subjects to extortion that could be out of a desire for profit and/or a concern for fairness [16, 17]. Despite the witting to gain the upper hand in IPD games, there still exists one open issue regarding ZD strategies: how will the payoff structure and the three parameters determine their pairwise dominance and extortion ability?

We have fully addressed the above question and revealed the unforeseen Achilles’ heel of ZD strategies. Most of all, we have found and characterized multiple general classes of strategies that are unbending to extortion and can trigger the backfire of being extortionate and even outperform an extortioner in certain conditions. A fixed unbending strategy will compel a self-interested and adaptive extortioner who tries to maximize their payoff to be fair and behave like Tit-for-Tat (TFT) [1, 18] ultimately by letting $\chi\to 1$. Examples of unbending strategies include general ZD strategies with the baseline payoff $O$ satisfying $P<O\leq R$ and the simple adaptive player Win-Stay Lose-Shift (WSLS) [19], of which the latter can even outperform an extortioner in interactions of more adversarial nature where $T+S<2P$ [20].

Remarkably, the insistence of an unbending player on fairness can rein in not only extortioners but also a much broader family of strategies. In the present work, we further illustrate such a steering role toward altruism by considering the adaptive learning dynamics of a focal reactive player whose move in the current round depends on what the co-player did in the last round [18] against another fixed unbending co-player in a donation game [13]. In so doing, reactive players not only are de facto ZD strategies but also can be conveniently visualized within a unit square. It turns out that unbending strategies from different classes either “train” the reactive player to behave like generous TFT [19] or a full cooperator. Therefore, unbending players can helm the adaptive learning dynamics of extortionate and reactive players to fairness even in the absence of population dynamics and evolution. Our findings help illuminate ideas for multi-agent optimization in computational learning theory.

In this section, we demonstrate the unilateral steering control by unbending players that use a fixed strategy from class A. Later on, we also show similar steering control by unbending players from class D. In order to visualize the learning dynamics, we choose to focus on the two-dimensional reactive strategies without loss of generality. Our analysis can be extrapolated for other memory-one strategies. In what follows, we explicitly prove the two major objectives of steering control over opponents, that is, characterize conditions for (i) yielding possible maximum payoffs and (ii) rendering fair and cooperative strategies, by using fixed unbending strategies in the adaptive learning dynamics.

Let $\bm{p}=[p_{1},p_{2},p_{1},p_{2}]$ be a reactive strategy and $\bm{q}=[1,q_{2},0,q_{4}]$ an unbending strategy from class A (as shown in Example 8). Using the method by Press and Dyson [7], we obtain the payoff $s_{X}(p_{1},p_{2})$ as a quadratic rational function of $p_{1}$ and $p_{2}$, resulting from the quotient of two determinants. If $p_{1}=1$, for example, $s_{X}(1,p_{2})=s_{Y}(1,p_{2})=b-c$. If $p_{1}=p_{2}=0$, for another example, $s_{X}(0,0)=bq_{4}/(1-q_{2}+q_{4})$. In general, we have

where the denominator $f_{A}(p_{1},p_{2})=(1-q_{2})[(1-p_{1})-q_{4}(p_{2}-p_{1})^{2}]+q_{4}(1-p_{2})$ is always positive and

Let $\bm{p}=[p_{1},p_{2},p_{1},p_{2}]$ be a reactive strategy and $\bm{q}=[q_{1},q_{2},q_{3},q_{4}]$ an unbending strategy from class D (as shown in Example 2). We obtain $s_{X}(p_{1},p_{2})$ as a linear rational function of $p_{1}$ and $p_{2}$. If $p_{1}=p_{2}=1$, for example, $s_{X}(1,1)=[(b-c)q_{3}-c(1-q_{1})]/(1-q_{1}+q_{3})$. If $p_{1}=1$ and $q_{1}=1$, for another example, the unbending co-player becomes a complier with $O=b-c$ and $s_{X}=s_{Y}=b-c$. In general, we have

where $(\ast)=q_{3}(1-p_{1})+(1-q_{1})(1-p_{2})$ and $(\ast\ast)=1-q_{1}+q_{3}-(q_{2}-q_{1})(p_{2}-p_{1})$.

Our search for unbending strategies is directly motivated by suppressing extortion, thereby requiring targeted interactions with ZD opponents. Nevertheless, our work is more broadly motivated by how to foster and enforce fairness and cooperation in pairwise interactions in the IPD games. We reveal the previously unforeseen unbending properties of many well-known strategies in that they actually have the ability to resist extortion. For example, class A contains “PSO Gambler”, an optimized memory-one strategy that is unbending when $T+S>2P$. Other well-known examples of unbending strategies include WSLS from class A when $T+S<2P$, “willing” from class C, and all the strategies from class D which are ZD players themselves but with a higher level of generosity than their opponents.

As such, unbending strategies can be used to rein in seemingly formidable extortionate ZD players, from whom a fair offer can ultimately be cultivated in their own interest. Our findings are in line with recent experimental evidence suggesting that human players often choose not to accede to extortion out of concern for fairness [16]. The conclusion can be further generalized and shed light on the backfire of extortion by unbending players in a broad sense. Our analysis shows that an adapting reactive player X after a higher payoff will cooperate more by increasing $p_{1}$ and in some cases $p_{2}$ to avoid potential punishment from a fixed unbending co-player Y in a donation game. Unbending strategies from class A would “train” the reactive player to behave like generous TFT ($p_{1}=1$) whereas those from class D would even “train” the reactive player to act as a full cooperator ($p_{1}=p_{2}=1$).

Under the influence of a co-player Y from class A of unbending strategies, the learning dynamics of a reactive player X may exhibit bistability along the direction of change of $p_{1}$, hence allowing two different learning outcomes, depending on both the original state of the focal player and the specific strategy of the co-player. After entering full cooperation with respect to $p_{1}$, player X will stay neutral along the direction of change of $p_{2}$. A reactive strategy can therefore converge to the full defector corner $(0,0)$ or otherwise to the generous TFT edge $(1,p_{2})$. Player X will always get the greatest payoff at the cooperative edge $(1,p_{1})$ if the ratio $b/c>(\sqrt{5}+1)/2$ (the ‘golden ratio’) or converge to the edge if player Y takes a strategy from a subset of class A. On the other hand, unbending strategies from class D are able to guarantee global convergence to the same cooperative edge, after which the baseline payoff $O$ of player Y can further decide the direction of change of $p_{2}$. Player X will increase the value of $p_{2}$ until reaching the full cooperator corner $(1,1)$ if player Y uses an intermediate ZD strategy with $P<O<R$ or remain neutral along the edge $(1,p_{2})$ if player Y uses the generous ZD strategy with $O=R$.

In a nutshell, we have found and characterized general classes of unbending strategies that are able to steer the learning dynamics of selfish players in head-to-head encounters. Unbending strategies can trigger the backfire of greedy exploitations, punish the extortion, and thus turn the interactions into an Ultimatum game: to “win each battle” or to “win the war”. These strategies helm the reactive learning dynamics of extortionate and reactive players to fairness without introducing population dynamics and evolution through generations.

Our work helps pave the way for the promising initiative of combining game theory with artificial intelligence to gain more analytical insights into computational learning theory. Of particular interest are extensions to multi-agent learning systems that are fraught with perception and implementation errors and beyond pairwise interactions in a changing environment [21]. In doing so, integrating theoretical and empirical approaches will help enhance our understanding of cooperation in various advanced AI systems besides social and biological systems [22].