Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions

In RP framework, each agent $i$ is assigned a randomly sampled role embedding $z^{i}$ in every trial and learns to effectively adapt to and play this role. The challenge lies in defining the role space and enabling agents to adapt efficiently to their assigned roles. To address this, we employ a reward shaping method and introduce a reward feature mapping function $\psi(r^{i},z^{i})$ that processes the reward $r^{i}$ and the role embedding $z^{i}$ to generate a new reward value for agent $i$. This function $\psi(r^{i},z^{i})$ is specifically designed to capture the role-specific reward information, which is crucial for the agent to adapt to its role. During training, agents interact with a variety of other randomly sampled roles, optimizing their strategies based on these interactions. Consequently, if the role space is sufficiently diverse to adequately represent the entire policy space adequately, agents are equipped to effectively handle interactions with previously unseen agents.

Each agent $i$ takes action without communications, based on its observation $o^{i}$ and role embedding $z^{i}$, receiving individual rewards $r^{i}$. Its goal is to maximize its expected cumulative discount reward

where $J(\pi(z^{i}),\bm{\pi}(\bm{z}^{-i}))$ denotes the expected sum reward achieved by agent $i$ using policy $\pi(z^{i})$ while other agents use policies $\bm{\pi}^{-i}(\bm{z}^{-i})$. The detailed form of $J(\pi(z^{i}),\pi(\bm{z}^{-i}))$ is given by

Given the vast size of the policy space, it is challenging for the mapped role space $\mathbb{Z}$ to fully represent the entire policy space one-to-one. Inevitably, this reduction results in some loss of information. To address this issue, we analyze the adaptability with respect to missing strategies using an $\epsilon$-close (Ko, , 2006; Zhao et al., , 2023) approximation.

Based on Definition 4.1, if the random policy $\pi^{\prime}$ is $\epsilon$-close to the role policy $\pi(z^{\prime})$, we can derive the following theorem.

Theorem 4.1 suggests if a random policy $\pi^{\prime}$ is $\epsilon$-close to the role policy $\pi(z^{\prime})$, the expected cumulative reward for the learning policy $\pi(z)$ in relation to the random policy $\pi^{\prime}$ is approximately the same as optimizing it in relation to the role policy $\pi(z^{\prime})$. This implies that the learning agent can obtain an approximately optimal policy by optimizing the expected cumulative reward with respect to a role-based policy. However, multi-agent games are complex and dynamic, where each agent’s actions can unpredictably influence others. This uncertainty makes it challenging for agents to optimize their behavior effectively. To address this issue, we propose a prediction method to estimate the $\epsilon$-close role policy.

We aim for the learning agent to accurately predict the joint role embeddings of other agents. To achieve this, we introduce a role predictor $q_{\phi}$, which is trained to predict the joint role embeddings of other agents, denoted $\hat{\bm{z}}^{-i}_{t}$, based on the history of the observation $o_{0:t}$ and its own role embedding $z^{i}$ at time step $t$. Specifically, the prediction model is formulated as:

This model architecture enables the agent to integrate observational history and predefined role to infer the joint role of other agents within the environment, enhancing its predictive accuracy and adaptability.

Utilizing the role predictor $q_{\phi}$ and theorem 4.1, the learning agent optimizes its policy with the role embedding $z^{i}$ by maximizing its expected cumulative discount reward

Once the learning agent is assigned a role, it approximates the dynamics of the MARL problem into a unified single-agent RL framework and optimizes its policy independently.

Up to this point, the learning agent can adaptively interact with any unseen policies for a given role, assuming that both the policy and the role predictor are effectively trained. Unfortunately, the vast space of joint policies from other agents presents significant challenges in terms of generalization and stability.

To address the challenges of generalization and stability highlighted in the preceding discussion, we introduce meta-learning techniques (Duan et al., , 2016; Finn et al., , 2017; Fakoor et al., , 2019) into RP. Meta-learning enables agents to generalize from a limited set of experiences to new, unseen scenarios effectively. In our approach, we model the interactions between the learning agent and other agents as meta-tasks, utilizing meta-learning techniques to develop the learning agent’s policy.

We treat the role embeddings of other agents as context variables that encapsulate relevant information about the task environment. These role embeddings provide the contextual backdrop for the meta-learning process, offering a rich representation of the dynamic interactions within the multi-agent system. The sampling method of role embedding generates a probability distribution over tasks. Thus, we view the entire learning process as meta-learning tasks, where the agent must adapt to various roles in the environment, guided by the context provided by the role embeddings.

In this subsection, we summarize the RP framework and provide detailed insights into its practical training phase. The algorithm is delineated in Algorithm 1.

In practical training, we employ the typically meta-learning algorithm, $RL^{2}$ (Duan et al., , 2016), to optimize the policy, as illustrated in Fig. 3. Throughout the iterative process, the algorithm dynamically assigns role embeddings $z^{i}$ to agents, enabling them to predict joint role embeddings of other agents $\hat{z}^{-i}_{t}$. This facilitates action selection $a_{t}^{i}$ based on these roles and observed interactions. Rewards are subsequently mapped using the defined feature function $\psi$. The role predictor $q_{\phi}$ is updated by minimizing the prediction error between the predicted role embedding and the true role embedding. The learning agent is trained to maximize the expected cumulative mapped reward by interacting with other agents.

Overcooked (Carroll et al., , 2019) is a two-player cooperative game intended to test the collaboration ability of agents. In this game, agents work collaboratively to fulfill soup orders using ingredients like onions and tomatoes. Agents can move and interact with items, such as grabbing or serving soups, based on the game state. To complete an order, agents must combine the correct ingredients in a pot, cook them for a specified time, and then serve the soup with a dish to earn rewards. Each order has distinct cooking times and rewards.

Experimental setup. In Overcooked, we employ an event-based reward map function, which is a sparse reward function that rewards agents for some events happening. We select certain events already implemented in Overcooked and assign three preferences levels-hate, neutral and like-for each event which will lead to a negative, zero and positive reward. The reward shaping function is defined as $\psi(r^{i},z^{i})=r^{i}+\sum_{k}(z^{i}_{k}*E_{k})$, where $z^{i}_{k}\in[-1,0,1]$ is the preference of event $k$, and $E_{k}$ is the reward of event $k$. Details of the experimental settings in Overcooked are provided in Appendix B.1.

Fig. 4 illustrates the three layouts considered: Asymmetric Advantages, Cramped Room, and Counter Circuit. The Asymmetric Advantages layout includes only onions, with two agents separated into distinct areas. Cramped Room also features an onion-only layout but places both agents in a shared activity space. Counter Circuit presents a more complex setup, with both onions and tomatoes; agents must collect ingredients and prepare the correct soup to earn rewards.

We evaluated the performance of all algorithms in tasks where the trained agent interacts with various scripted agents implemented using HSP (Yu et al., , 2023). In each task, policies engaged with the scripted agent over 100 episodes, and we calculated the mean and standard deviation of the episode rewards. Since the Asymmetric Advantages and Cramped Room layouts contain only onions, we assessed performance across the following tasks: Deliver Soup Agent, which the scripted agent only delivers soup; Place Onion in Pot Agent, which the agent only places the onion in the pot; and Place Onion in Pot and Deliver Soup Agent, which the agent performs both actions. Fig. 5 shows the evaluation results for these onion-only layouts. In the Asymmetric Advantages layout, AnyPlay and TrajeDi perform well in Deliver Soup Agent task but struggle in the Place Onion in Pot Agent task, as their policy pools from stage one rarely contain potting policies, which limits their adaptability. By contrast, both HSP and RP overcome this issue using reward shaping methods. While HSP is limited by the size of its policy pool, RP achieves higher rewards by generating a wider range of policies through role embeddings, allowing it to adaptively apply different reward shaping functions. In the simpler layout, Cramped Room, all algorithms achieve satisfactory performance. However, RP exhibits a higher level of adaptability by consistently adjusting its policies to suit varying tasks, outperforming others in maintaining robust results across different scripted agent interactions.

In the more complex Counter Circuit layout, where agents must gather onions and tomatoes to prepare correct soup, we evaluated performance across three additional tasks: Place Tomato in Pot Agent, where the agent only places tomatoes in the pot; Place Tomato in Pot and Deliver Soup Agent, where the agent both places tomatoes in the pot and delivers soup; and Mixed Order, where the agent places various ingredients in the pot. The results is shown in Fig. 6. AnyPlay, Trajedi and BRDiv obtain policy pools through an implicit way, which limits their ability to cover the diversity of strategies required in complex scenarios, resulting in poor performance across all tasks. In contrast, RP and HSP apply reward shaping methods (an explicit approach) to obtain a broader range of policies, leading to higher rewards. RP consistently outperforms HSP, highlighting its superior adaptability in challenging scenarios. Notably, some results exhibit significant standard deviations, due to the sparse reward structure, where collecting or delivering orders variably impacts the total score.

In this subsection, we evaluate the performance of the RP framework in two-player mixed-motive games, Harvest and CleanUp (Hughes et al., , 2018; Leibo et al., , 2021), which require agents to balance individual contributions against collective benefits. These games introduce a higher level of complexity due to inherent conflicting interests, providing a challenging test environment for agents. To clarify the mixed-motive game settings, we provide detailed explanations of these games.

Harvest: agents face a common-pool resource dilemma where apple resources regenerate more effectively when left unharvested in groups. Agents can choose to harvest aggressively, risking future availability, or sustainably, by leaving some apples to promote long-term regeneration. Additionally, agents can use beams to penalize others, causing the penalized agent to temporarily disappear from the game for a few steps.

CleanUp: a public goods game in which apple growth in an orchard is hindered by rising pollution levels in a nearby river. When pollution is high, apple growth stops entirely. Agents can reduce pollution by leaving the orchard to work in polluted areas, highlighting the importance of individual efforts in maintaining shared resources. Agents are also able to use beams to penalize others.

Experimental setup. We implemented the scenarios using the MeltingPot framework (Leibo et al., , 2021) and applied SVO (Murphy and Ackermann, , 2014) to categorize agent roles. Utilizing the ring formulation of SVO, we introduced SVO-based (role-based) rewards as intrinsic motivators for agents to learn role-specific strategies. The reward feature mapping function $\psi$ is defined as:

where $r^{i}$ denotes the individual reward for agent $i$, $\bar{r}^{-i}$ the average reward of other agents, and $w$ a hyperparameter adjusting the impact of role-based rewards. We use eight main roles $\left\{z^{i}=\frac{k\pi}{4}\mid k=-4,\dots,3\right\}$ for training, as introduced in 3.

In this experiment, we evaluate performance across three tasks for each game. Each task involves a policy pretrained with a specific reward function, with the learning agent interacting with this pretrained agent to assess performance. The tasks are:

• •

  Selfish Agent: the agent pretrained with its own reward function.

• •

  Prosocial Agent: the agent pretrained with the collective reward function.

• •

  Inequity-Averse Agent: the agent pretrained with the inequity-averse reward function, as described in (Hughes et al., , 2018).

Since the baseline algorithms were not designed for mixed-motive games, we tested them in a cooperative setting by training with collective rewards and evaluating collective rewards in both Harvest and CleanUp to ensure fairness. Further details on baseline algorithms and experimental settings in these mixed-motive games are provided in B.2.

As shown in Table 1, RP demonstrates comparable performance in both Harvest and CleanUp games, especially in CleanUp game. In Harvest, although AnyPlay and HSP ultimately learn a non-confrontational policy, they contribute little to the collective reward. BRDiv learns a resource-competing policy, leading to resource depletion in the Selfish Agent task, and performs moderately in the Prosocial Agent task. In the Inequity-Averse Agent task, all baselines perform poorly. Instead, RP shows good performance in all tasks, because RP learned to dodge and harvest in interactions with aggressive roles. When RP play an individualistic role ($z=0$), it tends to use the beam to penalize other agents to get a individual reward, which lowers the collective reward.

In CleanUp, all baselines failed to learn a robust policy that achieves high collective rewards in evaluation tasks, as the environment requires agents to divide the work and cooperate to achieve a high collective outcomes. The policy pools they generated in the first training stage are insufficient to help them learn effective policies. In contrast, RP can utilize the exploration data with cooperative roles to learn a good policy. In the Selfish Agent task, RP learns to clean pollution all the time. In the Prosocial Agent task, RP learns to clean the pollution while the other agent is harvesting, once the other agent leaves the orchard, RP will harvest apples to maximize collective reward. Since the pretrained inequity-averse agent has lower advantageous inequity aversion²²2Details are shown in B.2, it converges to harvest apples if they are available in the orchard. Therefore, in the Inequity-Averse Agent task, RP learns to clean the pollution continuously. We prefer readers to our project website³³3https://weifan408.github.io/role_play_web/ for visualizations of the results.

In this subsection, we examine the impact of the role predictor and the meta-learning method on the performance of the RP framework through comprehensive ablation studies. We assess the contributions of each component by systematically removing the role predictor and the meta-learning method. We analyze interactions by having each role engage separately with all eight distinct roles. For each unique role pairing, agents interact over a series of 1000 episodes. Given the considerable variation in rewards associated with different roles, we employ the mean episode reward of each role as a metric for assessing performance. The results are shown in Fig. 8. The figure clearly illustrates RP’s superior adaptability to diverse roles compared to configurations lacking the role predictor and meta-learning method. The inset provides a detailed view of RP’s roles that achieve positive average rewards. Roles commonly seen in real-life settings—such as Altruistic, Prosocial, Individualistic, and Competitive—yield positive rewards. In CleanUp, the Sadistic role also achieves a small positive reward due to its exploitative tendency, though this behavior lacks a virtuous cycle, resulting in only minimal positive gains in average. For the Individualistic role ($z=0$), RP achieves the highest positive rewards. By contrast, in the ablation configuration without the role predictor and meta-learning method, the rewards are significantly lower even negative. Notably, RP attains positive average rewards when adopting prosocial roles and incurs negative rewards with antisocial roles, whereas policies from the ablation studies consistently produce either negative rewards or considerably lower positive rewards across all roles.

In our analysis of the RP framework, we investigated the principal role behaviors exhibited in the Harvest and CleanUp games. Experimental setting is the same with 5.3. Fig. 9 visualizes our findings for four main roles that reflect real-life behavioral patterns: Competitive ($z=-\frac{\pi}{4}$), Individualistic ($z=0$), Prosocial ($z=\frac{\pi}{4}$), and Altruistic ($z=\frac{\pi}{2}$).

The Individualistic role, marked by a significantly larger area on the radar chart, suggests an advantageous position in interpersonal dynamics, a phenomenon supported by findings in individualism studies (Triandis, , 2018). In contrast, the Competitive role, adept at penalizing others and monopolizing resources such as apples, frequently triggers retaliatory actions from other agents, often culminating in considerable negative rewards. The radar chart shows a large number of blank spaces in the lower half, indicating frequent failures in interactions with antisocial roles.

In the more competitive Harvest game, the Altruistic role demonstrates vulnerabilities as it primarily focuses on the welfare of others, often to its own detriment. Meanwhile, the Prosocial role, which aims to harmonize self-interest with the collective good, shows robust performance in this environments by fostering optimal group outcomes.

Conversely, in the CleanUp game, the Prosocial and Altruistic roles engage in environmentally beneficial behaviors, such as pollution removal. While these actions yield lower direct rewards for themselves, they facilitate greater social welfare for other agents. However, roles driven by self-interest, such as the Competitive and Individualistic roles, often reap benefits from these altruistic actions without bearing any of the associated costs. Additionally, when interacting with the more self-sacrificing Martyr role, both the Prosocial and Altruistic roles experience enhanced social welfare.

Overall, the role behavior analysis demonstrates the adaptability of the RP framework across diverse roles, with each role maintaining its distinct SVO.

In the RP framework, the role predictor is utilized to estimate the joint role embeddings of other agents. Fig. 10 displays the prediction outcomes of the role predictor $q(\phi)$ within both the Harvest and CleanUp games. Each heatmap represents the prediction results of a role interacting with all roles. The colors in the heatmap indicate the proportion of predictions that align with actual roles, with colors closer to the diagonal representing higher accuracy. This visualization method clearly illustrates that the most accurate predictions are those where the color intensity peaks along the diagonal, indicating a strong alignment between predicted and actual roles. These results confirm the efficacy of the role predictor in accurately forecasting the role embeddings of other agents. Notably, the predicted roles align with the actual roles or their close counterparts. Although prediction performance is slightly reduced in the CleanUp game due to its more complex environmental dynamics compared to the Harvest game, the role predictor occasionally outputs incorrect roles that exhibit similar behaviors to the actual roles, which still aids the learning agent in adapting to the environment.