Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration

Multi-Agent Reinforcement Learning (MARL) has emerged as a powerful paradigm for training groups of AI agents to collaborate and solve complex tasks [6, 7, 8, 9, 10, 24, 25, 26]. QMIX [27] addresses the challenge of credit assignment by computing joint Q-values from individual agents as a value-based method, while MADDPG [12] extends the policy gradient method DDPG [28] to multi-agent settings using centralized critics and decentralized actors. Different from the off-policy MADDPG, MAPPO [20], based on on-policy PPO [21], achieves strong performance in various MARL benchmarks. Despite significant progress, challenges in MARL persist, including generalizability for unseen scenarios, sample efficiency, non-stationarity, and agent communication [29, 30, 31, 32]. In this paper, we mainly focus on zero-shot scalability as one of the generalizability issues.

The first step to generalize MARL is to scale the learned MARL models to different population sizes without further training, which is particularly challenging when each agent’s policy is independently modeled [14, 33]. Some works tackle this by designing population-invariant [34], input-length-invariant [35], and permutation-invariant [36] networks. Graph neural networks [37] and transformers [17, 38] are adopted to handle the varying populations. UPDeT [16] further decouples the observation and action space, using an importance weight determined with the aid of the self-attention mechanism. In addition, curriculum learning [34], multi-task transfer learning [39], and parameter sharing [19] are also applied to scale MARL. SePS [18] models all the policies as $K$ parameter-shared networks, and learns a deterministic function to map the agent to one of the policies, which cannot be adaptively updated during the task. In sum, however, these methods all ignore the heterogeneity or limit the heterogeneity to predefined and fixed $K$ policies when scaling MARL, which may lead to ineffective labor division of the agent teams.

To implement heterogeneity without separately modeling each agent like HAPPO [14], HetGPPO [40] and CDS [15], existing works have delved into this much to learn the role assignment in MARL [41, 42, 43, 44]. ROMA [45] tailors individual policies based on roles and exclusively depends on the present observation to formulate the role embedding, which may not fully capture intricate agent behaviors. On the other hand, RODE [46] links each role with a specific subset of the complete action space to streamline learning complexity. LDSA [47] introduces heterogeneity from the subtask perspective, but it is hard to determine the total number of subtasks by prior knowledge. Despite these efforts, the existing methods encounter difficulties when scaled to new scenarios with varied population sizes as the number of roles is hard to pre-determine to fit new scenarios, and taking the IDs of agents as inputs to learn the roles makes the integration of a new agent unfeasible. Therefore, achieving a balanced trade-off between scalability and heterogeneity remains an ongoing challenge in MARL.

We begin by providing an overview of the background related to Markov Decision Processes (MDP) in multi-agent systems. A Markov game [48] is defined by a tuple $<N,S,A,P,r,\gamma>$, where $N=\{1,\ldots,n\}$ represents the set of agents, $S$ denotes the finite state space, $A=\prod_{i=1}^{n}A_{i}$ is the product of finite action spaces for all agents, forming the joint action space. The transition probability function $P:S\times A\times S\rightarrow[0,1]$ and the reward function $r:S\times A\rightarrow\mathbb{R}$ describe the dynamics of the environment, while $\gamma\in[0,1)$ is the discount factor. The agents engage with the environment according to the following protocol: at time step $t\in\mathbb{N}$, the agents find themselves in state $s\in S$; each agent $i$ selects an action $a_{i}\in A_{i}$, drawn from its policy $\pi_{i}(\cdot|s)$; these individual actions collectively form a joint action $\boldsymbol{a}=(a_{1},\ldots,a_{n})\in A$, sampled from the joint policy $\pi(\cdot|s)=\prod_{i=1}^{n}\pi_{i}(\cdot_{i}|s)$; the agents receive a joint reward $r=r(s,\boldsymbol{a})\in\mathbb{R}$ and transition to a new state $s^{\prime}$ with probability $P(s^{\prime}|s,\boldsymbol{a})$. The joint policy $\pi$, transition probability function $P$, and initial state distribution $\rho_{0}$ collectively determine the marginal state distribution at time $t$, denoted by $\rho_{t}^{\pi}$.

However, in naive MARL, scalability is hampered by the curse of dimensionality, as the state-action space grows exponentially with the number of agents. Besides, there is the non-stationary problem when agents individually update their policies. To address these two issues, the Centralized Training with Decentralized Execution (CTDE) paradigm [12] is proposed. CTDE allows agents to utilize central information during training, yet during execution each agent acts independently.

When agents are unable to observe the concise state of the environment, commonly referred to as partial observation (PO), a Partially Observable Markov Decision Process (POMDP) extends the MDP model by incorporating observations and their conditional probability of occurrence based on the state of the environment [49, 50]. Within the POMDP and CTDE paradigm, the decentralized partially observable Markov decision processes (DEC-POMDP) [51] have become a widely adopted model in MARL. A DEC-POMDP is defined by $<N,S,A,O,r,P,\gamma>$. $o_{i}=O(s;i)$ represents the local observation for agent $i$ at the global state $s$. Each agent utilizes its policy $\pi_{\theta_{i}}(a_{i}|o_{i})$, parameterized by $\theta_{i}$, to produce an action $a_{i}$ from the local observation $o_{i}$. And agents jointly aim to optimize the discounted accumulated reward $J(\theta)=\mathbb{E}_{\boldsymbol{a},s}[\sum_{t}\gamma^{t}r(s,\boldsymbol{a})]$, where $\boldsymbol{a}$ is the joint action.

Policy Gradient (PG) reinforcement learning offers the distinct advantage of explicitly learning a policy network, setting it apart from value-based reinforcement learning methods. PG methods aim to optimize the policy parameter $\theta$ to maximize the objective function $J(\theta)=E_{S}[V_{\pi_{\theta}}(s)]$.

However, choosing an appropriate learning rate in reinforcement learning proves challenging due to the variance of environments and tasks. Suboptimal learning rates may lead to either slow optimization or value collapse, where updated parameters rapidly traverse the current region in policy space.

To address this challenge and ensure stable policy optimization, Trust Region Policy Optimization (TRPO) [52] introduces a constraint on the parameter difference between policy updates. This constraint restricts parameter changes to a small range, preventing value collapse and facilitating monotonic policy learning.

Denote the advantage function as $A(s,\boldsymbol{a})=Q(s,\boldsymbol{a})-R$, with $R=\sum_{t}\gamma^{t}r(s,\boldsymbol{a})$. In the training episode $k$, the parameter update in TRPO follows $\mathop{\theta}_{k+1}={\arg\max}_{\theta}L(\theta_{k},\theta)~{}s.t.\bar{D}_{KL}(\theta||\theta_{k})\leq\delta$, where $L(\theta_{k},\theta)=E[\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{k}}(a|s)}A_{\pi_{\theta_{k}}}(s,a)]$ approximates the original policy gradient objective $J(\theta)$ within the constraint of KL divergence.

Building upon TRPO, Proximal Policy Optimization (PPO) [21] offers a simplified version that maintains the learning step constraint while being more computationally efficient and easier to implement.

In PPO, the objective function is given by:

forcing the ratio $\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{k}}(a|s)}$ to be within the interval $(1-\epsilon,1+\epsilon)$. This ensures that the new parameter $\theta$ remains close to the old parameter $\theta_{k}$.

Multi-agent PPO (MAPPO) extends PPO to the multi-agent scenario [20], specifically considering DEC-POMDP. MAPPO employs parameter sharing among homogeneous agents, where agents share the same network structure and parameters during both training and testing. MAPPO is a CTDE framework, with each PPO agent maintaining a parameter-shared actor network $\pi_{\theta}$ for policy learning and a critic network $V(s,\phi)$ for value learning, where $\theta$ and $\phi$ are the parameters of the policy and value networks, respectively. The value function requires the global state and is used during training to reduce variance.

MAPPO faces challenges when dealing with heterogeneous agents or tasks and lacks the monotonic improvement guarantee present in trust region learning. To address these limitations, [14] introduced Heterogeneous-Agent TRPO (HATRPO) and Heterogeneous-Agent PPO (HAPPO) algorithms. These algorithms leverage the multi-agent advantage decomposition lemma and a sequential policy update scheme to ensure monotonic improvement. HATRPO/HAPPO extend single-agent TRPO/PPO methods to the context of multi-agent reinforcement learning with heterogeneous agents.

Building on the multi-agent advantage decomposition lemma, HAPPO calculates the actor loss for each agent as follows:

where $i_{1:m}$ denotes the permutation of updated $m$ agents, $\theta_{k}^{i_{m}}$ is the network parameter of agent $i_{m}$ in training episode $k$, and $M^{i_{1:m}}(s,\boldsymbol{a})=\prod_{i=1}^{i_{1:m}}\frac{\pi_{\theta^{i}}^{i}(a^{i}|s)}{\pi_{\theta_{k}^{i}}^{i}(a^{i}|s)}(V(s,\phi)-R)$ represents the weighted advantage in HAPPO.

Both MAPPO and HAPPO calculate the critic loss as follows:

In our work, we adopt the parameter-shared HAPPO as the baseline and backbone for constructing our approach, SHPPO.

To design a heterogeneous layer for each agent, we first need to represent the agent’s strategy pattern as a low-dimensional latent variable $l$ as shown in Fig. 2(a). Take agent $i$ as an example, we take a 3-layer MLP as the Encoder to convert the observation $o_{i}$ and the memory from the RNN $h^{t-1}_{i}$ at time step $t$ to a latent distribution. Specifically, the Encoder outputs the mean $\boldsymbol{\mu}_{i}$ and the standard deviation $\boldsymbol{\sigma}_{i}$ of the multi-dimensional Gaussian $\boldsymbol{\mathcal{D}}_{i}$. Like humans, the agents’ strategy patterns may not be exactly the same under similar situations, so the Encoder learns a distribution instead of a fixed vector. The final latent variable $l_{i}$ is generated by sampling to keep the randomness as

Inspired by the classic Actor-Critic architecture [11], an inference net (InferenceNet) is introduced as a critic to guide the learning of the LatentNet (see Fig. 2(b)). Though the LatentNet generates the latent variable individually for each agent, the InferenceNet centrally learns an overall value function to evaluate the whole team. Specifically, the InferenceNet takes $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}$ from all the agents as the inputs, together with the joint global observation $o_{g}$, and then computes $V_{I}$ as the evaluation for the LatentNet. The InferenceNet is trained by supervised learning to minimize the difference between $V_{I}$ and the return from the environment $R$. In another way, it judges whether the learned latent distributions are good enough to help with decision-making based on global observations. Thus, the LatentNet’s goal is to maximize the $V_{I}$.

It is noteworthy that all the agents share the same parameters. Thanks to the CTDE architecture, during the execution and transfer, we only forward the LatentNet and ActorNet, namely, the Agent part in gray color in Fig. 2(b). Though the input dimensions of the InferenceNet and CriticNet vary with the population sizes, this will not influence the transfer to the new scenario to execute. In this way, our method is scalable to fit scenarios with different agent numbers. Please refer to Appendix Table IV for the detailed network settings.

With the generated latent variable for each agent, we can further design the heterogeneous layer for each agent described in Fig. 2(c, d). The heterogeneous layer is a linear layer in the ActorNet. The ActorNet for agent $i$ takes the observation $o_{i}$ and the hidden states of the RNN $h^{t-1}_{i}$ to get the action $a_{i}$. Here, in order to make better sequential decisions, the RNN is introduced to keep the memory from the previous steps by the hidden states. The ActorNet for different agents shares the parameters, except for the heterogeneous layer, whose parameters are generated with each agent’s $l$. The heterogeneous layer computes $hete\_f_{i}$ as Equation 5.

Here, the weights $w_{i}$ and the bias $b_{i}$ are distinct for each agent, resulting in heterogeneous decision-making policies. A decoder, implemented as an MLP, along with two separate linear decoders (denoted as w_decoder and b_decoder), generates $w_{i}$ and $b_{i}$ based on $l_{i}$. In sum, the LatentNet learns to represent the strategy patterns, and the heterogeneous layer turns the latent variable into part of the ActorNet’s parameters. This allows agents to exhibit unique action styles, despite sharing the majority of parameters. Furthermore, they can adapt their styles in response to evolving tasks or changes in team size. The execution of one time step is shown in Algorithm 1.

In this subsection, we formulate the losses to train the networks deployed in our framework. A good latent representation should be helpful for decision-making, and also identifiable and diverse. Therefore, we design three distinct loss items to guide the learning of the LatentNet.

We first employ the value $V_{I}$ from the InferenceNet $\mathcal{L}_{v}$ to evaluate the impacts of the latent variables on decision-making:

where $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}$ are the concatenation of each agent’s $\boldsymbol{\mu}_{i}$ and $\boldsymbol{\sigma}_{i}$. Note that when calculating $\mathcal{L}_{v}(\theta_{L})$ to update the LatentNet, the parameters of the InferenceNet are fixed and will not be updated. To maximize $\mathcal{L}_{v}$, we can expect that the learned latent variables will be improved to help the agents choose a better heterogeneous layer.

Then, we incorporate two additional unsupervised losses as regularization terms to facilitate the learning of identifiable and diverse latent variables.

The first item is $\mathcal{L}_{e}$, the mean entropy of the $\boldsymbol{\mathcal{D}}_{i}$, defined as follows:

where $n$ is the team size, $\mathcal{H}$ is the entropy function of the distribution, $\theta_{L}$ is the parameters of the LatentNet. To minimize the $\mathcal{L}_{e}(\theta_{L})$, the latent distributions tend to be sharper and thus more identifiable.

The second item is the distance among the agents’ latent variables $\mathcal{L}_{d}(\theta_{L})$:

where $cos\_sim$ is the cosine similarity between two vectors. $Norm$ is the normalization of all the distances to ensure the distance distribution :

By maximizing the $\mathcal{L}_{d}(\theta_{L})$, the latent variables will be diverse to encourage the agents to have heterogeneous styles. Note we implement the sampling function as rsample() in Pytorch to get reparameterized samples, so the sampling is differentiable.

Up to this point, we have the overall loss for the LatentNet $\mathcal{L}_{L}(\theta_{L})$ to be minimized as follows,

where $\lambda_{e}$ and $\lambda_{d}$ are the weights of the regularization items. Please see Appendix Table V for the details of the weights.

The InferenceNet is learned by a mean squared error (MSE) loss between the value $V_{I}$ and environment return $R$:

where the MSE loss is calculated as

where $B$ is the minibatch size, $x_{k}$ and $y_{k}$ are the samples in the minibatch. The InferenceNet will be able to infer how good the latent distributions are when $\mathcal{L}_{I}$ converges. Then, The value $V_{I}$ will be a good estimation of the return.

Besides, the ActorNet and the CriticNet are updated following the HAPPO loss Equation 2 and Equation 3, including the parameters of the decoder, w_decoder, and b_decoder. The loss of the ActorNet is a Temporal Difference (TD) loss, $\mathcal{L}_{TD}$:

note that here $\boldsymbol{l}$ is taken as an input without gradients. $V_{C}$ is the value from the critic.

The loss for the critic is $\mathcal{L}_{C}$:

In this subsection, we further explain how the latent learning works to adaptively represent the agents’ heterogeneous strategies in a low-dimension space as the latent variables.

In one episode, the first phase is the interaction with the environment to collect data. Every agent gains a latent variable and then decides its action decision at each step (see Algorithm 1). Data such as observations, hidden states, latent variables, actions, and rewards are saved in the buffer. After the agents succeed, fail, or the step limit is reached, the second phase is training (see Algorithm 2). The agents update their ActorNets one by one, following the procedure in HAPPO [14]. Of course, here all the agents share the same ActorNet. Thus, the ActorNet will be updated for $n$ times based on different agents’ observations, actions, and rewards. After that, the CriticNet, the LatentNet, and the InferenceNet update their parameters. Please refer to Appendix Table IV and V for the detailed network and hyperparameter configurations.

To demonstrate the effectiveness of our method, especially when scaled to unseen scenarios, we adopt two classic and hard MARL environments - Starcraft Multi-Agent Challenge (SMAC) [22] and Google Research Football (GRF) [23], and further modify them to design scalability tasks.

StarCraft II is a real-time strategy game serving as a classic benchmark in the MARL community. In the Starcraft Multi-Agent Challenge (SMAC) tasks, a team of allied units aims to defeat the adversarial team in various tasks.

Google Research Football (GRF) contains diverse subtasks in a football game. The agents cooperate as part of a football team to score a goal against the opposing team under scenarios such as counterattack, shooting in the penalty area, etc.

In order to directly zero-shot transfer the learned models to tasks with varied populations, we modify the original environments to fix the number of allies and enemies that one agent can observe. This way, the observations and actions length is the same before and after the transfer. Hence, we can concentrate on transferring strategies without the need to map observations, which is not the primary focus of this paper. More details of the modified tasks can be found in Appendix A. All the experiments in our paper are conducted on the new scalable tasks.

As for metrics, we adopt training rewards to evaluate the training progress. We also test the performance on the test environments to calculate the win rate for SMAC or score rate for GRF.

We compare the performance of our approach with four other baselines. As SHPPO is built based on parameter-shared HAPPO, we take both original HAPPO where each agent has an individual network, and HAPPO (share) where every agent shares the same parameters as baselines. HATRPO [14] is another heterogeneous MARL method, and for a fair comparison, we adopt its parameter-shared version as a baseline. MAPPO [20] is a classic homogeneous MARL method, and in this paper, we also use parameter-shared MAPPO.

We first evaluate SHPPO on the original tasks. The first task is a heterogeneous task MMM2, where each side has three kinds of different units - Marine, Marauder, and Medivac (Fig. 3(a)). They have different health points, hit points, and even special skills. Therefore, heterogeneous strategies will improve the team’s performance by maximizing each agent’s advantages. In Fig. 4(a), HAPPO performs best as it models every agent individually. However, thus HAPPO consumes more parameters and lacks scalability. In contrast, our approach, SHPPO, outperforms all the other parameter-shared baselines without modeling each agent. The training reward and win rate of SHPPO are very close to those of HAPPO, showing that SHPPO has a similar heterogeneous representation ability to HAPPO (See Table I). The performance of the two heterogeneous methods, HAPPO and HATRPO, is the worst when parameters are shared.

We also illustrate the screenshots with the corresponding latent variables (Fig. 5). Surprisingly, the locations of the latent variables can explain the agents’ strategies. At the early stage of the task (Step 5), the agents form two groups, the Marines move forward and attack in the front (green circle) to cover the Marauders and the Medivac behind (red circle). At step 11, the two armies encounter and the strategies alter accordingly. The Marauders get close to the enemies to attract the fire while the Medivac keeps curing them (blue circle). Some Marines (Agent 2, 5, and 8) are hurt with low health points, thus they retreat to be covered (red circle), while the other Marines attack the enemies (green circle). In the final stage, some agents are dead and the enemies are almost defeated. The only Marauder keeps attracting the fire in the front. However, the Marines attack the enemies regardless of their health points to achieve final victory. We can see that the latent variables successfully learn the heterogeneous strategy patterns in this task, which are located diversely in the latent space. These latent variables contribute to the adaptive policy updates during the task through the HeteLayer in the ActorNet, so the agents can not only have distinct strategies but also change their strategies as the task progresses.

Moreover, we evaluate the proposed method on a homogeneous task 8m_vs_9m (Fig. 3(b)), where each side has the same kind of unit - Marine. Though the agents are all the same, they can have different strategies and adaptively alter during the task. As shown in Fig. 4(b) and Table I, SHPPO outperforms all the baselines including HAPPO. The training process of SHPPO also converges faster than other baselines.

We test the performance on two tasks of GRF. In task 3_vs_1_with_keeper (Fig. 3(c)), our algorithm can control three football players against one adversary player and the keeper. SHPPO is the most effective method among all the methods in Fig. 4(c) and Table I.

In another task counterattack_easy (Fig. 3(d)), four agents need to cooperate together to do a counterattack. They need to pass the ball against one adversary player and the keeper to score. In Fig. 4(d), SHPPO reaches a similar performance with HAPPO after training, though SHPPO does not explicitly model each agent. Still, SHPPO outperforms all the parameter-shared baselines (also see Table I).

The proposed method SHPPO not only performs well on the original tasks on SMAC and GRF, but also has extraordinary zero-shot scalability. We construct a series of new tasks with different numbers of agents and enemies to test the scalability.

In the original MMM2 task, the ally team has 7 Marines, 2 Marauders, and 1 Medivac, while the enemy team has 8 Marines, 3 Marauders, and 1 Medivac. This task is labeled as 721_831. To adjust the difficulty of the new task appropriately, we either add one ally unit or remove one ally unit and one enemy unit together. This ensures that the task is neither too difficult nor too easy, avoiding situations where any method would have a win rate of 0% or 100%.

When transferring the learned model from the original task to the unseen tasks, SHPPO does not require additional training. While HAPPO cannot be directly transferred due to each agent having a distinct model, we address this by randomly selecting a model of the same type for the added agent, or removing extra models when there are fewer units in the new task.

From Table II, SHPPO performs the best on most of the unseen zero-shot tasks. Even in cases where SHPPO is not optimal, it still performs comparably to the best. In tasks 821_831 and 731_831, there is one more Marine and Marauder, respectively. Therefore, the homogeneity increases after the transfer, while MAPPO is trained with a shared network and may work better when the task is more homogeneous. It’s worth noting that although we attempt to transfer HAPPO, its performance on all the new tasks is worse due to over-fitting to the role assignments in the original task, despite having a better win rate on the original task.

In Fig. 6, we illustrated the latent variables after the transfer (821_831), where one new Marine is added while the enemies are the same, making the new task easier. Thus, the agents inherit the previous strategies and roles in the team, but further adaptively align them with the easier scenario. At step 7, the added Marine (Agent 9) joins its fellow Marines to attack the enemies (green circle), while other Marines are moving forward to attract and disperse the enemies’ fires, together with the Medivac and Marauders (blue circle). Note that no agents keep staying back to be covered as the red circle in Fig. 5, because they are taking a more aggressive strategy in easier tasks. At step 16, all the Marines stop moving and focus on attacks (green circle). The Marauders go on moving to the front with the support of the Medivac (blue circle). At the final stage (step 33), Marauder 1 dies, and almost at the same time, Marine 3 changes its strategy to replace the dead Marauder, moving to the front (blue circle). With the good representation ability of the latent variables, the HeteLayer can inherit the learned strategies in the original task. More importantly, thanks to the adaptive inference of latent variables based on the observations, the agents are able to update collaboration strategies accordingly when the population size and task difficulty vary.

On another task 8m_vs_9m of SMAC, we also create new tasks with both fewer and more agents, such as 6m_vs_7m and 10m_vs_11m. SHPPO is superior in all the unseen tasks. Surprisingly, HAPPO almost cannot win after the transfer. We suppose that though the agents can have distinct roles in a homogeneous task, their strategies may not be extremely different as they belong to the same unit type. HAPPO possibly overfits every agent’s heterogeneous policy to task 8m_vs_9m, which may not be suitable for the new tasks. Therefore, when we try to transfer HAPPO, the learned strategies do not work on the unseen tasks.

Similarly, we test scalability by removing or adding one agent in the two tasks of GRF. SHPPO maintains the best performance after transfer on the 3_vs_1_with_keeper series. For the counterattack_easy series, despite HAPPO’s best performance on the original task, SHPPO achieves the superior score rate on 3_vs_1_counterattack_with_keeper after the transfer. When adding a new agent, SHPPO also has comparable performance.

We further conduct ablation studies to examine the effects of different parts of latent learning.

In order to test the effectiveness of the latent variables, we first only mask the inputs of the LatentNet with zeros so that the latent variables cannot be adaptively updated according to the new observations. Moreover, in another experiment, we replace all the latent variables with zeros to totally remove the latent learning. In Fig. 7(a, b), when the inputs of LatentNet are zeros, the performance on two tasks of SMAC reduces significantly, especially in task 8m_vs_9m. When the latent variables are set as zeros, it is equal to the removal of the latent learning part, and we can see even worse performance in such a case. Therefore, flexible adjustment of heterogeneous strategies is important for the agents to cooperate.

Next, we test the effectiveness of the losses of latent learning. We first remove the guidance from the InferenceNet $\mathcal{L}_{v}(\theta_{L})$ while keeping the other losses $\mathcal{L}_{e}(\theta_{L})$ and $\mathcal{L}_{d}(\theta_{L})$. Results in Fig. 7(c, d) show that without $\mathcal{L}_{v}(\theta_{L})$, there will be a performance decline on both MMM2 and 8m_vs_9m. It suggests that the InferenceNet is able to assess the value of latent variables to help with agents’ decision-making.

Similarly, in Fig. 7(c, d), we then keep $\mathcal{L}_{v}(\theta_{L})$ but remove $\mathcal{L}_{e}(\theta_{L})$ and $\mathcal{L}_{d}(\theta_{L})$, respectively. The two experiments have similar performance reductions on task MMM2. Nevertheless, on the homogeneous task 8m_vs_9m, the experiment without the entropy loss $\mathcal{L}_{e}(\theta_{L})$ has the lowest win rate. We believe this is due to the fact that, while strategies for a homogeneous task may exhibit some degree of diversity for labor division, they cannot be excessively random or divergent, as the agents belong to the same unit type. While the entropy loss $\mathcal{L}_{e}(\theta_{L})$ can make sure that the learned latent variables are relatively deterministic and identifiable.