UNMAS: Multi-Agent Reinforcement Learning for Unshaped Cooperative Scenarios

In this paper, we focus on the unshaped cooperative multi-agent scenarios, in which the number of agents and the size of action set change over time. Our contribution to the uncertain challenge of MARL is twofold. (i) The self-weighting mixing network is proposed with the network input dimension adapting with the number of agents, so that the joint action-value is estimated more accurately. (ii) The action set of an agent is divided into two subsets: environment-oriented subset and unit-oriented subset. We use two network streams, the environment-oriented stream and unit-oriented stream, to calculate the Q values of corresponding actions. Finally, we introduce the training algorithm of UNMAS and conduct experiments to compare with several other MARL algorithms in the StarCraft II micro-management scenarios. The results show that UNMAS achieves the highest winning rates on a wide variety of StarCraft II micro-management maps.

Research on independent learning begins with the work of [27], in which the author tries to execute Q-learning independently for each agent to achieve cooperation. With the use of deep learning, IQL [12] introduces Deep Q-Network (DQN) [28] into MARL to cope with high-dimensional observations. Some other research that tackles the multi-agent problem with independent learning can be found in [13, 11, 29, 30]. DyAN [25] divides the observation of the agents into environment-oriented and unit-oriented subsets, and employs a GNN to adapt to the change in the number of agents in the expansion from small-scale scenarios to large-scale scenarios. However, independent learning suffers from the non-stationarity of environment, which leads to the difficulty of learning convergence [20].

Centralized learning treats the multi-agent system as a whole. Grid-Net [31] employs an encoder-decoder architecture to generate actions for each grid agent. [32] and [33] propose methods for the multi-agent system in the framework of actor-critic. [34] formulates the problem of multitask into a multi-agent system. However, the centralized learning method is hard to be scaled to larger scenarios, so the CTDE framework becomes popular as a compromise between independent and centralized learning. One spectrum of the CTDE method is the actor-critic method. MADDPG [35], which is developed from DDPG [36], provides a centralized critic for the whole system and a group of decentralized actors for each agent. COMA [14] computes a counterfactual baseline to marginalize out a single agent’s action while the other agents’ actions are fixed. COMA also provides a centralized critic to estimate the joint Q-function.

The other spectrum is the value-based method. The challenge for value-based methods in the framework of CTDE is how to factorize the joint action-value correctly [20]. VDN [18] factorizes the joint action-value by summing up the individual action-values of each agent. QMIX [5] combines the individual action-values in a non-linear way by a mixing network whose weights and biases are generated according to the multi-agent global state. ASN [26] improves the individual action-value network in VDN and QMIX by considering action semantics between agents. QTRAN [23] guarantees more general factorization by factorizing the joint action-value with a transformation. Q-DPP [20] introduces DPP into MARL tasks to increase the diversity of agents’ behaviors. ROMA [19] and RODE [37] allow agents with a similar role to share similar behaviors. Qatten [38] designs a multi-head attention network to approximate joint action-value function. However, such methods do not consider the problem of unshaped scenarios. They assume that the number of agents and the size of action set are fixed, and this assumption limits the applications of these methods.

This paper is organized as follows. Section II introduces the background knowledge about multi-agent reinforcement learning and factorization of the joint action-value. Section III describes UNMAS from three aspects: joint action-value function, individual action-value function, and training algorithm. Section IV shows the experiments and results, and analyzes the learned strategies. Finally, Section V gives the conclusion.

We consider a fully cooperative multi-agent task with partially observable environment, in which agents observe and take actions individually. This task is also called the decentralized partially observable Markov decision process (Dec-POMDP) [39]. It can be defined as a tuple $\mathscr{U}=\{\mathbb{D},\mathbb{S},\mathbb{U},\mathbb{T},\mathbb{O},R,\gamma\}$. $\mathbb{D}=\{1,...,n\}$ is the set of agents, the number of which is n. The Dec-POMDP extends POMDP by introducing the set of joint actions $\mathbb{U}$ and joint observations $\mathbb{O}$. The multi-agent system takes the joint action $\textbf{u}_{t}=\{u_{1,t},...,u_{n,t}\}$ according to the joint observation $\textbf{o}_{t}=\{o_{1,t},...,o_{n,t}\}$ and gets the immediate reward $r_{t}$ from environment according to the function $R:\mathbb{S}\times\mathbb{U}\rightarrow\mathbb{R}$. Then, the global state of multi-agent system $s_{t}\subseteq\mathbb{S}$ is produced according to the transition function $\mathbb{T}$, which specifies Pr$(s_{t+1}|s_{t},\textbf{u}_{t})$. Finally, $\gamma$ in tuple $\mathscr{U}$ denotes the discount factor of discounted cumulative reward: $G_{t}=\sum_{j=0}^{\infty}\gamma^{j}r_{t+j}$.

In the Dec-POMDP, we consider a joint policy $\boldsymbol{\pi}$, which is composed of the policies $\pi_{i}(u_{i,t}|o_{i,t})$ of every agent $i\in\mathbb{D}$. The joint policy has a joint action-value function: $\textbf{Q}(\textbf{o}_{t},\textbf{u}_{t})=\mathbb{E}_{\textbf{o}_{t+1:\infty},\textbf{u}_{t+1:\infty}}[G_{t}|\textbf{o}_{t},\textbf{u}_{t}]$. The purpose of fully cooperative multi-agent task is to maximize this return.

As mentioned before, value-based methods with CTDE framework need to find an efficient and adaptable way to factorize the joint action-value. A common requirement in the field of CTDE is the Individual Global Max (IGM) condition.

Since the IGM condition is difficult to be verified in practice, the following monotonicity condition is mostly used as its substitute:

If $\textbf{Q}(\textbf{o}_{t},\textbf{u}_{t})$ is factorized monotonically as (2), then this way of factorization meets the IGM condition. Under the IGM condition, maximizing the joint action-value is equivalent to maximizing the action-value of each agent. Thus, the multi-agent system pursues the same goal as every agent and achieves cooperation.

We propose UNMAS in the next section to adapt to the number and size changes in the unshaped scenario.

Fig. 1 presents the self-weighting mixing network, which approximates the joint action-value function. Since UNMAS adopts the CTDE framework, the joint action-value function is only used in training. The weights and biases of the self-weighting mixing network are produced by a group of hyper networks represented by the yellow blocks in the architecture, and the input of the hyper networks is the global state $s_{t}$.

Instead of following the original definition of the joint action-value function that takes joint observations and joint actions as input, CTDE decomposes it as a mapping from the individual action-values of each agent. Such way greatly decouples the complicated interplay between different agents.

The biggest difference between UNMAS and other CTDE methods lies in the ways of dealing with the input size of joint action-value function. In the other CTDE methods, the input size of joint action-value function is determined beforehand and keeps constant during the whole training and execution phase. Since they still provide action-values for dead agents, the joint action-value may be miscalculated and reduce the performance of agent policies. However, the joint action-value function represented by self-weighting mixing network can change its input size with the number of agents. It outputs two scalars $q_{i,t}^{{}^{\prime}}$ and $k_{i,t}$ for each agent:

where $q_{i,t}^{\prime}$ is the result after nonlinear mapping by self-weighting mixing network, and $k_{i,t}$ is the individual weight, which describes the contribution of agent $i$ to the joint action-value. By dynamically evaluating the contribution of each agent according to $k_{i,t}$, the estimation of the joint action-value could be more accurate. Thus, the cooperation of the multi-agent system becomes better. Elu [40] is a nonlinear activation function, which is also used in the mixing network of QMIX. $Q_{i}(o_{i,t},u_{i,t};\theta_{i})$ is the action-value of agent $i$, which is estimated by the individual action-value network. $W_{q}^{1}$, $W_{q}^{2}$, and $W_{k}$ are weights of the self-weighting mixing network as shown in Fig. 1, and $b_{q}^{1}$, $b_{q}^{2}$, and $b_{k}$ are the corresponding biases. $h_{o}^{i}$ and $h_{s}$ are outputs of networks $\textbf{NN}_{o}$ and $\textbf{NN}_{s}$, which take observation $o_{i,t}$ of agent $i$ and the global state $s_{t}$ as input, respectively. Finally, the concatenation of $h_{o}^{i}$ and $h_{s}$ is taken as the input of $[W_{k},b_{k}]$ to calculate the weight $k_{i,t}$.

By employing self-weighting mixing network to evaluate the contribution of each agent, UNMAS can adapt to the change in the number of agents in training and improve the accuracy of the estimation of joint action-value. Taking the StarCraft II micro-management scenario as an example, the number of agents decreases due to the death caused by enemies attack. The weights and biases of the self-weighting mixing network are obtained through the global state. Therefore, even if the $q_{i,t}^{{}^{\prime}}$ of each agent do not take the information of other agents into account, the self-weighting mixing network can still estimate the joint action-value accurately. The joint action-value function is formulated as follows:

where $n$ is the number of agents in the current timestep $t$, and $v_{t}$ is a scalar output from the hyper network. It is a bias of the joint action-value, which is generated from a fully-connected network with $s_{t}$ as input. The ablation result of QMIX [5] shows that adding $v_{t}$ to the joint action-value is able to achieve better results, which is also confirmed in our ablation experiments. In (4), in order to adapt to the change in the number of agents, we use $n$ to average the weighted sum $\sum_{i=1}^{n}q_{i,t}^{{}^{\prime}}\cdot k_{i,t}$. Besides, the weights of self-weighting mixing network take the absolute values to make sure that the factorization meets the IGM condition. The proof that the self-weighting mixing network meets the IGM condition is provided in Theorem 1.

Through the special design of network structure and averaging weighted sum with $n$, self-weighting mixing network can adapt to the change in the number of agents. In the next subsection, we introduce the architecture of the individual action-value network, which can adapt to the change in the size of action set.

The individual action-value network approximates the action-value function of each agent, and it shares the parameters among agents. In the framework of CTDE, an agent takes an action according to its local observation without communicating with the other agents.

In order to help agents adapt to the change in the size of action set, we divide the action set into two subsets. One is the environment-oriented subset, and the other is the unit-oriented subset. The first action subset represents the interactions between the agent and the environment, and keeps constant during the whole running phase. The second represents the interactions between the agent and other agents, and its size varies with the number of agents in the current environment. Taking StarCraft II micro-management scenario as an example, the environment-oriented actions include stop and four movement actions in the directions of up, down, left, and right, while the unit-oriented action is the attack action aiming at an enemy unit.

Based on the division of the action set, we propose the individual action-value network as shown in Fig. 2. Two network streams are constructed for the two action subsets. The environment-oriented stream takes the environment-oriented observation as input. It contains three Fully-Connected (FC) layers $\textbf{FC}_{j}^{e},j=1,2,3$, and one Gated Recurrent Unit (GRU) [41] layer $\textbf{GRU}_{1}^{e}$.

The GRU layer, employed to solve the problem of POMDP, can better estimate the current state. At each timestep $t$, the GRU layer additionally inputs a hidden state $h_{i,t}$, which contains historical information and outputs the next hidden state. Its outputs are the Q values of the environment-oriented actions as follows:

where $Q_{i,t}^{e}$ are the Q values of agent $i$ executing the environment-oriented actions. $o_{i,t}^{e}$ is the environment-oriented observation, which describes the information observed by agent $i$ from the environment. $h_{i,t+1}$ is the hidden state of the next timestep $t+1$. $\textbf{NN}_{i}^{e}$ is the short form of the environment-oriented stream.

The unit-oriented stream takes the unit-oriented observations as input. Its outputs are the Q values of the unit-oriented actions. The unit-oriented observation $o_{(i\to j),t}$ differs from the environment-oriented observation in the fact that it describes the information from agent $i$ to the target unit $j$. Taking the StarCraft II micro-management scenario as an example, if there are 3 enemies on the map at timestep t, the unit-oriented stream takes 3 observations in terms of these enemies to evaluate the Q values of attacking them. Due to the number of enemies changes over time in unshaped scenarios, the number of attack targets also changes. Besides, it should be noted that if the agent has a healing action, the target units here will be other agents, not enemies. Furthermore, we concatenate the first layer’s output of two streams to form a new vector, and feed it into the second layer of the unit-oriented stream. The Q values of the unit-oriented actions are calculated as follows:

where $vector$ is the concatenation of $h_{e,t}$ and $h_{(i\to j),t}$, which are the output of the first part of environment-oriented stream and unit-oriented stream. $Q_{(i\to j),t}(o_{(i\to j),t},u_{(i\to j),t})$ is the Q value of agent $i$ executing the unit-oriented action on target unit $j$. $\textbf{FC}_{2}^{unit}$ is the second layer of the unit-oriented stream.

Due to the concat operation, the output of the unit-oriented stream also contains the historical information provided by $h_{i,t}$, so there is no need to provide a GRU layer for it. For the sake of brevity, the environment-oriented observations and unit-oriented observations of agent $i$ are simplified into one variable $o_{i,t}=\{o_{i,t}^{e},o_{(i\to j),t}\}$. Finally, we concatenate all $Q_{i,t}^{e}$ and $Q_{(i\to j),t}$ to form the individual action-value function $Q_{i}(o_{i,t},\cdot)$.

It should be noted that the individual action-value networks of other methods like QMIX fail in adapting to the change in the size of action set. They use a single network to evaluate the Q values of all kinds of actions, which include both valid and invalid actions.

In this subsection, we introduce the training algorithm of UNMAS. The detail is shown in Algorithm 1. The agents take $\epsilon$-greedy policy to explore the environment in training, that is, choose a random action with probability $\epsilon$, otherwise choose the action with the highest Q value:

where the exploration rate $\epsilon$ decays along the training steps. The transitions and rewards generated by the interaction between the multi-agent system and the environment are stored into the replay buffer D.

When the replay buffer D is full, we sample a minibatch from replay buffer D to update the networks. They are trained by minimizing the loss:

where $b$ is the size of minibatch. $\theta=\{\boldsymbol{\theta}_{joint},\theta_{i}\}$ is the parameter of all networks involved in training. $\boldsymbol{\theta}_{joint}$ and $\theta_{i}$ are the parameters of self-weighting mixing network and individual action-value network respectively. $y_{k}^{joint}$ is the joint update target of the multi-agent system, which is defined as follows:

where the target action $\bar{\boldsymbol{u}}_{k+1}$ is obtained by maximizing the individual action-value function of agents. $\boldsymbol{\theta}^{-}_{joint}$ and $\theta_{i}^{-}$ are parameters of the target network for the self-weighting mixing network and the individual action-value network. Throughout the training phase, they are replaced by the current parameters every fixed episodes, which is defined as target update interval.

The multi-agent system is treated as a whole in the training phase, so the self-weighting mixing network and individual action-value network can also be seen as one neural network, which possesses the parameter $\theta$. At each iteration, $\theta$ is updated by the gradient that is obtained by minimizing the loss function in (11). Since the self-weighting mixing network only appears in the training phase, this learning law indicates that the role of the self-weighting mixing network and its parameter is to help the update of the individual action-value network.

The micro-management scenario is an important aspect of StarCraft II. It requires the player to control a group of agents to win a combat against enemies. The micro-management scenario with a decentralized setting is a proper experimental scenario to test MARL methods because the agents are partially observable and can perform decentralized actions. We use StarCraft Multi-Agent Challenge (SMAC)[42] as the environment and evaluate our method on various maps.

We consider combats with two groups of units, one of which is using UNMAS and the other is the enemy. This scenario is an unshaped scenario, as the number of agents and enemies will decrease due to the attack. In this scenario, each agent is partially observable, which means that it could only get the information within its sight range. The observation mainly contains the following attributes for both allies and enemies: distance, relative_x, relative_y, health, shield, and unit_type. The global state mainly contains the above information for all units, both agents and enemies. Details of the observation and state are provided in the Appendix A. Each agent has the following actions: move[direction], stop, and attack[enemy_id]. Each enemy is controlled by the built-in StarCraft II AI, which uses hand-crafted scripts. We set the difficulty of built-in AI as “very difficult”.

We take the scenario in Fig. 3(a) as an example to show more details about the uncertainty in StarCraft II micro-management. In this scenario, there are three agents and three enemies at the beginning. Each agent receives two kinds of observations. One kind is environment-oriented represented by a tensor of length 42. It contains the information of agent self (8), agent allies and enemies ($2*5+3*5$), and the one-hot coding of agent last action (9). The other kind is unit-oriented represented by a collection of 3 tensors of length 5. Each tensor describes the information of an enemy (5) observed by this agent. Then, agents select actions from their action sets according to these observations. Their action sets contain the following actions: move[up], move[down], move[left], move[right], stop, attack[enemy_0], attack[enemy_1], and attack[enemy_2].

Each agent evaluates actions by the individual action-value network described before. This network takes environment-oriented observations (42) and unit-oriented observations ($3*5$) as input, and outputs the values of actions ($6+3$). The values of 6 environment-oriented actions and 3 unit-oriented actions are computed by the separate streams. If an enemy dies in combats, the size of unit-oriented observations will become $2*5$, and thus the unit-oriented stream just outputs the values of attack actions towards the remaining two enemy units. In this way, the output dimension of the individual action-value network becomes $6+2$.

During training, UNMAS samples from the replay buffer and uses the self-weighting mixing network shown in Fig. 1 to compute the joint action-value of the multi-agent system. The input dimension of this network is 3, which corresponds to the number of agents on the map. Furthermore, if an agent dies later in combat, the individual action-value network provides only the action-values of the remaining two agents. Therefore, the input dimension of self-weighting mixing network becomes 2.

At each timestep, agents perform their actions in the decentralized way and receive a global reward from the environment. SMAC provides positive rewards by default, that is, the damage done by agents to enemy units. In addition, the positive (+10) reward is also provided after an enemy is killed. We evaluate the method by running 32 episodes every 10000 timesteps to get the test winning rates. The agents use $\epsilon$-greedy to choose their action and turn it off in testing. The exploration rate $\epsilon$ decays from 1 to 0.05.

We compare UNMAS with VDN [18], QMIX [5], QTRAN [23], and ASN [26]. VDN factorizes the joint action-value $\textbf{Q}(\textbf{o}_{t},\textbf{u}_{t})$ by the summation of the individual action-value function $Q_{i}(o_{i,t},u_{i,t};\theta_{i})$:

where $\theta_{i}$ is the parameter of the newtork representing the individual action-value function. QMIX employs a mixing network to represent the joint action-value function. This network takes the action-value $Q_{i}(o_{i,t},u_{i,t};\theta_{i})$ of all agents as input and the joint action-value $\textbf{Q}(\textbf{o}_{t},\textbf{u}_{t};\boldsymbol{\theta}_{joint})$ as output, where $\boldsymbol{\theta}_{joint}$ is its parameter. By being taken absolute values, the weights of the mixing network keep non-negative. QTRAN factorizes the joint action-value by designing a special learning objective. Therefore, it can transform the original joint action-value function into a new, easily factorizable one with the same optimal actions in both functions. ASN considers the action semantics between agents to compute the action-values and reconstruct the observation of agent $i$ at timestep $t$ as follows: $o_{i,t}=\{o_{i,t}^{env},m_{i,t},o_{i,t}^{1},...,o_{i,t}^{i-1},o_{i,t}^{i+1},o_{i,t}^{n}\}$, where $o_{i,t}^{env}$ is the observation about environment, $m_{i,t}$ is the private information about agent $i$ itself, and $o_{i,t}^{j}$ is the observation of agent $i$ to other agent $j$.

Experiments are performed on the following symmetric maps, where agents and enemies have the same numbers and types: 3 Marines ($3m$), 8 Marines ($8m$), 2 Stalkers and 3 Zealots ($2s3z$), 3 Stalkers and 5 Zealots ($3s5z$). Besides, the asymmetric maps including 5 Marines against 6 Marines ($5m\_vs\_6m$), 3 Stalkers and 5 Zealots against 3 Stalkers and 6 Zealots ($3s5z\_vs\_3s6z$) are also chosen for comparison. The description of these units is provided in Table I.

These maps are provided by SMAC and could test the performance of UNMAS on StarCraft II micro-management scenario. VDN [18], QMIX [5], and QTRAN [23] are commonly used for comparison in CTDE methods and have already been implemented in SMAC. The code of ASN [26] is also open-source. Besides, the detailed implementation of networks and hyper parameters are provided in Appendix Table B.1.

We apply these methods on the maps, and choose the test winning rate as the comparison metric. Each experiment is repeated three times for average results, and the learning curves are shown in Fig. 4.

1. 1)

   Symmetric Scenarios: The maps with symmetric setting include: $3m$, $8m$, $2s3z$ and $3s5z$.

   1. a)

      Homogeneous Scenarios: $3m$ and $8m$. In these scenarios, there is only one type of unit: Marine. Therefore, the individual action-value network only needs to represent the policy of one type of unit. As shown in Fig. 4(a) and Fig. 4(d), all of the methods quickly achieve a winning rate close to 100%. The curve of UNMAS starts rising slowly, but still achieves a higher winning rate faster than other methods.

   2. b)

      Heterogeneous Scenarios: $2s3z$ and $3s5z$. In these scenarios, there are two types of units: Zealot and Stalker, so the individual action-value network needs to represent the policies of different types of units. Different types of units have different roles in combat, so heterogeneous scenarios are more difficult than homogeneous scenarios. As shown in Fig. 4(b) and Fig. 4(e), UNMAS achieves the highest winning rate on these maps, especially $3s5z$. The $3s5z$ map is more difficult than $2s3z$, because the increase in the number of units makes the multi-agent system harder to control. In the $3s5z$ experiment, VDN and ASN can not perform well, and QTRAN fails to win. Only UNMAS and QMIX can win combats stably.

2. 2)

   Asymmetric Scenarios: The maps with asymmetric setting include: $5m\_vs\_6m$ and $3s5z\_vs\_3s6z$. It means that the number of enemies is greater than that of own agents.

   1. a)

      Homogeneous Scenarios: $5m\_vs\_6m$. In this scenario, 5 Marines controlled by the algorithms have to combat against 6 enemy Marines. Even if there is only one difference in the number, the control difficulty in order to win increases considerably. Therefore, the policies of agents are required to be more elaborate in order to win. A more elaborate policy means that the agents make fewer mistakes. As shown in Fig. 4(c), no method achieves 100% winning rate. Among the tested methods, UNMAS achieves the highest winning rate, which means that the multi-agent system using UNMAS is more possible to achieve cooperation and choose correct actions.

   2. b)

      Heterogeneous Scenarios: $3s5z\_vs\_3s6z$. It is an asymmetric and heterogeneous scenario, the most difficult of all. With an extra Zealot, the enemy is better able to block our Zealots and attack our important damage dealer Stalkers. Therefore, the multi-agent system not only needs to ensure the stability of its policy but also explores a better policy to win. As shown in Fig. 4(f), UNMAS achieves the highest winning rate.

Throughout the plot in Fig. 4, it is shown that UNMAS achieves uniform convergence on all experimental maps, and is especially superior on the most difficult scenario 3s5z_vs_3s6z. The final winning rates of different methods are listed in Table II. Compared with QMIX and other methods, we provide a more flexible network structure to represent the joint action-value function, so it is more suitable for training on unshaped scenarios. As for the design of the individual action-value network, ASN uses the GRU layer in unit-oriented stream, while we remove it. Since we are considering a Dec-POMDP especially with limited sight range, a unit wandering close to an agent’s sight range will cause it to receive the unit’s observations intermittently. Therefore, it may lead to a wrong computation of the hidden state for GRU layer and have a negative effect on the strategy of agent.

Besides, for other state-of-the-art MARL methods, due to the limits of reproduction, we only compare with the results provided in their literature, and show them in Table II. MADDPG [35] provides a centralized critic for the whole system and a group of decentralized actors for each agent. Qatten [38] designs a multi-head attention network to approximate joint action-value function. RODE [37] allows agents with a similar role to share similar behaviors. QPD [22] employs integrated gradients method to factorize the joint action-value of multi-agent system into individual action-values of agents. QPLEX [21] takes a duplex dueling network to factorize the joint action-value.

The results shown in Table II are the average winning rates after 2 million timesteps. In the article of RODE, the experiment on most maps is the result of training 5 million timesteps, we run RODE on each map and test its winning rate at 2 million timesteps. Since MADDPG does not experiment on StarCraft II micro-management scenarios, we compare with the results provided by [43], in which the authors implement several MARL methods. Although these methods achieve similar performance to us in symmetric maps, UNMAS still has obvious advantage in the most difficult scenario 3s5z_vs_3s6z. These results indicate that UNMAS is still competitive among those state-of-the-art methods.

In order to investigate the effect of (i) the weight $k_{i,t}$, (ii) the value $v_{t}$ in the self-weighting mixing network, and (iii) the concat operation in the individual action-value network, we conduct ablation experiments on three maps: $3s5z$, $5m\_vs\_6m$ and $3s5z\_vs\_3s6z$.

1. 1)

   The effect of weight $k_{i,t}$. We remove the weight $k_{i,t}$ in the self-weighting mixing network and replace the joint action-value with the following equation:

   $$\textbf{Q}(\textbf{o}_{t},\textbf{u}_{t};\boldsymbol{\theta}_{joint})=\frac{1}{n}\sum_{i=1}^{n}q_{i,t}^{{}^{\prime}}+v_{t},$$

   (14)

   which denotes that the weight $k_{i,t}$ of each agent is set to 1 in any case. We refer to this method as UNMAS-ADD. Fig. 5 shows that the winning rate of self-weighting mixing network decreases under the condition of fixed weights $k_{i,t}$. One possible explanation is that because the weights are fixed, the self-weighting mixing network cannot correctly estimate the true joint action-value based on the contribution of each agent to the joint action-value.

   

   

   In order to illustrate the effect of $k_{i,t}$ specifically, we provide two screenshots of the combat on the $3s5z\_vs\_3s6z$ map and mark the weight ratio $k_{i,t}/\sum_{j}k_{j,t}$ of each agent as shown in Fig. 6. The screenshots indicate that agents have different contribution to the joint action-value. When there is no close combat between two sides as shown in Fig. 6(a), the long-range unit Stalker is more important. Its average weight of 0.186 is larger than Zealot’s 0.073. Once they are fighting hand-to-hand as shown in Fig. 6(b), the short-range unit Zealot becomes vital. Its average weight of 0.120 is larger than Zealot’s 0.092. Besides, within the same type of units, agents get larger weights if there are more enemies around them. Taking the Zealots in Fig. 6(b) as an example, the unit surrounded by enemies has larger weight (0.162) than the one that is alone (0.095).

2. 2)

   The effect of value $v_{t}$. In the second experiment, we remove the value $v_{t}$ in the joint action-value. Therefore, the joint action-value can be calculated by the following formula:

   $$\textbf{Q}(\textbf{o}_{t},\textbf{u}_{t};\boldsymbol{\theta}_{joint})=\frac{1}{n}\sum_{i=1}^{n}q_{i,t}^{{}^{\prime}}\cdot k_{i,t}.$$

   (15)

   We refer to this method as UNMAS-NV. Fig. 5 shows that the introduction of $v_{t}$ increases the efficiency of approximation, which leads to higher winning rate. It indicates that the lack of a bias term makes it more difficult for the self-weighting mixing network to approximate the joint action-value function.

3. 3)

   The effect of the concat operation. In the third experiment, we remove the concat operation in the individual action-value network. It means that the Q values used to evaluate unit-oriented actions only depend on the observations of target units rather than the observation history. We refer to this method as UNMAS-NCAT. Fig. 5 shows that the concat operation is critical to the performance of the agent. The winning rate of UNMAS-NCAT in ablation experiments is much lower than UNMAS.

According to the ablation experiments, we figure out the importance of the weight $k_{i,t}$, value $v_{t}$, and the concat operation. The reason that UNMAS achieves the highest winning rate becomes clear. In the factorization of joint action-value, UNMAS provides the weights for each alive agent and ignores the agents that are killed by enemies, which leads to a more proper factorization. The bias term also helps approximate the joint action-value function. In estimating the action-value of an agent, UNMAS evaluates the actions in two different subsets respectively with the help of the concat operation, making the evaluation more accurate.

To understand what UNMAS has learned to achieve the performance, we analyze the strategies of agents using UNMAS according to the combat replay in this subsection.

1. 1)

   Homogeneous Scenarios: $3m$, $8m$ and $5m\_vs\_6m$. In these scenarios, the combat is conducted between Marines. Next is the analysis of the strategies executed by the multi-agent system.

   1. a)

      More to Fight Less: The multi-agent system adjusts the formation to form a situation where more agents attack fewer enemies in the combat. It is a basic strategy for micro-management scenario. Taking an example, as shown in Fig. 7(a), agents try to concentrate on attacking an enemy Marine. The red line indicates that the agent is attacking. By adopting this strategy, the multi-agent system is able to eliminate the enemies as quickly as possible to reduce the damage caused by them.

   2. b)

      Damage Sharing: The agents with higher health share the damage for agents with lower health. Since only alive agents can cause damage to enemies, it is important to ensure the survival of agents who is in danger. As shown in Fig. 7(b), when the health of the agent is low, it retreats a distance from the enemies to allow the other agents to share the damage. The white line indicates that the agent is moving. At the same time, the agents around it step forward in the enemy’s direction to ensure successful damage sharing. By adopting this strategy, the agents are able to survive longer to maximize the damage they cause.

In the homogeneous scenarios, the multi-agent system performs the above strategies to defeat the enemies. Although other methods like QMIX can also learn these strategies, the agents using UNMAS make fewer mistakes.

1. 2)

   Heterogeneous Scenarios: $2s3z$, $3s5z$ and the most difficult map $3s5z\_vs\_3s6z$. In these scenarios, the combat is conducted between two types of units: Zealot and Stalker. The former is short-range and the latter is long-range. For scenarios where units have different roles, the multi-agent system not only needs to execute the strategies analyzed above but also needs new strategies to accommodate the change. Next is the analysis of the strategies executed by the multi-agent system.

   1. a)

      Block Enemies: The Zealots try to block the enemy Zealots to protect the Stalkers. Since the Stalker is a long-range unit, it is able to cause damage to the enemies without being attacked. As shown in Fig. 7(c), the multi-agent system using UNMAS learns the strategy of placing the Zealots between Stalkers and enemies to ensure the safety of Stalkers. The yellow line is the defensive line formed by Zealots. Through this strategy, the multi-agent system can maximize its damage as much as possible.

   2. b)

      Defensive Counterattack: Agents focus on eliminating enemy Zealots who break through the defensive line as a defense, and then continue their previous attack as a counterattack. In the actual combat, the defensive line may not completely separate the enemies from Stalkers. As shown in Fig. 7(d), the multi-agent system using UNMAS learns to attack the enemy who breaks through the defensive line. They also perform a more to fight less strategy to eliminate enemies as quickly as possible. In addition, sometimes one Stalker chooses to attack the enemies at the edge of the battlefield to disperse pressure from other allies.

It should be noted that, in the heterogeneous scenarios, the multi-agent system not only learns the strategies mentioned above, but also learns more to fight less and damage sharing as in homogeneous scenarios. Other methods all fail in $3s5z\_vs\_3s6z$ because they just perform the more to fight less strategy as in homogeneous scenarios, while UNMAS achieves 28% winning rate because of the above strategies.