LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates

As Figure 2 show, The LINDA framework focuses on learning awareness of teammates based on local information in each agent’s individual network. For agent $i$, LINDA uses a GRU [5] cell to encode historical observations and actions into trajectory $\tau^{i}$. $\tau^{i}$ is fed into an awareness encoder $f_{i}$ with parameter $\bm{\theta}_{c}^{i}$ to build awareness of each agent. The awareness encoder learns a decomposition mapping for each agent, and outputs $n$ multivariate Gaussian distributions $\mathcal{N}(\bm{\mu}_{1}^{i},\bm{\sigma}_{1}^{i}),\cdots,\mathcal{N}(\bm{\mu}_{n}^{i},\bm{\sigma}_{n}^{i})$, where $n$ is the number of agents, and $\bm{\mu}_{j}^{i},\bm{\sigma}_{j}^{i}$ are the awareness mean and awareness variance for agent $j$ respectively. Awareness representation embeddings $\bm{c}_{1}^{i},\cdots,\bm{c}_{n}^{i}$ are sample from the corresponding multivariate Gaussian distributions, where $\bm{c}_{j}^{i}$ denotes agent $i$’s awareness for agent $j$. To ensure the gradient is tractable for the sampling operation, we apply the reparameterization trick [14]. To sample from a Gaussian distribution $z\sim\mathcal{N}(\mu,\sigma)$, it converts the random variable $z$ into $z=\mu+\sigma\epsilon$, where $\epsilon\sim\mathcal{N}(0,1)$, such that the gradient descent can be backpropagated through the sampling operation. Formally, for agent $i$, its $n$ awareness embeddings are generated by:

where $\odot$ is element-wise production, $\bm{c}^{i}_{j}$ denotes agent $i$’s awareness for agent $j$, and $\bm{\theta}_{c}^{i}$ is the parameters of the awareness encoder $f_{i}$. The awareness encoder $f_{i}$ inputs local trajectory $\tau^{i}$ and outputs the mean and variance of $n$ awareness distributions, $(\bm{\mu}^{i},\bm{\sigma}^{i})=(\bm{\mu}^{i}_{1},\cdots,\bm{\mu}^{i}_{n};\bm{\sigma}^{i}_{1},\cdots\bm{\sigma}^{i}_{n})$.

Since the awareness is designed for agents, awareness alone will cause the loss of environmental information. Therefore we concatenate agent $i$’s awareness embeddings $\bm{c}_{1}^{i},\cdots,\bm{c}_{n}^{i}$ together with its trajectory $\tau^{i}$ to compute the local action value $Q_{i}$ by the local utility network. During the centralized training, action values of all the agents together with the global state $s_{t}$ are fed into a mixing network to produce the global action value $Q_{tot}$ and compute TD error for gradient descent. In our implementation, we try three different kinds of mixing networks, VDN [32], QMIX [27], and QPLEX [36] for their monotonic approximation. To facilitate awareness learning, our approach additionally applies an information-theoretic regularization loss $\mathcal{L}_{c}(\bm{\theta}_{c}^{i})$ for agent $i$’s local network. The overall objective to minimize is

where

($\bm{\theta}^{-}$, are the parameters of a periodically updated target network) is the TD loss, $\bm{\theta}$ is the all parameters of the framework, and $\lambda$ is a scaling factor. We will then discuss the definition and optimization of the regularization loss $\mathcal{L}_{c}(\bm{\theta}_{c}^{i})$.

The latent awareness representations for each agent are hard to be learned automatically. Therefore an auxiliary loss function is necessary. Intuitively, we expect the learned awareness to be informative, which means that the learned awareness needs to incorporate actual information about others. Since our framework works under the CTDE paradigm, other agents’ trajectories are accessible during training but inaccessible during execution. We can utilize other agents’ trajectories for awareness centralized training, but awareness representations are entirely generated from the individual local trajectory for decentralized execution. To this end, we establish the relationship between agent $i$’s awareness for agent $j$, denoted as $\bm{c}^{i}_{j}$, and agent $j$’s actual trajectory $\tau^{j}$, by maximizing their mutual information conditioned on agent $i$’s local trajectory $\tau^{i}$. For agent $i$, the objective for optimizing its awareness for all the agents is to maximize

During centralized training, the overall objective for awareness learning is:

By unfolding the mutual information term and swapping the summation order, we can rewrite the objective Eq. 8 into:

For the same target agent $j$, the objective maximizes all the agents’ entropy of their awareness distributions for agent $j$ conditioned on their local trajectories, while minimizing the entropy conditioned on local trajectories and the target trajectory $\tau^{j}$. Due to the partial observability, awareness for agent $j$ is not entirely precise. Conditioned on the local trajectories alone, the uncertainty of the awareness distributions for agent $j$ should be large. By pushing the entropy higher, it prevents the awareness embeddings from collapsing to the same point in the representation space. It means that conditioned on multiple views from different agents, the awareness for the same object should be diverse and complementary.

The second term suggests that when the target trajectory $\tau_{j}$ is given, the awareness distributions for agent $j$ should get more deterministic. It means that conditioned on sufficient information, the agents’ awareness for the same object should get aligned and consistent in the representation space.

However, directly optimizing the objective Eq. 7 is difficult because computation involving mutual information is intractable. Inspired by [3], we introduce a variational estimator to derive a lower bound for the mutual information term:

where $q_{\xi}(\bm{c}_{j}^{i}|\tau^{i},\tau^{j})$ is the variational posterior estimator with parameter $\xi$, and ${\mathcal{CE}}$ is the Cross-Entropy operator. A detailed derivation is shown in A. By using a replay buffer $\mathcal{D}$, we can rewrite the lower bound in Eq. 10 and derive its loss function:

where ${\mathrm{KL}}$ is the Kullback-Leibler divergence operator. Finally, the overall loss function is

where $\lambda$ is an adjustable hyper-parameter to achieve a trade-off between the temporal difference loss and the summation of all agents’ awareness learning loss. Over the course of training, global information including trajectories of all the agents is used to compute the mutual information loss. During execution, the awareness learning module is removed, and each agent infers awareness for others conditioned on its local trajectory.

We choose two multi-agent benchmark environments: Level-based Foraging (LBF) [1] and StarCraft II micromanagement (SMAC)^†††Our experiments are all based on the PYMARL framework which uses SC2.4.6, note that performance is not always comparable between versions. [28] as the testbed.

Level-based foraging (LBF): LBF[1] is a grid-world game that focuses on the coordination of the agents (Figure 5). It consists of agents and foods initialized at different positions at the beginning of an episode. Both agents and foods are assigned random levels. The goal is for agents to collect foods to achieve the maximum team score. The food collection is constrained by the level. Food is successfully collected only when the sum of the levels of the agents who pick up the food simultaneously exceeds the level of the food. The action set of agents is composed of picking up action and movement in four directions: up, down, left, and right. Agents receive a team reward only when they successfully collect food, which means the environment has a sparse reward. Besides, the environment is partially observable, where the agents observe up to two grid cells in every direction. The agents’ observation includes the positions of other agents and foods in the visible range. For LBF, we test the algorithms in different sizes of exploration space, with $8\times 8,10\times 10,16\times 16$ grid sizes, and all with $2$ agents and $1$ food.

StarCraft II micromanagement (SMAC): SMAC[28] is an environment for collaborative multi-agent reinforcement learning based on Blizzard’s StarCraft II RTS game (Figure 5). It consists of a set of StarCraft II micro scenarios, each of which is a confrontation between two armies of units. The allied agents learn coordination to beat the enemy units controlled by the built-in game AI. It is a partially observable environment, where agents are only accessible to the status including positions and health of teammates and enemies in their limited field of view. For SMAC, we test the algorithms on maps of different difficulties and classify them as easy, hard and super hard (see Table 1 for detail).

To study the effectiveness of LINDA framework, we try three different mixing networks in: (i) VDN [32], (ii) QMIX [27], (iii) QPLEX [36]. Their mixing network structures are from simple to complex, with increasing representational complexity. The methods applied with LINDA are denoted as LINDA-VDN, LINDA-QMIX, and LINDA-QPLEX, respectively. To show the effectiveness of LINDA applied methods, We compare with RODE [40], which is the state-of-the-art method, CollaQ [46], which uses manual local information decomposition, and LIAM [25], which uses autoencoders for opponent modelling.

Because VDN suffers from structural constraints and limited representation complexity [27], it tends to fail in complex environments like SMAC. We evaluate LINDA-VDN in LBF, which is simpler for

agents to achieve goals. We configure three different grid sizes, $8\times 8$, $10\times 10$, and $16\times 16$, all with $2$ agents and $1$ food. For LINDA-QMIX and LINDA-QPLEX, we test them on multiple SMAC maps. For evaluation, we carry out each experiment with $5$ random seeds, and the results are shown with a $95\%$ confidence interval.

Figure 6 shows the learning curves of LINDA-VDN and VDN in different sizes of exploration space of the LBF environment. As the grid size grows, each agent needs to search a larger space to collaborate with others for food, and it takes more timesteps for agents to converge to a high reward. We show that the application of LINDA, i.e., LINDA-VDN, improves the performance of VDN. For smaller grid size, e.g. $8\times 8$, LINDA-VDN accelerates the learning speed and stabilizes the performance curve when converged. For larger grid sizes, the improvement is rather more significant.

In the SMAC environment, we test LINDA-QPLEX, LINDA-QMIX and other methods on six super hard maps, three hard maps and six easy maps. The details of the SMAC maps are described in C Table 1. From Figure 5, we can find LINDA-QMIX and LINDA-QPLEX outperform the vanilla QMIX and QPLEX, respectively, which indicates the effectiveness of LINDA for improving the coordination ability in complex sceneries. The superiority of QPLEX over other methods except for QPLEX-LINDA before 0.8M is for the improved network representation of QPLEX makes it good at some maps such as 3s5z_vs_3s6z. LIAM can solve the POMDP and improve the coordinate ability of QMIX in some way, and we find it is not competitive with LINDA. CollaQ only has a slight performance advantage over QMIX. We guess it is because CollaQ only uses local observation to capture the relationship between agents, which may not solve complex coordination problems.

We plot the averaged median test win rate across the six super hard maps in Figure 5. Compared with QMIX and QPLEX, the application of LINDA yields better learning performance and outperforms other methods. LINDA improves the collaboration for QMIX and QPLEX in all the six super hard maps, especially in tough scenarios. For example, in 6h_vs_8z, a map requiring strong coordination ability, all other methods failed, but LINDA-QPLEX achieved a high winning rate. In D, we further show the learning curves on the six easy scenarios and the hard maps. The LINDA framework still achieves slight performance improvement on most easy maps, where micro-tricks and cohesive collaboration are unnecessary. On the contrary, previous methods such as RODE perform worse than QMIX because the additional role learning module needs more samples to learn a successful strategy [40]. The overall experiments show that the LINDA framework is robust to both easy and tough scenarios and effectively enhances the learning efficiency, especially on challenging tasks.

To understand the superior performance of LINDA, we carry out ablation studies to verify the contribution of awareness learning. To this end, we add the LINDA structure to QMIX without the mutual information loss $\mathcal{L}_{c}(\bm{\theta}_{c})$ during training and denote it as LINDA-QMIX w/o $\mathcal{L}_{c}(\bm{\theta}_{c})$. Besides, to test whether the superiority of our method comes from the increase in the number of parameters, we also test QMIX with a similar number of parameters with LIDA-QMIX and denote it as QMIX-LARGE. As shown in Figure 8, we compare LINDA-QMIX with LINDA-QMIX w/o $\mathcal{L}_{c}(\bm{\theta}_{c})$, QMIX-LARGE, and QMIX on three SMAC maps: MMM2, 2c_vs_64zg and 3s_vs_5z. The experimental results indicate that the mutual information loss plays a significant role in enhancing learning performance. Besides, it also proves that larger networks will not definitely bring performance improvement, and the superiority of LINDA does not come from the larger networks.

To further analyze the learned awareness in the representation space, we visualize the awareness embeddings on the SMAC map MMM2. MMM2 is a heterogeneous scenario, in which 1 Medivac, 2 Marauders and 7 Marines face 1 Medivac, 3 Marauders and 8 Marines. In this task, different types of agents should cooperate well to fully exert the advantage of each unit type. We collect the awareness embeddings of all the agents at two different timesteps in one episode. Since there are $10$ agents, each with $10$ awareness embeddings for teammates, there are $100$ embeddings in total at each timestep. We reduce the dimension of each awareness embedding by t-SNE [34] to show them in a 2-dimensional plane.

As shown in Figure 9(a) and 9(b), the awareness embeddings generated by LINDA-QMIX w/o $\mathcal{L}_{c}(\bm{\theta}_{c})$, which is without awareness learning, distributed almost randomly in the representation space. Contrarily, with the proposed mutual information loss $\mathcal{L}_{c}(\bm{\theta}_{c})$, in Figure 9(c) and 9(d), agents’ awareness embeddings automatically form several clusters in the representation space. According to the positions of self-to-self awareness embeddings, we divide the agents into $K$ groups $G=\{G_{1},\cdots,G_{K}\}$. We color the awareness embeddings by the group each agent belongs to. That is, for each group $G_{k}$, the representations $\{\bm{c}^{i}_{j}\mid 1\leq j\leq n,i\in G_{k}\}$ are painted the same color, where $n$ is the number of agents. In the video frame of the same timestep, we see the correspondence between the agent groups formed in the game and in the awareness representation space. The agents in the same group tend to build similar awareness and achieve more cooperation. Besides, In the early timestep (Figure 9(c)) and the late timestep (Figure 9(d)), the formed groups dynamically changed and adapted according to the battle situation. As shown in Figure 9(d), when the $7$-th agent was dead, its awareness embeddings collapsed to a small region because the observation inputs of a dead agent are all zeros in the environment implementation. Other agents formed new groups and conducted new tactics for cooperation.

The phenomenon reveals the relationship between awareness representations and agents’ cooperation strategies. The aggregation of the learned awareness makes the group members share similar latent embeddings. Such consistency may encourage neural networks to output stable and consistent action value functions, thus reaching a consensus on the strategies among the group members and achieving better collaboration. Further, the pairwise relationship of the learned awareness implicitly forms groups among agents, which is similar to role learning [31, 16, 39, 40] in multi-agent systems. It indicates that pairwise awareness representations implicitly incorporate roles, and can be degraded into role-based methods by further processing the relationships of awareness among agents.

We conduct visualization on the dynamic learning process of the awareness distributions. For each agent $i$, we visualize its average of the variance of the multivariate awareness distribution for agent $1$, which is denoted as $\bar{\sigma}^{i}_{1}$. Figure 10 presents $\bar{\sigma}^{i}_{1},1\leq i\leq 6$ (for the first $6$ agents) over the course of training. The blue bold curve, which represents $\bar{\sigma}^{1}_{1}$, is nearly the lowest among all the curves during training. The variance of a self-to-self awareness distribution is relatively lower than the variance of other-to-self awareness distribution. It means that agents build more certain awareness for self than others, which is consistent with our intuition.

To further demonstrate whether a lower variance indicates a more precise awareness embedding, we visualize the variance and the difference of mean among the agents. To clearly show the relationship, we conduct visualization on a two-agent scenario, 2c_vs_64zg. In Figure 11, the red and blue solid curve represents the average of the awareness variance, which is denoted as $\bar{\sigma}^{1}_{1}$ and $\bar{\sigma}^{2}_{1}$ respectively. The red dashed curve represents the L$1$-norm of the difference between the awareness mean, which is denoted as $\|\bm{\mu}^{2}_{1}-\bm{\mu}^{1}_{1}\|$. We find that the peaks of $\|\bm{\mu}^{2}_{1}-\bm{\mu}^{1}_{1}\|$ and $\bar{\sigma}^{2}_{1}$ are highly overlapping, especially after $1.2$ millions of timesteps when the learning gradually converges. The peak of $\bar{\sigma}^{2}_{1}$ means that at that timestep, A$2$ (i.e., agent 2) is uncertain about the state of A$1$ probably because A$1$ is out of the view of A$2$. The peak of $\|\bm{\mu}^{2}_{1}-\bm{\mu}^{1}_{1}\|$ means that A$1$ and A$2$ have a huge difference in their awareness for A$1$. The phenomenon of peak alignment indicates that the inference uncertainty is highly related to the proximity of awareness. The awareness variance may serve as an indicator for agents to estimate the confidence of awareness. It suggests that for further research, we can put different degrees of emphasis on different awareness according to the confidence. Higher-level modules such as the attention mechanism can be built based on awareness confidence.