On the Role of Emergent Communication for Social Learning
in Multi-Agent Reinforcement Learning

Implicit communication conveys information to other agents that is not intentionally communicated (grupen2022multi). Implicit signaling conveys information to other agents based on one’s observable physical position (grupen2022multi). Implicit signaling may be a form of implicit communication such as through social cues (jaques2019social; ndousse2021emergent) or explicit communication such as encoded into the MDP through “cheap talk” (sokota2022communicating). Unlike implicit signaling, explicit signaling is a form of positive signaling (li2021learning) that seeks to directly influence the behavior of other agents in the hopes that the new information will lead to active listening. Multi-agent emergent communication is a type of explicit signaling which deliberately shares information. Symbolic communication, a subset of explicit communication, seeks to send a subset of pre-defined messages. However, these symbols must be defined by an expert and do not scale to particularly complex observations and a large number of agents. Emergent communication aims to directly influence other agents with a learned subset of information, which allows for scalability and interpretability by new agents.

Several methodologies currently exist to increase the informativeness of emergent communication. With discrete and clustered continuous communication, the number of observed distinct communication tokens is far below the number permissible (discreteComm). As an attempt to increase the emergent “vocabulary” and decrease the data required to converge to an informative communication “language”, work has added a bias loss to emit distinct tokens in different situations (EcclesBiases). More recent work has found that the sample efficiency can be further improved by grounding communication in observation space with a supervised reconstruction loss (MA_autoencoder). Information-maximizing autoencoders aim to maximize the state reconstruction accuracy for each agent. However, grounding communication in observations has been found to easily satisfy these input-based objectives while still requiring a myriad more samples to explore to find a task-specific communication space (karten2022sparse). Thus, it is necessary to use task-specific information to communicate informatively. This will enable learned compression for task completion rather than pure compression for input recovery. Other work aims to use the information bottleneck (tishby2015deep) to decrease the entropy of messages (wang2020learning). In our work, we use contrastive learning to increase representation similarity with future goals, which we show optimally optimizes the Q-function for messages.

The properties of the tokens in emergent communication directly affect their informative ability. As a baseline, continuous communication tokens can represent maximum information but lack human-interpretable properties. Discrete 1-hot (binary vector) tokens allow for a finite vocabulary, but each token contains the same magnitude of information, with equal orthogonal distance to each other token. Similar to word embeddings in natural language, discrete prototypes are an effort to cluster similar information together from continuous vectors (discreteComm). Building on the continuous word embedding properties, VQ-VIB (tucker2022towards), an information-theoretic observation grounding based on VQ-VAE properties (van2017neural), uses variational properties to provide word embedding properties for continuous emergent tokens. Like discrete prototypes, they exhibit a clustering property based on similar information but are more informative. However, each of these message types determines a single token for communication. Tokens are stringed together to create emergent “sentences”.

The policy network is defined by three stages: Observation Encoding, Communication, and Action Decoding. The best observation encoding and action decoding architecture is task-dependent, i.e., using multi-layer perceptrons (MLPs), CNNs (lecun1995convolutional), GRUs (chung2014empirical), or transformer (vaswani2017attention) layers are best suited to different inputs. The encoder transforms observation and any sequence or memory information into an encoding $H$. The on-policy reinforcement learning training uses REINFORCE (williams1992simple) or a decentralized version of MAPPO (yu2021surprising) as specified by our experiments.

Our work focuses on the communication stage, which can be divided into three substages: message encoding, message passing (often considered sparse communication), and message decoding. We use the message passing from (karten2022sparse). For message decoding, we build on a multi-headed attention framework, which allows an agent to learn which messages are most important (graphMA). Our compositional communication framework defines the message encoding, as described in section 4.

Mutual information, denoted as $I(X;Y)$, looks to measure the relationship between random variables,

which is often measured through Kullback-Leibler divergence (kullback1997information), $I(X;Y)=D_{KL}(p(x,y)||p(x)\otimes p(y))$. The message encoding substage can be defined as an information bottleneck problem, which defines a trade-off between the complexity of information (compression, $I(X,\hat{X})$) and the preserved relevant information (utility, $I(\hat{X},Y)$). The deep variational information bottleneck defines a trade-off between preserving useful information and compression (alemi2017deep; tishby2015deep). We assume that our observation and memory/sequence encoder provides an optimal representation $H^{i}$ suitable for sharing relevant observation and intent/coordination information. We hope to recover a representation $Y^{i}$, which contains the sufficient desired outputs.

In our scenario, the information bottleneck is a trade-off between the complexity of information $I(H^{i};M^{i})$ (representing the encoded information exactly) and representing the relevant information $I(M^{j\neq i};Y^{i})$, which is signaled from our contrastive objective. In our setup, the relevant information flows from other agents through communication, signaling a combination of the information bottleneck and a Lewis game. We additionally promote complexity through our compositional independence objective, $I(M_{1}^{i};\ldots;M_{L}^{i}|H^{i})$. This is formulated by the following Lagrangian,

where the bounds on mutual information $\hat{I}$ are defined in equations 1, 2, and 10. Overall, our objective is,

We derive an upper bound for the interaction information between all tokens.

The proof is in Appendix A.1.

Since we want the mutual information to be minimized in our objective, we minimize,

In order to induce complexity in the compositional messages, we additionally want to minimize the mutual information $I(H;M)$ between the composed message $\hat{m}$ and the encoded information $h$. We derive an upper bound on the mutual information that we use as a Lagrangian term to minimize.

The proof is in Appendix A.1. Thus, we have our Lagrangian term,

Conditioning on the input or observation data is a decentralized training objective.

Compositional communication necessitates an adaptive limit on the total length of the sequence.

Suppose one predicts two arbitrary tokens, $w_{k}$ and $w_{l}$. Given equation 1, it follows that there is low or near-zero mutual information between $w_{k}$ and $w_{l}$.

A trivial issue is that the message generator will predict every available token as to follow the unique token objective. Since the tokens are imbued with input-oriented information (equation 2), the predicted tokens will be based on relevant referential details. Thus, it follows that tokens containing irrelevant information will not be chosen.

A nice optimization objective that follows from corollary 4.3 is that one can use self-supervised learning with an end-of-sequence (EOS) token to limit the variable total length of compositional message sequences.

Now, we can define the pipeline for message generation. The idea is to create an architecture that can generate features to enable independent message tokens. We expand each compressed token into the space of the hidden state $h$ (1-layer linear expansion) since each token has a natural embedding in $\mathbf{R}^{|h|}$. Then, we perform attention using a softmin to help minimize similarity with previous tokens and sample the new token from a variational distribution. See algorithm 1 for complete details. During execution, we can generate messages directly due to equation 1, resolving any computation time lost from sequential compositional message generation.

To evaluate our methodology, we compare our method to the following baselines: (1) no-comm, where agents do not communicate; (2) rl-comm, which uses a baseline communication method learned solely through policy loss (ic3net); (3) ae-comm, which uses an autoencoder to ground communication in input observations (MA_autoencoder); (4) VQ-VIB, which uses a variational autoencoder to ground discrete communication in input observations and a mutual information objective to ensure low entropy communication (tucker2022towards).

We provide an ablation of the loss parameter $\beta$ in table 1 in the blind traffic junction scenario. When $\beta=0$, we use our compositional message paradigm without our derived loss terms. We find that higher complexity and independence losses increase sample complexity. When $\beta=1$, the model was unable to converge. However, when there is no regularization loss, the model performs worse (with no guarantees about referential representation). We attribute this to the fact that our independence criteria learns a stronger causal relationship. There are fewer spurious features that may cause an agent to take an incorrect action.

In order to understand the effect of the independent concept representation, we analyze the emergent language’s capacity for redundancy. A message token $m_{l}$ is redundant if there exists another token $m_{k}$ that represents the same information. With our methodology, the emergent ‘language’ converges to the exact number of observations and intents required to solve the task. With a soft discrete threshold, the independent information loss naturally converges to a discrete number of tokens in the vocabulary. Our $\beta$ ablation in table 1 yields a bijection between each token in the vocabulary and the possible emergent concepts, i.e., the enumerated observations and intents. Thus for $\beta=0.1$, there is no redundancy.

Due to coordination in MARL, grounding communication in referential features is not enough. Finding the communication utility requires grounding messages in ordinal information. Overall, figure 3 shows that our compositional, contrastive method outperforms all methods focused on solely input-oriented communication grounding. In the blind traffic junction, our method yields a higher average task success rate and is able to achieve it with a lower sample complexity. Training with the contrastive update tends to spike to high success but not converge, often many episodes before convergence, which leaves area for training improvement. That is, the contrastive update begins to find aligned latent spaces early in training, but it cannot adapt the methodology quickly enough to converge. The exploratory randomness of most of the early online data prevents exploitation of the high utility $f^{+}$ examples. This leaves further room for improvement for an adaptive contrastive loss term.

In order to test the heterogeneous alignment ability of our methodology to learn higher-order concepts from high-dimensional data, we analyze the performance on the Pascal VOC game. We compare our methodology against ae-comm to show that concepts should consist of independent information directly from task signal rather than compression to reconstruct inputs. That is, we show an empirical result on pixel data to verify the premise of the information bottleneck. Our methodology significantly outperforms the observation-grounded ae-comm baseline, as demonstrated by figure 4. The ae-comm methodology, despite using autoencoders to learn observation-grounded communication, performs only slightly better than no-comm. On the other hand, our methodology is able to outperform both baselines significantly. It is important to note that based on figure 4, our methodology is able to guess more than two of the four labels correctly across the two agents involved, while the baseline methodologies struggle to guess exactly two of thew four labels consistently. This can be attributed to our framework being able to learn compositional concepts that are much more easily discriminated due to mutual independence.

Critics of emergent communication may point to the increased sample complexity due to the dual communication and action policy learning. In the social shadowing scenario, heterogeneous agents can learn to generate a communication policy without learning the action policy of the watched expert agents. To enable social shadowing, the agent will alternate between a batch of traditional MARL (no expert) and (1st-person) shadowing an expert agent performing the task in its trajectory. The agent only uses the contrastive objective to update its communication policy during shadowing. In figure 5, the agent that performs social shadowing is able to learn the action policy with almost half the sample complexity required by the online reinforcement learning agent. Our results show that the structured latent space of the emergent communication learns socially benevolent coordination. This tests our hypothesis that by learning communication to understand the actions of other agents, one can enable lower sample complexity coordination. Thus, it mitigates the issues of solely observing actions.