AIIR-MIX: Multi-Agent Reinforcement Learning Meets Attention Individual Intrinsic Reward Mixing Network

Deep Reinforcement Learning (DRL) is an crucial branch of machine learning. It utilizes neural networks to approximate the optimal action decision or value function of the agent, realizing the generalization of the representation ability. By reason of the powerful fitting ability of deep learning, DRL can be employed as an effective way to tackle agent decision-making problems in complicated environments, such as group decision-making Nguyen et al. (2020), speech recognition Mousavi et al. (2016); Shen et al. (2019), autonomous driving Shen et al. (2019), natural language processing Young et al. (2018), and intelligent control Carlucho et al. (2020).

A multi-agent environment, where multiple agents are present for interaction and learning, is an emerging hot topic in recent years. Unfortunately, single-agent reinforcement learning is not very effective when used in a multi-agent environment, because the joint action space of the agent resulting from fully centralized learning is too large to learn the optimal policy. Therefore, Multi-Agent Reinforcement Learning (MARL) is an extension of deep reinforcement learning from single-agent to multi-agent, and has a list of methods to solve this problem. Centralized Training and Decentralized Execution (CTDE) Foerster et al. (2016); Wang* et al. (2020), an effective and widely applied method in this list. In this paradigm, the central controller manages all the agents’ observations, actions, and rewards during training. And the central controller and its value networks are not utilized during execution. Due to the above features and superiority, CTDE has become a widely applied paradigm in MARL, such as COMA Foerster et al. (2018), VDN Sunehag et al. (2017), QMIX Rashid et al. (2018) and QTRAN Son et al. (2019). In our research, we will also follow the CTDE paradigm in our proposed method.

In MARL, the reward function is extremely significant. However, it is difficult to go about formalizing all situations as reward functions in some real-world tasks. For exploring new environments in reinforcement learning, often only sparse rewards can be set. And yet, in practical application scenarios, sparse rewards still face problems such as inefficient samples and difficulty in exploration. This problem is more evident in MARL. Because of the inherent challenges of MARL, such as unstable environment and dimensional catastrophe, extending MARL to sparse reward settings will further increase the difficulty of policy learning. A good method for solving the incentive issue in a multi-agent environment is to assign extrinsic rewards and construct an intrinsic reward for each agent. In recent years, many researchers have started to focus on these directions. LIIR Du et al. (2019) learns an intrinsic reward function for each agent and continuously updates it to maximize the expected accumulated team reward from the environment. GIIR Wu et al. (2021) solves the lazy agent problem by using an intrinsic reward encoder to generate a separate intrinsic reward for each agent. OpenAI Berner et al. (2019) uses artificially set intermediate rewards to accelerate learning. FTW Jaderberg et al. (2019) learns the agent’s intrinsic reward through two layers of optimization. However, both approaches overlook the following points: (a) building dependencies (i.e., attention mechanisms) between agents can induce more precise rewards; (b) integrating intrinsic and extrinsic rewards to facilitate policy learning better.

In this paper, we propose Attention Individual Intrinsic Reward Mixing Network (AIIR-MIX) method to fill this gap. AIIR-MIX includes the generation of precise intrinsic reward network (AIIR) and a non-linear Mixing network (MIX) to combine the intrinsic and extrinsic rewards. In terms of generating intrinsic reward, we propose an intrinsic reward network based on the attention mechanism. We assume that each agent has a separate intrinsic reward. The agents’ observations and actions are extremely similar when they perform teamwork. Based on the above, the contribution of each agent in teamwork is calculated from each agent’s observation and action using the attention mechanism, and a more accurate intrinsic reward is generated for each agent. The intrinsic reward network is updated to maximize the standard cumulative discounted extrinsic rewards from the environment. In terms of combining intrinsic and extrinsic rewards, we propose a Mixing network that allows intrinsic and extrinsic rewards to be combined in a non-linear manner. The extrinsic reward is fed to the hyper network to generate the weights of the Mixing network. The Mixing network combines weights and intrinsic rewards to output global rewards for each agent. In addition, we apply an intrinsic reward function to the Actor-Critic algorithm, where each agent’s individual policy is updated under the direction of the corresponding proxy critic. Benefitting from these improvements, AIIR-MIX generates a more appropriate global reward and reduces artificial intervention in reward function design.

We evaluate the AIIR-MIX method on StarCraft II micromanagement benchmark  Samvelyan et al. (2019). The experimental results show that the AIIR-MIX performs better than the mainstream algorithms such as LIIR and QMIX in both homogeneous and heterogeneous maps. We conduct ablation experiments and demonstrate that both AIIR and MIX perform better than the baseline algorithm when they are used individually. Moreover, we visualize the training process and show the dynamic change process of attention weights and intrinsic reward as time advances in the complete trajectory. The results demonstrate the effectiveness and importance of the intrinsic reward based on the attention mechanism.

In generally, a fully cooperative multi-agent problem can be described as a decentralized partially observable Markov decision process (Dec-POMDP) Oliehoek and Amato (2016) consisting of a tuple $G=<\mathcal{N},S,U,P,Z,O,r,\gamma,\rho_{0}>$. $\mathcal{N}=\left\{1,2,\cdots,n\right\}$ denote the set of $n$ agents and $i\in\mathcal{N}$. $s\in S$ is the state of the environment which includes global information for all agents. $U$ is the set of actions. $Z$ is the agent’s observation set, each agent gets its own observation $o_{i}\in Z$ according to the observation function $O(s,i):S\times\mathcal{N}\rightarrow Z$. At each timestep, each agent $i\in\mathcal{N}\equiv\left\{1,\cdots,n\right\}$ chooses an action $u_{i}$ through the parameterized policy network $\pi_{i}\left(o_{i}\right)$ according to the current observation $o_{i}$, forming a joint action $u_{i}\in U\equiv U^{n}$ and leading to next state $s^{\prime}$ according to the transition function $P(s^{\prime}|s,u):S\times U\times S\rightarrow\left[0,1\right]$. $\boldsymbol{\pi}=\left\{\pi_{1},\pi_{2},\cdots,\pi_{n}\right\}$ denotes the joint policy consists of the policy of each agent. In order to distinguish different rewards, we denote the team reward from the environment as extrinsic reward $\mathbf{r}^{\textrm{ex}}$. The intrinsic reward set that will be learned as $\mathbf{r}_{t}^{\textrm{in}}={\left\{r^{\textrm{in}}_{i}\right\}}_{i=1}^{n}$, where $t$ is the index of the timestep. $\mathbf{r}^{\textrm{ex}}(s,u_{i}):S\times U\rightarrow\mathbb{R}$ is the team reward for each agent $i$ from environment. $\rho_{0}:S\to\mathbb{R}$ is the distribution of the initial state $s_{0}$. In a fully cooperative multi-agent problem, each agent receives the same $\mathbf{r}^{ex}$ to promote cooperative behavior.

The learning objective of the cooperative multi-agent problem is that $n$ agents learn a policy network $\pi_{i}$ parameterized by $\theta_{i}$ to maximize the global cumulative discounted reward set $\mathbf{r}_{t}^{\textrm{total}}$. That is, when $\mathbf{r}_{t}^{\textrm{total}}$ is the largest, the optimal joint policy of all agents is obtained $\boldsymbol{\pi^{*}}=\mathop{\textrm{argmax}}\limits_{\pi}\mathbb{E}_{s_{0},u_{0},...,s_{n},u_{n}}\left[\sum_{t=0}^{T}\gamma^{t}\mathbf{r}^{\textrm{ex}}_{t}\right]$, where $\gamma\in\left[0,1\right)$ is a discount factor and $T$ is the maximum number of steps.

In this Section, we introduce a new method called AIIR-MIX, which is based on the Actor-Critic framework. The main contribution of AIIR-MIX is to generate set $\mathbf{r}_{t}^{\textrm{in}}$ (w.r.t., the timestep is $t$) and combine it with $\mathbf{r}^{\textrm{ex}}$ to generate set $\mathbf{r}_{t}^{\textrm{total}}$ nonlinearly. We first present the overall AIIR-MIX framework in Fig. 1, and then we specifically show the relevant details about AIIR-MIX in Fig. 2.

As shown in Fig. 1, the total AIIR-MIX framework consists of five parts: extrinsic Critic network, total Critic network, total Actor network, AIIR, and Mixing network. AIIR and Mixing network make up a module for generating $\mathbf{r}_{t}^{\textrm{total}}$, which is the most important part of AIIR-MIX. In particular, the total Actor network is made up of $n$ Actor networks, each Actor network with parameter $\theta_{i}$ generates the policy of its corresponding agent. And we denote the current states $s_{i}$ as the input and the agent’s policy $\pi_{\theta_{i}}(u_{i}|s_{i})$ as the output of each Actor network. Similarly, each Critic network in total Critic with parameter $\omega_{i}$ evaluates an agent’s policy, which can update the Actor network end-to-end. Specifically, each Critic network updates the value function parameters $\omega_{i}$ depending on $Q(s_{i},u_{i};{\omega_{i}})$ and each Actor network updates the policy parameters $\theta_{i}$ for $\pi_{\theta_{i}}(u_{i}|s_{i})$ in the direction suggested by its corresponding Critic network. Following the CTDE paradigm, the total Critic network is only utilized during training. Like Lowe et al. (2017); Foerster et al. (2018), the total Critic network is updated by the Temporal-Difference error (TD-error) as well. But we utilize $\mathbf{r}^{\textrm{total}}_{t}$ generated from Mixing network by combining $\mathbf{r}^{\textrm{ex}}$ and $\mathbf{r}_{t}^{\textrm{in}}$ instead of $\mathbf{r}^{\textrm{ex}}$ to calculate TD-error. To better facilitate teamwork among the agents, we designed AIIR based on an attention mechanism to generate $\mathbf{r}_{t}^{\textrm{in}}$. Due to the powerful representational abilities of AIIR-MIX, it can update the total Critic network and the total Actor network better.

Based on the Bellman equation, the loss function of a Critic network can be defined as:

where $i\in\mathcal{N}\equiv\left\{1,\cdots,n\right\}$ and $y_{i}^{\textrm{total}}=r_{i}^{\textrm{total}}+\gamma Q(s_{i}^{{}^{\prime}},u_{i}^{{}^{\prime}};\omega_{i}^{{}^{\prime}})$. Of particular note is our definition of $r_{i}^{\textrm{total}}$ is the global reward of agent $i$, which is an element in $\mathbf{r}_{t}^{\textrm{tota}l}$. Each Actor network is updated by the policy gradient:

where $A_{\pi_{\theta_{i}}}\left(u_{i}|s_{i}\right)$ is the advantage function based on $A_{\boldsymbol{\pi}}(s,\textbf{u})$ (have been introduced in Section 3.1) and $A_{\pi_{\theta_{i}}}\left(u_{i}|s_{i}\right)=r^{ex}\left(s_{i},u_{i}\right)+\gamma V\pi_{\theta_{i}}\left(s_{i}\right)-V\pi_{\theta_{i}}\left(s_{i}^{\prime}\right)$. With a learning rate of $\alpha$ for the Actor network, the update of the parameter $\theta_{i}$ can be defined as:

By referring to Fig. 1, all agent share the extrinsic Critic network. The loss function of this extrinsic Critic network with parameter $\eta$ can be defined as:

Then, the generating procedure of $\mathbf{r}_{t}^{\textrm{in}}$ and $\mathbf{r}_{t}^{\textrm{total}}$ is described in detail. To promote cooperation amongst agents and meet the objective of maximizing the global reward for $\mathbf{r}_{t}^{\textrm{in}}$, we design an intrinsic reward generation framework based on an attention mechanism.

As shown in Fig.2c, we first define a feature extractor, to process the state $s_{i}^{t}$ and the last action $u_{i}^{t}$ of the agent $i$ at timestep $t$, where the feature extractor is a sequence of consecutive ReLU-FC-ReLU-FC. Then, $\mathbf{v}^{t}_{i}$ appears both as an output of the feature extractor and as an input to the attention mechanism, representing the local attention embedding of the agent. With the help of the attention mechanism, we get the correlation of all agent pairs. Commonly, the correlation of agent pairs is obtained by a distance metric function, and it can be denoted in three forms as follows:

since cosine similarity has the ability to normalize, we apply it to calculate the correlation between the attention embeddings of the agents. In addition, we perform softmax function on the calculated correlations:

The global attention embeddings $\mathbf{z}_{t}^{i}$ of agent $i$ is obtained by weighting and summing the attention embeddings of all agents according to the weighting coefficients:

where $\mathbf{z}_{i}^{t}$ contains the degree of similarity in the states and actions of the agents, learns the correlations between agents and promotes teamwork. After that, the local attention embedding $\mathbf{v}^{t}_{i}$ of agent $i$ and the global attention embedding $\mathbf{z}^{t}_{i}$ obtained through the attention mechanism are simultaneously provided as inputs to the fully connected layer. Finally, $r_{i}^{\textrm{in}}$ of agent $i$ and the set of all intrinsic rewards $\mathbf{r}_{t}^{\textrm{in}}$ are output:

As shown in Fig. 2a, $\mathbf{r}_{t}^{\textrm{in}}$ output by AIIR is non-linearly combined with $r^{\textrm{ex}}$ through the Mixing network to output $\mathbf{r}_{t}^{\textrm{total}}$. In previous studies, both LIIR Du et al. (2019) and GIIR Wu et al. (2021) combined $\mathbf{r}_{t}^{in}$ and $\mathbf{r}^{\textrm{ex}}$ by weighted summation:

In AIIR-MIX, we apply a non-linear approach to combine $\mathbf{r}_{t}^{\textrm{in}}$ and $r^{\textrm{ex}}$ to dynamically generate $\mathbf{r}_{t}^{\textrm{total}}$, so that the agent can obtain a more accurate reward at each timestep, thus facilitating the agent to select the optimal policy more readily. The weights $\{\mathcal{W}_{1},\mathcal{W}_{2}\}$ and biases $\{\mathbf{b}_{1},\mathbf{b}_{2}\}$ of the Mixing network are generated by a separate hyper network, as shown in the left of Fig. 2a. The $r^{\textrm{ex}}$ is utilized as input to the hyper network, which generates same weights $\{\mathcal{W}_{1},\mathcal{W}_{2}\}$ and biases $\{\mathbf{b}_{1},\mathbf{b}_{2}\}$ through different linear layers and outputs them as weights and biases of the Mixing network, respectively. Combining the gradient descent method and the hyper network to update the Mixing network, allows for the dynamic adjustment of the weights of the Mixing network, thus enabling to combine $\mathbf{r}_{t}^{\textrm{in}}$ and $r^{\textrm{ex}}$:

Then $\mathbf{r}_{t}^{\textrm{total}}$ is obtained by AIIR-MIX, which is introduced in Fig. 2b in more detail.

In this Section, we evaluate our AIIR-MIX method on the StarCraft Multi-Agent Challenge (SMAC) environment Samvelyan et al. (2019), which has become a benchmark for evaluating MARL methods. We compare AIIR-MIX with the state-of-the-art MARL methods such as LIIR Du et al. (2019), QMIX Rashid et al. (2018), COMA Foerster et al. (2018), QTRAN Son et al. (2019). We conduct ablation experiments to demonstrate the effectiveness and rationality of AIIR and Mixing network. Finally, in order to analyze the learning process of the agents more clearly, we visualize the attention weights and intrinsic reward at each timestep.

This paper presents AIIR-MIX, a novel multi-agent RL algorithm that learns an individual intrinsic reward through an attention mechanism for each agent so that the agent can obtain different rewards to facilitate the learning of the agent. Besides, we design a non-linear Mixing network to combine intrinsic and extrinsic rewards instead of a simple linear summation for the first time, thereby dynamically generating global rewards for each agent according to the changing environment. Our empirical results of the experiments carried out on the battle games in StarCraft II demonstrate that our approach induces trained policy compared with a few state-of-the-art MARL methods better. We further conduct ablation studies to confirm the effectiveness of the intrinsic reward network based on the attention mechanism and the non-linear Mixing network. And we visualize the intrinsic rewards and attention weights to illustrate how the intrinsic reward network assigns each agent an appropriate reward and promotes cooperation amongst agents.

This work was supported in part by the Aeronautical Science Foundation of China under Grant 20200058069001 and in part by the Fundamental Research Funds for the Central Universities under Grant 2242021R41094.