Decomposed Soft Actor Critic Method for Cooperative Multi-Agent Reinforcement Learning

Many real-world tasks can be modeled as multi-agent systems. Developing AI system for playing multi-agent games has raised much attention. Recent years, deep multi-agent reinforcement learning (MARL) algorithms [1] have presented impressive results in many challenging multi-agent systems, such as the coordination of autonomous vehicles [2], the challenging StarCraft II game [3], etc. Maybe the simplest way to solve multi-agent system problems is, treating everything else as the environment for each individual agent, and learning concurrently based on the global reward. However this will face the issues [4]: (1) non-stationarity: when an agent is learning, the policies of other agents are also changing simultaneously, which means that the dynamic of environments is non-stationary; (2) scalability: the joint state and action space grows exponentially as the number of agents increases. To cope with these issues, most recent advanced algorithms adopted the paradigm of centralized training with decentralized execution (CTDE)[5], in which they learn a centralized critic conditioned on joint action and observation history and take decentralized execution by learning different local actor (value functions or policies) for each individual agents.

Following CTDE paradigm, there are two main popular and promising research lines in MARL, one is the value function decomposition approach, another is multi-agent policy gradients. Value Decomposition Network (VDN) [6] represented joint Q value $Q^{tot}$ as a sum of individual Q-values $q^{i}$ that condition only on individual actions and observations. Each decentralized policy arise simply from its local Q values $q^{i}$ (selects actions greedily by $q^{i}$). Afterwards, QMIX [7] employed a network to estimate joint action-values as a non-linear combination of per-agent values that condition on local observations. The representative work of multi-agent policy gradient method is COMA [8] method, which explicitly used a counterfactual baseline to address the challenges of multi-agent credit assignment and a critic representation to compute the counterfactual baseline efficiently.

Recent work [16] points out that multi-agent Q-learning with linear value decomposition implicitly implements a classical multi-agent credit assignment method called counterfactual difference rewards, which draws a connection with COMA. However, value function decomposition is hard to apply in off-policy training and potentially suffers from the risk of unbounded divergence. In single-agent problems, to achieve sample efficiency and robust performance, [6] proposed the soft actor-critic algorithm, which is an off-policy actor-critic RL algorithm based on the maximum entropy reinforcement learning framework and achieves state-of-the-art performance on many challenging continuous control benchmarks.

To attain both stability and good final performance in CTDE paradigm, how to effectively incorporate soft actor critic paradigm with multi-agent value function decomposition would be important. Following the research line of [14], our key insight is, to efficiently compute the expected joint Q values, only when this linear condition — the joint Q value $Q^{tot}$ is the linear mixture of the individual Q value ${q^{i}}$ satisfy, the following equation holds (Detailed proof can be found in Appendix),

where the $Q^{tot}$ represents neural networks that consist of agent networks ${q^{i}}$ and the mixing network ${q^{mix}}$. Note that, in our method, to make the aforementioned equation holds, the mixing network ${q^{mix}}$ is not a complex non-linear way but linear weights that is produced by the hyper-network only conditioned on global state information.

Motivated by these insights, in this paper, we present a novel multi-agent soft actor-critic (mSAC) method, which is based on the following assumption: the joint Q value $Q^{tot}$ is the linear mixture of the individual Q values ${q^{i}}$. mCSAC contains three main components: decomposed soft Q network architecture, decentralized probabilistic policy, and counterfactual advantage function. This method incorporates the idea of the soft actor-critic and multi-agent value function decomposition effectively.

We empirically investigate the performance of our algorithm mSAC and analyze the influence of these components by ablation studies in StarCraft II micromanagement cooperative multi-agent tasks. Experiment results demonstrate that mSAC significantly outperforms current advanced policy-based algorithms (e.g. COMA) and achieves comparable performance with value-based approaches (e.g. Qmix) on most tasks. In addition, the variant method mSAC achieves pretty good results in large action space tasks, like 2c_vs_64zg and $MMM2$ task.

To sum up, here are our contributions:

• •

  We propose the novel mSAC method to effectively incorporate soft actor critic with value function decomposition method and investigate its practical performance on StarCraft II cooperative multi-agent benchmark.

• •

  We conduct extensive performance test of different mSAC variants to show the effect of soft value iteration, counterfactual advantage function, probabilistic policy, respectively.

In this section, we first introduce the definition and notation of the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) and then introduce the multi-agent policy gradient decomposed architecture. Afterwards we present the three variant method: multi-agent soft actor-critic (mSAC) method, multi-agent counterfactual soft actor-critic (mCSAC) method, multi-agent counterfactual actor-critic (mCAC) method, respectively.

In this section, we present our experimental results and some analysis. First, we describe the decentralised cooperated StarCraft II micromanagement benchmark to which we apply our proposed method mSAC and the variant methods we consider. Then we present the performance comparison between mSAC, the ablation variant algorithms mCSAC, mCAC and the representative value decomposed algorithm—Qmix and policy gradient algorithm—COMA in aforementioned discrete action environments. We used the Qmix implementation from this open-source code ¹¹1https://github.com/starry-sky6688/StarCraft with the same hyper-parameters as [7]. To be consistent with previous work, our implementation ²²2https://github.com/puyuan1996/MARL almost use the same network architecture and hyper-parameters across all the tasks. More experimental details can be found in the Appendix.

Experimental Setup We focus on the StarCraft II decentralized micromanagement tasks [13] ³³3We use StarCraft 2 Version SC2.4.10 in our experiments., in which each of the agents controls an individual army unit and each agent receives a global shared reward. We use StarCraft Multi-Agent Challenge (SMAC) environment [15] as our APIs, which has become a common-used benchmark for evaluating state-of-the-art MARL approaches such as COMA, QMIX and other baseline algorithms. In this paper, our algorithm learns multiple agents (or called policies) to control allied units to beat the enemy, while the enemy units are controlled by a built-in handcrafted AI, which make use of the handcrafted heuristics. Two representative StarCraftII micromanagement scenarios ($3m$ and $2c\_vs\_64zg$) are shown in Figure 2.

For comparing each method’s performance justly as much as possible, our Qmix implementation also use target Q networks that are obtained as an exponentially moving average of the Q function weights, which was different from the hard update manner in the original paper [7]. In addition, we adopt the same evaluation procedure as in [7]. For each run of a method, we pause training every 100 episodes and run 20 independent episodes where each agent performing greedy decentralised action selection (for Qmix chosen the action with the largest local Q values, for other methods chosen the action with the largest probability value). The percentage of these episodes in which the method defeats all enemy units within the (different) time limit is referred as the test win rate.

The magnitude of x-axis is 100 episodes, and for different maps, there are different episode length limits according to the difficulty level of different tasks. The shaded region indicates the one quarter of standard deviation.

Algorithm Details of Variant Methods The policy network of all agents includes a recurrent layer composed of GRUs with 64-dimensional hidden states, and a fully connected MLP layer before and after this. After the team is defeated or the time step limit is reached, one episode ends. The mixed network part of the value function consists of a single hidden layer of 64 units, and the ELU nonlinear activation function is not used. Its weights and biases are generated by an additional hyper-network composed of a single hidden layer of 64 units without the ReLU nonlinear activation function.

Similar to Qmix, the mSAC algorithm training is also carried out in mini-batch mode, the batch size is 32, the target smoothing coefficient used to update the two target Q networks is 0.005, and the discount coefficient is set to 0.95. Due to parameter sharing, all agents will be processed in parallel, and the information of each agent at each time step of each episode occupies one entry of the mini-batch. Once a new episode of trajectory is added to the replay buffer, the algorithm will update the network parameters of the actor and critic. Specifically, after collecting a episode of trajectory, 32 episodes were sampled from the replay buffer as a mini-batch to train the actor and the critic, fully expand the recurrent network part of the actor and the critic at all time steps and backpropagate the gradients, then apply the summarized gradient update to the neural network. For clarity, the hyperparameter settings of the mSAC algorithm are summarized in Table 1.

The learning performance of the mSAC method and its variant methods mCSAC and mCAC on the StarCraft II micro-operation task were tested separately to study the impact of off-policy update, soft Q value, probability distribution policy, counterfactual advantage function and other modules on the multi-agent policy gradient algorithm. In all maps, all algorithms used reward standardization techniques for stability purposes,

For clarity, we briefly outline the key differences of our different variant methods in Table 2.

Experimental Results

In this part, we compared the performance of our method mSAC, its variant methods mCSAC, mCAC, and advanced Qmix and COMA methods in the different maps, including maps with homogeneous agents ($3m$, $8m$), and maps with heterogeneous agents ($1c3s5z$, $3s5z$, $3s\_vs\_5z$), maps that agent contains a large action space ($2c\_vs\_64zg$, $MMM2$, $bane\_vs\_bane$, $27m\_vs\_30m$). The performance test win rate learning curves are shown in Figure 3, 4 and 5 respectively. Through the experimental results, it can be found that the method mSAC proposed in this chapter is similar to policy-based method COMA on the map of homogeneous agents, and is significantly better than COMA on other maps. In the maps $8m$, $1c3s5z$, $3m$, $3s5z$, compared with the current value-based method Qmix, it has similar asymptotic performance. In the map with a large action space for the agent ($2c\_vs\ _{6}4zg$, $MMM2$, $bane\_vs\_bane$), its performance is significantly better than Qmix. In some relatively difficult tasks, such as $3s\_vs\_5z$, the performance of all policy-based methods is worse than Qmix but not much. After carefully analyzing the experimental results, the following observations and conclusions can be drawn:

1. Soft policy iteration paradigm is also effective in multi-agent scenarios. From the comparison of the results of all maps we found that mCAC behaves worse than the other methods both in stability and asymptotic performance, which indicates that soft policy iteration paradigm is usually beneficial to the robust policy improvement in multi-agent policy gradient setting. We conjecture that this is because simultaneously maximizing expected return and entropy can make the agent explore more widely and efficiently, and can capture multiple modes of near-optimal behaviours.

2. It is important to jointly optimize the entire policy distribution On tasks where the agent has a relatively large action space. For example, in the map $2c\_vs\_64zg$, the Colossi unit has a large action space $|A|=70$. In the Qmix method, all agents are executed in a decentralized manner. Each agent greedily selects actions based on its local action value function. In a certain state, there is only one action that maximizes the local action value, and the others actions are given the same selection probability $\epsilon$. At a certain time, in map $2c\_vs\_64zg$, the probability of ally unit attacking a specific enemy among all the 64 enemy units is very high, while the The probability of attacking the other 63 units is the same $\epsilon$. While in the mSAC method, each agent executes actions according to its own strategy, that is, the learned categorical distribution, and can choose different areas of the action space in a planned way. By jointly optimizing the entire probability distribution to maximize the sum of expected returns and strategy entropy, intuitively speaking, this is more reasonable and effective than Qmix’s $\epsilon$-greedy paradigm exploration on tasks with large action spaces.

3. Counterfactual advantage functions are not always effective, and are more important in relatively complex tasks. For easy environments, like map $8m,1c3s5z,3s5z$, the performance of mCSAC and mSAC is similar, but in harder environment, like map $3s\_vs\_5z$ , the performance gap of mCSAC and mSAC is larger, which indicates that attribution of global reward is critical for solving this harder task, the counterfactual advantage function partially addresses the issue, could gradually learn a reasonable credit assignment during training in some tasks but is not always effective. Moreover, we carefully analyzed the performance of each seed and found that after training of 80000 episodes, in some seeds the test win rate can perform up to 90%, and some seeds are zero. We speculate that this may because it’s more difficult to explore good strategies in difficult maps, which indicates that effective exploration would be a important research problem.

In this paper, we presented the new decomposed multi-agent soft actor-critic method (mSAC) that incorporates value function decomposition, soft policy iteration, and counterfactual advantage function (optional), which supports efficient off-policy learning and addresses the issue of credit assignment partially. mSAC learns the distributional policy for each agent simultaneously which seems like a guided distributional exploration implicitly, which is especially important in large action space task through the experimental results.

In addition, we empirically investigate the performance of mSAC and its variant methods in StarCraft II micromanagement cooperative multi-agent benchmark. Experimental results demonstrate that mSAC can achieve relatively stable and efficient multi-agent off-policy learning and outperforms, or is competitive with, current main policy-based algorithms and value-based approaches (e.g. COMA, and Qmix) on most tasks, and achieves very good results in large action space task like 2c_vs_64zg and $MMM2$.

However, in this paper, we only study the effect of counterfactual multi-agent soft actor-critic paradigm on the discrete domain. Experiments under continuous domain need to be studied. In addition, The more solid theoretical analysis of the algorithm will be needed, and at the same time, how to explore more efficiently will be a valuable future work.

Some Equation Proof Details

The important equation for efficiently calculating the expectation of the joint Q values using the expectation of the local Q values following local policies:

The detailed proof is as follows:

we could approximate the above equation using following equation,

if we use a different mixing network for entropy term, the equation becomes: