Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning

Intrinsic rewards help agents learn useful policies across a wide variety of tasks and environments, even sometimes with sparse environmental rewards. At each time step, the agents receive not only environmental rewards but also intrinsic rewards. Intrinsic rewards can help agents improve their exploration ability and enhance social influence. In reinforcement learning, all kinds of information have various signals as intrinsic rewards, such as empowerment (Mohamed and Rezende 2015), model surprise (Blaes et al. 2019), information gain (Houthooft et al. 2016), and learning progress (Blaes et al. 2019). In MARL, some existing approaches utilize the correlation or influence of agents as intrinsic rewards to facilitate collaboration. EITI (Wang et al. 2020) leverages MI to capture the influence between an agent’s current trajectory and the next states of other agents in the environment, which is used as an intrinsic reward to encourage agents to explore cooperatively. SI (Jaques et al. 2019) proposes a method that utilizes social influence as an intrinsic reward, measured by the MI between an agent’s current action and the estimated next action of other agents, to achieve coordination and communication in MARL. VM3-AC (Kim et al. 2023) is similar to the idea of coordinating inter-agent behavior in SI-MOA, except that additional latent variables are introduced to induce nonzero mutual information between multi-agent actions, and the policy iteration algorithm is modified based on MI. PMIC (Li et al. 2022) utilizes the MI between global states and joint actions as a new standard. Based on this standard, it maximizes the MI related to behaviors that are conducive to collaboration and minimizes the MI related to behaviors that are not conducive to collaboration, breaking the current suboptimal collaboration and learning higher-level collaborative behaviors. Comparatively, our approach tackles the challenge of collaboration among agents from a causal inference perspective, which detects situation-dependent causal relationships among agents as intrinsic rewards to facilitate the exploration of inter-agent coordination.

The combination of reinforcement learning and causality has achieved some progress. The work (Schölkopf et al. 2021) utilizes causal modeling to achieve better state abstraction, enabling the agent to concentrate on key aspects and indirectly improving sampling efficiency. The work (Seitzer, Schölkopf, and Martius 2021) proposes integrating measures of causal influence into reinforcement learning algorithms to address the problems of exploration and learning in the robot manipulation environment. The work (Lee et al. 2021) utilizes causal intervention to identify the most relevant state variables for completing a task, thereby reducing the dimensionality of the state space. The work (Pitis, Creager, and Garg 2020) utilizes influence detection to create counterfactual data to enhance the training of RL agents. (Maes, Meganck, and Manderick 2007) extends causal Bayesian networks to multi-agent models, which inspires us to develop a causal graph model for MARL.

In this work, we consider the fully cooperative multi-agent game in the partially observable setting, which can be modeled as a decentralized partially observable Markov Decision Process (Dec-POMDP) (Oliehoek and Amato 2016). The Dec-POMDP is formally defined as a tuple $\langle\mathcal{I,S,U,O,P,R,\gamma}\rangle$, where $\mathcal{I}$ = $\{1,2,....N\}$ denotes the finite set of agents, $s_{t}\in\mathcal{S}$ denotes the set of joint states that cannot be observed by agents at the time step $t$. At each time step $t$, each agent $i$ $\in\mathcal{I}$ can only observe its local observation $o_{t}^{i}$ from observation function $\mathcal{O}(s_{t},i)$ and chooses an action $u_{t}^{i}\in\mathcal{U}^{i}$ according to its policy $\pi(\dot{|}o_{t}^{i})$, forming a joint action $u_{t}\in\mathcal{U}$. After executing $u_{t}$ in environment, each agent $i$ achieves a shared extrinsic reward $r_{t}$ from the reward function $\mathcal{R}(s_{t},u_{t})$ with a discount factor $\gamma\in[$0,1$)$ and the next state $s_{t+1}$ according to the transition function $\mathcal{P}(s_{t+1}|s_{t},u_{t})$. The goal of a collaborative team is to find a joint policy $\pi^{*}$ that can maximize the expected extrinsic discount return ${E}[\sum_{k=1}^{\infty}\gamma^{t}r_{t}]$ under the setting of fully-cooperative MARL.

In our approach, we adopt a CTDE paradigm, which has been a widely considered training paradigm in recent efforts in MARL (Sunehag et al. 2018; Rashid et al. 2018; Lowe et al. 2017; Iqbal and Sha 2019). During training, each agent can access to full information including the states, actions, rewards, and actions of other agents, while decentralized execution is conditioned solely on individual observation $o_{t}^{i}$.

A causal graphical model (CGM) (Shanmugam 2001) is commonly represented as directed cyclic graphs (DAGs) $\mathcal{G}=(V,E)$ and is defined by a distribution $\mathcal{P}$ over the set of random variables $\mathcal{X}$. The graph $\mathcal{G}$ consists of nodes $V$ and edges $\mathcal{E}\in V^{2}$ with $(v,v)$ for any $v\in V$. Each node $v_{i}$ is associated with a random variable $\mathcal{X}^{i}$ and each edge $(v_{i}\rightarrow v_{j})$ represents that $\mathcal{X}^{i}$ is a direct cause of $\mathcal{X}^{j}$, i.e., $\mathcal{X}^{i}$ is called a parent of $\mathcal{X}^{j}$. The set of parents of $\mathcal{X}^{j}$ is denoted by $\mathbf{PA}_{j}^{\mathcal{G}}$. The distribution $\mathcal{P}$ can be represented as

where $\mathbf{PA}_{j}^{\mathcal{G}}\subset\left\{X^{1},...,X^{v}\right\}\backslash\left\{X^{j}\right\}$ denotes the set of parents of $X^{j}$. The CGM models the structure of the causality. To reveal the causal structure from the data distribution, we assume that CGM satisfies the Markov condition and the faithfulness assumption, which makes the independence consistent between the joint distribution $P(X^{1},...,X^{v})$ and the graph $\mathcal{G}$.

Intervention sampling is a typical operation in causal discovery. Different from standard sampling, it sets the distribution of variables that require intervention in the causal graph model to a uniform distribution or fixed value, and then conducts sampling to obtain intervention data. Intervention data has more causal information than observational data, which is beneficial to measure causal relationships between random variables. In this work, we need to measure the influence of the current time step’s action of an agent on the state of other agents in the next time step, which can be obtained by intervening in the current behavior distribution.

A key component of SCIC is the intrinsic reward mechanism that brings each agent to a state where it can influence interested agents as much as possible. Intuitively, an agent is more likely to influence interested agents in certain states, and making agents reach these states as much as possible is more likely to enhance the causal influence between agents. We represent agent $j$ being able to be influenced by agent $i$ as “agent $i$ being able to take control of agent $j$”. Through mutual incentives between agents, agents can achieve friendly interaction and efficient cooperation with their peers. Thus, the causal influence of other agents on an agent can be viewed as a necessity for learning intrinsic reward mechanisms. We detect the causal influences between agents in particular situations using causal intervention and conditional mutual information. It is worth noting that the computation of causal influence is only executed during intensive training, where each agent can know the policies and actions of other agents.

With the causal influence estimations introduced in the previous subsection, the goal of the following is to learn the joint policy that maximizes the expected discounted reward, using causal influence between agents as an intrinsic reward. Specifically, agent $i$ receives a joint reward, combining the extrinsic team reward and the intrinsic reward from peer causal influence, that can be represented as

where $CI^{i,j}$ represents the causal influence of $A_{i}^{t}$ on $S_{t+1}^{j}$. $\alpha$ is a hyper-parameter, which is utilized to balance the intrinsic reward and the extrinsic reward. The intrinsic reward $r_{t,in}^{i}$ of agent $i$ is represented as $r_{t,in}^{i}=\sum_{j\neq i}CI^{i,j}$. In particular, each agent needs to learn a policy to maximize the conventional objective $J(\pi):J(\pi_{\theta}^{i})=\mathbb{E}[\sum\limits_{t=0}^{\infty}\gamma^{t}r_{t,total}^{i}]$. By maximizing the conventional objective, agent $i$ can take control of other interested agents. In principle, our proposed intrinsic reward mechanism can be combined with different CTDE-based MARL algorithms. In this work, our method is built upon MADDPG. The joint action-value function of each agent $i$ is approximated by the joint action-value network $\mathcal{Q}_{\phi}$ by minimizing the following loss

where $\hat{y}=r_{t}+r_{t,in}^{i}+\gamma\mathcal{Q}_{\phi_{i}^{{}^{\prime}}}(s_{t+1},\pi_{\theta^{{}^{\prime}}}(s_{t+1}))$, $\phi_{i}$ indicates the parameters of critic network, and $\phi_{i}^{{}^{\prime}}$ and $\theta^{{}^{\prime}}$ are the parameters of target networks. Then, the policy of agent $i$ is updated by minimizing the loss

where $\theta=(\theta_{1},...,\theta_{n})$ are the parameters of actor networks. The training algorithm is described in Algorithm 1.

We evaluate our proposed approach on three benchmark multi-agent tasks: Partial Observation Cooperative Predator Prey, Cooperative Navigation, and Cooperative Line Control. The benchmarks’ environment is implemented in a Multi-Agent Particle Environment ((Lowe et al. 2017)), where agents can follow a dual integrator dynamic model to move in 2D space. Like Cooperative Navigation, Cooperative Predator Prey is a well-known evaluation task for MARL. The Cooperative Line environment is a complex task environment, where there are $M$ agents and 2 targets, with the goal of allowing agents to evenly distribute themselves on the line between two targets. All algorithms are trained in a Linux server with a 2.30 GHz Xeon(R) CPU and two Nvidia 4090 graphics cards. The learning rates of the critic network and the actor network are set to 0.001. The discount factor $\gamma$ is set to 0.95. Each episode lasts up to 25 timesteps. To estimate the transition marginal distribution $p(s_{t+1}^{j}|s_{t}^{i})$, the number $K$ of per Monte-Carlo sample is set to 64.

To verify the superiority of our method, we employed three baseline algorithms for comparison, namely PMIC, MADDPG and SI, with a relaxation that all SI agents are equipped with the social influence reward. Among them, PMIC and SI are currently state-of-the-art MARL algorithms using intrinsic rewards in Multi-Agent Particle Environment (MPE). Since our proposed algorithm is based on MADDPG, MADDPG is also utilized as a baseline algorithm for comparison. Additionally, to verify the effectiveness of the components of the proposed method, some ablation studies also will be provided. Furthermore, considering the significant role played by the temperature parameters of intrinsic reward in balancing the relative importance between extrinsic and intrinsic reward, we also provide an experimental study on the temperature parameters.

Since our approach is based on MADDPG with centralized training and decentralized executing, we name it SCIC-MADDPG in the experiments. First, we evaluate the performance of our approach and other baseline approaches on Cooperative Predator Prey with 3, 4, and 5 predators, in which the policy of predators needs to be trained and the policy of prey is fixed. Figure 3(a), Figure 3(b), and Figure 3(c) illustrate the comparison results of rewards for SCIC-MADDPG, PMIC, SI, and MDDDPG on Predator-Prey tasks with 3, 4, 5 predators, respectively. We can observe that our SCIC-MADDPG demonstrates better performance compared to other algorithms, although there are relatively minor advantages when there are 4 predators in the environment. SCIC-MADDPG converges to a better reward than MADDPG, which indicates that the intrinsic reward is indeed conducive to improving the cooperation among agents. Both PMIC and SI receive lower rewards, suggesting that promoting agents to reach significant states is more beneficial for cooperation between agents than merely focusing on behavior coordination. From another perspective, enhancing the causal influence between agents is evidently more conducive to cooperation between agents than solely focusing on enhancing their correlation.

Moreover, we further perform evaluations on the Cooperative Navigation task with 3, 4, and 5 robots, which requires each agent to reach a distinct landmark to achieve the shortest total distance traveled. From Figure 3(d), Figure 3(e), and Figure 3(f), we can observe that our SCIC-MADDPG consistently achieves higher rewards than the other baseline methods in the Cooperative Navigation tasks with varying numbers of agents. Furthermore, we evaluate the performance of SCIC-MADDPG on more challenging Control Line tasks involving 3 and 5 agents, representing an exceptionally difficult task scenario. The experimental findings illustrated in Figure 3(h) reveal that SCIC-MADDPG outperforms other methods in the context of a 5-agent scenario. Additionally, Figure 3(g) illustrates that SCIC-MADDPG achieves superior performance in comparison to SI and MADDPG, while achieving comparable performance to PMIC. In summary, the results of these experiments collectively indicate that SCIC-MADDPG proves to be an effective approach, consistently outperforming other methods across various tasks.

To further evaluate our proposed approach, we further conducted a series of ablation experiments in this subsection. Specifically, to evaluate the effectiveness of intervention sampling, we implemented the SCIC w/o Intervention method, which obtains the action set when estimating causal effects between agents, not through intervention with a uniform distribution, but by sampling from the replay buffer. The ablation study was conducted on the Predator Prey task with 5 agents. As shown in Figure 4(a), SCIC-MADDPG performs significantly better than MADDPG and SCIC w/o Intervention. The key reason behind this achievement can be attributed to the fact that the sampling actions in the off-policy reinforcement learning algorithm originate from a mixture of different policies, which cannot be utilized to estimate causality. The results reveal two facts: 1) Using causal influence between agents as a Reward Bonus contributes to enhancing the performance of MARL algorithms by facilitating cooperation among agents; 2) The adoption of intervention leads to more accurate estimates of causal influence.