Enabling Multi-Agent Transfer Reinforcement Learning via Scenario Independent Representation

SMAC is constructed on top of the StarCraft II Learning Environment[26] and provides a wide selection of multi-agent micromanagement challenging scenarios where the goal is to eliminate opponents controlled by the built-in StarCraft AI.

Our work focuses on the micromanagement of a team of individual agents in decentralized decision-making with a shared interest, rather than the macromanagement of the army as a whole in a centralized way. SMAC has various homogeneous and heterogeneous scenarios, including symmetric and asymmetric distribution in each group. In each episode, SMAC provides a set of observations with state rewards and accepts a list of actions that make it ideal for RL algorithms to apply. The SMAC scenarios are named following a specific convention where the alphabetic character signifies the type of the agents, such as marines being abbreviated as $m$, stalkers as $s$, zealots as $z$, etc. The numeric value represents the number of active agents in each team. Our experiments utilize multiple scenarios with different difficulty levels including homogeneous scenarios $3m$, $8m$, $25m$ for evaluating intra-agent knowledge transferring and a heterogeneous scenario $2s3z$ for inter-agent knowledge reusing.

We model the SMAC scenarios as Markov games, which is a multi-agent extension of Markov Decision Processes (MDP) [27, 28, 29]. A Markov game includes a set of states denoting the agents’ and environment’s status, and a set of actions and observations for each agent involved. $A_{1},A_{2},...,A_{N}$ denotes the actions and $O_{1},O_{2},...,O_{N}$ denotes the observations where $N$ is the number of agents in each episode. The ally and enemy units, surrounded by the environment, are modeled for individual agents in the Markov game. The decision of action is taken by the agents’ current observation. In a Markov game where each agent acts in accordance with a policy $\pi$ at each environmental step and earns a shared reward $r$, $r$ represents the state transition from $S$ to $S^{\prime}$ ($S\times\{A_{1},...,A_{N}\}\rightarrow S^{\prime}$).

For the scenario-independent state representation, we consider two types of observation space in our experiments. The first one is local observation, where each agent has access to only a limited range of surrounding information that the agent can observe. The other one is shared global abstraction, where all the information of the game environment is accessible from a holistic point of view. The local observation of an agent contains a list of properties including hitpoints, unit type, relative positions, and distance for both allied and enemy agents within the observation range. This information is received from SMAC without customizing the default value. For the shared global abstraction, all agents on the map are collected along with agents’ features from the local observations, in addition to the weapon cooldown and previous action of each agent in the game. All features used in our experiments are normalized to $[0,1]$ for both the local and abstracted global observation spaces. Fig. 2 shows the default sight and shooting range of an individual agent in a SMAC environment. In SMAC, the default achievable reward in a single episode is scaled to a value between 0 and 20 [30], ignoring the negative rewards by default. The reward is distributed to the entire team and takes into account factors such as inflicted damage on enemy units, points earned from killing units, and a bonus for achieving victory, rather than being individually assigned to each agent. Equation 1 shows the shared reward computation for a full game episode,

where $t$ is the current environmental step, $T$ is the terminal step, $n$ is the agent identifier, $N$ is the maximum number of agents, $D_{n}$ is the damage dealt to enemy units by agent $n$ in step $t$, $k$ is the number of enemies that have been defeated in step $t$, $R_{max}$ is the maximum possible reward amount, and $w$ is 1 on a win and 0 on a loss. The episode reward is calculated when the game is over and a discounted reward with a decay of $0.9$ is used for training our RL models.

Our MARL policies were evaluated on various SMAC scenarios, each with $31$ different random seeds for statistical analysis. After tuning the parameters representing the unified input and output, each of the experiments was conducted for $2000$ episodes. For statistical analysis, we evaluate the proposed model on $31$ random seeds in different SMAC scenarios. Each experiment runs for a total number of $2000$ episodes considering the large number of tuning parameters introduced in our neural network architecture. We built our neural network on top of the standard Advantage Actor-Critic (A2C) algorithm for both actor and critic networks [25] utilized $\varepsilon$-soft, which combines the greedy and softmax methods with an initial value of 1 and gradually decreases as the process continues. Equation 2 is used to calculate $\varepsilon$ at a specific time step where $\varepsilon_{0}$ is 0.0001 and the total number of environmental steps denoted by $\gamma$ is $30,000$.

We use $\varepsilon$-soft instead of $\varepsilon$-greedy in order to select an action according to the softmax behavioral policy, which randomly selects an action, where the likelihood of selecting an action is the proportion of the action’s estimated value relative to the sum of the estimated values for all actions. This methodology was chosen because it promotes continued intelligent exploration of the state space, even when $\varepsilon$ itself becomes arbitrarily small. Exponential Linear Unit (ELU) has been used for the activation function where the value for $\alpha$ is 1. The learning rate used for each of the episodes is $0.0001$.

Reusing existing knowledge can greatly speed up the MARL learning process and make complex tasks more achievable for agents in MAS. This involves mapping a knowledge space to a policy, where the knowledge space is composed of samples from the current task, previous tasks, and other agents. To acquire an optimal policy, model-free MARL techniques require a considerable number of samples to be trained. It is challenging to learn optimal policies due to the large state and action spaces. For instance, there are various situations involving distinct units and terrain conditions in SMAC, and finding efficient strategies for each scenario from scratch can be time-consuming. Creating a generalized representation for the states independent of specific scenarios is a big challenge in SMAC scenarios. In the default state representation provided by SMAC, the states are represented based on the number of units in each scenario shown in Fig. 3a. For scenarios $3m$, $8m$, and $25m$, the size of local state is provided in the form of $30$, $80$, and $250$ one-dimensional vectors respectively which is marked with a dotted rectangle of red color in Fig. 3a. The default state representation is compact and carries precise agent information for training MARL. However, it limits the knowledge transfer over multiple scenarios due to the scenario-dependent state size. To mitigate this problem, we further focused on constructing a fixed-size state representation that is unified in such a way that the state size remains constant, irrespective of the number of agents in the scenario.

Curriculum learning is a specific type of transfer learning that arranges a series of tasks according to their increasing level of complexity [31]. This approach involved training on how to play a game against simulated opponents who became progressively more competent, allowing the agents to learn useful strategies and gain knowledge that they could apply to real-world challenges [32]. We evaluated the performance of Curriculum Transfer Learning (CTL), which merges curriculum learning and transfer learning to facilitate faster convergence of the learning process and lead to better optimum outcomes within the fixed time steps. We delve into the intriguing question of how the transfer of winning strategies among Marines learned in scenarios $3m$ and $8m$ can positively impact the behavior of Stalkers and Zealots in an extended heterogeneous environment $2s3z$. Fig. 5 shows the curricular process flow of training our policy from the simplest scenario, $3m$, retrain the learned model in a medium-level scenario, $8m$, and finally carries the knowledge learned from $3m$ and $8m$ to tackle a much more complex scenario $2s3z$ with different unit types and heterogeneous team units that the policy has never seen in prior training scenarios.

In the first step, we trained each of our selected homogeneous scenarios for 2000 epochs and 31 independent instances in parallel. The agents exploit the information from the ongoing simulation and train the neural networks accordingly. Performance comparison of these models was carried out based on different input state resolutions. Table I represents the outcomes of $3m$, $8m$, and $25m$ with varying local state dimensions ranging from $19\times 19$ to $55\times 55$ resolutions. Results showed that the agents for $3m$ and $8m$ achieved a score of $20$ which represents the agent successfully defeated their opponent team in the given scenario. However, for $25m$, the highest score obtained by one of the best-trained models was $15.97$, with a $37\times 37$ dimension of local state representations. Upon considering the overall stability of the performance, the $37\times 37$ dimension was chosen for local state reformulation in a unified manner. Consequently, the best-performing RL model from each of the $3m$, $8m$, and $25m$ scenarios is selected as a seed for training our shared neural networks on other scenarios, utilizing the transfer learning technique.

The seeds selected from different homogeneous scenarios have been utilized for training other scenarios to evaluate the performance of transferring knowledge across scenarios in SMAC. Fig. 6 illustrates the performance improvement in $3m$, $8m$, and $25m$ scenarios using TL based on different seed policies. In Fig. 6a, we can see that the performance of $3m$ is improved significantly when the knowledge of $8m$ and $25m$ is used instead of learning from the beginning without any shared information. The average running episode reward of $3m$ is enhanced by $9.7\%$ and $19.97\%$ when the pre-trained policy of $8m$ and $25m$ is utilized, respectively. The reason behind the improvement of decision making is that both $8m$ and $25m$ have a larger number of Marines actively learning, and extracting those information boosts the learning performance compared to $3m$ where only three active Marines are fighting against the enemy units. Fig. 6b reflects the performance of $8m$ on the scale of maximum average running episode reward. The performance curve of TL-$3m$ depicts the impact of the seed policy $3m$ on $8m$ scenario with an overall $45.3\%$ average score improvement. We can see that the pre-trained models performed poorly at the beginning of the training due to the change in the unseen scenario and then outperformed other models after 1000 iterations, proceeding toward the winning strategy in each episode afterward. It took However, reusing the pre-trained $25m$ model boosts the average performance with an overall improvement of $8.01\%$. Though our model couldn’t ensure stable winning for $25m$, it tweaked with a better representation to deal with enemy units and slightly surpassed the base units’ performance. Fig. 6c shows the impact of seeds $3m$ and $8m$ on the $25m$ scenario, where the average running episode reward is enhanced by $36.4\%$ and $41.3\%$ respectively compared to the learning from scratch approach.

Table II reflects the statistical analysis of the TL model on extended homogeneous SMAC environment. To examine the effectiveness of our proposed approach, four primary metrics are considered: the maximum, minimum, average, and standard deviation (STD) of the average of the running average episode reward with respect to various thresholds. All these evaluations are performed across a complete set of 31 seeds for statistical analysis. The highest minimum average running episode reward score is $5.21$, $11.5$, and $6.54$ for $3m$, $8m$, and $25m$, respectively, when the pre-trained seed policies $8m$, $3m$, and $8m$ are incorporated in the training phase instead of learning from scratch. Similarly, the maximum and average of the running episode outperformed the base scenarios due to the effect of TL and taking action based on the knowledge of other scenarios. The fluctuation in the score is minimized, depicted by the STD value. The smaller the value of STD, the more stable the model is in terms of performance. The boldly marked values are the highest values highlighted in different SMAC scenarios.

As of yet, our study investigates the impact of TL on homogeneous scenarios for evaluating intra-agent knowledge transfer, and the experimental results demonstrate a significant improvement in MARL performance. We further aim to extend our evaluation to complex heterogeneous scenarios evaluating not only the intra-agent knowledge transfer within the same unit type, but also the inter-agent knowledge transfer among different types of units. To this end, we have selected the $2s3z$ testing bed and compared the performance using seeds from prior experiments. We scrutinized the outcome of CTL in which the $3m$ model was trained at first, as this is one of the simplest maps in SMAC. After training each $3m$ instance for $2000$ episodes, we saved the best-performing model and utilized it for $8m$, which is comparatively difficult to train. The knowledge learned to control Marines from $3m$ and $8m$ is then carried out for the most challenging scenario of our test environment, $2s3z$ containing Stalkers and Zealots that the model has never seen in the previous training scenarios.

Fig. 7 depicts the outcome of the average running episode reward for $2s3z$ model in different TL and CTL scenarios. The graph shows the upward trend of our curriculum model when the information is carried out from $3m$ to $8m$ and then $8m$ to $2s3z$. The TL model using pre-trained seeds $3m$ and $8m$ improved the average reward of $2s3z$ by $38.6\%$ and $38.2\%$ compared to the performance of learning from scratch on $2s3z$. The performance of all these simulations was surpassed by the performance of CTL with an overall improvement of $72.2\%$. Table III demonstrates the statistical representation of the performance metrics. The curriculum transfer $3m\rightarrow 8m\rightarrow 2s3z$ works best with the maximum average reward of $17.2$ and the highest average of $13.83$ in our simulation environment. This ensures our curriculum architectures’ robustness in intra- and inter-agent knowledge transfer.