Enhancing Two-Player Performance Through Single-Player Knowledge Transfer: An Empirical Study on Atari 2600 Games

Transfer learning refers to using knowledge gained from one or multiple source domains to improve a learner in a target domain. Transfer learning strategies are applied to both traditional machine learning techniques and deep neural networks. Using transfer learning could result in more efficient training and better performance, since the source domain(s) has already useful features that could be leveraged for the new task (Zhuang et al. 2020).

Reinforcement learning is a subfield of machine learning in which an agent aims to maximize the cumulative reward while interacting with its environment (Sutton and Barto 2018). Utility values for states or Q values for state-action pairs are updated through trial and error. RL can be used to train agents that play games such as chess, poker, or Go at levels that surpasses the capabilities of the human players or is comparable to the best of them. (Silver et al. 2016). A DQN uses a deep neural network to approximate the Q values and is able to handle complex learning environments such as those found in video games by using the entire screen of pixels as inputs (Justesen et al. 2019). The use of DQN and its successors have allowed researchers to tackle a vast set of video games, from first-person shooters to sports games (Justesen et al. 2019). Deep RL approaches, however, are not without drawbacks: among them, interpretability (Eberhardinger, Maucher, and Maghsudi 2023; Sieusahai and Guzdial 2021) and long training time.

Combining transfer learning and RL allows for quicker learning and learning more complex problems (Braylan and Miikkulainen 2016). This concept has been applied to a relational regression algorithm used to learn a generalized Q-function (Ramon, Driessens, and Croonenborghs 2007). More recently, Gamrian and Goldberg (2019) show that transfer knowledge between slightly modified RL environments can be improved using Generative Adversarial Networks (GANs) with imitation learning. Anwar and Raychowdhury (2020) demonstrate the effectiveness of transfer learning in deep RL for autonomous navigation in the Unreal game engine. Pleines et al. (2022) propose sim-to-sim transfer, transferring knowledge learned in a limited Unity engine implementation of Rocket League to the full game, showing that knowledge quickly learned in a simplified environment can be useful in a more complex environment. Balla and Perez-Liebana (2022) present research on the effectiveness of pre-training Successor Features in transferring their knowledge to new target tasks without further training. The survey by Muller-Brockhausen et al. (2021) discusses the difficulty of transfer learning in generalizing to different problem variations and the need for a united benchmark to test transfer learning in RL. This is an ongoing challenge requiring further studies.

A multi-agent environment refers to a setting where multiple autonomous agents learn to interact with each other and the environment simultaneously. This framework is often used to model complex, dynamic systems with multiple decision-makers such as multiplayer games. In a multi-agent environment, each agent has its own goals and learning process. They may cooperate, compete, or do both, depending on the situation (Tampuu et al. 2017). Their actions affect not only their own future state but also those of the other agents, creating a complex, intertwined learning problem.

There are many different approaches in multi-agent environments. Boeda (2021) proposes planning using Goal Oriented Action Planner to facilitate cooperation among multiple agents. Lei and Zhang (2017) study cooperative game theory with a service composition method.

One common method for training agents in multi-agent environments is RL. Multi-Agent RL (MARL) extends RL to multi-agent settings, where multiple agents learn to optimize their behaviors while interacting with each other. Chakraborty and Stone (2014) examine convergence to Nash equilibrium and targeted-optimality in learning. Booth and Booth (2019) present Marathon Environments, a suite of continuous control benchmarks for multi-agent RL in the Unity game engine. Ferreira et al. (2022) propose a framework for developing multi-agent cooperative game environments to facilitate the process and improve agent performance during RL. MARL has also been examined in StarCraft (Khan, Ahmed, and Sukthankar 2022), where a transformer-based joint action-value mixing network is shown to be superior to other benchmarks.

Won, Gopinath, and Hodgins (2021) present a learning framework that generates control policies for physically simulated athletes in two-player games, boxing and fencing. The framework uses a two-step approach for learning basic skills and learning bout-level strategies. The authors develop a policy model based on an encoder-decoder structure that incorporates an autoregressive latent variable and a mixture-of-experts decoder. The pre-trained model with basic skills is transferred over and trained on the two-player environment.

In our work, we explore the use of knowledge transfer from a single-agent environment to its corresponding multi-agent environment, sidestepping the complications from training directly in a multi-agent environment. Training in a single-agent environment has several potential advantages over a multi-agent environment, mainly due to its stablility and efficiency. Single-agent environments are often more stable and predictable because there is no other learning entity to introduce non-stationarity. In multi-agent settings, each agent’s policy is changing as it learns, which can lead to a constantly shifting environment that is challenging to learn from. Training can be more computationally efficient in single-agent settings. Each additional agent in a multi-agent setting increases the complexity of the state and action spaces, which can slow down learning.

We deploy the concept of self-play in our empirical study. Self-play in RL is defined to be a method of deploying an algorithm against copies of itself to learn and test in various stochastic learning environments. Self-play has been well-tested in many applications. AlphaGo Zero and AlphaZero (Silver et al. 2018) famously use self-play to achieve superhuman performance in chess, shogi, and Go. Self-play has also seen success in fighting game AI (Takano et al. 2019), “Big 2”, a four-player game of imperfect information (Charlesworth 2019), among others. Liu et al. (2021) introduce a critic into the policy gradient method to form a self-play actor-critic method for training agents and demonstrate its success in a few simple environments.

In our empirical study, we have chosen ten different Atari environments provided by Gymnasium (single player) (Towers et al. 2024) and PettingZoo (two players) (Terry et al. 2021). Both of these libraries use the Arcade Learning Environment (ALE) which is a platform for building intelligent agents across different Atari games (Shao et al. 2019). There have been numerous attempts at training AI agents to play Atari games using RL. One of the most notable achievements is when DQN reached human-level performance across 49 Atari games (Mnih et al. 2015). After this breakthrough, other algorithms and improvements of DQN such as Rainbow DQN and Ape-X DQN have shown promising results and in some cases surpassing DQN results (Shao et al. 2019). More recently, a novel approach of creating an auxiliary reward using instructions extracted from the manuals of the Atari games have shown improvement in performance and training speed (Wu et al. 2024). For the purpose of this study, we focus on the regular DQN algorithm that is common with Atari games. We use DQN with a couple of improvements such as prioritized experience replay and double structure to aid the training process.

For developing our Atari transfer benchmark, we focus on the games that are present in both Gymnasium and PettingZoo libraries, with single-player and two-player versions. Gymnasium provides single agent Atari environments while PettingZoo offers a library to facilitate multi-agent Atari training. Atari environments are simulated via the Arcade Learning Environment. We use a total of nine Atari games in ten Atari environments. The names of the environments sorted alphabetically are: Boxing, Double Dunk, Entombed competitive, Entombed cooperative, Flag Capture, Mario Bros, Pong, Space Invaders, Surround, and Tennis. Entombed is a game with two different environments, cooperative and competitive, based on the strategy of the player. Throughout the rest of this paper we use “Atari games” as a shorthand for Atari game environments. For training, the Atari RAM is used instead of image pixels to optimize computational power and time. RAM observations are a total of 128 bytes.

In single-player environments, all games except Mario Bros, Entombed, Flag capture and Surround use version 4 (v4) environments with default frame skip of one and no action repeating. The four mentioned games use version 5 as it yields better results. In version 5 there is a default frame skip of 4 and repeat action probability of 0.25.

Single-player training is conducted using the Tianshou platform (Weng et al. 2022). In addition to the environment’s settings, the default processing of the DQN Atari benchmark in Tianshou is used. These include sampling initial states by taking random number of no-ops on reset, returning every 4th frame (frame skipping) with maxing over 2 most recent raw observations, making the end of life the same as end of episode (bootstrapping values) without resetting to help with the value estimation, normalizing the observation between 0 and 1, and clipping the reward to be between -1 and 1.

For two-player environments, the same reward clipping and observation normalization is used to ensure that the network can comprehend values after transfer. Based on the work of Lee et al. (2022), we also add the following preprocessing techniques:

1. 1.

   Limiting the max number of steps in each episode to 200.

2. 2.

   Frame skip of 4.

3. 3.

   Sticky actions with a probability of 0.25.

Pre-processing in two-player Atari games is done using the SuperSuit library (Terry, Black, and Hari 2020) and training is done using the Machin library (Li 2020). In two-player training, no-op resets are performed on the first 130 frames for Space Invaders, and the first 60 frames for Pong (Lee, Ganapathi Subramanian, and Crowley 2022). Two-player environments reset upon termination of first-player. In Mario Bros, the game can continue even if player two is dead, so we ensure that the game resets when player two is terminated as well.

A double deep Q-network is the main RL algorithm to train on the games for both single-player and two-player environments. Prioritized experience replay is used alongside this to ensure that experiences with a higher importance get prioritized during training. A fully connected deep neural network with the same structure for both single-player and two-player games is used to map observations of 128 bytes to the number of actions that varies based on the environment but is consistent for both single-player and two-player versions of the same game. The details of this neural network and hyper-parameters used for training can be found in Table 1.

For playing the opponent in two-player versions of games, self-play is used. During training, only the network of player 1 (also called “the player”) is trained while the network of player 2 (also called “the opponent”) is only updated at certain time-steps by copying the weights of player 1’s network. This ensures that the opponent’s network is always updated with the previous generations of the player’s network. The network of player 1 is solely trained using player 1’s experience. Since we aim to not train the opponent’s network separately, we need to ensure that the network understands the difference between players 1 and 2 when playing. For this purpose, we use our method of agent indication in two-player Atari that is discussed in the next section.

In Atari, all agents have the same observation from the environment, which could be a problem when there is a need to indicate which player is which. A common method of agent indication is by adding a separate channel to the observation to indicate the agent’s turn (Gupta, Egorov, and Kochenderfer 2017). However, adding a new channel to a network could make the training more difficult as it could make it harder for transfer learning to discover parts of the observation that are completely new. To avoid this problem, we annotate the parts in the Atari two-player RAM that are directly related to player 1 or 2. The annotation has been done using both PettingZoo and PCAE Atari emulator. Annotations of four environments are inspired by the annotations from the Atari Annotated RAM Interface (Anand et al. 2019). Our focus is mainly on elements that could affect a player’s gameplay such as player avatars’ locations, scores, and the number of remaining lives. After annotating the desired parts, we implement a PettingZoo custom wrapper that would reconstruct the observation of player 2 by swapping the related parts of player 2 with player 1. As an example, the bytes related to locations of player 1 and 2 are swapped. Meaning that the specific byte that contains the location of player 1 now contains the location of player 2 and the network is able to decide based on this location. Essentially, the observation of player 2 is reconstructed as if player 1 were the one playing instead. The code used for training two-player environments including this custom wrapper is publicly available in our GitHub repository at https://github.com/Justkim/Two-player-atari-RL .

For the transfer of knowledge from a single-player game to the two-player version of the same environment, the agent is first trained in single-player setting. After this training, all the weights are transferred to a new deep neural network to be trained in a two-player setting. To make the overall training more efficient and decrease the total running time while leveraging knowledge from the transferred network, the first two layers are frozen and do not get updated during two-player training. We have conducted preliminary experiments using different numbers of layer freezing, and found that freezing the first two layers produced the most desirable results.

As part of our research, we conduct an empirical study in the ten Atari games. For each game, two sets of training are conducted. For the transferred agents, first the network is pre-trained on the single-player game for 10 million steps. Then, the weights of this network is transferred to the two-player version of the same game for further running of 20,000 episodes. An episode refers to a complete sequence of steps that starts from an initial state and ends when the environment reaches a terminal state. Five different sessions with different random seeds are used in the two-player environments. For the training-from-scratch agents, five sessions with different random seeds in each game are done from scratch in the two-player version without any transfer for 20,000 episodes. The results of these experiments in the two-player versions are compared (transferred vs. from scratch) using the rewards gathered by the player in each episode. Across the different games and experiments, the same hyper-parameters are used to ensure consistency and avoid bias.

Table 1 reports the set of hyper-parameters for training single-player and two-player environments. The experiments (two-player training) were conducted using a cluster with 7 cores of 2x Intel(R) Xeon(R) Gold 5320 CPU @ 2.20GHz (Ice Lake) and a maximum of 10G RAM.

One advantage of transfer learning and layer freezing is the running time saved. Table 3 shows the average time taken for each of the ten games in the two-player versions. In each case, the transferred version took less time than the version from scratch. This makes sense since layer freezing allows a network to be trained with less parameters. The results show that the transferred version took, on average, 27% less time, a substantial saving.

Figure 1 shows the average reward over five training sessions with different random seed values in transferred vs. from scratch settings, again both in the two-player versions. The graphs show a running average with a window of 10 and random sampling to present the trend of the training process more clearly. Table 2 presents the mean and standard deviation of average total rewards in five different points in time: At the start of training, 5000 episodes in, 10,000 episodes in, 15,000 episodes in, and lastly the final episode. These provide a sample of the training process for each agent in each game.

In each graph in Figure 1, the blue curve indicates the transferred version while the orange curve indicates the version from scratch. It is interesting to observe that while in some games (in particular, Double Dunk, Space Invaders, and Tennis), the transferred version clearly outshines the version from scratch, this is not consistent across all games. While we do not observe any environments where the version from scratch is better than the transferred version, there are cases where they are closely matched. This addresses our first research question: we see that the transferred version is at least as good as the version from scratch in terms of rewards. However, it still begs the question: could the transferred agent performance be predicted with a simple predictor derived from the input (the Atari RAM)?

There are different ways to provide a notion of “complexity” of the RAM usage. For our study, we focus on the temporal variation in the RAM data, as we feel that temporal information best captures a game in progress. Variation indicates the change in each byte of the RAM, which can serve as an indicator of the RAM observation’s complexity. More frequent changes in each byte suggest that specific bytes are more dynamic, reacting to different actions. Additionally, the emphasis is on temporal changes, as substantial changes in a byte value over time may be misleading when considered on a global scale. To mitigate this, RAM Complexity is calculated by analyzing specific time windows.

A dataset of 50,000 RAM instances is gathered by a random agent playing every two-player game. In each of these datasets, a difference is calculated between the actual data in each byte and average of the values that are neighbors of the data using a kernel of size 11 across the time dimension. Each difference value is squared. The average of these squared differences is considered the final value for a specific byte in the RAM values. We repeat this process for every byte in the RAM dataset and obtain values that show the temporal variation for each byte separately. We cap these values at a maximum of 3000 to reduce chances of outliers. As an Atari RAM has 128 bytes, this provides 128 separate values.

We present a method of visualizing the magnitude of RAM data variations in an Atari game environment. Using the 128 values gained from the previous step we can produce a heatmap for a game. We give this heatmap a dimension of 16 by 8 for better visualization. Figure 2 shows the resulting heatmaps. This visualization allows us to intuitively observe how differently each game utilized its RAM space - for some games, activities are concentrated in a few bytes, while in other games, activities are spread out in the RAM.

We present a measure for RAM complexity and analyze the Atari RAM of each game by this measure. From these 128 byte variation values, we can calculate one number that we define as our ”RAM Complexity”, by taking the average of the 128 values. The second column of Table 4 shows our defined RAM Complexity for each of the ten game environments.

The last column of Table 4 shows a normalized difference of means based on the rewards of the last 100 episodes in each game. As the scales of the average total rewards vary by environment, a direct comparison across environments is not possible. Normalization involves rescaling the data to a common scale without distorting differences in the ranges of values, thus allowing for a comparison between the environments. To calculate the normalized difference of means, we first use a 90% winsorization to remove extreme outliers in rewards. Then the rewards of the last 100 episodes in each environment are scaled to a [0, 1] range by performing a Min-Max normalization (using the minimum and maximum reward per each environment). Lastly, the difference between the mean of the scaled last 100 episodes rewards in transferred and non-transferred versions is calculated.

We observe that generally there is a trend that as RAM complexity goes down, the advantage of the transferred agent goes down as well, with Surround being an obvious outlier. The Pearson Correlation Coefficient between the two sets of values is 0.44, indicating a weak positive relationship. If the outlier Surround is removed, the Pearson Correlation Coefficient becomes a stronger 0.71. This suggests that a higher RAM complexity (based on our measure of temporal change) could be connected to the transferred agent performing better in this environment compared to agents trained from scratch, and that the benefits of transferred knowledge become less relevant in simpler environments. This addresses our second research question: in the absence of other information, we could use the RAM complexity as a weak predictor of the effectiveness of transfer. To our knowledge, we are the first to conduct this type of analysis.