MultiRoboLearn: An open-source Framework for Multi-robot Deep Reinforcement Learning

After the emergence of reinforcement learning in robotics, OpenAI Gym [8] has become one of the standard interfaces for robot learning. Then It has been further combined with Mujoco [4] to provide few standardized training environments for the simulation of single robots. However, it does not have direct connections with real-world robotics applications. Also, the same problems have also been presented in the Roboschool [5]. In addition, as a physical simulation engine for games, robotics and RL, Pybullet [9] provides a set of benchmarks for comparing the performance of different algorithms. However, it can only be used in simulations with a single robot. OpenAI-ros [10] inherits the interface of Gym and Gazebo, but also has the same problem as the Pybullet. Unity Robotics [11] provides a realistic platform where researchers can combine AI technologies to make robots complete complex tasks, however, it has not considered multi-robot version without providing any multi-robot APIs. Although Isaac Gym [12] has high training performance comparing with other toolkits training on CPU, it has the same issues as aforementioned toolkits which are lack of utilities linking multi-agent reinforcement learning methods to multi-robot applications.

The DeepMind Control Suite [13] provides RL environments in the simulation without physical support. It is well known that gym-gazebo2 [14] presents an upgraded, real-world application version of gym-Gazebo, this toolkit uses a new ROS2-based software structure, but it only focuses on specific industrial robot arms, cannot be extended for other robots, and does not provide a MADRL algorithmic environment. Recently, Deepbots[15] has designed an environment where the community can easily develop RL methods in Webots [16], however, they also have the above mentioned problems. Additionally, they do not consider MADRL algorithm environments or lacks the ability to be applied on real-world robots. PyRoboLearn [17] integrates RL and imitation learning environments. However, it exclusively focuses on the simulation without consideration on the application of real robots. The toolkit, robo-gym [18] introduces a freely available framework allowing the use of DRL in the simulations and real-world robots. The framework is tailored for the single mobile robot and manipulator as it cannot be extend into multi-robot systems in both simulation and the real world.

It is important to note that there is a growing body of work that focuses on MADRL from a theoretical perspective without considering how to apply these algorithms to real-world robots. Unlike the learning-based single-agent framework, to the best of our knowledge, there are no general algorithmic environments for theoretical research on MADRL or any toolkits to implement MADRL for multi-robot systems. Specifically, PettingZoo [19] aims to provide a python library for multi-agent reinforcement learning akin to multi-agent version of Gym, however, this toolkit only integrates some of classic and highly-simplified environments including Atari, Butterfly, etc. In the game domain, Unity ML-Agents toolkit [20] is a popular framework to design intelligent characters in the game scenarios by providing extensive, scalable, sufficient python APIs where researchers can easily train agents using reinforcement learning, imitation learning, curriculum learning, and any other methods according to the need of multiple tasks in the game. However it has the same issues as PettingZoo. Although OpenAI has recently released some MADRL environments, such as the multi-agent particle environment [21], hide and seek [22] and Arena [23], these environments cannot support the MADRL study of multi-robot systems in the real world.

We introduce our framework module by module, starting from master manager, then continuing with simulated environments, real-world environments and algorithm libraries. The relationship of these modules is illustrated in Fig. 2.

1. 1.

   Master Manager: This framework primarily uses a central control structure to drive simulated and real-world systems. The use of ROS communication structure allows for easy establishment of IP networks. This control structure facilitates users to stifle the training or testing process of simulated and real-world robots according to their experimental needs. On the other hand, this structure makes it easy to control both simulated and real-world robots with trained models.

   The mManager has two functions. The first is to connect to a simulation of multi-robot systems. This makes the studying of various MADRL algorithms possible. There are two steps regarding this function. On the one hand, the task scenarios and multi-robot models are loaded through Gazebo. In this step, the mManager sets up namespaces for different robots in order to control all robots efficiently in the same simulated environment. In addition, mManager automatically sets up various services to facilitate sending and receiving of system data. On the other hand, the mManager provides an algorithmic environment by building a virtual environment using Python. Researchers can design different algorithms in this container without considering the compatibility of the Gazebo environment and the algorithmic environment. The second function is to connect and configure the physical platform. When using a multi-robot system in the real world, there are three steps to perform. First, mManager automatically establishes a local network by setting static IP addresses to add different robots and migrate them to various scenarios without having to consider network configurations. Secondly, namespace, rostopic, and rosservice are configured for each real robot autonomously. The third step in the training model, obtained from the simulation, provides a corresponding control policy for each robot to allow the real-world multi-robot systems to perform the complex tasks.

2. 2.

   Algorithm Environment: This is one of the most important components in our framework, which inherits the interface of OpenAI Gym. Our implementation can be easily combined with various algorithm libraries, helping users to focus on designing the algorithm itself, without considering difficulties of connecting algorithms and environments. The main function of this component is to give some definitions and application programming interface (API). It defines collections of joint states, actions, and other necessary parameters. Also, it defines the interface to connect Gazebo to implement Markov Decision Process (MDP).

   The simulated environment provides two API modules, namely multi-robot module and task scenarios module. Task scenarios module uses Gazebo as a simulator, facilitating task configuration using the model library and worlds developed by the ROS community. For robot developers, it is easy to develop different task scenarios by designing different models and worlds. For algorithm developers who are unfamiliar with Gazebo, they also can easily combine different worlds and models to implement desired tasks. Our implemented mManager already provides a number of task types, including tasks for homogeneous and heterogeneous robots.

3. 3.

   Real-Robot Environment: The purpose of this environment is to provide real-world multi-robot systems to test, and apply the designed algorithms. In order to achieve modularity, flexibility and simplicity, we built a real multi-robot system using real-world SPARK robot, and we designed a local network system to support their communications. Due to the loose coupling of ROS communication structure, this network can be easily connected to other robots that support ROS interface, achieving high extensibility and ease of use.

   In this environment, an emitter replay buffer and a receiver reply buffer are implemented to discretize real world signals. This is mainly due to that in the real-world, time passes by in a continuous manner, and these buffers are needed to serve as a bridging mechanism. We designed the transmitter replay buffer to send the actions of all robots in a time interval. It is essentially a list of buffers equipped on each robot to store the commands in a queue. This list of buffers will then send the corresponding command to each robot at a fixed frequency during an operation (See Emitter Reply Buffer in Fig. 3 for reference).

4. 4.

   Algorithm Library: To achieve modularity and generality, nManager provides a series of algorithm interfaces to support different Python-based DL libraries, such as Tensorflow, Pytorch, keras, Numpy, Pandas, etc. Also, to help MADRL researchers apply algorithms in the multi-robot system with minor modifications, we build pipelines that supports currently popular RL libraries, such as Baselines [24], Spinning up [25] and rllab [26]. For generality, the framework can support both discrete action space and continuous action space environments.

This subsection describes the overall process of our framework, including simulation and real-world parts, as shown in Fig. 3. In the simulation, our framework first receives states of the previous step, and then subsequently sends them to the algorithms. After that, the algorithms select some actions and then send them to the corresponding robot in real world applications. After actions are executed, the next states can be obtained from the environment. In real world, the algorithm selects the actions and passes them to the emitter replay buffer in order, then passes the actions to the corresponding robots in the RoboEnv. The RoboEnv collects the state of each robot at the same time by using receiver replay buffer and packages all states together into a buffer. All the process mentioned above is called a cycle.

The difference between our framework and single-robot frameworks is that we have to consider the current state of all robots in a cycle time. For example, we have to collect the current states of all robots, which is called a joint state of multi-robot system. According to robots’ relationships (e.g. cooperative, competitive or semi-cooperative, semi-competitive), we will also have to set a joint rewards structure for all the robots. In real-world multi-robot systems, concurrency is a key to the smooth execution of the command, data collection, and collaboration. Commands will be executed in a continuous period of time, which is allowed for the single robot. However, for a multi-robot system, each robot executes the corresponding command sequentially, which leads to inconsistencies and can cause communication delays or blockages. This framework builds a list of emitter and receiver replay buffers by setting an execution time for the commands. In operation, our framework sends the stored instructions to the corresponding transmitter buffer, which then sent them to the real robot. Each robot can send and collect data at the same time, enabling the multi-robot system to complete its tasks smoothly.

1. 1.

   Adding Robots in Simulation & Real World: In the simulation, the new robot model can be easily integrated into the simulation through URDF. The multi-robot system’s network can be configured with only minor code changes by using the mManager. Regarding the real world scenario, robots can be added into multi-robot systems by creating IP in the local network and creating namespaces for new robots. Also, each robot requires a specific transmitter and receiver replay buffer. However, these hardware settings can be added by making a small modification to the original code.

2. 2.

   Using Algorithms: New Reinforcement learning algorithms, including single-agent and multi-agent ones can be easily developed and utilized due to our framework’s integration with most of the deep reinforcement learning libraries. For MADRL, our framework supports various training methods, such as independent network training, centralized training and decentralized execution, etc.

1. 1.

   Mobile Navigation of SPARKs in obstacle-free environment: In this scenario, the goal is that three SPARKs collaborate to reach each goal position while avoiding collisions during navigation, as shown in Fig. 4 (a). All three SPARKs have the same configuration in the dynamic model and the same parameter settings in the hardware. On the one hand, in order to detect static and dynamic obstacles, SPARK is equipped with a laser sensor that scans 360 degrees in a two-dimensional plane. In addition, to determine the current position of each robot, SPARK is equipped with an odometer that provides the local coordinates of each robot with an average error of 0.03 m over a distance of 1 m. The whole scene is placed in a room of $4\times 6$ square meters. The joint observations are designed as a list container that collects each robot’s observation with 17 values. The first two values from odometers are the robot’s position in the global coordinates of the scenario. Note that all the robots are positioned in the same global coordinates over episodes. The remaining 15 values are the measured distances of the laser scanner evenly placed around each robot. The sparse processing of the laser scans may lead to the inability to perceive other dynamic robots. We deal with this problem in two ways. The first way is that in the simulation, we enlarge the collision model volume of each robot’s laser scanner. The other way is to preprocess the received data from the laser scanner of real robots. To ensure the consistency between simulation and real-world laser data, we add some Gaussian noise to the laser scanner in the simulation. Also, to reduce the complexity of learning, these data from both simulation and hardware are downsampled.

   In the case of discrete action space, the action is a series of discrete numbers, essentially the speed controller of the mobile robot. In the case of continuous action space, the action consists of two continuous values, namely linear velocity and angular velocity.

   The difference between RL and MADRL in our case is the reward function. It should be noted that MADRL needs to consider agents’ relationships, such as cooperation, competition, and half-cooperation-half-competition. In these scenarios, we consider a scenario where robots need to collaborate to perform tasks. The base reward received by the robot at each step is propositional to the distance the robot moved during last step. For all robots that reaches their goal areas simultaneously, they will all receive a large positive reward. If the robot reaches its goal area by itself, it will receive a small positive reward. Moreover, in the case of the collision, they will receive a small negative reward.

2. 2.

   Mobile Navigation of SPARKs in obstacle environment: In this scenario task, the difference, comparing with the previous scenario, is that there are some static obstacles. In robotics, navigation and obstacle avoidance are common research topics in the community, so the purpose of this task scenario is to demonstrate that this framework is applicable to a wide range of task scenarios and can solve common problems in the field of robotics.

In this subsection, we provide a series of cases showing a proof of concept by using two kinds of algorithms in two cases to demonstrate the capability of this framework. In the case of continuous action space, SAC is chosen by improving SAC algorithm [27]. In the case of discrete action space, multi-DQN same as [28] is well employed.

The SAC algorithm, contains prioritized experience replay, double DQN network to improve the original algorithm’s performance. Hyperparameters are chosen according to the benchmark for [27] with some minor modifications. The multi-DQN algorithm adopts Dueling-DQN. This algorithm includes the proposed prioritized experience replay, and sum tree [28] to improve performance. Hyperparameters are mainly derived from [28] with minor modification.

1. 1.

   Computer Setup for Training and Test: As seen in Fig. 1, mManager (essentially a computer with 20 CPU cores) provides the functionality of switching between specific modes. When switching to the simulation mode, mManager can be utilized in the learning process. When switching to the real-world mode, mManager serves to provide control policies for all robots during test process.

2. 2.

   Real-world Setup: In the real-world experiments, we construct two task scenarios in the laboratory, which resembles the simulated scenarios to reduce the gap between simulation and the real world, improving the deployment accuracy of trained model in the real-world robots. In Fig. 5, standing barriers are utilized as walls and restricted areas. In addition, two boxes are used as obstacles. The whole scene maps to $4\times 6$ square meters. The trained model, receiver and emitter replay buffer are running on the mManager and connected to SPARKs’ local network using Wi-Fi. To ensure consistency of all robots, commands are sent in 20 ms and executed in 100 ms.

In this subsection, our framework is evaluated using two types of algorithms in two scenarios. During the experiment, three agents are first trained in the same simulated environment. When the training process is finished, the trained model, namely control policies, is tested in both simulated and real-world environments. We show that the trained model can be directly deployed in real-world scenarios without any modification or any further training in real-world robots. See the accompanying video for the demonstration of the two scenarios in both simulation and the real world.

1. 1.

   Results for SAC in two Scenarios: Three agents are all trained in an obstacle-free environment and obstacle environment by using a SAC algorithm with the same hyperparameters. In the obstacle-free scenario, three agents are trained to perform the task of navigation after 2000 episodes in the simulation. However, after 600 episodes, reward curves starts to converge (see Fig. 6 (a) for reference). The convergence criterion for multi-robot systems is different compared to the single-robot case. In our experiments, for a single robot, the model is considered to have converged when the success rate exceeds 80% for 20 consecutive episodes. However, a multi-robot system is considered converged only when all the robots in the multi-robot system converge. In an obstacle-free scenario, the success rate remained between 80% and 100% after more than 600 episodes of training. In the other scenario with obstacles, we noticed that it was more difficult to reach convergence than in the obstacle-free scenario. In fact, it is after 800 episodes that the reward curve starts to converge and the final success rate stayed between 80% and 100%.

   To have a final evaluation, the trained models are tested in both simulated for 200 episodes as well as real-world environments for 50 episodes. In the obstacle-free scenario, the trained model had 91.5% success rate in the simulation and had 82.0% success rate in the real world. In the second scenario, the trained model had 92.0% success rate in the simulation and had 75.0% success rate. Although the success rate of a real-world experiment is lower than that of a simulation experiment, it is well demonstrated that this framework can be directly applied in a multi-robot system.

2. 2.

   Results for Multi-DQN in two Scenarios: The multi-DQN algorithm is also trained in the two task scenarios with the same task-related hyperparameters. All the agents are trained over 2000 episodes in the simulation. In the obstacle-free scenario, the training converged after 900 episodes with a success rate between 80 % and 100% (see Fig. 6 (c)). In the other scenario with obstacles, after training over 1000 episodes, all the agents can reach goal areas with a success rate between 80% and 100% (see Fig. 6 (d)). A final evaluation of 200 episodes is also performed in simulation and the evaluation of 50 episodes is performed in the real-world scenarios. In the obstacle-free environment, the trained model reached 82% success rate in the simulation and had 52.0% success rate in the real world. In the second environment, the trained model had 76.5% success rate in the simulation and had 40.0% success rate.