Adaptable Automation with Modular Deep Reinforcement Learning and Policy Transfer

Advances in machine learning, robotics, and computing technologies are giving rise to a new generation of “intelligent” automation technologies that can transform various industries such as manufacturing, healthcare, defense, transportation, and agriculture, among others. Automation and robotics are a central building-block of the above industries—from processing and part transfer robots in factories to assistive and patient monitoring robots in hospitals, unmanned aerial drones/vehicles in agricultural/defense fields, and mobile robots in warehouses and fulfillment centers [1, 2, 3, 4, 5, 6]. The need for “intelligence” in such automation systems stems from the fact that most robotic operations in industry are currently limited to rote and repetitive tasks performed within structured environments. This leaves an entire swath of more complex tasks with high degrees of uncertainty and dynamic environments [7] difficult or even impossible to automate. Examples include maintenance and material handling for producing the desired product in manufacturing systems [8], robot surgeries and pharmacy automation in healthcare systems [9], safe working environments in disaster management for deep-sea operation, and nuclear energy [10], fruit picking, crop sensing, and selective weeding in agriculture systems [11] .

A fundamental question concerning the notion of intelligent automation in this context then becomes: How can we enable adaptable industrial automation systems that can analyze and act upon their perceived environment rather than merely executing a set of predefined programs? Adaptability is among the key characteristics of industrial automation systems in response to unpredictable changes or disruptions in the process [12]. Recent developments in deep learning techniques along with the emergence of collaborative robots (cobots) as flexible and adaptive automation “tools” [13] have enabled the preliminaries for incorporating intelligence and learning capabilities into the current fixed, repetitive, task-oriented industrial manipulators [14]. The challenge, however, is to design efficient control algorithms and architectures that enable the robots to modify their behavior to cope with uncertain situations and automatically improve their performance over time, with minimal need for reprogramming or manual tuning of the robot’s behavior.

Reinforcement Learning (RL) has achieved tremendous success in recent years in enabling intelligent control of robotic manipulators in simulated environments or physical lab settings [15, 16]. RL allows agents to learn behaviors through interaction with their environment and generalize the learned policies to new unseen variations of the same task without changing the model itself [17]. Yet, current studies are predominantly focused on simplified tasks that cannot be scaled up to challenging, real-life problems [18, 19]. This is caused by a number of major limitations of current RL algorithms for continuous control which we refers some of them below:

• 1.

  Task specialization. Deep RL algorithms often require reprogramming and relearning on new tasks from scratch. This limitation is due to the lack of efficient mechanisms for transfer of learned policies between tasks. For instance, a cobot can only move objects of similar shape and size in a defined environment or assemble a particular set of chips on a board. Task specialization is one of the critical gaps in RL for continuous control, which has been extensively investigated by many researchers in recent years [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30].

• 2.

  Sample inefficiency. Deep RL algorithms often require massive training datasets [24, 27], even for learning tasks that are relatively simpler than complex, real-life tasks; e.g., playing Atari games [31], Go [32], and poker [33]. However, there is still the problem of handling multi-tasks systems. The other critical challenge for the RL algorithms is the problem of stability - especially in high-dimensional continuous action spaces [34] - and being sensitive to hyperparameter settings [35].

• 3.

  Stability. Hyperparameters must be set carefully for different problem settings to obtain good results. However, most deep RL algorithms, such as Deep Deterministic Policy Gradient (DDPG) [36] and Twin Delayed Deep Deterministic Policy Gradient (TD3PG) [37] are too sensitive to hyperparameter tuning [21]. In those algorithms, a minor variation in hyperparameters can lead to completely different performance and results. Haarnoja et al. [38] propose an off-policy maximum entropy actor-critic-based method, Soft Actor-Critic (SAC), that addresses both sample efficiency and stability issues. Nevertheless, the problem of balancing between exploitation and exploration still remains to be addressed.

• 4.

  Inductive bias. Training an appropriate model requires training with a large number of samples. Meta RL tackles this limitation by training the agent on a series of interrelated tasks drawn from a given distribution or a prior [21, 39, 40, 27, 41] to leverage the agent’s inductive bias for learning new tasks rapidly or adapting to new environments.

Motivated by the aforementioned limitations, this article aims at building and testing a new deep RL framework, based on the actor-critic[38], to enable a robot to efficiently adapt to new variations of its previously learned tasks by sharing parameters across tasks with both parametric and non-parametric variations. The proposed framework is built upon the notions of task modularization [42] and Transfer Learning (TL) [43], and the idea of training a general neural network at both module and task levels. The underlying idea is that transferring the learned knowledge across different interrelated tasks can potentially alleviate the problems of task specialization through multi-task learning [44, 45, 46] and sample efficiency through parameter sharing between tasks and/or task modules [47]. The proposed framework contributes the following features to realize the aforementioned goal of the study:

• 1.

  Task modularization. In complex industrial applications of robots, adaptability can be achieved by modularizing the learning tasks, learning optimal policies for each individual module, and then transferring learned policies across modules of different interrelated tasks with both parametric and non-parametric variations. Therefore, we build on the notion of task modularity for faster learning of a series of manipulation tasks. Using the reward function of each task, smaller modules of the task can be obtained. These modules are also used in order to enhance the training of the task by training their modules and feeding them to the task actor in proposed actor-critic architecture.

• 2.

  Parameter sharing. In order to increase learning speed and decrease memory usage, we propose the idea of hyper-networks (Hyper-Actor) to automatically transfer the knowledge between the modules of the task and between a sequence of interrelated tasks during training with modifying SAC algorithms.

• 3.

  Multi-task learning. We generalize the idea of our modularized training to the multi-task level. Hence, two types of hyper-networks trained on all tasks and modules simultaneously. This forms the meta-learning structure of our actor-critic design architecture by obtaining a multi-task modularized trained actor-network, that can be used for testing a completely new task.

• 4.

  Exploration-exploitation trade-off. We define and employ a new replay buffer in addition to the existing one to enable the SAC algorithm to achieve a stable trade-off between training the neural networks and memory requirement, and faster training.

This article is to propose a deep neural network architecture for transferring knowledge to test a new task from different trained tasks with employing based methods on one of the recent robotics benchmark named Meta-World [23]. This benchmark contains different robotic manipulation tasks for a single task and multi-task learning. However, the critical issue is how to adapt the state space and reward function of tasks defined in the Meat-World with the proposed algorithms because Meta-World, by default, has a sparse reward function for each task manipulation based on the distance.

Section 2 provides an overview of background and related work in the areas of transfer leaning, actor-critic-based RL, and meta learning. Section 3 presents the theoretical and modeling preliminaries. Section 4 discusses the proposed framework in detail. Section 5 presents the experimental results and analyses. Section 6 provides conclusions and a summary of limitations and directions for future research.

RL algorithms attempt to optimize the behavior of an agent to solve a given task. The agent interacts with the environment by taking an action. Any phenomenon that is beyond the control of the agent to manipulate is considered as a part of the environment [17]. The limited scalability of RL algorithms to problems with large state and action spaces, due to the so-called “curse of dimensionality”, has hindered the implementation of RL algorithms to complex, industrial problems [48]. In recent years, deep RL achieved incredible success by reaching super-human performance in learning how to play Atari games [31]. The results lead to the development of more challenging tasks by DeepMind such as AlphaGo [49] in 2016 and AlphaStar [50] in 2019. In 2020, García and Shafie [51] proposed a safe RL algorithm to teach humanoid robots how to walk faster.

An RL problem can be modeled as a Markov Decision Process (MDP), which is a stochastic process that satisfies the Markov property [17] defined by a tuple $\langle\mathcal{S},\mathcal{A},p,r,\gamma\rangle$, where $\mathcal{S}$ is state space, $\mathcal{A}$ is action space, $p:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{P}_{\mathbf{s}}$ is state transition distribution, $r$ is reward function, and $\gamma$ is a discount factor. RL algorithms can be value-based (i.e., learning from off-policy experiences), policy-based (i.e., directly optimizing actions), or both. Value-based RL is guided by a $Q$-value function that indicates the goodness of taking an action given a specific state, guiding the agent in taking actions that maximize the $Q$-value [89]. In policy-based RL, however, the policy is updated directly, based on the gradient calculated from the reward parameters [90], which is useful for problems with continuous action space.

The proposed Hyper-Actor Soft-Actor Critic (HASAC) framework is presented in this section. HASAC aims at improving the adaptability of the RL agent (e.g., a robot) to a wide range of tasks through task modularization, parameter-sharing, multi-task learning, and enhanced exploration-exploitation trade-off. The proposed framework builds on SAC as the state-of-the-art continuous control RL algorithm, and also utilizes SAC as a baseline for evaluating HASAC. SAC operates by updating three sets of parameters as follows: (1) the state value function network, $\psi$, (2) the soft $Q$-function network parameter, $\theta$, and (3) the policy network parameter, $\phi$. As a result, the tractable policy $\pi_{\phi}(\mathbf{a}_{t}|\mathbf{s}_{t})$ will also be updated. In the actor, the policy parameter, $\pi_{\phi}$, is learned directly by minimizing the expected Kullback-Leibler (KL) divergence:

where $\mathcal{D}$ is the distribution of previously sampled states and actions, called replay buffer.

To efficiently balance the exploration and exploitation in the proposed HASAC, another replay buffer named elite reply buffer ($\mathcal{D}_{e}$), is defined to collect states and actions that pass the reward function threshold ($\zeta$). The proposed mechanism leads to faster learning by improving the exploration and exploitation trade-off (for more detail, see the mechanism of replay buffers in Algorithm 1).

To tackle complex tasks with continuous and high-dimensional action spaces, such as industrial collaborative robotics operations, HASAC aims at enabling the RL agent to automatically learn new tasks and transfer optimal policies across different tasks. These characteristics can be achieved by employing the aggregation concept defined based on the notions of modularity [57] and transfer learning [55]. This can be achieved either by sharing parameters across modules of a task or through multi-task learning. As the task becomes complex, current deep RL methods require to proportionally increase the training sample size to ensure desirable performance. However, question remains on how to train a single network that allows the sharing of some features across different robotic manipulation tasks, especially in real-world environments.

Yu et al. [23] present a new robotic benchmark, Meta-World, to investigate the ideas of multi-task RL and meta RL on 50 different robotic manipulation tasks. They findings reveal that training agents on diverse robotic tasks affects the final performance compared to training them separately. We therefore speculate that defining a new general network for parameter sharing across tasks is expected to be a beneficial mechanism, allowing to update policy parameters while not affecting the optimization process of the task modules. This idea builds on the notion of transfer learning [43], which aims at transferring the knowledge from learning an old task to a new task, to help the agent master new task faster. It is also applicable to learning at the task level when a task can be decomposed into several reusable modules [57].

The underlying idea of HASAC is based on the relationship between the actor and the critic networks (Figure 2). For each task, the critic updates the action-value function by computing the value function in an iterative fashion. Therefore, each task has a specific critic network. Since the output of the critic depends on the task, a separate critic is required for each task [95]. In addition, each task module has its own critic, which helps increase the performance of the agent in learning the task faster.

The actor also follows the standard actor-critic procedure, which involves learning the $Q$-value function for each state-action and updating the policy accordingly. Doing so, training a new single actor-network on all tasks, along with the current networks in the SAC, allows sharing the mutual features across different tasks. This is the idea which builds on the notion of transfer learning. Same as the critic network, the idea also can be applied on the module level. We refer to this new general actor as the “hyper-actor”. In Algorithm 1, $\mathcal{T}$ denotes the hyper-actor network that is trained on all interrelated tasks in multi-task learning and $\mathcal{M}_{i}$ denotes the hyper-actor network for all modules of the task $i$. As mentioned before, each module or task also has its specific actor-network from the baseline structure. Because, the trained actor on the baseline algorithm may receive different updates for the parameters. Therefore, two sets of parameters are introduced: $\mathbf{\varrho}^{\mathcal{M}_{i}}$ (Hyper-Actor-Module parameters) and $\mathbf{\varrho}^{\mathcal{T}}$ (Hyper-Actor-Task parameters). For more detail regarding the algorithm, see Algorithm 1. See also the Nomenclature for notations.

Consider, a an example, the “open window” task of Meta-World [23] which contains “reach” and “pull lever” modules. At the module level, when the training phase for module “reach” finishes, the actor and critic parameters of the “reach” module will be reset. However, when the training process of the Hyper-Actor-Module network finishes, the memory will not be reset, and Hyper-Actor-Module parameters, $\mathbf{\varrho}^{\mathcal{M}_{reach}}$, will be transferred to the second module, “pull lever”. Thus, the Hyper-Actor-Module is trained with the pull module’s loss function, and then it will be trained on other modules. Since each module of a task has a specific loss function for the critic, it is an efficient way for only training a hyper-actor network in the test phase. Now suppose the “open window” task to be the first task in the task sequence of a multi-task learning problem. The actor of the “open window” is updated using the “open window” critic loss function, and then the value function of the “open window” critic will be used to update the Hyper-Actor-Module parameters ($\mathbf{\varrho}^{\mathcal{M}_{open-drawer}}$). Then, the updated $\mathbf{\varrho}^{\mathcal{M}_{open-drawer}}$ will be used for updating the loss function of the “close window” critic. Next, at the task level, the updated “close window” critic will be used to update the actor parameters of the Hyper-Actor-Task ($\mathbf{\varrho}^{\mathcal{T}}$) (see Figure 2).

The main difference between the Hyper-Actor-Module and Hyper-Actor-Task networks and the other networks is in the backpropagation process. The Hyper-Actor-Module and Hyper-Actor-Task networks are backpropagated cumulatively. In multi-task learning, this implies that those networks are not freed within their training loop. Therefore, backpropagation will be continued for the multi-task network until the end of training tasks in the sequence of multi-task learning, which is the final Meta-Action design for our testing experiment. Likewise, at the task level, the hyper-network is not freed within a task loop, and it will only be freed and saved once all modules of a task are trained. Hence, the hyper-network with parameter set $\mathbf{\varrho}^{\mathcal{M}_{i}}$ is the final action design for a particular task $i$.

The backpropagation process of HASAC is similar to the process used for training the actor and critic networks in the SAC algorithm. Hence, two sets of networks, $\mathcal{M}_{i}$ and $\mathcal{T}$, are trained to enable a comprehensive learning framework which can be used for generalizing to new tasks at test time (see Algorithm 1). Further, for the tasks that can be divided into different modules, the actor and critic networks parameters will be initialized with the policy parameters of the most related module from the past ($\phi\leftarrow\varrho^{\mathcal{M}_{i}}$).By applying the proposed idea based on the relationship between the actor and critic networks as well as the notions of task modularization and transfer learning, HASAC can achieve significant improvements in both sample efficiency and final performance over previous state-of-the-art approaches, as demonstrated next.

This section evaluates the performance of the proposed HASAC framework on a set of robotic manipulation tasks from the Meta-World benchmark [23]. The Meta-World benchmark contains 50 different robot arm control tasks enabled by the MuJoCo physics simulator [88]. All of the Meta-World tasks involve moving a simulated Sawyer robot arm to interact with objects by moving them from an initial state to a goal state. The action space consists of four degrees of freedom (three translational, one rotational) for the end-effector of the robot arm with observation space of 3D Cartesian positions of the object, the robot end-effector, and the goal position [96]. One of the essential steps in designing architecture is to properly modularize the tasks. Without loss of generality, our experiments are focused on four tasks “open window”, “close window”, “open drawer”, and “close drawer”. Although these tasks are symbolic and relatively simple, we believe analysis using the HASAC framework can be informative for solving realistic and more complex industrial problems. To modularize the aforementioned tasks, “reach” and “pull lever” are selected as the respective task modules (see Figure 3).

This article addresses one of the key research questions associated with the adaptability of industrial automation systems to the increasing variability of tasks assigned to automated equipment such as collaborative robots. A novel modularized actor-critic based deep RL framework is developed and tested on multiple simulated robotic manipulation tasks. The main goal is to improve performance and sample efficiency for training robotic manipulation tasks and multi-task learning through task modularization and transfer of policy parameters using “hyper-actor” networks at both module level and task level. Experiments indicated that incorporating task modularization in actor-critic based RL methods is an effective mechanism to learn complex tasks faster and more efficiently. Besides, the experimental results concurred that the effectiveness of this framework increases with increasing task complexity, and also it enables the agent to learn simple tasks faster.

The experimental results also show that the proposed HASAC framework achieves better success rate and converges faster with more stability than the baseline SAC algorithm by a relatively large margin. Moreover, results show that the trained Hyper-Actor-Module and Hyper-Actor-Task networks improve the success rate of the final task across the sequence of multi-task learning. Indeed, the hyper-network architecture allows efficient sharing and reuse of network elements across tasks, which creates future opportunities for developing more robust and efficient meta RL algorithms.

A major limitation of this work is the lack of experimentation on more complex and realistic tasks, both in simulated environments and on a real robot in a laboratory setting. This should be one of the foremost future research directions in this area. Future research should also investigate more meaningful industrial use-cases where the use of RL and AI in general is preferred to traditional approaches for programming automation equipment. One important example from industrial, collaborative robotic applications is human-robot collaboration where the presence of the human in the loop both justifies the incorporation of AI in robotic control and facilitates the task modularization and policy transfer processes. Future research should also investigate the implications of the proposed framework for a variety of industrial sectors including manufacturing, healthcare, defense, and agriculture, among others.