Reinforcement Learning with Adaptive Curriculum Dynamics Randomization
for Fault-Tolerant Robot Control

We focus on the standard reinforcement learning problem [22], in which an agent interacts with its environment according to a policy to maximize a reward. The state space and action space are expressed as $\mathbf{S}$ and $\mathbf{A}$, respectively. For state $s_{t}\in\mathbf{S}$ and action $a_{t}\in\mathbf{A}$ at timestep $t$, the reward function $r:\mathbf{S}\times\mathbf{A}\rightarrow\mathbb{R}$ provides a real number that represents the desirability of performing a certain action in a certain state. The goal of the agent is to maximize the multistep return $R_{t}=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r(s_{t^{\prime}},a_{t^{\prime}})$ after $T$ steps, where $\gamma\in[0,1]$ is the discount coefficient, which indicates the importance of the future reward relative to the current reward. The agent decides its action based on the policy $\pi_{\theta}:\mathbf{S}\times\mathbf{A}\rightarrow[0,1]$. The policy is usually modeled using a parameterized function with respect to $\theta$. Thus, the objective of learning is to identify the optimal $\theta^{*}$ as

where $J(\theta)$ is the expected return defined as

where

represents the joint probability distribution on the series of states and actions $\tau=(s_{0},a_{0},s_{1},\dots,a_{T-1},s_{T})$ under policy $\pi_{\theta}$.

The actuator failure is represented through the variation in the torque characteristics of the joint actuators. By denoting $o_{t}\in\mathbb{R}$ as the actuator output at timestep $t$, we simply represent the output value of the failure state, $o_{t}^{\prime}$, as follows:

where $k\in[0,1]$ is a failure coefficient. Failure coefficients of $k=1.0$ and $k\not=1.0$ indicate that the actuator is in the normal or failed state, respectively. A larger deviation of $k$ from $1.0$ corresponds to a larger degree of robot failure, and $k=0.0$ means that the actuator cannot be controlled. Figure 1 illustrates an example of failure of an 8-DOF robot that had learned to walk. The top row in Fig. 1 shows a normal robot with $k=1.0$, and the bottom row in Fig. 1 shows a failed robot (red leg damaged) with $k=0.0$. Because the conventional reinforcement learning does not take into account environmental changes owing to robot failures, the robot falls in the middle of the task and cannot continue walking, as shown in the bottom row in Fig. 1.

To implement fault-tolerant robot control the ACDR algorithm employs dynamics randomization by randomly sampling the failure coefficient $k$ and curriculum learning by adaptively changing the interval of $k$. The process flow of the ACDR algorithm is presented as Algorithm 1 and can be described as follows.

For dynamics randomization, one leg $l$ of the quadruped robot is randomly chosen as

where $\mathrm{Uni}\{0,1,2,3\}$ is the discrete uniform distribution on the possible leg index set $\{0,1,2,3\}$ of the quadruped robot. The leg $l$ is broken at the beginning of each training episode, and the failure coefficient $k$ is randomly chosen as

where $\mathrm{Uni}(L,U)$ is the continuous uniform distribution on the interval $[L,U]$. According to Eq. (4), the broken actuator of the leg $l$ outputs $o^{\prime}_{l,t}=ko_{l,t}$ in each training episode. As the actuator output changes, the dynamics model $p(s_{t+1}|s_{t},a_{t})$ also changes during training. This dynamics randomization means that a policy can be learned under varying dynamics models instead of being learned for a specific dynamics model. Under dynamics randomization, the expected return is determined by taking the expectation over $k$, as indicated in Eq. (2), as

By maximizing this expected return, the agent can learn a policy to adapt to the dynamics changes owing to robot failures.

For curriculum learning, the interval $[L,U]$ in Eq. (6) is adaptively updated according to the return earned by the robot in training. As mentioned previously, a large interval $[L,U]$ is likely to result in a policy with large variance  [13] or a conservative policy [14]. Thus, a small interval $[L,U]$ is assigned to Algorithm 1 as the initial value. The interval $[L,U]$ is updated as follows: First, a trajectory of states and actions $\tau=(s_{0},a_{0},\dots,s_{T})$ is generated based on a policy $\pi_{\theta}$, where $\theta$ is the policy parameter optimized using a proximal policy optimization (PPO) algorithm [23]. Subsequently, the return $g$ of the trajectory $\tau$ is calculated by accumulating the rewards as

By repeating the parameter optimization and return calculation $m$ times, the average value of the returns $\bar{g}$ can be determined. This value is used to evaluate the performance of the robot and determine whether the interval $[L,U]$ must be updated. If the average $\bar{g}$ exceeds a threshold $g_{th}$, the interval $[L,U]$ is updated, i.e., the robot is trained with the updated failure coefficients.

In this study, we consider two methods to update the interval $[L,U]$. The manner in which the interval is updated during curriculum learning is illustrated in Fig. 2. In the first method, the upper and lower bounds of the interval $[L,U]$ gradually decrease as

given the initial interval $[1.5,1.5]$. In all experiments, we set $\Delta_{U}=\Delta_{V}=0.01$. This decrease rule represents an easy2hard curriculum because smaller values of $L$ and $U$ correspond to a higher walking difficult. For example, when $k=0$, the leg does not move. The second method pertains to a hard2easy curriculum, i.e.,

We set the initial interval as $[0.0,0.0]$. In this state, the robot cannot move the broken leg, and its ability to address this failure is gradually enhanced. Notably, although the easy2hard curriculum is commonly used in curriculum learning, the hard2easy curriculum is more effective than the easy2hard configuration in accomplishing the task. The difference between the two frameworks is discussed in Section IV-D.

In the analysis, we introduce a slightly modified reward function instead of using the commonly applied function described in [24], because the existing reward function [24] is likely to result in a conservative policy in which the robot does not walk but remains in place. The existing reward function [24] can be defined by

where $v_{\mathrm{fwd}}=\frac{x^{\prime}-x}{\Delta t}$ is the walking speed, $u$ is the vector of joint torques, and $f_{\mathrm{impact}}$ is the contact force. The reward function in Eq. (11) yields a reward of $1$ even when the robot falls or remains stationary without walking. Therefore, this reward function is likely to lead to a conservative policy. To ensure active walking of the robot, we modify the reward function as

where

Preliminary experiments demonstrate that the modified reward function in Eq. (12) can ensure more active walking of the robots.

For reinforcement learning, we employ the actor and critic network consisting of two hidden layers of $64$ and $64$ neurons with $\tanh$ activation. All networks use the Adam optimizer [27]. The related hyperparameters are listed in Table I.

The ACDR algorithm is compared to the following three algorithms. Baseline: The baseline is a basic reinforcement learning algorithm for normal robots that implements neither dynamics randomization nor curriculum learning. Uniform Dynamics Randomization (UDR): UDR is a dynamics randomization algorithm that is based on the uniform distribution $\mathrm{Uni}(0.0,1.5)$ and does not implement curriculum learning. Linear Curriculum Dynamics Randomization (LCDR): In the LCDR algorithm, the interval $[L,U]$ is linearly updated. Detail of the LCDR can be found in A. Similar to the ACDR, the LCDR uses the easy2hard and hard2easy curricula. The progress of the two curricula, the legends of which are ”LCDR_e2h” and ”LCDR_h2e”, is shown in Fig. 2.

We evaluate the algorithms in terms of the average reward and average walking distance in two conditions: plain and broken. In the plain, the robots function normally, i.e., the failure coefficient is fixed at $k=1.0$. In the broken, the robots are damaged, and $k$ randomly takes a value in $[0.0,0.5]$. We implement each algorithm on the two conditions, with five random seeds for each framework. The average reward and average walking distance are calculated as the average of 100 trials for each seed. The average walking distance is one that is suitable for evaluating the walking ability of the robot.

The average reward and average walking distance for all algorithms are shown in Fig. 3 and Fig. 4, respectively. In both the plain and broken conditions, UDR is inferior to the Baseline, which does not consider robot failures. This result indicates that the simple UDR cannot adapt to robot failures. The hard2easy curriculum of LCDR, denoted by LCDR_h2e, outperforms the Baseline in terms of the average reward, as shown in Fig. 3. However, LCDR_h2e corresponds to an inferior walking ability compared to that of the Baseline for both plain and broken conditions, as shown in Fig. 4. This result indicates that LCDR_h2e implements a conservative policy that does not promote walking as active as that generated by the Baseline. For both LCDR and ACDR, the hard2easy curriculum outperforms the easy2hard curriculum, as discussed in Section IV-D. Moreover, ACDR_h2e achieves the highest performance in all cases. This finding indicates that ACDR_h2e can avoid the formulation of a conservative policy by providing tasks with difficulty levels that match the progress of the robot.

Figure  5 shows the average reward for each failure coefficient $k$ in the interval $[0,1]$. ACDR_h2e earns the highest average reward for any $k$, and ACDR_h2e achieves the highest performance in any failure state. Moreover, the performance of UDR, LCDR_e2h, and ACDR_e2h, deteriorate as $k$ increases. This result indicates that the robots implementing these strategies cannot learn to walk. Therefore, even when $k$ increases and it is easy to move the leg, the robots fall and do not earn rewards.

The experiment results show that the hard2easy curriculum is effective against robot failures. To evaluate whether the hard2easy curriculum contributes to an enhanced performance, we compare this framework with a non-curriculum algorithm in which the robot is trained with a fixed failure coefficient $k$. The values of $k$ are varied as $\{0.0,0.2,0.4,0.6,0.8,1.2,1.4\}$. For each $k$, Fig. 6 shows the average reward earned by the robot on interval $[0,1]$ of the failure coefficient. ACDR_h2e still achieves the highest performance over the interval $[0,1]$. Therefore, the hard2easy curriculum can enhance the robustness against robot failures.

The hard2easy curriculum gradually decreases the degree of robot failure and makes it easier for the robot to walk. In contrast, the easy2hard curriculum gradually increases the degree of difficulty, and eventually, the leg stops moving. Figures 3 and 4 show that the hard2easy curriculum is more effective than the easy2hard curriculum. We discuss the reason below.

Figure 6 shows that the average reward tends to increase as $k$ increases and deteriorates as $k$ decreases. For example, $k=1.4$ corresponds to the highest average reward over the interval $[0,1]$, whereas $k=0.0$ or $0.2$ yield low rewards. This result suggests that it is preferable to avoid training at $k=0.0$ to allow the robot to walk even under failure. Hence, we can hypothesize that the easy2hard curriculum, in which training is performed at $k=0.0$ at the end of training, deteriorates the robot’s ability to walk. To verify this hypothesis, we perform the same comparison as described in Section IV-B for robots trained on the interval $k\in[0.5,1.5]$; in particular, in this framework, the robots are not trained at $k=0$. Figures 7 and 8 show the average reward and average walking distance, respectively, and Fig. 9 shows the average reward for each $k$ over the interval $[0,1]$. Comparison of these results with those presented in Figs. 3, 4, and 5, demonstrates that by avoiding training in the proximity of $k=0$, the average rewards of LCDR_e2h and ACDR_e2h are increased. In summary, the easy2hard curriculum deteriorates the learned policy owing to the strong degree of robot failure introduced at the end of training, whereas the hard2easy curriculum does not.

Furthermore, the average rewards of UDR, LCDR_h2e, and ACDR_h2e increase, as shown in Figs. 7, 8, and 9. These results indicate that training in the proximity of $k=0$ is not necessary to achieve fault-tolerant robot control. Thus, interestingly, the robot can walk even when one of the legs cannot be effectively moved, without being trained under the condition.