Consolidated Adaptive T-soft Update for Deep Reinforcement Learning

First of all, the basic problem statement of DRL and an actor-critic algorithm, which can handle continuous action space, is natural choice [7] as one of the basic algorithms for robot control. Note that the proposed method can be applied to other algorithms with the target network.

In DRL, an agent interacts with an unknown environment under Markov decision process (MDP) with the current state $s$, the agent’s action $a$, the next state $s^{\prime}$, and the reward from the environment $r$. Specifically, the environment implicitly has its initial randomness $p_{0}(s)$ and its state transition probability $p_{e}(s^{\prime}\mid s,a)$. Since the agent can act on the environment’s state transition through $a$, the goal is to find the optimal policy to reach the desired state. To this end, $a$ is sampled from a state-dependent trainable policy, $\pi(a\mid s;\theta_{\pi})$, with its parameters set $\theta_{\pi}$ (a.k.a. weights and biases of DNNs in DRL). The outcome of the interaction between the agent and the environment can be evaluated as $r=r(s,a,s^{\prime})$.

By repeating the above process, the agent gains the sum of $r$ over the future (so-called return), $R=\sum_{k=0}^{\infty}\gamma^{k}r_{k}$, with $\gamma\in[0,1)$ discount factor. The main purpose of DRL is to maximize $R$ by optimizing $\pi$ (i.e. $\theta_{\pi}$). However, $R$ cannot be gained due to its future information, hence its expected value is inferred as a trainable (state) value function, $V(s;\theta_{V})=\mathbb{E}[R\mid s]$, with its parameters set $\theta_{V}$. Finally, DRL optimizes $\pi$ to maximize $V$ while increasing the accuracy of $V$.

To learn $V$, a temporal difference (TD) error method is widely used as follows:

where $y$ denotes the pseudo-supervised signal generated from the target network with the parameters set $\bar{\theta}_{V}$ (see later). By minimizing $\mathcal{L}_{V}$, $\theta_{V}$ can be optimized to correctly infer the value over $s$.

To learn $\pi$, a policy-gradient method is applied as follows:

where $a$ is sampled from the alternative policy $b$, which is often given by the target network with the parameters set $\bar{\theta}_{\pi}$. The sampler change is allowed by the importance sampling, and the likelihood ratio is introduced as in the above loss function. By minimizing $\mathcal{L}_{\pi}$, $\theta_{\pi}$ can be optimized to reach the state with higher value.

The target network with $\bar{\theta}_{V,\pi}$ is briefly introduced together with the latest update rule, T-soft update [15]. First, with the initialization phase of the main network, the target network is also created as a copy with $\bar{\theta}_{V,\pi}=\theta_{V,\pi}$. Since the copied $\bar{\theta}_{V,\pi}$ is given independently of $\theta_{V,\pi}$, and not updated through the minimization problem of eqs. (1) and (3). Therefore, the pseudo-supervised signal $y$ has the same value for the same input, which greatly contributes to the stability of learning by making the minimization problem stationary.

However, in practice, if $\bar{\theta}_{V,\pi}$ is fixed at its initial value, the correct $y$ cannot be generated and the task is never accomplished. Thus, $\bar{\theta}_{V,\pi}$ must be updated slowly towards $\theta_{V,\pi}$ as in alternating optimization [20]. When the target network was first introduced, a technique called hard update was employed [8], where $\theta_{V,\pi}$ was updated a certain number of times and then copied again as $\bar{\theta}_{V,\pi}=\theta_{V,\pi}$. Afterwards, soft update shown in the following equation has been proposed to make $\bar{\theta}_{V,\pi}$ asymptotically match $\theta_{V,\pi}$ more smoothly.

where $\tau\in(0,1]$ denotes the update rate.

The above update rule is given as an exponential moving average, and all new inputs are treated equivalently. As a result, even when $\theta_{V,\pi}$ is incorrectly updated, its adverse effect as noise is reflected into $\bar{\theta}_{V,\pi}$. This effect is more pronounced when $\tau$ is large, but mitigating this by reducing $\tau$ causes a reduction in learning speed [16].

To tackle this problem, T-soft update that is robust to noise even with relatively large $\tau$ has recently been proposed [15]. It regards the exponential moving average as update of a location parameter of normal distribution, and derives a new update rule by replacing it with student-t distribution that is more robust to noise by specifying degrees of freedom $\nu\in\mathbb{R}_{+}$. T-soft update is described in Alg. 1. Note that $\sigma^{2}$ and $W$ must be updated as new internal states.

With mathematical explanations, the issues of T-soft update are again summarized as below.

1. 1.

   Since $\nu$ must be specified as a constant in advance, it must be tuned for each task to provide the appropriate noise robustness to maximize performance.

2. 2.

   The larger $\Delta_{i}\sigma_{i}^{-2}$ is, the more the update is suppressed. However, the simple calculation of $\Delta_{i}$ as mean square error makes it easier to hide the noise hidden in the $i$-th subset.

3. 3.

   If $\tau_{i}$ is frequently close to zero (i.e. no update), there is a risk that $\bar{\theta}_{V,\pi}$ will not asymptotically match $\theta_{V,\pi}$.

The first two of the three issues mentioned above are resolved by deriving a new update rule, so-called AT-soft update. To develop AT-soft update, the formulation of AdaTerm [17], which is a kind of stochastic gradient descent method with the adaptive noise robustness, is applied. In this method, by assuming that the gradient is generated from student-t distribution, its location, scale, and degrees of freedom parameters, which are utilized for updating the network, can be estimated by approximate maximum likelihood estimation. Instead of the gradient as stochastic variable, the parameters of the main network are considered to be generated from student-t distribution, and its location is mapped to the parameters of the target network. With this assumption, AT-soft update obtains the noise robustness as in the conventional T-soft update. In addition, the degrees of freedom can be estimated at the same time in this formulation, so that the noise robustness can be automatically adjusted according to the faced task.

Specifically, the $i$-th subset of $\theta_{V,\pi}$ (e.g. a weight matrix in each layer), $\theta_{i}$ with $d_{i}$ the number of dimensions, is assumed to be generated from $d_{i}$-dimensional diagonal student-t distribution with three kinds of sample statistics: a location parameter $\bar{\theta}_{i}\in\mathbb{R}^{d_{i}}$; a scale parameter $\sigma_{i}\in\mathbb{R}_{+}^{d_{i}}$; and degrees of freedom $\nu_{i}\in\mathbb{R}_{+}$. With $\tilde{\nu}_{i}=\nu_{i}d_{i}^{-1}$ and $D_{i}=d_{i}^{-1}\sum_{j=1}^{d_{i}}(\theta_{i,j}-\bar{\theta}_{i,j})^{2}\sigma_{i,j}^{-2}$, its density can be described as below.

where $\Gamma$ denotes the gamma function. Note that the conventional T-soft update simplifies this model as one-dimensional student-t distribution for the mean of $\theta_{i}-\bar{\theta}_{i}$, but here we treat it as $d_{i}$-dimensional distribution with slightly increased computational cost.

With this assumption, following the derivation of AdaTerm, $\bar{\theta}_{i}$, $\sigma_{i}$, and $\nu_{i}$ are optimally inferred to maximize the approximated log-likelihood. The important variable in the derivation is $w_{1}$, which indicates the deviation of $\theta_{i}$ from $\mathcal{T}$, and is calculated as follows:

That is, since $D_{i}$ represents the pseudo-distance from $\mathcal{T}$, the larger $D_{i}$ is, the closer $w_{1}$ is to $0$. In addition, the smaller $\tilde{\nu}_{i}$ is, the more sensitive $w_{1}$ is to fluctuations in $D_{i}$, leading to higher noise robustness. Using $w_{1}$, $w_{2}$, which is used only for updating $\tilde{\nu}_{i}$, can be derived as follows:

These $w_{1,2}$ are used to calculate the update ratio of the sample statistics, $\tau_{1,2}$.

where $\tau\in(0,1]$ denotes the basic update ratio given as a hyperparameter. To satisfy $\tau_{1,2}\in(0,1]$, the upper bounds of $w_{1,2}$, $\overline{w}_{1,2}$, are employed.

where $87.3365$ means the negative logarithm with the tiny number of float32.

The update amounts for $\bar{\theta}_{i}$, $\sigma_{i}^{2}$, and $\tilde{\nu}_{i}$ are respectively given as follows:

where $\epsilon\ll 1$ denotes the small value for stabilizing the computation and $\underline{\tilde{\nu}}$ denotes the lower bound of $\tilde{\nu}$ (i.e. the maximum noise robustness) given as a hyperparamter.

Using the update ratios and the update amounts obtained above, $\bar{\theta}_{i}$, $\sigma_{i}^{2}$, and $\tilde{\nu}_{i}$ can be updated.

As a result, AT-soft update enables to update the parameters set $\bar{\theta}_{i}$ of the target network adaptively (i.e. depending on the deviation of $\theta_{i}$ from $\mathcal{T}$), while automatically tuning the noise robustness represented by $\sigma_{i}^{2}$ and $\tilde{\nu}_{i}$.

However, if $\tau_{1}\simeq 0$ continues in the above update, $\theta_{i}$ will deviate from $\bar{\theta}_{i}$ gradually, and the target network will no longer be able to generate pseudo-supervised signals since the assumption $\bar{\theta}_{i}\simeq\theta_{i}$ is broken. In such a case, parts of $\theta_{i}$ would be updated with the minimization of eqs. (1)–(3) in a wrong direction, causing $\tau_{1}\simeq 0$ as outlier. Hence, to stop this fruitless judgement and restart the appropriate updates, reverting $\theta_{i}$ to $\bar{\theta}_{i}$ and holding $\theta_{i}\simeq\bar{\theta}_{i}$ would be the natural effective way. To this end, a heuristic consolidation is designed as below.

Specifically, the update ratio from the main to target networks, $\tau_{c}$, is designed to be larger when the update ratio of $\bar{\theta}_{i}$ is smaller (i.e. when $\theta_{i}$ deviates from $\mathcal{T}$).

where $\lambda\in[0,1]$ adjusts the strength of this consolidation, which should be the same as or weaker than the update speed of the target network.

Next, since consolidating all $\theta_{i}$ would interfere with learning, the consolidated subset of outliers, $\theta_{i}^{c}$, should be extracted. The simple and popular way is to use the $q$-th quantile $Q$ with $q\in[0,1]$. Since the component that contributes to making $w_{1}$ small is with large $(\theta_{i,j}-\bar{\theta}_{i,j})^{2}\sigma_{i,j}^{-2}=:\Delta_{i,j}$, $\theta_{i}^{c}$ is defined as follows:

Thus, the following update formula consolidates $\theta_{i}^{c}$ to the corresponding subset of the target network, $\bar{\theta}_{i}^{c}$.

A rough sketch of this consolidation is shown in Fig. 1. Although loss-function-based consolidations, as proposed in the context of continual learning [21, 22], would be also possible, a more convenient implementation with lower computational cost was employed.

The pseudo-code for the consolidated adaptive T-soft (CAT-soft) update is summarized in Alg. 2. Note that, although $\underline{\tilde{\nu}}$ must be specified as a new hyperparameter in (C)AT-soft update, $\nu$ specified in T-soft update is already conservatively set for noise, and we can inherit it (i.e. $\underline{\tilde{\nu}}=1$). Therefore, the additional hyperparameters to be tuned are $\lambda$ and $q$. $q$ can be given as $q\simeq 1$ so that a few parameters in $i$-th subset are consolidated without interfering with learning. $\lambda$ can be given as the inverse of the number of parameters to be consolidated. In other words, we can decide whether to make $q$ closer to $1$ and consolidate fewer parameters tightly, or make $q$ smaller and consolidate more parameters slightly.

For the statistical verification of the proposed method, the following simulations are conducted. As simulation environments, Pybullet [18] with OpenAI Gym [24] is employed. From it, InvertedDoublePendulumBulletEnv-v0 (DoublePendulum), HopperBulletEnv-v0 (Hopper), and AntBulletEnv-v0 (Ant) are chosen as tasks. To make the tasks harder, the observations from them are with white noises, scale of which is $0.001$. With 18 random seeds, each task is tried to be accomplished by each method. After training, the learned policy is run 100 times for evaluating the sum of rewards as a score (larger is better).

The implementation of the base network architecture and DRL algorithm is basically the same as in the literature [23]. However, it is noticeable that the stochastic policy function is modeled by student-t distribution for conservative learning and efficient exploration [14], instead of normal distribution. Hyperparamters for the implementation are summarized in Table I. Note that the learning difficulty is higher because the specified value of the learning rate is higher than one suitable for DRL, revealing the effectiveness of the target network for stabilizing learning.

The following three methods are compared.

• •

  T-soft update: $(\tau=0.1,\nu=1)$

• •

  AT-soft update: $(\tau=0.1,\underline{\tilde{\nu}}=1)$

• •

  CAT-soft update: $(\tau=0.1,\underline{\tilde{\nu}}=1,\lambda=1,q=1)$

Here, $q$ is designed to consolidate only one parameter in each subset as the simplest implementation. Correspondingly, $\lambda$ is given as the inverse of the (maximum) number of consolidated parameters. $\tau=0.1$ is set smaller than one in the literature [15], but this is to counteract the negative effects of the high learning rate (i.e. $10^{-3}$) set above.

The learning behaviors are depicted in Fig. 2. As pointed out, the deviation by (C)AT-soft updates were larger than that of the conventional T-soft update since (C)AT-soft update have better outlier and noise detection performance and the target network update is easily suppressed. This was pronounced in the early stage of training when the noise robustness is high and the update of the main network is unstable. However, CAT-soft update suppressed the deviation in the early stage of training. As the learning progresses, CAT-soft update converged to roughly the same level of deviation as AT-soft update because the consolidation was relaxed with the weakened noise robustness.

The scores of 100 runs after learning are summarized in Table II. AT-soft update slightly increased the performance of T-soft update on Ant, but decreased it on Hopper. In contrast, CAT-soft update outperformed T-soft update in all tasks.

As a demonstration, a simulation closer to the real robot experiment, MinitaurBulletDuckEnv-v0 (Minitaur) in Pybullet, is tried. The task is to move a duck on top of a Ghost Minitaur, a quadruped robot developed by Ghost Robotics. Since this duck is not fixed, careful locomotion is required, and its states (e.g. position) are unobserved, making this task a partially observed MDP (POMDP). Note that the default setting for Minitaur tasks is unrealistic, as pointed out in the literature [25]. Therefore, it was modified as shown in Table III (arguments not listed are left at default).

T-soft and CAT-soft updates are compared under the same conditions as in the above simulations. The learning curves of the scores for 8 trials and the test results of the trained policies are depicted in Fig. 3. The best behaviors on the tests can be found in the attached video. As can be seen from Fig. 3, only the proposed CAT-soft update was able to acquire the successful cases of the task (walking without dropping the duck). Thus, it is suggested that CAT-soft update can contribute to the success of the task by steadily improving the learning performance even for more practical tasks.