Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rollout

During interaction with the environment, the higher-level agent makes a plan using subgoals and receives entire feedback by accumulating all external rewards within the planning horizon:

The lower-level agent is intrinsically motivated in the form of internal reward that evaluates subgoal-reaching performance:

Where $\eta$ denotes a Boolean hyper-parameter whose value is 0/1 for the relative/absolute subgoal scheme.

Experience replay has been the fundamental component for off-policy RL algorithms, which greatly improves the sample efficiency by reusing previously collected experiences. Here, there is no dispute that the lower-level agent can collect the experience $\tau_{lo}=\left(\left\langle s_{t},sg_{t}\right\rangle,a_{t},r^{lo}_{t},\left\langle s_{t+1},sg_{t+1}\right\rangle\right)$ by using the behavioral policy to directly interact with the environment. The higher-level agent interacts indirectly with it through the lower-level proxy and then stores a series of state-action transitions as well as a cumulative reward, i.e., $\tau_{hi}=\left(\left\langle s_{t:t+c-1},g\right\rangle,sg_{t:t+c-1},r^{hi}_{t},\left\langle s_{t+c},g\right\rangle\right)$, into the high-level replay buffer. The lower- and higher-level policies can be trained by sampling transitions stored in these experience replay buffers $\mathcal{D}_{lo}$, $\mathcal{D}_{hi}$. The aim of optimization is to maximize the expected discounted reward $\mathbb{E}_{L\in\left\{lo,hi\right\}}\left[\sum^{\infty}_{t=0}\gamma^{i}r^{L}_{t}\right]$, where $\gamma\in\left[0,1\right]$ is the discount factor. In practice, we instantiate lower- and higher-level agents based on the TD3 algorithm [12], each having a pair of online critic networks with parameters $\phi_{1}$ and $\phi_{2}$, along with a pair of target critic networks with parameters $\phi^{\prime}_{1}$ and $\phi^{\prime}_{2}$. Additionally, TD3 has a single online actor parameterized by $\theta$ and a target actor parameterized by $\theta^{\prime}$. All target networks are updated using a soft update approach. Then, the Q-network can be updated by minimizing the mean squared temporal-difference (TD) error over all sampled transitions. To simplify notation, we adopt unified symbols $o^{L}_{t}$ and $a^{L}_{t}$ to indicate the observation and performed action, where $\left\langle o^{L}_{t},a^{L}_{t}\big{|}_{L=lo}\right\rangle=\left\langle\left\langle s_{t},sg_{t}\right\rangle,a_{t}\right\rangle$ for lower-level while $\left\langle o^{L}_{t},a^{L}_{t}\big{|}_{L=hi}\right\rangle=\left\langle\left\langle s_{t},g\right\rangle,sg_{t}\right\rangle$ for higher-level. Hence, the Q-learning loss can be written as follows:

Where $y^{L}_{t}$, i.e., $y^{lo}_{t}$ or $y^{hi}_{t}$, is dependent on $\theta_{lo}$ or $\theta_{hi}$ correspondingly because target policies map states to the "optimal" actions in an almost deterministic manner:

Where $\sigma$ is the s.d. of the Gaussian noise, $a_{c}$ defines the range of the auxiliary noise, and $o^{L}_{t^{\prime}}$ refers to the next obtained observation after taking an action. It is noteworthy that $t^{\prime}=t+c$ with respect to the higher-level while $t^{\prime}=t+1$ for the lower-level. Drawing support from Q-network, the policy can be optimized by minimizing the following loss:

As mentioned above, we outline the common actor-critic approach with the deterministic policy algorithms [32, 12]. For more details, please refer to [12].

For the high-level subgoal generation, reachability within $c$ steps is a sufficient condition for facilitating reasonable exploration. Zhang et al. [60, 50] mined the adjacency information from trajectories gathered by the changing behavioral policy over time\chdeleted during the training procedure. In that study, a $c$-step adjacency matrix was constructed to memorize $c$-step adjacent state-pairs appearing in these trajectories. To ensure this procedure is differentiable and can be generalized to newly-visited states, the adjacency information stored in such matrix was further distilled into an adjacency network $\psi$ parameterized by $\Phi$. Specifically, the adjacency network approximates a mapping from a goal space into an adjacency space. Subsequently, the resulting embeddings can be utilized to measure whether two states are $c$-step adjacent using the Euclidean distance. For example, the $c$-step adjacent estimation (or shortest transition distance) of two states $s_{i}$ and $s_{j}$ can be calculated as: $d_{st}(s_{i},s_{j};\Phi)\approx\frac{c}{\zeta_{c}}||\psi_{\Phi}(\varphi(s_{i}))-\psi_{\Phi}(\varphi(s_{j}))||_{2}$, where $\zeta_{c}$ is a scaling factor and the $\varphi$ function maps states into the goal space. Such an adjacency network can be learned by minimizing the following contrastive-like loss: $\mathcal{L}_{{\rm adj}}(\Phi)=\mathbb{E}_{s_{i},s_{j}\in\mathcal{S}}[l\cdot\max(||\psi_{\Phi}(\varphi(s_{i}))-\psi_{\Phi}(\varphi(s_{j}))||_{2}-\zeta_{c},0)+(1-l)\cdot\max(\zeta_{c}+\delta_{\rm adj}-||\psi_{\Phi}(\varphi(s_{i}))-\psi_{\Phi}(\varphi(s_{j}))||_{2},0)]$, where $\delta_{\rm adj}>0$ is a hyper-parameter indicating a margin between embeddings and $l\in\{0,1\}$ is the label indicating whether $s_{i}$ and $s_{j}$ are \chreplaced$c$$k$-step adjacent.

Graph-based navigation has become a popular technique for solving complex and sparse reward tasks by providing a long-term horizon. The relevant frameworks [43, 19, 9, 8, 53, 23, 58, 28, 24] commonly contain two components: (a) a graph built by sampling landmarks and (b) a graph planner to select waypoints. In a graph, each node corresponds to an observation state, while edges between nodes are weighted using a distance estimation. Specifically, a set of observations randomly subsampling from the replay buffer are organized as nodes, where high-dimensional samples (e.g., images) may be embedded into low-dimensional representations [43, 9, 33, 53, 58]. However, operations over the direct subsampling of the replay buffer will be costly. The state aggregation [8] and landmark sampling based on farthest point sampling (FPS) were proposed for further sparsification [19, 23, 28, 24]. Our study follows prior works of Kim et al. [23, 24], in which FPS was employed to select a collection of landmarks, i.e., coverage-based landmarks ${LM}^{\rm cov}$, from the replay buffer. In addition to this, HIGL [23] used random network distillation [5] to explicitly sample novel landmarks, i.e., novelty-based landmarks ${LM}^{\rm nov}$, a set of states rarely visited in the past. Hence, the final collection of landmarks was ${LM}={LM}^{\rm cov}\cup{LM}^{\rm nov}$. Once landmarks are added to the graph, the edge weight between any two vertices can be estimated by a lower-level value function [19, 9, 23, 28, 24] or the (Euclidean-based or contrastive-loss-based) distance between low-dimensional embeddings of states [43, 53, 33, 60]. Following prior works [19, 23, 24], in this study, we estimated the edge weight via the lower-level value function, i.e., $-V_{lo}(s_{i},\varphi(s_{j}))\approx-Q(s_{i},\varphi(s_{j}),\pi\left(a|s_{i},\varphi(s_{j});\theta_{lo}\right);\phi_{lo}),\forall s_{i},s_{j}\in LM$. After that, unreachable edges were clipped by a preset threshold [19]. In the end, the shortest path planning algorithm was run to plan the next subgoal, the very first landmark in the shortest path from the current state $s_{t}$ to the goal $g$:

In HIGL, after finding a landmark through landmark-based planning (see Equation 6), the raw selected landmark was shifted towards the current state $s_{t}$ for reachability: $sg_{t}^{\rm pseudo}=sg_{t}^{\rm plan}+\delta_{\rm pseudo}\cdot\frac{sg_{t}^{\rm plan}-\varphi(s_{t})}{||sg_{t}^{\rm plan}-\varphi(s_{t})||_{2}}$, where $\delta_{\rm pseudo}$ denotes the shift magnitude. Then, the higher-level policy was guided to generate subgoals adjacent to the planned landmarks and the landmark loss of HIGL was formulated as:

Here, Kim et al.\chadded[23] employed the adjacency constraint to encourage the generated subgoals to be in the $c$-step adjacent region to the planned landmark. However, the entanglement between the adjacency constraint and landmark-based planning limited the performance of HIGL.

Inspired by PIG [24], we proposed a disentangled variant of HIGL. We only made minor modifications to the landmark loss (see Equation 7) of HIGL:

The former term is the adjacency constraint and the latter is the landmark-guided loss, so the proposed method was called ACLG. The hyper-parameters $\lambda_{\rm adj}$ and $\lambda^{\rm ACLG}_{\rm landmark}$ were introduced to better balance the adjacency constraint and landmark-based planning. \chaddedThe illustrations (see Fig. 1) visually demonstrate the differences between HIGL and ACLG.

Training a hierarchy using an off-policy algorithm remains a prominent challenge due to the non-stationary state transitions [38, 30, 61]. Specifically, the higher-level policy takes the same action under the same state but could receive markedly different outcomes because of the low-level policy changing, so the previously collected transition tuple is no longer valid. To address the issues, HIRO [38] deployed an off-policy correction to maintain the validity of past experiences, which relabeled collected transitions with appropriate high-level actions chosen to maximize the probability of the past lower-level actions. Alternative approaches, HAC [1, 30], replaced the original high-level action with the achieved state (projected to the goal space) in the form of hindsight. However, HAC-style relabeled subgoals are \chdeletedonly compatible with the past low-level policy rather than the current one, deteriorating the non-stationarity issue.

Our work is related to HIRO [38], and the majority of modification is that we roll out the off-policy correction using learned transition dynamics to suppress the accumulative error. The closest work is the MapGo [61], a model-based HAC-style framework in which the original goal was replaced with a foresight goal by reasoning using an ensemble transition model. Our work differs in that we screen out a faithful subgoal that induces rollout-based action sequence similar to the past transitions, while the MapGo overwrites the subgoal with a foresight goal based on the model-based rollout. Meanwhile, our framework proposes a gradient penalty with model-inferred upper bound to prohibit the disturbance caused by relabeling to the behavioral policy.

The promises of model-based RL (MBRL) have been extensively discussed in past research [36]. The well-known model-based RL algorithm, Dyna [49], leveraged a dynamics model to generate one-step transitions and then update the value function using these imagined data, thus accelerating the learning. Recently, instantiating environment dynamics\chdeletedmodel using an ensemble of probabilistic networks has become quite popular because of its ability to model both aleatory uncertainty and epistemic uncertainty [7]. Hence, a handful of Dyna-style methods proposed to simulate multi-step rollouts by using ensemble models, such as SLBO [35] and MBPO [20]. Alternatively, the model-based value expansion methods performed multi-step simulation and estimated the future transitions using the Q-function, which helped to reduce the value estimation error. The representative \chdeletedvalue estimation algorithms include MVE [10] and STEVE [4]. Besides, in fact, the estimated value of states can directly provide gradients to the policy when the learned dynamic models are differentiable, like Guided Policy Search [29] and Imagined Value Gradients [6]. Our work differs from these works since we use the higher-level Q-function to estimate the value of future lower-level transitions. As stated in a recent survey [34], there have been only a few works [40, 61] involving the model exploitation in the goal-conditioned RL. To our knowledge, there is no prior work studying such inter-level planning.

A bootstrapped ensemble of dynamics models is constructed to approximate the true transition dynamics of environment: $f(s_{t+1}|s_{t},a_{t})$, which has been demonstrated in several studies [7, 26, 20, 47, 55, 56]. We denote the dynamics approximators as $\Gamma_{\xi}=\{\hat{f}^{1}_{\xi},\dots,\hat{f}^{B}_{\xi}\}$, where $B$ is the ensemble size and $\xi$ denotes the parameters of models. Each model of the ensemble projects the state $s_{t}$ conditioned on the action $a_{t}$ to a Gaussian distribution of the next state, i.e., $\hat{f}^{b}_{\xi}(s_{t+1}|s_{t},a_{t})=\mathcal{N}(\mu^{b}_{\xi}(s_{t},a_{t}),\Sigma^{b}_{\xi}(s_{t},a_{t}))$, with $b\in\{1,\dots,B\}$. In usage, a model is picked out uniformly at random to predict the next state. Note that, here, we do not learn the reward function because the compounding error from multi-step rollouts makes it infeasible for higher-level to infer the future cumulative rewards. As for the lower-level agent, the reward can be computed through the intrinsic reward function (see Equation 2) on the fly. Finally, such dynamics models are trained via maximum likelihood and are incorporated to encourage inter-level cooperation and stabilize the policy optimization process.

With well-trained dynamics models, we expand the vanilla off-policy correction in HIRO [38] by using the model-generated state transitions to bridge the gap between the past and current behavioral policies. Recall a stored high-level transition $\tau_{hi}=\left(\left\langle s_{t:t+c-1},g\right\rangle,sg_{t:t+c-1},r^{hi}_{t},\left\langle s_{t+1:t+c},g\right\rangle\right)$ in the replay buffer, which is converted into a state-action-reward transition: $\tau_{hi}=\left(\left\langle s_{t},g\right\rangle,sg_{t},r^{hi}_{t},\left\langle s_{t+c},g\right\rangle\right)$ during training. Relabeling either the cumulative rewards or the final state via $c$-step rollouts, resembling the FGI in MapGo [61], substantially suffers from the high variance of long-horizon prediction. In essence, both the final state $s_{t+c}$ and the reward sequence $R_{t:t+c-1}$ are explicitly affected by the action sequence $a_{t:t+c-1}$. Hence, relabeling the $sg_{t}$, instead of the $s_{t+c}$ or $r^{hi}_{t}$, with an action sequence-based maximum likelihood is a promising way to improve sample efficiency. Following prior work [38], we consider the maximum likelihood-based action relabeling:

Where $\tilde{sg}_{t}$ indicates the candidate subgoals sampled randomly from a Gaussian centered at $\varphi(s_{t+c})$. Meanwhile, the original goal $sg_{t}$ and the achieved state (in goal space) $\varphi(s_{t+c})$ are also taken into consideration. Specifically, according to Equation 9, the current low-level policy performed $c$-step rollouts conditioned on these candidate subgoals. These sub-goals maximizing the similarity between original and rollout-based action sequences will be selected as optimal. Yet, we find that the current behavioral policy cannot produce the same action as in the past, so the $s_{t+1}$\chreplaced may not be revisited. will not be reached at all.Therefore, the vanilla off-policy correction still suffers from the cumulative error due to the gap between the $s_{t+1:t+c}$ and the unknown transitions $\hat{s}_{t+1:t+c}$. In view of this fact, we roll out these transitions using the learned dynamics models $\Gamma_{\xi}$ to mitigate the issue. Besides, we employ an exponential weighting function along the time axis to highlight shorter rollouts and \chreplacedslowly shift the original states to the rollout-based ones.eliminate the effect of cumulative error. Then Equation 9 is rewritten as:

Where $t\leq i\leq t+c-1$ and $\hat{s}_{i}\big{|}_{i=t}=s_{t}$. $\rho\in\mathbb{R}$. $\rho\in\mathbb{R}$ is a hyper-parameter indicating the base of the exponential function, where in practice, we set $\rho$ to 0.95.

Soft-Relabeling: Inspired by the pseudo-landmark shift of HIGL [23], instead of an immediate overwrite, we use a soft mechanism to smoothly update subgoals:

Where $\chreplaced{\delta_{sg}}{\delta_{g}}$ represents the shift magnitude\chreplaced from the original subgoals updated at the constant speed $\epsilon$. The soft update is expected to be robust to outliers\chdeleted and protect the Q-learning, which is highly sensitive to noise.

Apparently, from the perspective of the lower-level policy, the subgoal relabeling implicitly brings in a distributional shift of observation. Specifically, these relabeled subgoals are sampled from the goal space but are not executed in practice. The behavioral policy is prone to produce unreliable actions under such an unseen or faraway goal, resulting in ineffective exploration. Motivated by \chreplaced[2, 25, 15][25, 15], we pose the Lipschitz constraint on the Q-function gradients to stabilize the Q-learning of behavioral policy. To understand the effect of the gradient penalty, we highlight the \chreplacedLipschitzLlipschitz property of the learned Q-function.

Now, we propose an approximate upper-bound approach grounded on the learned dynamics $\Gamma_{\xi}$. Fortunately, the lower-level reward function is specified in the form of L2 distance and is immune to environment stochasticity. Naturally, the upper bound of reward gradients w.r.t. input actions can be estimated as:

In practice, we approximate the upper bound using a mini-batch of lower-level observations independently sampled from the replay buffer $\mathcal{D}_{lo}$, yielding a tighter upper bound and, in turn, more forcefully penalizing the gradient:

Then, following prior works [15], we plug the gradient penalty term into the lower-level Q-learning loss (see Equation 3), which can be formulated as:

Where $\lambda_{gp}$ is a hyper-parameter controlling the effect of the gradient penalty term. Because the gradient penalty enforces the Lipschitz constraint on the critic, limiting its update, we had to increase the number of critic training iterations to 5, a recommended value in WGAN-GP [17], per actor iteration. Considering the computational efficiency, we apply the gradient penalty every 5 training steps.

In a flat model-based RL framework, model-based value expansion-style methods [10, 4] use dynamics models to simulate short rollouts and evaluate future transitions using the Q-function. Here\chadded, as shown in Fig. 2, we steer the behavioral policy towards globally valuable states, i.e., having a higher higher-level Q-value. Specifically, we perform a one-step rollout and evaluate the next transition using the higher-level critics. The objective is to minimize the following loss:

Where $\lambda_{osrp}$ is a hyper-parameter to weigh the planning loss. Note that the $sg_{t}$ is not determined by higher-level policy solely. Meanwhile, considering that the higher-level policy is also \chdeletedconstantly changing over time, the $sg_{t}$ is sampled randomly from a Gaussian distribution centered at $\pi(s_{t},g;\theta_{hi})$. In practice, a pool of $s_{t}$ and $g$ is sampled from the buffer $\mathcal{D}_{hi}$, and then they are repeated ten times with shuffling the $g$. Next, these samples are duplicated again to accommodate the variance of $sg_{t}$. On the other hand, the next step’s subgoal $sg_{t+1}$ is also produced by the fixed goal transition function or by the higher-level policy conditioning on the observation. But, from the perspective of lower-level policy, the probability of such two events is equal because of the property of Markov decision process, i.e.,

Hence, the Equation 17 is instantiated:

Obviously, the second term $ⓐ$ is too dependent on current higher-level policy. The TD3 [12] seeks to smoothen the value estimate by bootstrapping off of nearby state-action pairs. Similarly, we add clipped noise to keep the value estimate robust. This makes our modified term $ⓐ$:

Where the hyper-parameters $\sigma$ and $a_{c}$ are common in the TD3 algorithm (see Equation 4). In the end, $\mathcal{L}_{osrp}$ is incorporated into lower-level actor loss (see Equation 5) to guide the lower-level policy towards valuable highlands with respect to the overall task. Here, in the same way, we employ the one-step rollout-based planning every 10 training steps.

First, we conduct experiments on Ant Maze (U-shape) to explore the effect of hyper-parameters in ACLG: (1) the number of landmarks and (2) the balancing coefficient $\lambda^{\rm ACLG}_{\rm landmark}$.

Next, we highlight the impact of the hyper-parameters in GCMR: the shift magnitude $\delta_{sg}$ in soft-relabeling, the penalty coefficient $\lambda_{gp}$, and the coefficient $\lambda_{osrp}$ for the rollout-based planning. We deploy ACLG+GCMR in multiple environments.

To validate the effectiveness of the GCMR, we plugged it into the ACLG, the disentangled variant of HIGL, and then compared the performance of the integrated framework with that of ACLG, HIGL [23], HRAC [60], DHRL [28], as well as the PIG [24]. Note that the numbers of landmarks used in these methods are different. In most tasks, HIGL employed 40 landmarks, ACLG used \chreplaced60120 landmarks, DHRL utilized 300 landmarks, and PIG employed 400 landmarks (see Table 3 in the "Supplementary Materials" for details). This is in line with prior works. Also, consistent with prior research, our experiments were performed on the above-mentioned environments with sparse reward settings, where the agent will obtain no reward until it reaches the target area.

But we also present a discussion about dense experiments based on AntMaze (U-shape), as shown in Fig. 8. Note that the comparison on dense reward setting did not involve DHRL and PIG due to the scope and limitations of their applicability. In the implementation, we did not use the learned dynamics model until we had sufficient transitions for sampling, which would avoid a catastrophic performance drop arising from inaccurate planning. It means that our method was enabled only if the step number of interactions was over a pre-set value. Here, the time step limit was set to $20K$ for maze-related tasks and $10K$ for robotic arm manipulation. After that, the dynamics model was trained at a frequency of $D$ steps. In the end, we evaluate their performance over 5 random seeds \chadded(the seed number is set from 0 to 5 for all tasks), conducting 10 test episodes every $5K^{\rm th}$ time step. All of the experiments were carried out on a computer with the configuration of Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz, 8-core, 64 GB RAM. And each experiment was processed using a single GPU (Tesla V100 SXM2 32 GB). We provide more detailed experimental configurations in the "Supplementary Materials".

We also investigate the effectiveness of different crucial components in our method.

In prior works, there has been a strong emphasis on optimizing the higher-level policy, as global planning is determined by the higher-level policy. The lower-level policy is overlooked, merely seen as an "unintelligent" subordinate to the higher level. Yet, in fact, the behavior of lower-level policy significantly influences the effectiveness of exploration and the stability of the hierarchical system because the lower-level policy interacts directly with the environment. Reinforcing the robustness of the lower-level policy can prevent catastrophic failures, such as collisions or tipping over, thereby accelerating reinforcement learning. This study reemphasizes the importance of the lower-level policy from the perspective of inter-level cooperation, drawing attention to related research on optimizing lower-level policies.

For facilitating inter-level cooperation and communication, we propose a novel goal-conditioned HRL framework, which mainly consists of three crucial components: the model-based off-policy correction for the data efficiency, the gradient penalty on the lower-level policy for the robustness, and one-step rollout-based planning for the cooperation. The experimental results indicate that the gradient penalty and one-step rollout-based planning achieved the expected effects, significantly enhancing the performance of the HRL framework. However, the model-based off-policy correction did not yield significant effects. The reason might be that these predicted states in correction could potentially suffer from compounding errors in long-horizon rollouts. Although soft-relabeling can mitigate this error, it is unstable due to the introduction of sensitive hyper-parameter $\delta_{sg}$ (see Fig. 5).

This study has certain limitations. First, our experiments show that the GCMR achieved significant performance improvement, and such improvement came at the expense of more computational cost (see Table 4 in the "Supplementary Materials" for a quantitative analysis of computational cost). However, the time-consuming issue only occurs during the training stage and will not affect the execution response time in the applications. Second, we need to clarify that the scope of applicability is off-policy goal-conditioned HRL. The effectiveness in general RL tasks or online tasks \chreplacedstill needs to be validated in future workhad not been validated. Third, the experimental environments used in this study have 7 or 30 dimensions. Our network architecture of transition dynamics models is relatively simple, leading to limited regression capability. Applications in complex environments that closely resemble real-world scenarios with high-dimensional observation, like the large-scale point cloud environments encountered in autonomous driving, might face limitations. This issue will be investigated in our future work. \chaddedBesides, related research on cooperative multi-robot HRL has successfully coped with extremely complex environments by enabling multiple robots to learn through the collective exchange of environmental data[45, 13, 46]. Integrating the proposed algorithm into multi-robot HRL systems is expected to enhance the performance in complex environments further, and this will be investigated in future work. Finally, we can observe that the effect of one-step rollout-based planning is more pronounced in an environment with a higher maximum distance within a single step. This observation inspires us to delve into multi-step rollout-based planning in the future to broaden its application.