Trajectory Planning for Teleoperated Space Manipulators Using Deep Reinforcement Learning

According to Figure 1, a space robotic system typically consists of a spacecraft (base) and a manipulator with $n$ degrees of freedom (DOF).

The vector of joint angles of the manipulator is denoted as $q=[\theta_{1},\theta_{2},\cdots,\theta_{n}]^{T}$, and correspondingly, its velocity can be represented as $\dot{q}$. Additionally, $(v_{i},w_{i})^{T}$ ($i=b,e$) represent the velocities of the base and the robotic arm, respectively. Referring to [47] , the following equation can be obtained:

where $J_{b}$ and $J_{m}$ are the Jacobian matrices of the base and the manipulator respectively. Without any external force or torques executed, the space robot is in free-floating mode and its linear momentum $P$ and angular momentum $L$ are conserved. Therefore, it holds that:

where $H_{b}$ and $H_{bm}$ are inertia matrix and coupling inertia matrix, respectively. Further, assuming the initial values of $P$ and $L$ are both zero, the velocity of the base can be expressed as:

where $J_{bm}$ is the Jacobian matrix of the base. Substituting Eq. (3) into Eq. (1) yields the following expression:

where $J_{g}$ is referred to as the generalized Jacobian matrix, which is not only related to kinematic parameters but also to dynamic parameters.

Reinforcement learning is a sequential decision-making process grounded in the theoretical framework of Markov decision processes (MDPs). Typically, an MDP is defined by a 6-element tuple $(\mathscr{S},\mathscr{A},\mathscr{P},\mathcal{R},\rho,\gamma)$, where $\mathscr{S}$ and $\mathscr{A}$ represent the state space and action space, respectively; $\mathscr{P}$ is the state transition function and $\mathcal{R}$ denotes the reward function while $\rho$ represents the initial distribution of states; $\gamma$ signifies the discount factor for future rewards. At time step $t$, the agent observes the environmental state $s_{t}$. Subsequently, based on its current policy $\pi$, it selects an action $a_{t}\sim\pi(\cdot|s_{t})$ to apply to the environment. The environment then returns a new state $s_{t+1}\sim\mathscr{P}(\cdot|s_{t},a_{t})$ along with a reward $r_{t}=R(s_{t},a_{t})$. The objective of reinforcement learning is to maximize the cumulative reward $G_{t}=\sum_{k=t}^{T}\gamma^{k-t}r_{k}$ to obtain an optimal policy $\pi^{*}(\cdot|s_{t})$, where $T$ is the maximum number of steps the agent interacts with the environment in an episode.

The trajectory planning problem of space robots can be modeled using MDPs, described by a tuple of six elements. Specifically, the setup for the state space, action space, and reward function is as follows:

• •

  State $\mathscr{S}$.

  Due to the crucial value of the information contained in the state for agent decision-making, it is imperative to consider as many relevant features influencing decisions as possible when designing the state. In the trajectory planning process of space robots, factors such as the joint angles $q\in\mathbb{R}^{n}$, joint angular velocities $\dot{q}\in\mathbb{R}^{n}$, end-effector pose $p_{e}\in\mathbb{R}^{6}$ and velocity $v_{e}\in\mathbb{R}^{6}$, target pose $p_{target}\in\mathbb{R}^{6}$, and the distance $d\in\mathbb{R}$ between the end-effector and the target point all play pivotal roles. Particularly, when the space robot operates in a free-floating mode, the presence of non-holonomic constraints between the base and the manipulator can affect the manipulator’s planning. Therefore, the base pose $p_{b}\in\mathbb{R}^{6}$ and velocity $v_{b}\in\mathbb{R}^{6}$ should also be included in the state space design. Consequently, for the fixed-base mode, the state is designed as follows:

  $$s_{t}=(q,\dot{q},p_{e},v_{e},p_{target},d)\in\mathbb{R}^{2n+19};$$

  (5)

  By contrast, for the free-floating mode, the state is designed as follows:

  $$s_{t}=(q,\dot{q},p_{e},v_{e},p_{b},v_{b},p_{target},d)\in\mathbb{R}^{2n+31}.$$

  (6)

• •

  Action $\mathscr{A}$.

  The action is designed as a set of torques applied to the manipulator joints, clipped to a certain range: $a_{t}\in\mathbb{R}^{n}$ and $a_{ti}\in[0,max(torque_{i})](i=1,\cdots,n)$.

• •

  Reward function $\mathcal{R}$.

  The reward function is defined as follows:

  	$\displaystyle r_{t}=$	$\displaystyle\omega_{1}d_{t}+\omega_{2}log(d_{t}+1e^{-6})+\omega_{3}(||(v_{e})_{t}||+||(v_{b})_{t}||)+\omega_{4}||\dot{q_{t}}||$		(7)
  		$\displaystyle+\delta(d_{t}\leq d_{threshold})\omega_{5}(T-t)log(d_{threshold}),$

  where the episode is terminated when the distance $d$ is small than the threshold $d_{threshold}$. Each term in Eq. 7 serves a distinct purpose. The first term incentivizes the robotic arm’s end effector to approach the target point as closely as possible. The second term addresses the scenario where the end effector lingers near the target point as the distance nears the threshold without progressing further, ensuring continual advancement towards the target. The third and fourth terms aim to mitigate excessive velocity fluctuations in the base, end-effector, and joints of the robotic arm during trajectory planning, promoting smooth transitions across these variables from a reward standpoint. Lastly, the fifth term functions as a terminal reward, activated only when the distance falls below or equals the threshold. The magnitude of this reward is directly proportional to the remaining steps at the end of an episode, $T-t$. As a result, it is expected that as the agent completes tasks more swiftly, the reward value associated with this term will increase proportionally.

In the context of space teleoperation, the time delay introduced by the data link disrupts the Markov property of the original decision-making process, significantly compromising the performance of reinforcement learning algorithms. Specifically, shown in Figure 2, once an operator issues an instruction at the master end, there is a delay before this instruction reaches the slave end, which constitutes the action delay as referenced in [34]. Meanwhile, when the space robot at the slave end actually interacts with the environment, the feedback from the environment, containing new states and rewards, is not promptly relayed to the master end, resulting in the observation delay and reward delay discussed in [34]. Since observation delay and action delay exert equivalent influences on the agent’s decision-making process, proven in [34], we solely focus on the observation delay, ensuring that its length aligns with that of the round time delay(RTD).

Remark. At time $t$, the state generated by the slave end is denoted as $s_{t}$, while the state actually observed by the master end is denoted by $x_{t}$.

In ideal circumstances, RTD remains constant as it is solely determined by the transmission distance. Figure 3 shows a constant delay scenario, assuming an RTD-induced delay of $d_{RTD}=1$. The state of the robot at the slave end evolves over time, while at the master end, due to the presence of RTD, the first observation is null, and subsequent observations lag behind those at the slave end by one time step, expressed as $x_{t}=s_{t-1}\delta(t>=1)+\Phi\delta(t<1)$, where $\Phi$ indicates the null, and the $\delta$ function is the Dirac delta function.

During the transmission, network congestion and other environmental factors often introduce randomness into the delay. Building upon the RTD, we assume that the random delay follows a uniform distribution with a parameter $\xi$ , denoted as $d_{random}\sim Uniform(\xi)$. Consequently, the total delay in data link is $d_{total}(t)=d_{RTD}+d_{random}(t)$. In Figure 4, a scenario with random delay is illustrated, where $d_{RTD}=1$ and $\xi=1$. At $t=0$, $d_{total}(0)=1$, indicating that the master end does not observe the state at $t=0$. Instead, it observes $s_{0}$ at $t=1$, resulting in $x_{0}=\Phi$ and $x_{1}=s_{0}$. Similarly, at $t=1$, $d_{total}(1)=2$, and at $t=2$, $d_{total}(2)=1$, implying that the master end does not observe the state at $t=2$, and both $s_{1}$ and $s_{2}$ are potentially observed at $t=3$. However, the old state is discarded, resulting in the observed state at $t=3$ being $s_{2}$, namely, $x_{3}=s_{2}$. Note that, regardless of a constant or random delay, the focus of this study is on making decisions at each time step based on the observations from the master end. The following sections will delve into a detailed analysis of this aspect.

The objective function at the master end is designed as follows:

where $x_{t}$ is the delayed state after processing in Section 3.1 and $D$ is the distribution of previously sampled states and actions, or a replay buffer. The optimal policy is obtained by maximizing the objective function. However, the end-to-end trajectory tracking for space robots poses a significant challenge due to its complex nature and the expansive state space dimension. For example, for Algorithm 1 and Algorithm 2, the dimension is $2n+19$ or $2n+31$, while the dimension extends to $2n+19+nd_{RTD}$ or $2n+31+nd_{RTD}$ for Algorithm 3. Furthermore, optimizing Eq. 12 yields a deterministic policy and often leads to local optima, making it difficult to find policy parameters that achieve high cumulative rewards. To address this, Eq. 12 is modified as follows:

where $\mathcal{H}(\pi(\cdot|x_{t}))$ represents the entropy of the policy, and $\alpha$ determines the relative importance of the entropy term against the reward. Eq. 13 not only encourages the agent to explore more effectively but also controls the stochasticity of the optimal policy.

In this study, Eq. 13 is optimized based on the SAC algorithm. Similar to traditional SAC algorithms, three neural networks $Q_{\theta}{(x_{t},a_{t})}$, $V_{\psi}(x_{t})$ and $\pi_{\phi}(a_{t}|x_{t})$ are designed to to approximate the state-action value function, state value function, and policy function, respectively. The parameters $\theta$ of the Q-function are updated by minimizing the soft Bellman residual:

where $V_{\overline{\psi}}$ represents the target value network for $V_{\psi}$ and is updated by the exponential moving average of the value network weights. The parameters $\psi$ of the soft value function are updated by minimizing the squared residual error:

The parameters $\phi$ of the policy network are updated by minimizing the KL-divergence:

where $Z_{\theta}(x_{t})$ is the partition function and can be ignored. We utilize the reparameterization trick to minimize Eq. 16 and construct the policy network:

where $\epsilon_{t}$ is a noise vector. Consequently, Eq. 16 can be rewritten as:

The only difference from the traditional SAC lies in structure of the replay buffer. In traditional methods, a quadruple is stored in the replay buffer after each step. However, in teleoperation, the master end may not have access to the true state. To ensure the reliability of the data stored in the replay buffer, such data cannot be included.

The pseudocode of the complete procedure is specified in Algorithm 4.

We introduce the simulation settings from three aspects: environment setup, neural network architecture, and operating platform.

Environment setup. Based on the 7-DOF redundant robotic arm model[26], illustrated in Figure 7, we construct a single-arm space robot model within the MuJoCo environment, as shown in Figure 8. The kinematic and dynamic parameters are detailed in Table LABEL:tab:_kinematic. In the constructed simulation environment, the length of each simulation time step $T_{m}$ is set to 0.01 s, with an action executed every 4 time steps. Throughout the experiments, the maximum number of environment steps per episode is $T=250$, resulting in a maximum simulation duration of $T\cdot T_{m}\cdot 4=10$ seconds. This experiment involves a random grasping task performed in teleoperation mode, aiming to position the end effector of the space manipulator within 5 mm of the target point. Given the end effector’s radius of 0.02 m, we set $d_{threshold}=0.025$ m, as outlined in Table LABEL:tab:_kinematic.

Based on the model in Figure 8, we built four distinct simulation environments, each corresponding to a specific scenario, provided in Table LABEL:tab:_environments.

To enhance the model’s generalization, we conduct uniform sampling within a cube in the workspace range of the robotic arm and treat them as the positions of target during training. Additionally, we introduce noise to the initial joint angles and angular velocities of the robotic arm. Specifically, in the world coordinate system, the position of the base is

and the position of the target star is

The initial joint angles of the robotic arm without noise are

and the noise added to the initial angles is represented as

Similarly, the initial angular velocities of the robotic arm without noise are

and the noise added to the initial velocities is denoted as

Consequently, the initial joint angles of the robotic arm are given by

and the initial angular velocities are given by

The termination condition for each episode is either reaching the maximum number of simulation steps $T$ or the distance between the end effector and the tracking target point being less than or equal to $d_{threshold}$.

The hyperparameters $\omega_{1},\cdots,\omega_{5}$ in Eq. 7 are set as follows: -0.01, -1, -0.1, -0.1, and 1.5, respectively.

Neural network architecture. The network architectures and hyperparameters of the generic SAC algorithm used in this paper are presented in Table 3.

When using Algorithm 2, the network architecture and parameters of the forward model are presented in Table 4.

Operating platform. All experiments are conducted on an NVIDIA GeForce RTX 3090 graphics card. The versions of gym, mujoco, and PyTorch used in the experiments are 0.21.0, 2.0.2.8, and 1.11, respectively.

For the four environments in Table LABEL:tab:_environments, we conduct experiments with fixed delays $d_{RTD}=1,2,4,6,8$ and corresponding random delays with $\xi=2$ for each algorithm. The state augmentation method based on SAC is named as SACAS. We abbreviate constant_Mapping as CM, and similarly constant_Prediction, constant_SACAS, random_Mapping, random_Prediction, and random_SACAS as CP, CS, RM, RP, and RS, respectively. Each data point in Figure 9, 11 - 16, and every entry in Tables LABEL:tab:_delay=1-LABEL:tab:_delay=8 are obtained from experiments with three different random seeds.