Learn to Grasp via Intention Discovery and its Application to Challenging Clutter

In the first phase of learning, we aim to learn an intention partitioning strategy with a neural network, as shown in Fig. 3A. The input is a given state from simulated grasp demos, and the output is a family of probability distributions indicating how likely the current state is to be divided into each intent.

Intent Segmentation and Data Collection: Considering a grasp demonstration $S=(s_{1},s_{2},\ldots,s_{l})$ represented by a sequence of states $s=(I,h)$, where $I$ is the camera observation of the environment, and $h$ is the gripper pose. The state $s\in S$ in a grasp demo can be naturally segmented into $n$ intents, denoted as $K=(k_{0},k_{1},\ldots,k_{n})$, according to the timing order and similarity. The index of $k$ indicates the timing order of intents, and an intent $k_{t+1}$ can only be reached after completing former intent $k_{t}$. The $h$ consists of $(x,y,z,\alpha,\beta,\gamma,\psi)$, where $(x,y,z,\alpha,\beta,\gamma)\in SE(3)$ is the 6D gripper pose, and $\psi$ is one hot vector representing the opening and closing of the gripper.

We now give a formal definition of the $k$-intent segmentation problem. If $k=l$, each gripper pose corresponds to a segment. Otherwise, despite the fact that humans can manually label segments, the following segmentation algorithm can be used to reduce labor costs. Let $T=(T_{0},T_{1},\ldots)$ denote the set of all possible ways of segmentation for a sequence $S$. The sequence $S$ of length $l$ contains $n$ non-overlapping contiguous sub-sequences, denoted as $T_{i}=(\tau_{1},\tau_{2},...,\tau_{n})$. Each state $s$ in segment $\tau_{i}$ belongs to the intent $k_{i}$. We denote the dissimilarity in a segment $\tau_{i}$ as $e_{i}$, then the error of segmentation $T_{i}$ is calculated as $E_{p}=\sum_{i=0}^{n}e_{i}$. Thus, we define the optimal segmentation as to find the minimize $E_{p}$ in $T$:

The $T_{\mathrm{opt}}(S,n)$ can be found by the dynamic-programming (DP) algorithm [25], and the main recurrence of the DP is

where $S[1,\dots,j]$ denotes the sub-sequence of $S$ that contains states in positions from $1$ to $j$. The function of the recurrence is to divide the segmentation problem into subproblems and combine their solutions to form the final segmentation. The dissimilarity $e_{i}$ in a segment is the sum of the dissimilarities $\Lambda$ between states, calculated by the following formula:

where $\lambda$ and $\mu$ are the hyper-parameters to adjust the influence of the gripper orientation and finger condition (i.e., open/close) change. The distance of the orientation is the relative difference of Euler angle changes and is normalized. After segmentation, each state $s$ in a demo is assigned to an intent $k_{i}=(1,2,\ldots,n)$ as supervision signals.

To learn an intention estimator, a set of grasp demonstrations needs to be collected and segmented using the above algorithm. We generate grasps in simulation by augmenting three human-encoded grasps (see Fig. 2(a)), using invariant and equivariant principles. Consider an encoded grasp $S$ for an object $o$ with a pose $o_{p}$. A new grasp $S^{\prime}$ can be augmented by the following procedures. First, we apply a set of transformations to the object pose $o_{p}$, including changing object positions and orientations. Then the new grasp $S^{\prime}$ is transferred from $S$ via homogeneous transformation by calculating the $SE(3)$ matrix between $o_{p}$ and $o^{\prime}_{p}$, as shown in Fig. 2(b). We also randomize the aspect ratio of objects at each new grasp generation. Although such augmentation of grasps leads to some imperfect grasps, these imperfect demos do not affect policy learning. We further analyze the influence of imperfect grasp demonstrations in Sec. III-D and Fig. 4(c).

Intention Estimator Learning: The goal of the intent estimator is to map the similarity between the given state and each intent. Operationally, we form it as a classification problem and use a neural network $p=f(s)$ to learn this mapping, where the given state $s$ contains a depth image $I$ of the environment and gripper pose $h$. The network processes grasp pose and visual observation in separate channels, and the output features are combined to feed into a feed-forward pipeline to calculate probabilities that the given state belongs to different intents. More precisely, the depth image $I$ and gripper pose $h$ are processed with a convolutional (Conv) encoder and a multilayer perceptron (MLP) encoder, respectively. Then the features are combined using concatenation operation and fed into three subsequent fully connected (FC) layers with 256, 256, and 128 neurons. The Conv encoder consists of one 1x1 convolutional layer followed by a global average pooling. The MLP encoder consists of one FC layer. We use the following cross-entropy loss to train the network, as shown in Eq. 5:

where $c=(1,2,3,\ldots,n)$ is the class index of intents and $k_{i}$ is the intent class label of the given state.

After we train an intention estimator that can discern the intent of the given state, we distill an intrinsic reward from its prediction. The intrinsic reward allows the robot to follow the intent during policy learning in RL and is detailed below.

Problem Formulation: We formulate the picking problem as a Markov Decision Process (MDP). The MDP is defined by a state-space $S$, action space $A$, a function of reward $R(s_{t},s_{t+1})$, and the transition probability $P(s_{t+1}|s_{t},a_{t})$. At time step $t$, a robot agent to pick objects observes the state $s_{t}$ and predicts an action $a_{t}$ based on current policy $\pi(a_{t}|s_{t})$. The rewards from the environment and intention estimator are provided to the agent afterward and then transition to a new state $s_{t+1}$. RL aims to learn an optimal policy $\pi$ that selects actions that maximize its cumulative reward.

Rewards with Intents: The output probabilities from the intention estimator are used to design a reward function $r^{\prime}$ as follows:

where $s_{t+1}$ is the state at time step $t+1$ and $p\{(k_{i}=t)|(s_{t+1})\}$ is a predicted probability between 0 and 1 representing the similarity between the current state and the $t^{th}$ intention $k_{t}$. The $k_{i}=t$ represents that we bundle time step $t$ with intent index $k_{i}$ to encourage the agent to follow the intents during training. Note the proposed method does not strictly limit agents to follow intents. In order to grasp in finite steps, the episode length is fixed to the number of intents.

Meanwhile, a task reward $r_{task}$ is given at the end of an episode, 10 for grasping one object successfully and 0 for otherwise. Thus the full reward function is defined as $r_{total}=r_{task}+r^{\prime}$. When the RL agent meets scenes that are never encountered in demonstrations, though the intrinsic reward $r^{\prime}$ from the intention estimator is almost zero, the agent can still explore on its own and obtain the task reward $r_{task}$ to learn the grasping policy in novel scenes.

Policy Architecture: The policy is trained with Proximal Policy Optimization (PPO) in simulation. PPO requires training a value network that forecasts the discounted sum of future rewards from the current state and a policy network that maps a current state to actions. The policy and value networks share the same state input, as shown in Fig. 3. The state is defined as $s_{t}:=(I_{t},h_{t})$, where $I_{t}$ is a depth image with a resolution of 120×120 from the camera, and $h_{t}$ is the gripper pose at time step $t$ including the position, orientation, and closure status of the gripper. The policy and value networks share the same front-end network. $I_{t}$ is sequentially processed by three convolutional layers with kernel sizes of $8\times 8$, $4\times 4$, and $3\times 3$, and $h_{t}$ is processed by one FC layer with eight neurons. Then, the concatenation of two extracted features is fed through the subsequent two FC layers with 64 neurons and split into two output layers: one for predicting the action and another for estimating the value. As a wrist-mounted camera on a real robot might capture things outside the workspace after acting $a_{t}$, to reduce the sim-to-real gap, we only update $h_{t}$ and always use the initial depth observation $I_{0}$ as the part of state $s_{t}$ instead of updating the depth observation over time.

Actions: In our environment, each policy action $a_{t}$ includes a gripper pose displacement and a vector to control the gripper closure. The gripper pose displacement is the difference between the initial pose and the desired one, encoded as $(x_{t}^{\prime},y_{t}^{\prime},z_{t}^{\prime},\alpha_{t}^{\prime},\beta_{t}^{\prime},\gamma_{t}^{\prime})$, where $(x_{t}^{\prime},y_{t}^{\prime},z_{t}^{\prime})$ is the relative displacement and $(\alpha_{t}^{\prime},\beta_{t}^{\prime},\gamma_{t}^{\prime})$ are the rotations of the gripper about its $x$-, $y$- and $z$-axes. The one-hot vector to control the gripper closure is denoted as $\psi_{t}^{\prime}$. We discretize each action’s coordinate according to the workspace. In addition, the episode will be terminated if $\psi_{t}$ is true. If terminated, the robot returns to its initial pose, receives new observations, and executes the next grasp, which provides a certain degree of ability for handling uncertainty and imperfect executions. For example, if objects slip from the hand during the last grasp trail, the robot can try again when it receives new observations after resetting to its initial pose.

We train the policy in the Pybullet simulator[26]. The training process consists of two stages. First, we learn the intention estimator from grasp demos, and then the RL agent explores and learns the grasping policy with the aid of the intrinsic reward constructed by the intention estimator. To train the intention estimator, we generate 10000 grasps in the simulation for each provided grasp type using the method described in Sec. III-A. Each encoded grasps have four poses. We use $n=3$ as the number of intents. Intuitively, when the gripper closes, it often represents a shift of intention, and thus we use $\mu=5$ to increase the influence of such activities in the dissimilarity calculation. A total of 30000 grasps were used to train the intention network with cross-entropy loss. The Adam optimizer was employed, starting with a learning rate of 0.001. One hundred epochs are performed, and the learning rate is halved every ten epochs during training.

During the RL policy learning phase, a pool of 64 robots generates training episodes by downloading the current policy parameters every 10 epochs from the optimizer. In each environment, random objects were placed in the workspace with random poses. Only cuboids, cylinders, and their variants with different aspect ratios are used during training, as shown in Fig. 4(a). The robot then collects the episodes in the simulation, during which the simulator automatically determines the task reward, and the estimator provides the intrinsic reward. If the workspace is empty or an object is dropped, the environment will be reset, at which point objects with random poses will fall into the workspace again. Finally, the resulting episodes are sent back to the optimizer. The Adam optimizer with a learning rate of $10^{-4}$ is used. We also deploy domain randomization to make the learned policy robust to a range of real-world conditions. Fig. 4(b) shows the learning curve for the final model training.

After training, the policy network alone is deployed to the robot in both simulation and the real world. In simulation experiments, we set up the following environments: a) Scenes in demonstrations (Similar Scene): environment constructed with a single object but in new configurations, including object friction and mass. b) Scenes not in demonstrations (Unseen Scene): a cluttered scene with multiple objects, where grasping policies can be found by self-exploration. Fig. 4(c) summarizes the result tested on similar and unseen scenes in simulation. The learned policy from the final model (denoted as $Ours$) is able to grasp the object with success rates of 98.3% in the similar scene and 91.2% in the unseen scene (row 3 in Fig. 4(c)). In contrast, removing the intent estimator from RL policy learning (denoted as $Ours$-$w/o$ $intent$), the success rates are considerably lower (row 5 vs. row 3 in Fig. 4(c) because pure RL exploration cannot find a successful grasping policy for thin objects such as cards. Meanwhile, the policy directly learned by behavior cloning (denoted as $BC$) using the same demonstrations performs better than $Ours$-$w/o$ $intent$ but lower than $Ours$. This validates our hypothesis that learning from intent (row3 in Fig. 4(c)) can help the agent learn complex and better policies while retaining its ability to explore unseen scenarios beyond directly cloning policies (row 1 in Fig. 4(c)) or exploring entirely on its own (row 5 in Fig. 4(c)). Moreover, the method of behavior cloning achieves poor performance in the unseen scene. In comparison, our model generalizes well to the novel scene.

We also investigate the impact of demonstration quality on policy learning. A total of 12.3% of demos for training intent estimators fail. We remove these imperfect demonstrations and use the remaining perfect demos to learn the policy using behavior cloning and our method (row 2 and row 4 in Fig. 4(c)). We observe that by using perfect demonstrations, the success rate of behavior cloning increases by more than 5% compared to using imperfect demonstrations (row 2 vs. row 1 in Fig. 4(c)). In contrast, our method does not rely on the quality of the demonstration. It achieves comparable performance with the one learned with perfect demonstrations (row 3 vs. row 4 in Fig. 4(c)). Because when the intent estimator’s guidance is biased due to imperfect demonstrations, the RL agent can revise the policy through self-exploration, illustrating the superiority of learning from intents rather than direct cloning atomic actions. Such ability also helps agents to learn in unseen scenes that the agent seamlessly switches to self-exploration when meeting novel scenes, allowing agents to obtain policies in scenes that are not demonstrated. Real-world experiment results are presented in Sec. IV.

As shown in Fig. 5, we deployed the learned policy on an off-the-shelf robotic grasping platform, including a 6-DOF robot arm equipped with a robotiq140 parallel gripper and an Intel L515 depth camera.

In this section, we quantitatively evaluated our picking system and other state-of-the-art methods with two protocols. In the first protocol, which we refer to as “isolated object grasping”, the robot attempted to grasp a single object lying in the workspace. We also used a second protocol where the robot cleaned a pile of mixed objects randomly dumped into the workspace. This test was more challenging as the robot had to avoid collisions with other objects while grasping. We used two metrics for evaluation: successful picks per attempt (Success Rate) and picks per hour (PPH). A successful grasp is grasping only one object and not pushing any other objects out of the workspace. Tab. I summarizes the results of the learned policy in the real world.

We first examined the performance of our policy with the first protocol (col. 1-8 in Tab. I) on the table environment. Our method (Ours) obtained success rates of over 90% for dominos, tubes, cans, and cosmetics. For the most challenging objects, including cards, user manuals, protractors, and Go stones, our method achieved success rates of over 80% for cards and over 60% for the other objects. In contrast, the state-of-the-art 6-DoF grasp synthesis method (VPN) [13] and the learning-based planar grasping method (Planar) [14] could not successfully grasp cards, user manuals, and protractors. Notably, when testing dominos, tubes, and Go stones using the VPN baseline, we manually select top-down grasp poses; otherwise, the VPN method cannot detect a feasible grasp for these objects. Meanwhile, the behavior cloning (BC) method performed worse with all test objects.

We then evaluated our learned policy on the cluttered table populated by multiple objects (column 9 in Tab. I). Our method stably obtained a success rate of 82% in the challenging dense clutter. This level of performance is beyond other baselines. Also, the success rates of behavior cloning dropped below 15% on mixed objects due to the inherent compounding error and distribution shift. From Tab. I, the protractor and the Go stone are the most challenging to grasp among test objects. We hypothesize that this happened because protractors and Go stone have complex geometries and dynamics different from the training objects, increasing the difficulty of generalization. The other methods also perform less effectively.

At last, to emphasize the generalization ability of our learned policy and the value of using an off-the-shelf parallel gripper alone to grasp objects in broad categories, we also focused on comparing the presented approach with other methods in a conveyor environment common in industry (see Fig. 6A). The belt on the conveyor has higher friction than the table environment and is elastic. In addition, the significant variation of material properties over the surface adds extra noise to the depth camera. Overall, our method reported in the third row still achieved a higher grasp success rate (except for the tube) and PPH in all conditions. The tube has a lower success rate as it has a different non-centrosymmetric shape, making it easier to grasp from the head rather than the object’s center. In contrast, all training objects are centrosymmetric and do not have such a characteristic.

The learned policy manifests a dexterous behavior, as shown in Fig. 6. The robot approaches the protractor and continues interacting to reach a state that is feasible to grasp (see Fig. 6C). This distinguishes the presented approach from other exploration-based methods, which confine the policy to a format of approaching the object with a certain pose, closing the finger, and avoiding interaction with objects (see Fig. 6D and 6E). We can also observe that the behavior cloning method performs poorly due to the distribution shift issue, further showing the significance of learning from intentions. Note also that the policy learned by our method is more robust and not tied to particular objects. Fig. 6B shows the learned policy responding to different poses of the same object. The policy identifies purely from observations and adopts different strategies. Such behavior is not specified during training in any way and is discovered by itself. Our training environment features only simple rigid objects, with no complex geometry or compliance, such as protractors and user manuals. Nevertheless, the learned policy successfully meets the diversity of real-world conditions encountered at deployment.

In this section, we investigate 1) the effect on the success rate of adding novel objects, which perform relatively poorly during real-world testing, into the policy training phase; 2) how well the intention estimator, trained on only cube and cylinder, generalizes to different objects. For the first question, we add models of the protractor, Go stone, and tube to the policy learning phase (i.e., Phase B in Fig. 3) and train with our proposed method (denoted as Ours-extra). Qualitative real-world and simulation results (row 2 vs. row 1) show that the success rates of these objects can be improved by adding their models to training. For the second question, we learn the policy with extra objects and without using the intention estimator, denoted as Ours-w/o intent in Tab. II, and compare its performance with Ours-extra. The results (row 3 vs. row 2) show that the intention estimator successfully generalizes to objects that differ from those used in the intention estimator training phase. Ours-extra achieves an over 90% success rate for grasping the protractor, while Ours-w/o intent, which only purely relies on self-exploration, cannot grasp the protractor successfully. Meanwhile, Ours-extra achieves higher performance for the Go stone than Ours-w/o intent. Both results show the successful generalization of the intention estimator to different novel objects. The results also show that without the guidance of the intention estimator, RL agents’ self-exploration cannot discover a successful policy to grasp challenging thin objects (e.g., protractor).