AsymDex: Leveraging Asymmetry and Relative Motion in Learning Bimanual Dexterity

Consider the problem of bimanual dexterous manipulation, in which two multi-fingered hands coordinate to manipulate up to two objects. Formally, this problem can be defined as a Partially-Observable Markov Decision Process (POMDP) $\mathcal{M}=(\mathcal{S},\mathcal{Z},\mathcal{A},\mathcal{R},\mathcal{P})$, where $\mathcal{S}\in\mathbb{R}^{n}$ is the state space, $\mathcal{Z}\in\mathbb{R}^{m}$ is the observation space, $\mathcal{A}\in\mathbb{R}^{u}$ is the action space, ${\mathcal{R}:\mathbb{R}^{m}\times\mathbb{R}^{u}\rightarrow\mathbb{R}}$ is the reward function, and ${\mathcal{P}:\mathbb{R}^{n}\times\mathbb{R}^{u}\rightarrow\mathbb{R}^{n}}$ is the environment dynamics. Note that we do not assume access to any demonstrations. Instead, we tackle of challenge of learning purely based on reinforcement. Given this formulation, the problem boils down to learning a policy $\pi:\mathcal{Z}\rightarrow\mathcal{A}$ that maximizes the expected discounted cumulative reward $E_{\pi}[\Sigma_{t=0}^{T-1}\gamma^{t}\mathcal{R}(z(t),a(t))]$.

Observation and Action Space: The observation space $\mathcal{Z}$ is composed of hand and object measurements. At step $t$, $z(t)$ contains $\xi^{b}_{1}(t)\in\mathbb{R}^{6}$, $\xi^{b}_{2}(t)\in\mathbb{R}^{6}$, $\xi^{h}_{1}(t)\in\mathbb{R}^{n_{1}}$, $\xi^{h}_{2}(t)\in\mathbb{R}^{n_{2}}$, $o_{1}(t)\in\mathbb{R}^{6}$, and $o_{2}(t)\in\mathbb{R}^{6}$, where $\xi^{b}_{1}$ and $\xi^{b}_{2}$ denote the first and the second hand bases’ $6D$ poses, $\xi^{h}_{1}$ and $\xi^{h}_{2}$ denote the first and the second hands’ joint states (e.g., the palm and fingers), and $o_{1}$ and $o_{2}$ represents the $6D$ poses of either two objects (e.g., stacking two cups), or two parts of the same object (e.g., bottle and bottle cap). The action at Step $t$ is given by $a(t)=[\hat{\xi}^{b}_{1}(t),\hat{\xi}^{b}_{2}(t),\hat{\xi}^{h}_{1}(t),\hat{\xi}^{h}_{2}(t)]$, composed of the target base poses ($\hat{\xi}^{b}_{1}$ and $\hat{\xi}^{b}_{2}$) and target joint positions ($\hat{\xi}^{h}_{1}$ and $\hat{\xi}^{h}_{2}$) for both hands. These actions are fed to a PD tracking controller to actuate both hands.

Note that our primary contributions pertain to the interaction phase of bimanual dexterous manipulation, in which two dexterous multi-fingered hands coordinate to complete the task after having grasped the necessary object(s). Most existing works focus solely on the interaction phase [40, 5, 41]. In Section. 3.6, we discuss how our approach can be extended to also tackle the acquisition phase, in which the hands learn to grasp objects before coordinating.

We begin by discussing the most straightforward approach one could take: a monolithic policy that utilizes all accessible environment states to plan actions for both hands: $\pi_{\mathrm{naive}}(\hat{\xi}^{b}_{1}(t),\hat{\xi}^{b}_{2}(t),\hat{\xi}^{h}_{1}(t),\hat{\xi}^{h}_{2}(t)|\xi^{b}_{1}(t),\xi^{b}_{2}(t),\xi^{h}_{1}(t),\xi^{h}_{2}(t),o_{1}(t),o_{2}(t))$. Training such a policy can be highly inefficient due to the high dimensionality of the observation and action spaces. Importantly, this monolithic implementation does not exploit the natural asymmetry found in most bimanual tasks. Below, we explain how AsymDex incorporates this insight.

When humans execute bimanual manipulation tasks, we tend to use our dominant hand to perform precise manipulations, and use a non-dominant hand to facilitate such manipulation [16, 17, 18]. For instance, when opening a bottle, we typically use our facilitating hand to move and reorient the bottle such that the bottle cap is closer to and oriented toward the dominant hand, which will then grasp the cap and uncap the bottle. During this cooperation process, our non-dominant hand moves and rotates the object with a firm grasp to facilitate the object manipulation by the dominant hand. Motivated by this, we assign different roles to each robot hand (i.e., a facilitating hand and a dominant hand) during bimanual manipulation.

Further, we make the observation that there tends to be no relative motion between the facilitating hand and the grasped object since the facilitating hand need only hold, move, and reorient the object (i.e., no in-hand reorientation). On the contrary, the hand dominant can interact freely with the object either directly or via another object. This observation suggests that the asymmetric manipulation and coordination strategy is neither dependent on nor influences the hand joints of the facilitating hand. As such, we can considerably reduce the observation and action spaces by accounting for asymmetry. Precisely, this observation allows us to define an asymmetric bimanual policy: $\pi_{\mathrm{asym}}(\hat{\xi}^{b}_{d}(t),\hat{\xi}^{b}_{f}(t),\hat{\xi}^{h}_{d}(t)|\xi^{b}_{d}(t),\xi^{b}_{f}(t),\xi^{h}_{d}(t),o_{1}(t),o_{2}(t))$ which use the facilitating hand to only to reposition and reorient the object. This considerable reduction in the dimensions of the observation and action spaces is likely to result in improved sample efficiency.

In addition to asymmetry, a key characteristic of bimanual dexterous manipulation is the synchronized motion of the two hands in which each hand moves in response to the other. In this section, we explain how we can further reduce the size of the observation and action spaces by defining relative and object-centric spaces that capture the relationships between the motions of two hands and the object(s) being manipulated. Indeed, the use of relative state spaces has shown to considerably benefit bimanual manipulation with simple end effectors [42, 43, 44, 23]. Furthermore, some of these prior work is limiting the relative space in a one-degree-of-freedom (1-DoF) action space [23], while AsymDex allows for complete 6-DoFs relative space.

Let $o_{f}=o_{1}$ be the state of the object being held by the facilitating hand, and let $o_{d}=o_{2}$ be the state of the object being manipulated by the dominant hand. We attach a coordinate frame to the object being held by the facilitating hand: $P_{f}$. Now, we can transform the observations $(\xi^{b}_{d}(t),\xi^{b}_{f}(t),\xi^{h}_{d}(t),o_{f}(t),o_{d}(t))$, which where originally defined in the world coordinate frame $P_{W}$, into the new coordinate frame $P_{f}$. Note that since there is no relative motion between the facilitating hand and the object that it’s holding, neither $o_{f}(t)$ nor $\xi^{b}_{f}(t)$ do not change in $P_{f}$. As such, both $o_{f}(t)$ and $\xi^{b}_{f}(t)$ can be neglected without losing any information. Now, transforming the remaining observations $(\xi^{b}_{d}(t),\xi^{h}_{d}(t),o_{d}(t))$ into the new coordinate $P_{f}$ yields $(\xi^{b}_{r}(t),\xi^{h}_{d}(t),o_{r}(t))$, respectively. Here, $\xi^{b}_{r}(t)$ denotes the 6D pose of the dominant hand base and $o_{r}(t)$ denotes the 6D pose of the object being manipulated by the dominant hand, both now defined relative to the object being held by the facilitating hand. Note that since $\xi^{h}_{d}(t)$ denotes the dominant hands’ joint states, it is not impacted by the change of coordinates. Similarly, we apply the same modifications to the action spaces of the asymmetry bimanual policy. This allows us to reduce the actions from $(\hat{\xi}^{b}_{d}(t),\hat{\xi}^{b}_{f}(t),\hat{\xi}^{h}_{d}(t))$ to $(\hat{\xi}^{b}_{r}(t),\hat{\xi}^{h}_{d}(t))$, where $\hat{\xi}^{b}_{r}(t)$ is the target relative pose of the dominant hand now defined relative to the object being held by the facilitating hand.

Incorporating the above change of coordinates in addition to leveraging asymmetry, allows us to define AsymDex’s policy as $\pi_{\mathrm{AsymDex}}(\hat{\xi}^{b}_{r}(t),\hat{\xi}^{h}_{d}(t)|\xi^{b}_{r}(t),\xi^{h}_{d}(t),o_{r}(t))$. Note that our formulation has significantly reduced the dimensions of both the state and action spaces, compared to the naive policy $\pi_{\mathrm{naive}}$ as defined in Section 3.2. See Appendix. A for RL algorithm and policy architecture.

To control the hand bases based on the target relative pose $\hat{\xi}^{b}_{r}(t)$ provided by $\pi_{\mathrm{AsymDex}}$, we designed a bimanual controller that computes both the target dominant hand base pose $\hat{\xi}^{b}_{d}(t)$ and the target facilitating hand base pose $\hat{\xi}^{b}_{f}(t)$ as follows

where $R^{o_{f}}_{world}$ denotes the rotational transformation from Frame $P_{f}$ to the world frame $P_{W}$, $dist(\cdot)$ denotes the difference between two 6D poses. and $\alpha$ is a hyperparameter that controls the involvement of each hand. Throughout our experiments, we used fixed $\alpha=0.5$. The pseudo-code of the training process is included in Alg. 1.

While our approach as explained thus far deals with the challenge of coordinating two hands to accomplish asymmetric dexterous manipulation tasks, it assumes the task begins with that the object(s) of interest have already been grasped. However, in practice, robots must be able to tackle the challenge of grasping the necessary objects before the interaction between the two hands and the objects can begin. We refer to this phase as the acquisition phase. Most recent works on bimanual dexterous manipulation often entirely ignore the acquisition phase and focus purely on the interaction phase [7, 5, 40]. In contrast, we demonstrate that our approach can seamlessly accommodate the acquisition phase by i) leveraging the observation that the acquisition phase doesn’t require the coordination of two arms, and ii) employing recent advances in learning to grasp. Specifically, we demonstrate that we can seamlessly integrate AsymDex with PDGM [45], which can efficiently learn multi-fingered grasping policies by leveraging pre-grasp poses (see Fig. 2). Details about the grasping reward design are available in Appendix. B. We begin by executing the grasping policy in isolation and then ”turn on” the asymmetric policy learned by AsymDex after the object has been firmly grasped by the facilitating hand. If the task requires the dominant hand to also grasp a second object, we employ the same method to train a grasping policy for dominant hand to acquire the object, but switch the control of the dominant hand’s joints over the asymmetric policy after the object has been grasped.

In this experiment, we focus on AsymDex’s effectiveness during the interaction phase. We initialize the environment such that the hands are at a pre-grasp pose around the objects using appropriate initial grasps (After the first timestep of the environment, the hand need to learn to catch and grasp the object from pre-grasp pose if it is not a facilitating hand). Note that this is a common assumption in recent methods that learn bimanual dexterous manipulation skills [5, 40]. Further, this allows us to isolate and examine AsymDex’s ability to learn to coordinate two multi-fingered hands. See Section 4.2 for the second experiment in which we also consider the challenge of acquiring the objects from a tabletop surface before interaction begins.

We compare AsymDex policy against the following baselines:

• $\bullet$

  Monolithic: This policy doesn’t make assumptions about the structure of bimanual manipulation (see Sec. 3.2). As such, this baseline allows us to examine the necessity and effectiveness of leveraging both the asymmetry in hand roles and the relative action and observation spaces.

• $\bullet$

  Asym-w/o-rel: This policy leverages asymmetry in hand roles, but learns over absolute observation and action places (see Sec. 3.3). As such, this baseline allows us to examine the necessity and effectiveness of relative action and observation spaces.

• $\bullet$

  Rel-w/o-asym: This policy leverages the relative observation and action places, but ignores asymmetry. As such, it allows us to examine the necessity and effectiveness of asymmetry.

We report the learning curves in Fig. 4 and success rates in Table 1. As can be readily observed, AsymDex is the only method that consistently performs well across all tasks, either performing comparably or outperforming all the baselines. While Rel-w/o-asym performs comparably to AsymDex on Stack, it is not able to match AsymDex’s performance on the other tasks. Both AsymDex and Rel-w/o-asym perform substantially better than the other two baselines across all tasks except Switch, where AsymDex performs much better than all three baselines. In fact, Asym-w/o-rel is able to learn an effective policy in only one task (Bottle cap), while Monolithic struggles on all four tasks. Curiously, Monolithic outperforms asym-w/o-rel on the Stack task. Qualitative analysis of rollouts reveals that, unlike Asym-w/o-rel, the Monolithic policy learns to use the facilitating hand’s fingers to orient the cup towards the bottom of the other cup.

Taken together, the above observations reveal a few insights. First, when used in isolation, neither asymmetry nor relative motion are sufficient across all tasks. In particular, while each might provide sufficient structure to accomplish some tasks, they prove to be less effective on other tasks. Second, the use of relative motion offers a larger boost in performance compared to asymmetry, likely due to the fact that relative spaces avoid unnecessary exploration (e.g., when the two hands move in parallel) while allowing the facilitating hand to exhibit more complex behaviors. Third, ignoring both asymmetry and relative motion hardly leads to success.

In this experiment, we evaluate AsymDex’s ability to incorporate the object acquisition phase in addition to the interaction phase. Specifically, we initialize the environment for each task such that the objects of interest are placed on a tabletop surface. As such, each method needs to learn both to grasp the necessary objects and to coordinate the two hands to complete the tasks.

For AsymDex, we follow the same strategy as introduced in Section 3.6. We compare AsymDex’s performance against the following baselines:

• $\bullet$

  1-stage-monolithic: This baseline uses a single policy to learn both the grasping and interaction phases for both hands, and thus allows us to investigate the benefits of AsymDex’s two-phase decomposition.

• $\bullet$

  2-stage-monolithic: This policy benefits from the same phase decomposition as AsymDex policy, but leverages neither asymmetry nor relative motion. As such, this baseline allows us to examine if this task can be solved merely with two-phase decomposition.

To ensure a fair comparison, we provide pre-grasp pose annotations to both baselines. Further, we ensure that the total number of env. interactions (the number of one stage or the sum of two stages) is the same across AsymDex and the baselines. See Appendix. B for details of the grasping learning.

We report the overall roll-out success rate of all methods for two tasks across five random seeds in Table. 2. We find that AsymDex significantly outperforms the other two baselines in both tasks, suggesting that combining phase decomposition with AsymDex’s other two design choices (asymmetry and relative state) results in policies that can effectively handle both the acquisition and the interaction phases of bimanual dexterous manipulation. The fact that 2-stage-monolithic baseline outperforms the 1-stage-monolithic baseline demonstrates the inherent benefits of phase decomposition. Our qualitative analysis of Block in cup task revealed that 1-stage-monolithic policy learns to tip the cup over and push the block towards the cup. In contrast, both the two-stage policies learn the expected behavior. This suggests that the phase decomposition nudges the grasping and interaction policies to learn reasonable behaviors that complement each other.