AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions

Given a set of interactions $\mathcal{I}=\{I_{1},I_{2},\cdots\}$ as inputs, the Adaptive Information Encoder $\mathcal{E}_{AAP}$ outputs a 128-dim hidden information representation $z_{\mathcal{I}}$ ($z$ for brevity). It first encodes each interaction $I_{i}$ using the input encoders mentioned before, and then uses an MLP network to encode the features into a 128-dim latent code $z_{I_{i}}$ representing the hidden information extracted from $I_{i}$. As different interactions contain different amount of hidden information, we use another MLP Network to predict an attention score $w_{I_{i}}\in\mathbb{R}$ for each interaction. To get a summarized hidden information from a set of interactions, we simply computes a weighted average over all $z_{I_{i}}$’s according to the weights $w_{I_{i}}$’s and use the resulting feature as $z_{\mathcal{I}}$. Formally, we have $z_{\mathcal{I}}\leftarrow\left(\sum_{i}z_{I_{i}}\times w_{I_{i}}\right)/\left(\sum_{i}w_{I_{i}}\right)$.

Given the object partial point cloud observation $O$, an arbitrary interaction point $p\in O$, an arbitrary gripper orientation $R\in SO(3)$ and the latent code $z$, the Adaptive Critic Network $\mathcal{C}_{AAP}$ predicts an AAP action score $s_{u|z}^{AAP}\in[0,1]$ indicating the likelihood for the success of the interaction action $u$ given the interaction information $z$. It first encodes the input $\{O,P,R\}$ using the input encoders as mentioned before and then employs an MLP network to predict AAP action score $s_{u|z}^{AAP}$, taking the concatenated features together with $z$ as inputs. A higher AAP action score $s_{u|z}^{AAP}$ for action $u$ indicates a higher chance for $u$ to succeed in accomplishing the given manipulation task.

Given the input object partial point cloud $O$, an arbitrary point $p\in O$, and the latent code $z$, the Adaptive Affordance Network $\mathcal{D}_{AAP}$ predicts an actionability score $a_{p|z}^{AAP}\in[0,1]$ at point $p$. It first encodes the input $\{O,p\}$ using the aforementioned input encoders and then uses an MLP network that takes the concatenated features together with $z$ as inputs and produces an actionability score $a_{p|z}^{AAP}$ as the output. A higher actionability score $a_{p|z}^{AAP}$ indicates a higher chance to successfully interact on point $p$.

Given the input object partial point cloud $O$, an arbitrary interaction point $p\in O$, an arbitrary gripper orientation $R\in SO(3)$, the latent code $z$, and the AAP action score $s_{u|z}^{AAP}$ produced by $\mathcal{C}_{AAP}$, the AIP Critic Network $\mathcal{C}_{AIP}$ predicts the AIP action score $s_{u|z}^{AIP}\in\mathbb{R}$ of $u$. It first encodes the inputs $\{O,p,R,s_{u|z}^{AAP}\}$ using the input encoders and then uses an MLP network that takes the concatenated features together with $z$ as inputs and generates an AIP action score $s_{u|z}^{AIP}$ for the action $u$. A higher AIP action score suggests that the action $u$ may query more unknown yet interesting hidden information and thus is worth exploring next.

Given the input partial shape observation $O$, an arbitrary interaction point $p\in O$, the latent code $z$, and the AAP actionability score $a_{p|z}^{AAP}$ at point $p$ estimated by $\mathcal{D}_{AAP}$, the AIP Affordance Network $\mathcal{D}_{AIP}$ predicts the AIP actionability score $a_{p|z}^{AIP}\in\mathbb{R}$ at point $p$. It first encodes the inputs $\{O,p,a_{p|z}^{AAP}\}$ using the aforementioned input encoders and then employs an MLP network to predict an AIP actionability score $a_{p|z}^{AIP}$, taking the concatenated features together with $z$ as inputs. A higher AIP actionability score at $p$ indicates more unknown yet helpful hidden information may be obtained by executing an interaction at $p$.

To propose an action $u_{t+1}=(p_{t+1},R_{t+1})$ for the next interaction, given the hidden information $z$ and the input shape partial point cloud $O$, we first obtain the AIP actionability heatmap $A_{p|z}^{AIP}$ for every point $p\in O$ predicted by the AIP Affordance Network $\mathcal{D}_{AIP}$ and then select the point $p_{t+1}\leftarrow p_{*}$ with the highest AIP actionability score $a_{p_{*}|z}^{AIP}$. Then, we sample 100 random actions $\{u_{1},u_{2},\cdots,u_{100}\}$ at $p$ using the Where2Act’s pre-trained Action Proposal Network, use our AIP critic network $\mathcal{C}_{AIP}$ to generate the AIP action scores $s_{u_{i}|z}^{AIP}$ for each action $u_{i}$, and then choose the action $u_{t+1}\leftarrow u_{*}$ with the highest AIP action score $s_{u_{*}|z}^{AIP}$.

The AIP procedure for generating few-shot interactions stops when a preset budget is reached or the maximal AIP actionability score is below a certain threshold (e.g., 0.05).

To supervise $\mathcal{C}_{AAP}$, we use a standard binary cross entropy loss, which measures the error between the prediction of $\mathcal{C}_{AAP}$ and target part’s ground truth motion $m$ of an interaction $I$. Specifically, given the hidden information $z$, a batch of interaction observations $\mathcal{I}=\{I_{1},I_{2},...,I_{B}\}$ where $I_{i}=\{O_{i},u_{i},m_{i}\}$, and the AAP action score prediction $s_{u_{i}|z}^{AAP}$ for each interaction $I_{i}$, the loss is defined as

where $r_{i}=1$ if $m_{i}>\tau$ (e.g., $\tau=0.01$) or $r_{i}=0$ rendering a binary discretization for each interaction outcome.

To train the Adaptive Affordance Network $\mathcal{D}_{AAP}$, we apply an $\mathcal{L}_{1}$ loss to measure the difference from the predicted score $a_{p|z}^{AAP}$ to the ground truth. To estimate the ground truth actionability score for $p$, we randomly sample 100 actions at $p$ according to the pre-trained Where2Act Action Proposal Network, predict the AAP action scores $s_{u|z}^{AAP}$’s of these actions $u$’s using the Adaptive Critic Network $\mathcal{C}_{APP}$, and take the average of the top-5 scores as the ground truth actionability score.

To supervise the AIP Critic Network $\mathcal{C}_{AIP}$, we use an $\mathcal{L}_{1}$ loss to measure the difference between our predicted AIP action score $s_{u|z}^{AIP}$ and the ground truth AIP action score $gt_{u|z}^{AIP}$. We design a greedy yet effective way to estimate the ground-truth scores. Given a set of interactions $\mathcal{I}_{T}=\{I_{1},I_{2},\cdots\}$, to generate $gt_{u_{i}|z}^{AIP}$ for an interaction action $u_{i}$, we respectively encode two interaction subsets $\mathcal{I}_{i-1}=\{I_{1},I_{2},\cdots,I_{i-1}\}$ and $\mathcal{I}_{i}=\{I_{1},I_{2},\cdots,I_{i}\}$ into latent codes $z_{\mathcal{I}_{i}}$ and $z_{\mathcal{I}_{i-1}}$. Then, we feed $z_{\mathcal{I}_{i}}$ and $z_{\mathcal{I}_{i-1}}$ as the conditional inputs to the Adaptive Critic Network $\mathcal{C}_{AAP}$ separately and count the difference of the AAP action scoring loss $\mathcal{L}_{\mathcal{C}}^{AAP}$ as the ground truth of AIP action score $gt_{u_{i}|z_{\mathcal{I}_{i-1}}}^{AIP}$. More concretely, let the AAP action scoring loss conditioned on $z_{\mathcal{I}_{i}}$ and $z_{\mathcal{I}_{i-1}}$ respectively be $\mathcal{L}_{\mathcal{I}_{i}}$ and $\mathcal{L}_{\mathcal{I}_{i-1}}$. We define the ground truth AIP action score $gt_{u_{i}|z_{\mathcal{I}_{i-1}}}^{AIP}\leftarrow\mathcal{L}_{\mathcal{I}_{i-1}}-\mathcal{L}_{\mathcal{I}_{i}}$. The AIP action score is trained to regress an estimated positive influence of executing $u$ on the AAP action score predictions, where an action giving more influence is preferred as it helps discover more hidden information useful to the task.

To train the AIP Affordance Network $\mathcal{D}_{AIP}$, we use another $\mathcal{L}_{1}$ loss. For each position $p\in O$, we sample 100 actions $u_{i}$’s using the pre-trained Where2Act Action Proposal Network, obtain the AIP action scores $s_{u_{i}|z}^{AIP}$’s of these actions $u_{i}$’s predicted by the AIP Critic Network $\mathcal{C}_{AIP}$, and use the average of the top-5 scores as the regression target.

We iteratively train the AAP module and AIP module until a joint convergence since the update of the subnetworks in one module will affect the training of the subnetworks in the other module. More specifically, the update of $\mathcal{C}_{AAP}$ and $\mathcal{D}_{AAP}$ in the AAP module will affect the ground-truth AIP action scores, while the update of $\mathcal{C}_{AIP}$ and $\mathcal{D}_{AIP}$ in the AIP module will change the proposed interactions used to generate $z$ in the AAP module. Therefore, our final solution is to train the AAP and AIP modules iteratively. The procedure starts with firstly training the AAP module using randomly sampled interactions. We then train the AIP module to learn to propose more efficient and effective proposals. Then, with the trained subnetworks in the AIP module, we finetune the AAP module with the proposed few-shot interactions. We iteratively alternate the training until both modules converge.

Following the settings of Where2Act [18], we conduct our experiments in the SAPIEN [37] simulator equipped with NVIDIA PhysX [22] simulation engine and the large scale PartNet-Mobility [20] dataset. We use 972 articulated 3D objects covering 15 object categories, mostly following Where2Act, to carry out the experiments. The dataset is divided into 10 training and 5 testing categories. The shapes in the training categories are further divided into two disjoint sets of training and test shapes. See supplementary for detailed statistics.

Following Where2Act [18], we perform experiments over all object categories under different manipulation action types. We train one network for each downstream manipulation task over training shapes from the 10 training object categories and evaluate the performance over test shapes from the training categories and shapes from unseen test categories. Besides, to further demonstrate the effectiveness of our method, we conduct two additional experiments under challenging tasks with clear kinematic ambiguity, each of which is conducted over a single object category: 1) pulling closed doors of cabinets that cannot be easily distinguished which side to pull open; 2) pushing faucets with uncertainties which direction to rotate (clockwise or counter-clockwise). These experiments are particularly interesting yet challenging cases on which previous work Where2Act [18] fail drastically and we hope to test our framework.

Following Where2Act, we abstract away the robot arm and only use a Franka Panda flying gripper as the robot actuator. The input shape point cloud is assumed to be cleanly segmented out. To generate the input partial point cloud scans, we mount an RGB-D camera with known intrinsic parameters 5-unit-length away pointing to the center of the target object.

To simulate manipulating shapes with uncertain dynamics, we randomly change the following three physical parameters in SAPIEN: 1) the friction of the target part joint, 2) the mass of the target part, and 3) the friction coefficient of the target part surface. For the ”pulling closed door” task, we manually select the cabinets whose doors have no clear handle geometry in the PartNet-Mobility dataset [37], and set the poses of those doors to be closed. The gripper cannot tell which side to pull open the door because it is impossible to tell whether the axis position is on the left or right of the door from passive visual observations. For the ”pushing faucet” task, we randomly set the rotating direction of the faucet switch to be in one of the following three modes: only clockwise, only counter clockwise, or both ways.

We compare our framework with several baselines (see supplementary for more detailed descriptions for the baseline designs):

• -

  Where2Act: the original method proposed in [18] where only the pure visual information is used for predicting the visual actionable information and no interaction data is used at all during test time;

• -

  Where2Act-interaction: the Where2Act method augmented with four additional interaction observations as inputs where the interaction positions are uniformly sampled over the predicted affordance heatmap using Furthest Point Sampling (FPS) and we train an additional encoding branch similar to the Adaptive Information Encoder to extract the additional input feature;

• -

  Where2Act-adaptation: the Where2Act method augmented with a heuristic based adaptation mechanism to replace the AAP module where given the interaction observations we locally adjust the predictions for similar points;

• -

  Ours-random: a variant of our proposed method that we use randomly sampled interaction trials over the geometry instead of the AIP proposals;

• -

  Ours-fps: a variant of our proposed method that we use FPS to sample over the predicted affordance for interactions instead of the AIP proposals.

We compare to Where2Act to show that the few-shot interactions indeed help to remove ambiguities and improve the performance. The Where2Act-interaction baseline uses FPS to sample interaction positions relying on the intuition that it may sparsely sample over all possible regions of uncertainties. Comparing to this baseline helps validate that our iterative framework learns smarter strategies to perform more effective interaction trials. Furthermore, the Where2Act-adaptation baseline helps substantiate the effectiveness of our proposed AAP module, while the Ours-random and Ours-fps baselines are designed to verify the usefulness of the proposed AIP module.

Besides, we compare to an ablated version of our method to verify the significance of iterative training between the AAP module and the AIP module.

• -

  Ours w/o iter: an ablated version that trains the whole framework without the iteratively training process.

Following Where2Act [18], we use the F-score, balancing the precision and recall, to evaluate the predictions of the Adaptive Critic Network $\mathcal{C}_{AAP}$, and use the sample-successful rate (Sample-Succ) to evaluate the performance of the Adaptive Critic Network $\mathcal{C}_{AAP}$ and the Adaptive Affordance Network $\mathcal{D}_{AAP}$. To compute the sample successful rate, we apply the learned test-time strategy to fill $\mathcal{I}$ and then use the extracted hidden information $z$ as the conditional input to $\mathcal{C}_{AAP}$ and $\mathcal{D}_{AAP}$. After that, we randomly select a point to interact from the group of points with the top-100 actionability scores $a_{p|z}^{AAP}$, sample 100 actions $u_{i}$’s at $p$, obtain the AAP action scores $s_{u_{i}|z}^{AAP}$’s of these actions $u_{i}$’s predicted by the AAP Critic Network $\mathcal{C}_{AAP}$, and then choose the action $u_{i}$ with the highest $s_{u_{i}|z}^{AAP}$ to execute. We perform 10 interaction trials per test shape and report the final sample-succ rate as the percentage of sampling successful interactions in simulation.

In Table 2, comparing against the ablated version of our method Ours w/o iter that trains the whole system without the interactive training process, we see that Ours-final achieves better results in most cases, which proves the effectiveness of the iterative training scheme. By iteratively alternating the training between the AAP module and the AIP module, the networks would be trained under the distribution of test-time interactions and thus achieve improved performance. Our method can generalize well to novel shapes and even shapes from unseen object categories through the scores in test category.

Finally, we perform real-world and real-robot experiments to show that our method can to some degree work beyond synthetic data. We use a Franka panda robot with a two-finger parallel gripper as the actuator to pull open a cabinet door in the real world. Fig. 6 presents the results that our system proposes two interaction trials to inquire more information about this real-world cabinet and successfully learns to adapt to the posterior predictions.

Please refer to the supplementary materials for a video better illustrating this example, more experiment settings, more example results, and more experiments with additional analysis.