Relational Learning for Skill Preconditions

Since the input scene contains variable number of objects we use two different input formats for our baseline methods. First, we only add masks for each object, i.e., each voxel is labeled with a value of 1 if it is occupied by any object, while unoccupied voxels are labeled with 0. Thus, this input format only requires one channel in its formulation. For our second input format we add another channel to the input. This additional channel contains labels for each object in the scene. Thus, if a voxel is occupied by $i^{\prime}th$ object we label it with integer $i$. This input format consists of 2 channels. Additionally, we found using colored voxel representations to be extremely noisy since many objects in our experiments were quite close to each other.

We next discuss some changes to the input format for each specific task. For the food cutting task we add labels that represent the knife object and the food object if they are present in the scene. For all other objects in the scene we add a separate distractor label. For the block unstacking task we use two different labels. The block to be removed is marked by one label, while all the other objects in the scene use the other label. As before, we experiment with object ids for each block. But for the block unstacking task this method performed much worse and did not generalize at all when tested on a larger set of blocks.

The input to our object relational model $f_{\text{rel}}$ consists of 3 channels, where the first channel contains binary masks for both anchor and referrant objects. The second and third channel contains the binary masks for the anchor and referrant object separately. When using the object relation model on real world manipulation data we take the object masks of two objects $i$ and $j$. We set $i$ as the anchor object and $j$ as the referrant object. We transform the view such that the anchor object $i$ is in the middle of the scene and centered at $(0,0,0)$. This is similar to the formulation used to train $f_{\text{rel}}$ in simulation.

To get discrete relations directly using the input scenes with voxels we use an approach similar to [4]. We divide the space around each object (anchor) into multiple regions. These regions estimated using the bounds of the voxel objects. Specifically, we use $26$ regions since a 3D object can be divided into $3^{3}$ regions and we do not include the middle region which contains the object. For every other object (referrant object) in the scene, we find the number of voxels of this referrant object which intersect with each of the above regions. The region with most voxels is used to represent the discrete relation between the anchor and referrant object. Once we have the discrete relation for the objects, we use it in exactly the similar way as a continuous relation, i.e., we use the same network architectures for the precondition classification model.

As shown in the main paper we evaluate our approach against the real2sim baseline. The real2sim baseline we use the same voxel representations as used by our approach and other baseline methods. As the first step to implement the real2sim baseline we need to import the voxel representations from the real world to VREP [15]. Since these voxel representations do not conform to any fixed shape we cannot really import these object representations directly. Hence we import each voxel for every object separately into VREP. Thus each voxel is initially added as cuboid object. Once all the voxels of a given object are imported we group them together to form one composite object. Once all the objects in a scene have been imported we use it to test our preconditions. In addition to this, we also tried convex decomposition of the non-convex object shapes, but we found this to be incredibly slow and inaccurate for our purpose.

Since the VREP simulator does not support cutting we use the sim2real baseline only for the sweeping objects in a line and block unstacking task. To verify the preconditions for the sweeping objects in a line task we additionally add a virtual robot with a long cuboid attached to it. This robot consists of prismatic joints which allow us to move in the $XYZ$ axes. As before we use a spring-damper system to move the robot. At the beginning of the task we move the robot to its start position. We set the initial $X$ coordinate as the minimum $X$-position of all object corners minus some threshold $(0.02m)$. The $Y$ and $Z$ coordinates are set as the median of all object centers. This allows us to sweep through the scene in a consistent manner. To verify if the objects are indeed along a line, we evaluate if all the objects lie on one side (along the $X$-axes) of the robot handle.

We verify the preconditions for the block unstacking task by verifying if the blocks in the scene minus the block to be removed are stable or not. To achieve this we import all the blocks except the one to be removed into the simulator. Initially, all blocks are kept static. Once all of them have been imported we convert them to dynamic and allow them to interact with each other. To verify if the blocks are indeed stable we note each block’s position before and after allowing dynamic interactions. If any of the blocks move by more than a certain specified threshold we assume that the blocks are not stable. To find the best threshold value we do a grid search in the set $(0.0005,0.001,0.0015,0.002,0.003,0.004)m$. We found $0.0015$ and $0.002$ to give the best and very similar results.

For our baseline architecture we use the resnet model which has been adapted to operate on 3D input. We follow the architecture proposed in [18]. However, we slightly tweak the ResNet architectures in [18] for our purposes. For instance, we do not use the BatchNorm layers in the ResNet architectures since we empirically observed that they showed much worse performance. Additionally, in contrast to [18] for the first convolution layer we use the same stride for all three input channels. Besides these changes, we follow the same architecture as [18]. For other specific architecture details please look at Table 1 in [18]. The output of the ResNet model is forwarded through 2 fully-connected layers with outputs of size 64 and 1 respectively.

We also use another baseline similar in style to VGG [26], i.e., our model consists of a series of convolutions and ReLU non-linearities. More-specifically our architecture consists of 6 convolution layers, each of which operates on 3D input. Table 6 lists the different parameters for each of these convolution layers. The output of each convolution layer is passed through ReLU non-linearity. The final output of the convolution is flattened to a 512 dimensional vector. This vector is then passed through multiple fully-connected layers with output size $[256,64,1]$.

Relational Network: We use the below form of relation network function,

$\displaystyle RN(S)=g_{\phi}\left(\sum_{i,j}f_{\theta}\left(f_{\text{rel}}(o_{i},o_{j})\oplus f_{\text{rel}}(o_{j},o_{i})\right)\right),$

(1)

where $\oplus$ represents the concat operation, the function $f_{\theta}$ consists of two fully-connected (FC) layers with ReLU non-linearity in-between, and $g_{\phi}$ also consists of 2 FC layers with ReLU non-linearity.

For our relation network model we implement the function $f_{\theta}$ using a two layer neural network with fully connected layers with outputs of size 128 and 32 respectively. We implement $g_{\phi}$, which acts as an accumulator, again by using two fully connected layers with outputs of size 16 and 1 respectively. We use the ReLU non-linearity for both $f_{\theta}$ and $g_{\phi}$.

Graph Neural Network: For our GNN model we use the following update model to process the output at each node,

$\displaystyle v^{(1)}_{i}=f_{\psi}\left(v_{i}\oplus\sum_{j}\left(f_{\text{edge}}(v^{(0)}_{i},v^{(0)}_{j},e_{ij})\right);\psi\right)$

(2)

We add separate node and edge models for our GNN model. Our edge model takes the edge input and passes it through a fully connected layer with output size of 128, followed by ReLU, and finally with another fully connected layer which outputs a continuous embedding for this edge. These edge embeddings are used for the next graph layer as well as for each node being processed. Our node model initially sums the edge embeddings for all the edges connected to this node. This sum of edge embeddings is then concatenated with the original node input and passed through a fully-connected layer with output size 128, followed by ReLU and then another fully conected layer with output size 128. This is similar in design to the edge update model. We stack two layers of the above GNN model for precondition learning. The first layer uses an input of size 256 and outputs representations of size 128. The next layer uses an input of size 128 and outputs representations of size 128. We sum the node and edge embeddings of the last GNN model and concatenate them together to form an input of size 256. This input is then passed through 2 fully conected layers with output size of 64 and 1 respectively.

Our object relation model initially uses the ResNet-18 model of [18]. As before, we make two changes to this model, we use a stride of $(2,2,2)$ for the first convolutional layer, and we do not use the BatchNorm layers. The convolutional layer output is flattened into a vector of size $1536$ which is then projected through 2 fully-connected layers with output size $512$ and $256$. Thus, we get the final relational embedding of size $256$. We concatenate this with the action vector and pass them through three fully connected layers of size 128, 64 and 9 respectively. The final output of size 9 consists of the predicted position changes of size 3, predicted orientation changes of size 3 and predicted mean contact position of size 3.

As noted previously, for our embedding network we use a 3D CNN with a ResNet [17] based architecture [18], specifically we use the ResNet-18 architecture. The output of the ResNet18 model is then downsampled using a linear layer to a size of 256. To train the above embedding model, we use the Adam optimizer [28] and set an initial learning rate of $3e-4$ and a batch size of 256. Our overall loss function for the

The total loss for the object relation model can be written as,

	$\displaystyle\mathcal{L}$	$\displaystyle=\lambda_{\text{pos}}^{\text{cont}}\times\mathcal{L}_{\text{pos}}^{\text{cont}}+\lambda_{\text{orient}}^{\text{cont}}\times\mathcal{L}_{\text{orient}}^{\text{cont}}$
		$\displaystyle+\lambda_{\text{pred}}\times\mathcal{L}_{\text{pred}}+\lambda_{\text{orient}}\times\mathcal{L}_{\text{orient}}+\lambda_{\text{contact}}\times\mathcal{L}_{\text{contact}},$

where the first set of losses $\mathcal{L}_{\text{pos}}^{\text{cont}}$, $\mathcal{L}_{\text{orient}}^{\text{cont}}$ are the position and orientation based contrastive losses. The next set of losses $\mathcal{L}_{\text{pred}}$, $\mathcal{L}_{\text{orient}}$, $\mathcal{L}_{\text{contact}}$ are the direct supervised losses. When training the object relation model with all the above losses we set, $\lambda_{\text{pos}}^{\text{cont}}=2$, $\lambda_{\text{orient}}^{\text{cont}}=2$. For the supervised losses we set $\lambda_{\text{pos}}=1$, since orientation values change less we set $\lambda_{\text{orient}}=10$, and finally $\lambda_{\text{contact}}=1$. Alternatively, when training the model with just contrastive losses we set $\lambda_{\text{pos}}^{\text{cont}}=\lambda_{\text{orient}}^{\text{cont}}=1$.

Contrastive Loss: To implement the contrastive loss we use a batch all strategy, i.e., instead of explicitly sampling anchor, positive and negative pairs separately, we sample a batch of data and compare all scene triplets. We compare normal and adaptive actions for each triplet pair separately, i.e., for each scene in the triplet we compare their adaptive action effects with the other scene’s adaptive action effects only. To compare normal actions with different action magnitudes we use a threshold of $0.04m$ to classify similar actions. Thus, scenes with difference in action magnitude greater than the above threshold are not compared directly in contrastive losses.

As noted in the paper, we use the following to compare the action effects on position changes,

$\displaystyle\Delta p^{r}=\frac{\Delta p^{\text{observed}}}{\Delta p^{\text{desired}}},$

(3)

where $\Delta p^{\text{observed}}$ is the observed change in referrant object center position, and $\Delta p^{\text{desired}}$ is the desired change i.e. action vector. We set $\Delta p^{\text{desired}}$ to 1 for normal actions, while for adaptive actions it is set to the voxel distance between objects centers. To find scenes with similar action effects we use a threshold of $0.2$. More precisely, given two scenes ($m$ and $n$) and their action effects in terms of $\Delta p^{r}$, we say these two scenes are similar if their action effects are below the threshold for all actions, $|\Delta p^{r}_{m}-\Delta p^{r}_{n}|<0.2$. Analogously, if $|\Delta p^{r}_{m}-\Delta p^{r}_{n}|>0.21$ we set these two scenes as dissimilar. For orientation changes we set $\Delta\theta_{\text{sim}}=0.004$ and $\Delta\theta_{\text{diff}}=0.008$. Finally, we set the contrastive loss margin $\gamma=2.0$ for both position and orientation changes. Table 7 lists the above parameters for contrastive losses.

Our Model: For our RN and GNN based models, we use the Adam optimizer [28] and set an initial learning rate of $3e-4$. For accelerated learning, we also test with a larger learning rate of $1e-3$. Since the precondition model is trained on less data we use a smaller batch of size 8 only. We also decay the learning rate with 0.995 after every epoch.

Baselines: For the baseline models we use the Adam optimizer as well. We experiment with both random initialization as well as kaiming initialization for the resnet based baseline methods [17]. We do a grid search to find the best learning rate between $[1e-3,1e-4]$ and report numbers with it. Also, given the small train data size we test with multiple batch sizes of 4, 8 and 16 and report numbers with the best results.

Table 6 lists the different parameters used to train the precondition models.

Table 8 shows the results for the food cutting experiment when we have 3 distractor objects in the test set, but only upto 2 distractor objects in the train set. As observed above, the GNN based model performs the best with a wt. F1-score of $0.94$. Additionally, the baseline models also perform well with a maximum wt. F1-score of $0.902$. The good performance of the baseline models on food cutting task can be attributed to the fact that there exist only two main objects in the scene (food and the knife). Since we provide separate labels to both of these objects, the baseline models only needs to focus on these objects and hence does not require complex compositional reasoning. Thus, even with increasing number of distractor objects, the baseline model still performs quite well.

For the block unstacking task we also experiment with the scenario where the train set contains larger number of objects as compared to the test set. Table 9 shows results when the train set contains 6 or 7 objects while the test set contains only 4 objects. Interestingly, the performance of our GNN based models also reduces in this scenario. We get a max. wt. F1-score of $0.825$ only, which is worse as compared to the previous train-test splits. However, our GNN based models still perform better than the visual baseline models, whose maximum wt F1 score is $0.784$. This result indicates that although structured representations are extremely useful, we cannot assume that training on a large set of objects will automatically generalize to fewer objects as well. The real2sim baseline performs the best (wt. F1 score: $0.889$) in this setting. This is because using 4 blocks, we can only create a limited number and types of block stacks. Hence, given only stacks with 3 blocks (since we remove 1 block before exporting into the simulator) the Real2Sim baseline is able to perform quite well. Additionally, a small number of blocks also result in fewer perception errors, these errors when transferred to simulation do not affect the precondition output significantly. This is also similar to the real2sim results for sweeping task (main paper) wherein we observe that larger number of objects lead to worse performance.

Figure 8 (Right) shows example scenes which show the complex reasoning required for solving the block unstacking task. The only difference between the two scenes in Figure 8 (Right) is in the position of the top block which results in different ground truth precondition labels. While our precondition model is correctly able to predict the preconditions in both scenes, the visual baseline model always predicts false. The above example clearly shows that our model is able to perform complex reasoning using the learned object relational embeddings.

Why DiscreteRel baseline performs poorly? We also look at why discrete relations are often insufficient to model relations for manipulation tasks. To show this we use the block unstacking task, in the simplest possible setting – with only 3 objects in the scene. Figure 9 plots multiple different pairs (A, B, C) of such scenes. Each of these scenes are part of the train set with 3 objects. Also, for each scene pair the extracted discrete relations are similar (aside from image A, which shows the same scene). This is because each referrant object occupies relatively similar amount of space around every anchor object. However, despite similar discrete relations, both scenes in each scene pair (A, B, C) have different precondition outputs (when the black block is removed). For instance, in Figure 9 (B) removing the black box from left figure (scene) results in a stable block configuration. While for the right scene in Figure 9 (B), this results in the orange block falling and hence not a stable configuration. These results show the inability of discrete relations to differentiate between seemingly similar scenes but with very object arrangements. Finally, each of the above scene only contains 3 objects. With increasing number of objects in the scenes, the ambiguity associated with discrete relations increases further, which results in an overall poor performance across all different task configurations.