Spatial Reasoning from Natural Language Instructions for Robot Manipulation

To evaluate the language network, we assume that the object positions and sizes are known. We have experimented with two datasets. We use the publicly available Blocks dataset [2]. Additionally we synthesize a diagnostic dataset to test our model performance for more diverse and complicated visual scenarios. We briefly explain the datasets as follows:

Blocks 2D: In the blocks dataset[2], each sample has a natural language instruction and the positions of all 20 blocks as the input and the labels are the position from which the block must be picked up (start position) and the location where the block must be placed (end position). A sample instruction: “Pick up block 9 and place it above block 8”. The corpus has a training / development / test distribution of 3712 / 699 / 705 instructions. We allow the start and end positions to be real valued locations and do not restrict them to be on the grid, but unlike [36] which considers 3D block structures, we retain the 2D world. We include only those instructions in the blocks dataset that have fewer than 40 tokens in the instruction.

Synthetic Dataset: The Blocks dataset has a few limitations: (a) All the blocks are uniquely numbered and only one instance of each block is in the scene, (b) In most of the cases, the instructions are such that the goal location can be obtained by finding the appropriate anchor block and a relative offset direction from a predefined set, and (c) The sizes of the blocks are identical. To diagnose whether the proposed model is capable of reasoning about a variety of other spatial relationships, object attributes (e.g. size), abstract concepts (e.g. row or column) as well as scenes with multiple instances of each object, we build a synthetic dataset. We follow a similar approach proposed in [37] and generate $\sim$42,000 unique instructions with varied scenes containing objects of sizes randomly chosen between 1.0 to 3.0 and divide it into train / dev / test distribution of 29465 / 4216 / 8416. Each scene contains a maximum of 12 distinct objects and up to a total of 24 objects. Some of the representative templates used to generate instructions are as follows: (i) Pick the largest / smallest $\langle$obj$\rangle$, (ii) Pick the leftmost / rightmost $\langle$obj$\rangle$ from the row of $\langle$obj$\rangle$s, (iii) Pick the $\langle$obj_pos$\rangle$ $\langle$obj$\rangle$ from top row, (iv) Pick the $\langle$obj1$\rangle$ above / below / to the left of / to the right of $\langle$obj2$\rangle$. For this dataset, we predict only the start location and ignore the end location.

We define two evaluation metrics to compare the baseline results quantitatively with ours.

Mean Squared Error (MSE): It is the average over the squared distances between the gold and predicted locations. Define the prediction and gold locations for $i^{th}$ instruction as $L_{P}$ and $L_{G}$ respectively. Then $MSE=\frac{\sum_{i=1}^{N}{\left\|L_{P}^{i}-L_{G}^{i}\right\|}^{2}}{N}$. The center of the simulated world is at $(0,0)$ and restricted in the range of $[-1,1]$ in both $x$ and $y$ directions.

Tolerable Accuracy (TA): In majority of the real world applications, it is acceptable even if the predicted and the target locations do not exactly match but the distance between them is within a certain tolerable (application-specific) range. To account for this fact, we propose a new metric Tolerable Accuracy. A prediction is considered to be correct if the distance error (in both $x$ and $y$ directions in simulated world) is less than a tolerance value tol. We count the number of correct prediction instances out of the total $N$ instructions to evaluate TA. Mathematically, $TA=\frac{\sum_{i=1}^{N}{\mathds{1}\{|L_{P}^{i}-L_{G}^{i}|_{x,y}<tol\}}}{N}$. $\mathds{1}\{z\}$ is an indicator function and it’s value is $1$ when $z$ is true, otherwise $0$.

We compare our proposed model with the following baseline algorithms:

Center: This model assumes complete knowledge about the start location and places the block at the middle of the table.

Random: The Random baseline decides both the start and end locations to be random. The Center and Random are two simple baselines taken from [1].

LSTM [37]: The word embeddings of the instructions obtained from an embedding layer is passed through a word-level multi-stage LSTM model. The output from the last layer of LSTM is the instruction embedding and passed through a MLP to obtain the final outputs. This model does not use the image information, hence can identify object positions only through the presence of any bias in the instructions. We included this baseline to show that the dataset is not biased in such a way that the instruction alone can predict the start and end positions.

LSTM+CNN [37]: This is an end-to-end approach that takes the image and the text instruction as input and directly predicts the start and end positions. As above, the instruction embedding is obtained from the last-layer hidden state of LSTM and the image is encoded using CNN features. They are concatenated and fed into a MLP to get the prediction of the start and end locations.

LSTM+CNN+SA [37]: The encodings for image and instruction are generated as above and then soft spatial attention is employed to get the final representation. A MLP takes this as input and produces the required output.

LSTM+UNet [7]: The Ling-UNet[7] is an end-to-end architecture that takes the RGB pixels from the camera and the instruction text and directly regresses the start and end positions. This is similar to the LSTM+CNN architecture above, but a UNet is used instead of the CNN and vectors derived from the instruction text using the LSTM are convolved with skip connection activations of the UNet. Similar to the proposed architecture, the start and end positions are extracted from the last layer of the UNet using the spatial-softmax layer.

RNN-NoAttn-NoGround [1]: The model architecture has a single-layer RNN at its heart. It takes as input the instruction and predicts the start object, anchor object, and chooses from 8 pre-defined offsets corresponding to the 8 adjacent positions (right, bottom-right, etc.). The start position prediction is simply the position of the predicted object to be picked up. The end position is obtained by adding the position of the predicted anchor object and the predicted offset. Note that this model is not grounded because the predictions of the start, anchor, and offset are invariant to the positions of the objects in the scene and depend only on the natural language instruction. Furthermore, it cannot distinguish between two or more instances of the same object in the scene. For fair comparison, we train this model on our dataset.

LangNet-Attn-NoGround: We extend the model in Bisk et. al [1] by introducing an attention mechanism over a multi-layer BiLSTM.

Lang-FCNet: This model differs from the proposed Lang-UNet model in that object positions are represented as a list of 2D points rather than as a binary image, and they are passed through fully connected layers rather than the convolutional hourglass network (U-Net).

Better generalisation with Attention: The language network is trained on the blocks dataset using the Adam optimizer with mean absolute error loss, with learning rate 1e-3, and weight decay 1e-9. A few sample predictions of the Lang-UNet (BiLSTM model with attention) model are visualized in Figs. 4 and 5. For one example, the attention weights for the different tokens of the instruction when predicting the start, anchor, and offsets are shown in Fig. 6. Table I compares the performance of the proposed approach with the baselines. We observe that the attention-based models - LSTM-Attn-NoGround and Lang-UNet perform significantly better than the RNN-NoAttn-NoGround model that has no attention component, especially for end co-ordinate prediction. Fig. 6 shows the attention mechanism is able to attend on the correct offset and target block and intuitively explains the reason behind improved performance for end co-ordinate prediction. Note that the accuracy of predicting the end location is worse than for the start location. This is because in most textual instructions, the start location is simple and unambiguous (“place block 4…” or “pick up block 3…”), whereas the target is more complex (last example in Fig. 4) and sometimes ambiguous (“the 14th block moved next to the 12th block”). It also suggests why attention is more important for predicting the end location than the start in case of Blocks dataset.

Mitigating the effect of bias through Attention: We noticed that the Blocks dataset is biased with some block numbers more frequently being associated with some offsets (such as “north of”) than others. Because of this, the RNN-NoAttn-NoGround model overfits and always predicts the same offset when some block numbers are present in the instruction and ignore the actual content of the text. In contrast, the LSTM-Attn-NoGround model and the Lang-UNet are forced to attend to the offset token in the instruction and gets it right. For example, all the models correctly predict the output for “Move block 4 above block 5”. But, when the block number is changed to “Move block 11 above block 5”, only Lang-UNet and LSTM-Attn-NoGround make the correct prediction. To quantify this, we selected 20 simple examples such as the above example from the validation set. The Lang-UNet, RNN-NoAttn-NoGround models correctly predicted 19 and 19 examples respectively. But when we randomized the block numbers in those instructions, the number of correct predictions were 19 and 6 respectively. Attention was necessary to retain performance and demonstrates its usefulness in such biased datasets.

Visual grounding is essential for diverse and complex data: We note that the ungrounded models which use the natural language instruction alone and do not use object positions perform reasonably on the Blocks dataset. However, they perform poorly on the synthetic dataset because it is not possible to predict the correct position for an instruction such as “Pick the apple to the left of the orange” without actually using the object positions. The proposed Lang-UNet peforms well on both the Blocks and the synthetic datset, but slightly underperforms RNN-NoAttn-NoGround on the Blocks dataset due to the quantization in representing the object positions. Moreover, the LSTM-Attn-NoGround and RNN-NoAttn-NoGround models use hard-coded offsets ($\{0,0.166,-0.166\}$ in both X and Y directions) that are added to the position of the anchor object to predict the end position and are thus specialized to the Blocks dataset, whereas the proposed LangUNet does not use such hard-coded offsets.

Benefits of Grid representation over List: The Lang-FCNet takes the output from the localisation network as a list of 2D points, whereas the Lang-UNet considers a 2D binary grid representation as explained in Sec. IV. Fig. 2 depicts the differences between the output formats from the localisation network. On examples such as “Pick up the banana from the row of bananas”, Lang-UNet is successful in predicting the location of the banana because the convolutional layers in the U-Net help in recognizing “rows”, whereas Lang-FCNet performs poorly. From Table I, we infer that the performance improvement in Lang-UNet over Lang-FCNet, particularly for the synthetic dataset, is due to the binary grid representation because it provides the model a better way to understand the object positions and the relative spatial relationships amongst each other compared to a list of 2D coordinates. Our empirical evaluation in Table I also suggests the superiority of the pipelined approaches (Lang-FCNet and Lang-UNet) over the end-to-end models (LSTM+CNN and LSTM+CNN+SA).

We demonstrate the complete pipeline using a Dobot Magician robot arm (Fig. 1). Playing cards are placed at random positions in front of the robot on the table. A camera mounted overhead captures the the top view of the table. The position of the cards is obtained using an object detector[38] that is fine tuned to detect playing cards. Based on the instruction and the positions of the playing cards on the table, the robot picks-and-places a card. Note that while we can switch out the playing cards and instead detect other objects, it is necessary to know a priori what objects will be referenced in the instruction text and to fine-tune the object detector to detect those objects. Out of 15 trials with playing cards, the robot successfully picks the right card in all the trials. In 14 cases, the card is placed within 1 cm of the target. In one case, the localization of the anchor is off by more than 1 cm. A video of the robot in operation is available at https://youtu.be/bfmDC-zoCFc.