Translating Natural Language Instructions for Behavioral Robot Navigation with a Multi-Head Attention Mechanism

Inspired by the success of the Transformer model [Vaswani et al., 2017] on encoding different relationships of multi-modal data [Tan and Bansal, 2019, Zhou et al., 2020], we propose to use its multi-head attention mechanism to blend information from the two representation sub-spaces, natural instructions and navigation graph, in a more useful way. That is, different heads will specialize in fusing different patterns between both information sources. We hypothesize that this capability might help the decoder to alleviate the performance hit in novel environments at test time.

The architecture (Figure 1(c)) considers an initial encoding layer, where each word of the instruction is encoded using pre-trained GloVe descriptors [Pennington et al., 2014], and each triplet set is one-hot encoded to indicate which of the $\mathcal{B}$ behaviors and $\mathcal{N}$ nodes constitute each triplet. Subsequently, the encodings are embedded using bi-directional Gated Recurrent Units (GRU) [Chung et al., 2014]. The multi-modal representations are then fused by the newly added multi-head attention mechanism. A fully connected layer downstream reduces the dimensionality of the fused information, which is used as context $C$ by a recurrent GRU decoder. The decoder takes the initial position and translates the instruction to a sequential behavioral plan, soft attending its context $C$ at each time step. The loss function is cross entropy with respect to correct translations.

We use the dataset introduced in [Zang et al., 2018a] with the original train and test splits, where the Test-Repeated split has environments that were seen by the agent at training time, and the Test-New split has previously unseen maps. In total, we consider 10,040 instructions (8,066 for training) distributed across 100 maps, each with 6 to 65 rooms. We also use the same performance metrics: F1 score, edit distance (ED) to ground truth, and M@k metrics, where we have a match if the translation is under $k$ moves away from the ground truth¹¹1A move adds, deletes, or swaps a behavior in the plan., with M@0 being an exact match. The model was trained for 200 epochs, with a batch size of 256. The multi-head attention layer was set to have 4 heads. The rest of the model parameters are as established in [Zang et al., 2018a].

Table 1 details the performance of our approach, along with the baseline reported by [Zang et al., 2018a] as well as by our own implementation of that model, which notably was not able to perform as expected on the Test-Repeated set. As a result of using our multi-headed approach, we see a clear performance gain (23.2%) on exact match for the Test-New set, which confirms an improved generalization capability by our translation model. However, for the Test-Repeated set we see a 8.5% decrease in exact match with respect to the original approach (although we do beat our own implementation of the baseline by 25.9% in this set, and by 18.4% in the Test-New set).

In this paper, we introduced multi-head attention as a useful mechanism for leveraging a knowledge base to improve natural language translations to a high-level behavioral language that is understandable and executable by robots, exhibiting a better performance on never-before-seen environments with respect to previous work. Future research efforts contemplate minimizing the lost performance over previously seen maps and doing a qualitative analysis of the resulting attention weights.

This work was partially funded by the Millennium Institute for Foundational Research on Data and Fondecyt grant 1181739.