The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigation

Figure 2 shows an unrolled illustration of the proposed Object-and-Room Informed Sequential BERT (ORIST). The ORIST is mainly composed of three parts: an object-level initial embedding module, a sequential BERT module and a room-and-direction multi-task module. All these modules are reused between steps. At step $t$, the initial embedding module takes as inputs an instruction $\mathcal{X}$, features of object regions $\mathcal{O}_{t}$, and navigable orientations $\mathcal{D}_{t}$. Then its outputs are fed into transformer layers of the sequential BERT module to adaptively fuse information from individual words and visual objects as well as each orientation via the self-attention mechanism. Finally, the fused features $\mathbf{H}_{t,U}$ are sent to two branches. One branch is the room-and-direction multi-task module, which predicts the room type $\mathbf{R}_{n,t}$ and $\mathbf{R}_{g,t}$, the direction $\mathbf{D}_{t}$, the next navigating action $\mathbf{a}_{t}$, and the navigation progress $\mathbf{p}_{t}$. Another branch is the LSTM of the sequtial BERT module, which produces a new temporal context $\mathbf{h}_{t}$ and connects to the next navigation step. The whole model is trained end-to-end.

Existing VLN methods typically use ResNet [13] features of each discrete view to represent visual information. However, this may weaken vision-and-language matching because the ResNet is trained for image classification which may represent only one salient object, leading to the same image feature being matched to a variety of different objects mentioned in instructions (e.g., laptop and table). To address this issue, we introduce an object-based representation to facilitate object-level cross-modality matching. The bounding boxes are adopted from REVERIE [28].

When designing the embedding module, we take into account the following three considerations. (I) The distinct characteristics of natural-language instructions and visual objects, e.g., word tokens vs. object regions, word order vs. object position. (II) The type of an input token (e.g., a visual or textual input). (III) The characteristic of VLN tasks, e.g., the orientation feature of each navigable location. To this end, for an instruction $\mathcal{X}$ and a candidate location in view $v_{t,i}$ composed of $N^{i}$ objects $\mathcal{O}_{t,i}=\{o_{t,i}^{1},\cdots,o_{t,i}^{N^{i}}\}$, we obtain the initial embedding via

where $\mathbf{E}_{t,w}\in\mathbb{R}^{(L+2)\times d_{h}}$ denotes the embedding of the expanded instruction $\bm{x}^{tok}=\{[\mathrm{CLS}],\mathcal{X},[\mathrm{SEP}]\}$. In addition, $\bm{x}^{pos}=[0,1,\cdots]$ is zero-based token position; $\mathbf{E}_{t,o}^{i}\in\mathbb{R}^{N^{i}\times d_{h}}$ denotes an embedding of $N^{i}$ object regions; $\bm{o}_{t,i}^{fea}\in\mathbb{R}^{N^{i}\times 2048}$ represents objects’ features extracted using FasterRCNN as described in [3]; and $\bm{o}_{t,i}^{pos}\in\mathbb{R}^{N^{i}\times 7}$ is the position feature of $N$ object regions, of which each row is composed of the $x,y$ location of the top-left and bottom-right points, and the region’s height, width, and area. In Eq. (1), $\bm{x}^{type}=\mathbf{0}$ and $\bm{o}_{t,i}^{type}=\mathbf{1}$ are token type vectors, and $\mathbf{E}_{t,d}^{i}\in\mathbb{R}^{1\times d_{h}}$ denotes the embedding of the candidate location’s orientation $\bm{d}_{t,i}\in\mathbb{R}^{1\times 4}=\mathcal{D}_{t,i}$. $\mathrm{Emb}(\cdot)$ is a projection layer that embeds tokens to feature vectors. $\mathrm{FC}(\cdot)$ is a fully-connected layer.

For each candidate location $i$, we obtain its initial embedding matrix $\mathbf{H}_{t,0}^{i}=[\mathbf{E}_{t,w},\mathbf{E}_{t,o}^{i},\mathbf{E}_{t,d}^{i}]\in\mathbb{R}^{C_{t}\times d_{h}}$, where $C_{t}$ is the maximum number of tokens at step $t$ across all candidates. The subscript $0$ in $\mathbf{H}_{t,0}^{i}$ denotes it is an initialisation before going through the sequential BERT module (detailed in the next section). The concatenation of all $G_{t}$ candidates at step $t$ constructs the initial embedding $\mathbf{H}_{t,0}=[\mathbf{H}_{t,0}^{1};\cdots;\mathbf{H}_{t,0}^{G_{t}}]\in\mathbb{R}^{G_{t}\times C_{t}\times d_{h}}$.

BERT-like models have demonstrated their effectiveness in learning textual-visual entity matching [9, 30]. However, most of them are designed for one-round decision tasks (e.g., VQA). VLN, as a multi-round task and partially observed Markov decision process, requires a model to be aware of which parts of an instruction have been completed and which parts have yet to be executed. As such, a direct application of BERTs to VLN does not perform well due to the loss of temporal context. To address these problems, in this work we propose a sequential BERT module.

The light blue panel in Figure 2 shows an unrolled illustration of our sequential BERT at step $t$ and $t+1$, which consists of two main components: transformer layers and an LSTM. Each transformer layer $\mathcal{F}_{j}$ is a basic transformer [34], which is a multi-head self-attention

where $\mathbf{H}_{t,j-1}\in\mathbb{R}^{G_{t}\times C_{t}\times d_{h}}$ is the output from a previous transformer layer, $\mathbf{M}_{t}$ is a mask matrix indicating whether a token can be attended, and $\mathbf{H}_{t,j}=[\mathbf{A}_{t,j,1},\cdots,\mathbf{A}_{t,j,n}]$ is a concatenation of outputs of $n$ attention heads. Concretely, the $k$-th head’s output $\mathbf{A}_{t,j,k}$ is obtained by

where $\mathbf{W}_{j}^{*}$ is a learnable parameter matrix, and $d_{j,k}$ is the embedding dimension of $\mathbf{A}_{t,j,k}$. Assume we have $U$ transformer layers in total, as was demonstrated experimentally in [6, 9, 21], then the embedding of the first token (i.e., [CLS]) of the last transformer layer, denoted as $\mathbf{H}_{t,U}[0]\in\mathbb{R}^{G_{t}\times d_{h}}$, is able to represent fused mutually attended information from textual and visual inputs.

To enable these transformer layers to model temporal context, an intuitive solution is to use a collection of object entities observed along the navigation path. In this way, one can expect that previously observed objects might be matched to completed instruction parts, and new visual perceptions could be matched to instruction parts to be executed. However, this will involve a huge computation cost (e.g., GPU memory and computation time) as the agent navigates. To tackle this problem, we propose to employ an LSTM to adaptively learn how to summarise seen instructions and visual perceptions as the agent navigates. Specifically, we first encode $\mathbf{H}_{t,U}[0]$, which contains mutually attended instruction and visual information, to get a deeper representation $\mathbf{E}_{t}\in\mathbb{R}^{G_{t}\times d_{h}}$ via

Then, we aggregate surrounding information from all the $G_{t}$ candidates via $\mathbf{I}_{t}=\textrm{AvgPool}(\mathbf{E}_{t}).$ Next, we obtain the new temporal context $\mathbf{h}_{t}$ by

where $\mathbf{h}_{t-1}$ and $\mathbf{c}_{t-1}$ represents previous temporal context and LSTM cell state, respectively. $\mathbf{h}_{0}$ and $\mathbf{c}_{0}$ are initialised with zeros. Thus, $\mathbf{h}_{t}$ contains the accumulated temporal context along the navigation and it is passed to the next navigation step as shown by the right panel of Figure 2.

Room types and directions can be useful information for navigation as they are often mentioned in instructions and provide strong navigation guidance. We note that a certain room type can usually be characterised by the presence of specific objects, such as bed for bedroom, TV/couch for living room, oven/microwave for kitchen. As our model takes objects as inputs, it is thus able to predict which room the agent should go to according to the current visual perception and instruction. Furthermore, it is also important for agents to understand the room that constitutes its final goal. This can help the agent make decisions based on the long-term goal. Additionally, we enable an agent to recognise the relative direction (i.e., left/right/front/back) of each candidate in order to better align with key direction cues (e.g., “turn left/right”) in instructions. The progress monitor [41] is also adopted to further facilitate analysis of progress. We use $\mathrm{MLP}$ layers to implement the above goals:

where $\mathbf{D}_{t},\mathbf{R}_{n,t},\mathbf{R}_{g,t},\mathbf{a}_{t},\mathbf{p}_{t}$ denote the relative direction prediction for each candidate (we evenly divide the surrounding 360^∘ into four directions representing left/right/front/back), the room type prediction for the next step and the goal location, the action logit, and the navigation progress, respectively. The ground truth room types are extracted from the Matterport3D dataset [7]. $\mathbf{E}_{t}$ is composed of embeddings of all candidate and $\mathbf{I}_{t}$ is the fused embedding of all candidates as described in Section 3.3.

We set parameters $\lambda_{1}$ and $\lambda_{2}$ to 0.2 to keep the associated losses at the same level with others, and set $\lambda_{3}$ to 0.2 following [32]. We use $U=12$ transformers in our sequential BERT module. To take advantage of existing BERT-based works, we initialise the transformers in our model with the weights of UNITER [9]. UNITER is originally trained on four image-text datasets (COCO [19], Visual Genome [17], Conceptual Captions [29], and SBU Captions [26]) for joint visual and textual understanding. Note that these datasets have no overlap with the VLN data.

Our model is trained separately for each task. We use the AdamW [20] optimiser with a learning rate of $1\times 10^{-6}$ and the model is trained on 8 Nvidia V100 GPUs. For the R2R task, following common practice, augmentation data from Envdrop [32] are used. For the REVERIE and NDH tasks, only the original training data are used. It is worth noticing that although Thomason et al. [33] show that using the complete dialogue leads to much better results on the NDH task (rather than using just the oracle’s answer), here we choose the oracle’s answer as instruction to avoid the additional computation cost, and we achieve the best results documented, as is demonstrated later.

In this section, we evaluate the impact of the key components of our model: the sequential design for BERT and the newly proposed three loss functions. The evaluations are conducted on the NDH and REVERIE datasets. NDH utilises detailed navigation instructions as in the R2R task, while REVERIE utilises much shorter and high level instructions. The results are shown in Table 4, where the symbols S, $\mathcal{L}^{\mathbf{D}},\mathcal{L}^{\mathbf{R}_{n}}$, and $\mathcal{L}^{\mathbf{R}_{g}}$ denote the sequential design for BERT, direction loss, room-type loss for the next step, and room-type loss for the goal location, respectively.

In Figure 3, we visualise one navigation trajectory on the R2R task to give an intuitive example of how our agent navigates. The visualisation is based on the attention visualisation tool BertViz [35]. The left panel shows the process of attention shifting as the agent navigates. Navigable locations at each step are marked using blue cylinders and the perceived views for each navigable location are bounded using an indexed rectangle. The green rectangle denotes the view associated with the next step predicted by our model. Objects in each view are marked with red bounding boxes. As shown in the left panel, the attention on the instruction shifts intuitively as the agent navigates. Concretely, at the first step, the agent focuses on “exit the room” and it goes forward to the door. Next, the agent becomes aware of “turn left” and performs the correct action. Then, at steps 3$\sim$5, the attention is updated from “the left room” to “proceed through the room”, and the agent behaves accordingly. At the last two steps, the attention focuses on the end of the instruction and the agent recognises the bathroom by virtue of the presence of the sink. Hence, it chooses the correct viewpoint and finally decides to stop.

To illustrate that the agent is on the road to “know where”, we take the decision progress of step 6 as an example and show details in the right panel. Specifically, the right panel shows two kinds of attention distributions under each navigation candidate: 1) Attention on the instruction. As shown by the pink color on the left column words (the darker the larger), the “bathroom” is focused on. 2) Self-attention on all tokens. In particular, the right column shows attention distribution of “bathroom” on all tokens. We can see “bathroom” is strongly connected to the object “[drawer#sink#table]” (a darker line between two tokens denotes a stronger connection). The 12 kinds of colors denote attention computed from 12 attention heads. Additionally, at the bottom of the right panel, we present the progress prediction, the next room and goal room type prediction, and the direction of each navigable viewpoint. The agent successfully infers these clues from the instruction and visual perceptions. All of these information assist the agent making the correct navigation decision.