Vision-Dialog Navigation by Exploring Cross-modal Memory

According to the NDH task, the dialogs often begin with an underspecified, ambiguous instruction (e.g., Go to find the table), which requires further clarification. A dialog prompt is a tuple ($S,t_{o},p_{0},G_{j}$) contains a house scan $S$, a target object $t_{o}$ to be found, a starting position $p_{0}$, and a goal region $G_{j}$. At each round of communication, the navigator asks a question $Q$ and get a response $R$ from the oracle, then predict the navigation action $A$. Each sample of VDN consists of a repeating sequence $<A_{0},Q_{1},R_{1},A_{1},...,Q_{k},R_{k},A_{k}>$ for $k$ rounds of interaction. For each dialog with prompt ($S,t_{o},p_{0},G_{j}$), a vision-dialog navigation instance is created for each of $0\leq i\leq k$. The input is a hint about the target $t_{o}$ and a dialog history $H_{t}=\{D_{1},...,D_{t-1}\}$ at the $t$-th round of dialog, where $D_{i}=(Q_{i},R_{i})$.

Given the problem setup, the proposed CMN for NDH can be framed as an encoder-decoder architecture: (1) an encoder that explores the language memory about the historical communication $H_{t}$ between the navigator and the oracle, generating contextualized representation for $D_{t}$. (2) a decoder that first looks back to previous views of the navigator to help resolve the current dialog, and then converts the representation enhanced by the cross-modal memory into the navigation action space $A_{t}$. Fig. 2 presents an overview of the architecture of CMN, which consists of L-mem and V-mem module. The L-mem learns to attend relevant previous dialogs to explore context information in a given dialog $D_{t}=(Q_{t},A_{t})$. The V-mem learns to recall the cross-modal memory at the previous step for the visual perception of the current scene.

We expect the navigator to make the current decision by remembering the previous cross-modal memory about the environment. Here we introduce V-mem to restore the previous cross-modal memory during navigation to help generate memory-aware representations for the current vision perception. First, we used the final cross-modal encoding $e^{vlm}_{t,s-1}$ of the previous step $s-1$ to attend on the panoramic features $V_{t,s}$ in step $s$, the resulted memory-aware features $V^{m}_{t,s}$ depicts the correlation between previous decision and current views. We first project the $e^{vlm}_{t,s-1}$ and $V_{t,s}$ to $c$ dimensions and compute soft attention $A^{vis}$ as follows:

$$\begin{split}X=f_{v}(e^{vlm}_{t,s-1})&\odot f_{vlm}(v_{t,s,i})\\ A^{vis}(e^{vlm}_{t,s-1},v_{t,s,i})&=\sigma(X)/\sqrt{c},\\ \end{split}$$

(2)

where $f_{v}(\cdot)$ and $f_{vlm}(\cdot)$ denote the two-layer multi-layer perceptrons which convert the input to $c$ dimensions. $\sigma$ represent softmax function. $\odot$ denotes hadamard product (i.e., element-wise multiplication). Then we compute the memory-aware representation which contains the information about the previous action decision based on the attention $A^{vis}$ as:

$$v_{t,s}^{mem}=\sum_{i=1}^{s}A^{vis}_{s,i}{v_{t,s,i}}.$$

(3)

The output representations of the V-mem, $v_{t,s}^{m}\in$ $\mathbb{R}^{K}$, is calculated by applying the attention between memory about previous actions and the current views.

In this section, we formally describe the Language Memory (L-mem) module. Given the current question-answer $D_{t}$ and the dialog history features, the L-mem module aims to attend to the memory about the most relevant dialogs in history with respect to the dialog at the current round. Specifically, we first compute scaled dot product attention (Attention) [33] in multi-head settings which are called multi-head attention. Let $d_{t}$ and $M_{t}$ = $\left\{h_{i}\right\}_{i=0}^{t-1}$ be the current dialog and the dialog history feature vectors, respectively. The trainable weights $W_{n}^{Q}$, $W_{n}^{K}$ and $W^{V}_{n}\in\mathbb{R}^{L}\times c$ are used to project the $d_{t}$ and $M_{t}$ into features of $c$ dimension. In our experiment the dimension $c$ is set to 512. Then we calculate the attention $A^{lan}_{n}$ of $d_{t}$ to each element of the dialog memory $M_{t}$ as:

$$\centering\begin{split}&A^{lan}_{n}(d_{t},h_{i})=\mathrm{softmax}((d_{t}W_{n}^{Q})(h_{i}W_{n}^{K})^{T})/\sqrt{c},\\ &{\hat{d}}_{t}=\mathrm{concat}_{n=1}^{N}\left\{\sum_{i=0}^{t}A^{lan}_{n}(d_{t},h_{i})W^{V}_{n}h_{i}\right\},\\ &{\hat{d}}_{t}=\mathrm{LayerNorm}({\hat{d}}_{t}+d_{t}),\end{split}\@add@centering$$

(4)

where the outputs of each of the $N$ attention heads are concatenated and the contextualized representation of current dialog ${\hat{d}}_{t}$ is computed by applying a residual connection, followed by layer normalization. Next, the ${\hat{d}}_{t}$ is fed into a two-layer nonlinear multi-layer perceptron ($f_{lan}$), followed by a layer normalization and residual connection as:

$$\begin{split}&{\hat{d}}_{t}=\mathrm{LayerNorm(}f_{lan}({\hat{d}_{t}})+{\hat{d}_{t}}),\\ &{d_{t}^{ctx}}=\mathrm{concat}\{{{\hat{d}}_{t}},{d_{t}}\}.\end{split}$$

(5)

We then obtain the memory-aware representations by concatenating the contextual representation ${\hat{d}}_{t}$ and the original dialog representation $d_{t}$, denoted as $d^{ctx}_{t}\in$ $\mathbb{R}^{2L}$. Building on the multi-head attention mechanism, the L-mem can be stacked in multiple layers to get a high-level abstraction of the context of dialog history.

After exploring the memory of visual perception and language interactions with attention modules respectively, we further introduce cross-modal attention to explore the semantic correlation between the language and visual memory. We first perform language-to-vision attention by leveraging the memory-aware representation of the last dialog $d^{ctx}_{t}$ to attend the visual memory $V^{m}_{t,s}$ via the scaled dot product attention as:

$$e_{t,s}^{vm}=\mathrm{Attention}(d^{ctx}_{t},\{v_{t,0}^{m},...,v_{t,s}^{m}\}).$$

(6)

The visual memory provides supplement information about previous views, which enables better scene understanding for the navigator. Then we calculate vision-to-language attention to generate the final cross-modal memory encoding $e_{t,s}^{vlm}$ as:

$$e_{t,s}^{vlm}=\mathrm{Attention}(e_{t,s}^{vm},\{d_{0}^{m},...,d_{t}^{m}\}).$$

(7)

Here the language memory is incorporated twice. The first time is in Eq. 5 and the second time is in Eq. 7. The differences between the two incorporation lie in three folds. Firstly, $d^{ctx}_{t}$ is the concatenation of the last dialog feature $d_{t}$ and the attention weighted feature of previous dialog history $H_{t}$. So the dominate semantics derives from the last dialog $d_{t}$, namely, $d^{ctx}_{t}$ provides the contextualized information for $d_{t}$. In contrast, the cross-modal memory-aware representation $e_{t,s}^{vlm}$ exploit to discover the correlation between visual memory and all the existing dialogues. Secondly, the goal of calculating $d^{ctx}_{t}$ is to help the navigator better understand the current response from oracle, while the $e_{t,s}^{vlm}$ aims to learn the alignment between visual memory and language instructions, capturing the temporal correlations for better visual grounding. Lastly, the language-to-vision attention in Eq. 6 and the vision-to-language attention in Eq. 7 construct a closed reasoning path between visual and language context, providing rich information of cross-modal memory for the action prediction.

In Fig. 4 we describe the cooperation of the language memory and visual memory of the navigator. The language memory is maintained for each instance of the NDH task, resolving the ambiguous instruction of oracle. In contrast, the visual memory is collected within each round to capture temporal visual cues, which could benefit visual grounding. As is shown in Fig. 2, the action for each step is predicted based on the encoding produced by exploring the cross-modal attention between the visual and language memory.

By performing memory-aware reasoning with the cooperation of both language instructions and visual views, the navigator is able to better understand the historical decisions from temporal alignment between dialog history and previous views, which provide rich context information for the action prediction for the current step $s$ as:

$$\begin{split}\hat{a}_{t,s}&=\sigma(f_{m}(e^{vlm}_{t,s})),\\ a_{t,s}&=\mathrm{softmax}(f_{a}(\hat{a}_{t,s}))\end{split}$$

(8)

where $f_{m}(\cdot)$ and $f_{a}(\cdot)$ are single-layer linear transformations to project the $e^{vlm}_{t,s}$ from $K+L$ dimension to $K$, and project the $\hat{a}_{t,s}$ from $K$ dimension to $M$ dimension which is the number of actions. Following [9], we employ the panoramic action space with panoramic features of images. The agent is required to choose a candidate from the panoramic features of the visual views in the next step.

Datasets: We evaluate our model on the CVDN dataset which collects 2050 human-human navigation dialogs and over 7k trajectories in 83 MatterPort houses [2]. Each trajectory corresponds to several question-answer exchanges. The dataset contains 81 unique types of household objects, each type appears in at least 5 houses and appear between 2 and 4 times per such house. Each dialog begins with an ambiguous instruction, and the subsequent question-answer interaction between the navigator and oracle will lead the navigator to find the target.

Evaluation Metrics: Following the previous works in visual language navigation and visual dialog navigation, we use four popular metrics to evaluate the proposed method from different aspects: (1) Success Rate (SR), the percentage of the final positions less than 3m away from the goal location. (2) Oracle Success Rate (OSR), the success rate if the agent can stop at the closet point to the goal along its trajectory. (3) Goal Progress (GP), the average agent progress towards the goal location. (4) Oracle Path Success Rate (OPSR), the success rate if the agent can stop at the closest point to goal along the shortest path. Note that this could be different from the OSR if the shortest path is not be used for supervision (i.e., mixed path or navigator path).

Different Supervision: The navigator paths in the CVDN dataset are collected from humans playing roles as the navigators, while the oracle paths are simultaneously generated by the shortest path planner. The typical supervision for the agent in the navigation task is defined by the shortest path, which is the same as the oracle path given in the CVDN dataset. However, even human demonstrations could be imperfect compared to the oracle path in realistic situations. Thus, the CVDN dataset also provides a new form of supervision called the mixed supervision path. The mixed supervision path is defined as the navigator path when the end nodes of the navigator and the oracle are the same, and the oracle path otherwise.

Implementation Details: We adopt RMSProp as the optimizer for the agent and set the learning rate to 0.0001. We train all agents with student-forcing for 20000 iterations of batch size 60, and evaluate validation performance every 100 iterations. The best performance across all epochs is reported for validation folds. The navigator moves its predicted action $\hat{a}$ at each time step. Then cross-entropy loss is applied to $\hat{a}$ and $a^{*}$ which is the next action along the shortest path to the target. The total training process costs 2 GPU days on a single Titan 1080Ti device.

Compared Models: We compare our proposed CMN with several baselines and the state-of-the-art method: (1) The Shortest Path Agent takes the shortest path to the supervision goal at inference time and represents the upper bound navigation performance for an agent. (2) The Random Agent chooses a random heading and walks up to 5 steps forward (as in [2]). (3) The Vision Only baseline where the agent considers visual input with empty language inputs. (4) The Dialog Only baseline where the agent considers language input with zeroed visual features. (5) The sequence to sequence model proposed in [31] where the historical dialogs are concatenated to form a single instruction as in visual language navigation models [2].

Comparison to previous methods: As is shown in Tab. 1, our proposed CMN outperforms the previous state-of-the-art method [31] on the Goal Progress (m) with different supervisions ( e.g., planner path (Oracle), player path (Navigator) and trusted path (Mixed)), demonstrating the ability of our method to grounding visual elements in the explored environments. When evaluated on the Val Unseen and Test Unseen data, the gap between our CMN and the seq-to-seq method is also significant, showing that CMN generalizes well for unexplored environments.

We ablate the V-mem and L-mem module on the Val sets in Tab. 2 and the Test set in Tab 3.

Baselines: In the first row, we conduct a baseline from VLN by directly using the concatenated dialog history as the language input. The difference between the VLN baseline and the Sequence-to-sequence model are two folds. First, the Seq-to-seq model uses a $1\times 2048$ feature vector to represent each panoramic image, while the dimension of the visual feature used in baseline VLN is $1\times 36\times 2048$ for all 36 views of the panoramic image. Second, the action space in the Seq-to-seq model is the low-level visuomotor space, where the predictions of actions are $3$-d logits. In contrast, the baseline VLN uses panoramic action space, where the agent has a complete perception of the scene and directly performs high-level actions. Our framework is built based on the baseline VLN with both the panoramic vision features and the panoramic action space.

Effect of different memory module: We disable the V-mem module by directly averaging the visual features of each panoramic view. In Tab. 2 and Tab. 3 we can see that the performance dropped when the model lost visual memory about previous navigation steps. The reason why we use averaged feature here is that the averaged features are more confused and useless than the last memory features since closer memory is more correlated to the current state than earlier memory, which helps us disable most of the functionality of memory modules in ablation studies.

To remove the L-mem module, we replace the memory-aware representation of the language interactions by the word-level context within the last question-answer sentences. It can be seen in Tab. 2 and Tab. 3 that the performance of our method dramatically decreases when discarding the language memory module (L-mem), indicating that the language memory is crucial for the understanding of the oracle instructions. In Tab. 3, we also set the output of our encoder ($e_{t,s-1}^{vlm}$ Fig. 2) to zero values to eliminate the cross-modal memory context (VL-mem) from the previous navigation step. The VL-mem can be regarded as a high-level abstraction of historical navigation and represents rich information about the previous action decision made by the agent. The results indicate that the performance of our model would drop when the cross-modal memory is lost.

Discussion about the temporal order in memory: As is shown in Fig. 2, CMN predicts the action at step $s$ by restoring the cross-modal memory $e^{vlm}_{t,s-1}$, which represents the memory about the decision making of the previous navigation action at step $s-1$. From this point of view, the information about the temporal order of the cross-modal memory is implicitly embedded in our CMN. We further consider a more explicit way that directly concatenates the embedding of the order and the memory features. The performance is comparable to the implicit way.

To see how our proposed CMN performs the visual dialog navigation task, we visualize two qualitative examples in Fig. 5. The first example contains one round of dialogue with eight steps of navigation. We can see that the agent successfully reaches the target with a comprehensive understanding of the natural language response from oracle, indicating that our method is also compatible with the VLN task. In the second example, there are five rounds of dialogue between the navigator and the oracle. In the first round of communication, the navigator moves two steps down to the hallway, which requires the navigator to correctly recognize visual elements, including bed, hallway, and stairs while understanding language instructions. In the third round of communication, the oracle suggests the navigator to “go back”. Our CMN explores to better understand this instruction by referring it to dialog history to resolve the specific meaning, that is, the inverse navigation operation of the previous steps. In step 1 of the third round, the navigator returns back to the hall. Finally, it finds the target “towel”. Since our proposed Cross-modal Memory could help restore the visual and language memory of previous steps and interactions, the navigator can resolve the ambiguous instruction “go back” and thus return to the previous location.