MemoNav: Selecting Informative Memories for Visual Navigation

We consider image-goal navigation (ImageNav) in this paper. As shown in Figure 1 (left), the objective of the agent is to learn a policy $\pi(a_{t}|s_{t},\mathbf{I}_{target})$ to reach a target, given an image $\mathbf{I}_{target}$ that contains a view of the target and a series of observations, $\left\{\mathbf{I}_{t}\right\}$, captured during the navigation. At the beginning of navigation, the agent receives an RGB image, $\mathbf{I}_{target}$, of the target. At each time step, it captures an RGB-D panoramic image, $\mathbf{I}_{t}$ of the current location and generates a navigational action. Following [20], any additional sensory data, e.g. GPS and IMU, are not available.

The MemoNav is mainly built upon VGM [20], which is briefly introduced below.

VGM incrementally builds a topological map $G=\left\{V,E\right\}$ from the agent’s past observations where $V$ and $E$ denote nodes and edges, respectively. The node features (STM) $\mathbf{V}\in\mathbb{R}^{d\times N_{t}}$ are generated from observations by a pretrained encoder $\mathcal{F}_{loc}$ where $d$ denotes the dimension of feature and $N_{t}$ the number of nodes at time $t$.

VGM updates the map in two steps: localization and graph update. Localization is implemented by calculating embedding similarity. VGM first uses $\mathcal{F}_{loc}$ to map the current observation $\rm{I}_{\emph{t}}$ to an embedding $e_{t}\in\mathbb{R}^{d}$, and then calculates the similarities $\left\{s_{i}\mid s_{i}=\frac{n_{i}\cdot e_{t}}{\left\|n_{i}\right\|\left\|e_{t}\right\|},i=1,2,\ldots,N_{t}\right\}$ between $e_{t}$ and each node feature $\mathbf{V}=\left\{n_{1},n_{2},...,n_{N_{t}}\right\}$. If the similarity of a node $s_{i}$ is higher than a threshold $s_{th}$, this node is seen as near the agent. Graph Update depends on three cases. (i) If the agent is localized at the last node, the map will not be updated. (ii) If the localized node is different from the last one, a new edge connecting these two nodes will be created, and the feature of the localized node will be replaced with the new embedding. (iii) If the agent fails to be localized, a new node with the current observation embedding will be created, and a new edge will connect this new node with the last node.

VGM utilizes a three-layer graph convolutional network (GCN) to process the topological map and obtain the encoded memory $\mathbf{M}$ (i.e., a sequence of feature vectors). Before the first layer, VGM fuses each feature of a node with the target image embedding $e_{target}=\mathcal{F}_{enc}(\mathbf{I}_{target})$ using a linear layer.

The encoded memory is then decoded by two Transformer decoders, $\mathcal{D}ec_{cur}$ and $\mathcal{D}ec_{target}$. $\mathcal{D}ec_{cur}$ takes the current observation embedding $e_{cur}=\mathcal{F}_{enc}(\mathbf{I}_{cur})$ as the query and the feature vectors of the encoded memory $\mathbf{M}$ as the keys and values, generating a feature vector $f_{cur}$. Similarly, $\mathcal{D}ec_{target}$ takes the target embedding $e_{target}$ as the query and generates $f_{target}$. Lastly, an LSTM-based policy network takes as input the concatenation of $f_{cur}$, $f_{target}$ and $e_{cur}$, and outputs an action distribution.

Single-goal tasks are not sufficient for thoroughly evaluating the potential of memory mechanisms, since finding a single goal may not need all navigation history. Hence, it remains unclear which fraction of navigation history is useful for finding the goal. To further investigate how memory mechanisms assist the agent in exploring the environment and locating the target, we borrow the idea of MultiON [41] and collect multi-goal test datasets (an example in Figure 1). In a multi-goal task, the agent navigates to an ordered sequence of $N$ destinations $\left\{\rm{T}\right\}_{1}^{N}$ in the environment. We place the final target within the vicinity of previous ones to form a loop. By letting the agent return to visited places, we are able to test whether memory mechanisms can help the agent plan a short path. If not, the agent will probably waste its time re-exploring the scene or traveling randomly.

The MemoNav comprises three key components: the forgetting module, the global node, and the GATv2-based memory encoder. We show the pipeline of the MemoNav in Figure 2 and describe these components in the remainder of this section.

All experiments are conducted in the Habitat [33] simulator with the Gibson [42] scene dataset. Following VGM [20], we train all models with 72 scenes, and evaluate them on a public 1-goal dataset [25] that comprises 1400 episodes for 14 unseen scenes. For a fair comparison, 1007 sampled episodes are used for evaluating our model on 1-goal tasks, while 1400 episodes are used for conducting ablation studies. In addition, we generate 700-episode datasets for a multi-goal task by ourselves (see the Appendix). The difficulty of an episode is indicated by the number of goals ¹¹1The 1-goal difficulty level here denotes the hard level in the public test dataset All baselines and our model are trained on the 1-goal dataset and tested on other difficulty levels.

The action space of the agent consists of four options: stop, forward, turn_left, and turn_right. When the agent performs a forward action, the agent moves 0.25m forward. A turning action rotates the agent by 10 degrees. The agent is allowed to take at most 500 steps in either 1-goal or multi-goal tasks. The agent succeeds in an episode if it performs stop within a 1m radius of each target location. The agent fails if it performs stop anywhere else or runs out of time step budget.

Following VGM [20], we first train the agent using imitation learning, minimizing the negative log-likelihood of the ground-truth actions. Next, we finetune the agent’s policy with proximal policy optimization (PPO) [34] to improve the exploratory ability of the agent. The reward setting and auxiliary losses remain the same as in VGM. All our models are obtained on a TITAN RTX GPU.

We compare the proposed model with baselines that also utilize memory mechanisms. Random is an agent taking random actions with oracle stopping. ANS [8] and Exp4nav [11] are metric map-based models proposed for the task of exploration. They are adapted for ImageNav in the experiments of VGM [20]. SMT [14] stacks past observations in chronological order and uses a Transformer decoder to further process them. Neural Planner [4] needs to explore the scene first to pre-build a topological map. Once the map is built, this method performs a hierarchical planning strategy to generate navigation actions. NTS [9] incrementally builds a topological map without pre-exploring. Its navigation strategy follows two steps: a global policy samples a subgoal, and then a local policy takes navigational actions to reach the subgoal. VGM [20] (see Section 3.2) is the baseline the MemoNav is built on. The quantitative results of these baselines on 1-goal tasks are from [20]. We re-evaluate VGM and report the new results.

In 1-goal tasks, the success rate (SR) and success weighted by path length (SPL) [2] are used. In a multi-goal task, we borrow two metrics from [41]: the Progress (PR) and progress weighted by path length (PPL). PR is the fraction of goals that are successfully reached, equal to the SR for 1-goal tasks. PPL indicates navigation efficiency and is defined as $PPL=\frac{1}{E}\sum_{i=1}^{E}Progress_{i}\frac{l_{i}}{\max\left(p_{i},l_{i}\right)}$, where $E$ is the total number of test episodes, $l_{i}$ and $p_{i}$ are the shortest path distance to the final goal via midway ones, and the actual path length taken by the agent, respectively.

Comparison on 1-goal tasks. Table 1 shows that the proposed method outperforms all the baselines in SR on the 1-goal dataset. Compared with the metric map-based methods (Exp4nav [11] and Neural Planner [4]), our model enjoys a large improvement in SR (from 0.47 to 0.78) and SPL (from 0.39 to 0.54). The two baselines require pre-built maps but exhibit lower SR and SPL. This is partly because the explored areas fail to cover the target place. Compared with VGM [20], our model exhibits a slight performance gain, increasing the SR by 0.03 while using 20% fewer node features. This result indicates that VGM uses redundant past observations that probably interfere with the agent’s navigation. In contrast, our agent uses the informative fraction of these observation, obtaining a higher success rate with less past information. The SPL slightly decreases, probably because the agent forgets a certain proportion of explored places and spends more time re-exploring.

Comparison on a multi-goal task. In Table 2, we compare the MemoNav with VGM [20] on a multi-goal task. As the difficulty level rises from 1 goal to 4 goals, both methods witness drops in the navigation success rate, but the MemoNav is more robust and more competent to complete a multi-goal task. At the 2-goal level, the MemoNav outperforms VGM in PR by 0.05 (11.1%). When the number of goals increases to 3, the performance gap widens: the improvement in PR rises to 0.09 (27.2%). At the 4-goal level, the two methods both perform worse, but our method still surpasses VGM in PR by a noticeable margin of 0.03 (10.7%).

The performance gain of each proposed component. We ablate the three key components described in Section 4.2, and show the results in Table 3. Comparing rows 2, 3, and 4 with row 1 (VGM), we can see that the LTM brings the largest improvement over the baseline at the 2, 3, and 4-goal levels among the three components. For example, the LTM brings an increase in PR by 0.056 at the 3-goal level. Although applying the forgetting module only achieves improvements at the 2 and 4-goal levels, its cooperation with the LTM witnesses a significant increase in the SR/PR at the 1, 2, and 3-goal levels. More importantly, compared with VGM (row 1), the synergy of the three components (row 7) increases the PR by 0.049 (10.9%), 0.092 (21.9%), 0.038 (13.8%) at the 2, 3 and 4-goal levels respectively. These results demonstrate that the three components are helpful for solving long-time navigation tasks with multiple sequential goals.

The importance of the LTM. Table 4 presents the results of ablation experiments that demonstrate the importance of the LTM (described in Section 4.2.2). The first ablation (row 2) is to replace the feature in the LTM with that of a randomly selected node in the STM once the GATv2 encoding is finished so that the LTM is disabled. Row 2 shows that the agent’s performance decreases at all difficulty levels compared with the full model (row 1). Furthermore, when the LTM feature is not incorporated into the WM (row 3), the performance also deteriorates. In summary, the LTM stores a scene-level feature essential for improving the success rate.

The correlation between the difficulty level and forgetting threshold. We evaluate our model with different forgetting thresholds $p$ used by the WM (see Section 4.2.1). The results are shown in Figure 4. For clarity, the figure shows the performance differences between our default model (row 7 in Table 3) and the variants. The four levels exhibit different trends: when $p$ increases (i.e., a larger fraction of STM is not incorporated into the WM), our model first witnesses increases and then drops in the SR/PR and SPL/PPL at the 3 and 4-goal levels while enjoying slight gains in these metrics at the 1 and 2-goal levels. At the 1-goal level, our model obtains the highest SR and SPL when $p=80\%$, which means that a large fraction of node features in the map are useless when the navigation task is easy. A similar trend can also be seen at the 2-goal level. In contrast, our model exhibits an increase in the PR (from 0.313 to 0.321) and PPL (from 0.049 to 0.050) at the 4-goal level when $p$ rises from 0% to 40%. However, if $p$ rises to 80%, the two metrics see a precipitous decline, and the agent takes more than 20 steps to complete the tasks. These results suggest that excluding unimportant STM from the WM improves navigation performance in multi-goal navigation tasks. However, excluding an excessive fraction forces the agent not to utilize what it has explored, thus reducing the navigation efficiency.

To observe how the MemoNav improves the navigation performance, we show example episodes of VGM and our model at the four difficulty levels in Fig 5 (more examples are provided in the Appendix). We can see that the MemoNav agent takes fewer steps and its trajectories are smoother. VGM tends to spend a large proportion of time going in circles in narrow pathways. For instance, the 1st and 3rd columns in Figure 5 show that the VGM agent is trapped in a bottleneck connecting two rooms, while our agent uses the time steps wasted by VGM to efficiently explore the scene. This comparison shows that the MemoNav is more capable of avoiding deadlock.

We next draw a histogram of the normalized path lengths of successful episodes for VGM and our model to further investigate to what degree our model avoids deadlock. The normalized path length is calculated as $l_{norm}=l_{i}/\max\left(p_{i},l_{i}\right)-1$ where $p_{i}$ and $l_{i}$ are defined in Section 5.4. $l_{norm}$ indicates how many extra steps the agent takes compared to the shortest path. A larger $l_{norm}$ represents a less efficient navigation episode. Figure 6 shows that the histogram of our model exhibits a less heavier-tailed distribution, especially on the 3-goal dataset. This distribution means that our model has fewer episodes with an extremely large number of steps. This result agrees with the visualization in Figure 5 , as it suggests that our model utilizes goal-relevant navigation history stored in the WM to reduce the number of wasted steps.

Analyzing these results, the reason why our model helps the agent escape from loops is twofold. Firstly, the WM excludes misleading node features and reduces the interference of irrelevant past observations. Secondly, the LTM provides the agent with a scene-level context that helps it locate the target from a global view.