VLFM: Vision-Language Frontier Maps
for Zero-Shot Semantic Navigation

In the development of robots capable of human-like navigation, learned foundation models that mimic this human reasoning process can be invaluable. Many methods, known as zero-shot methods, utilize these models to facilitate semantic navigation without any task-specific training or fine-tuning. Zero-shot methods are convenient because they can be easily adapted or repurposed for future robotic systems performing complex tasks, and they provide intermediate representations that improve interpretability. The remarkable performance of large language models (LLMs) and vision-language models (VLMs) has facilitated task-independent solutions for the semantic inference of out-of-view scene information [1, 2, 3].

In this work, we propose Vision-Language Frontier Maps (VLFM), a zero-shot approach for target-driven semantic navigation to an unseen object in a novel environment. VLFM builds occupancy maps from depth observations to identify frontiers of the explored map region. To find semantic target objects, VLFM prompts a pre-trained VLM to select which of these frontiers is most likely to lead to the semantic target. In contrast to prior language-based zero-shot semantic navigation methods [2, 4, 3], our method does not rely on object detectors and language models (e.g., ChatGPT, BERT) to evaluate frontiers using text-only semantic reasoning. Instead, VLFM uses a vision-language model to directly extract semantic values from RGB images in the form of a cosine similarity score with a text prompt involving the target object. VLFM uses these scores to generate a language-grounded value map that is used to identify the most promising frontier to explore. This spatially-grounded joint vision-language-based semantic reasoning increases computational inference speed and overall semantic navigation performance.

We demonstrate VLFM in photorealistic environments within the Habitat [5] simulator, where we achieve state-of-the-art results on the Object Goal Navigation (ObjectNav) task, even when compared to methods trained directly on the task. Specifically, we achieve absolute increases in success rates weighted by path length over prior state-of-the-art approaches of 12% on Gibson [6], 5% on Matterport 3D (MP3D) [7] and 3% on Habitat-Matterport 3D (HM3D) [8] datasets. We also demonstrate our approach in the real world on a Boston Dynamics Spot mobile manipulation platform by navigating efficiently to unseen semantic targets across a novel office building floor, without access to a pre-built map.

ObjectNav. Object Goal Navigation (ObjectNav) involves executing semantic target-driven navigation in a novel environment, where performance is primarily measured by the efficiency of the robot’s path to an instance of a given target object category. This is based on the premise that effective use of semantic priors should enable a robot to locate objects more efficiently [10]. Learning approaches to train robots with semantic navigation abilities have typically leveraged reinforcement learning [11, 12, 13, 14], learning from demonstration [15], or prediction of semantic top-down maps [16, 17, 18, 19, 20, 3] on which waypoint planners can be used.

However, these task-specific trained approaches only work with the closed-set of object categories that they were trained on, and are often trained exclusively on simulated data, which can impede deployment of these policies onto real-world platforms. In contrast, our work proposes a zero-shot method that can take in an open-set of object categories, uses models that were trained on large amounts of real-world data, and demonstrates successful semantic navigation in the real world.

Zero-shot ObjectNav. Recent works in zero-shot methods for ObjectNav involve adapting the frontier-based exploration method proposed by [21]. Frontier-based exploration involves visiting the boundaries between explored and unexplored areas on a map that is iteratively built by the agent as it explores. Many methods for choosing the next frontier to explore have been proposed, such as classical methods that select frontiers based on the expected amount of information the agent would gain [22, 23]. CLIP on Wheels (CoW) [1] adopts a straightforward approach in which the robot explores the closest frontier until the target object is detected using either CLIP [24] features or an open-vocabulary object detector. LGX [4] and ESC [2] use a large language model (LLM) that processes object detections presented in the form of text to identify which frontiers would most likely harbor an instance of the target object. Instead of an LLM, SemUtil [3] uses BERT [25] to embed the class labels of objects detected near frontiers, and then compare them to the text embedding of the target object to select the frontier to explore next. However, these methods introduce a bottleneck in which visual cues from the environment must be converted into text by an object detector before they can be used to semantically evaluate frontiers. Additionally, reliance on an LLM requires a large amount of compute that may require a remote server the robot must connect to. In contrast, VLFM uses a vision-language model that can be easily loaded onto a consumer laptop to generate semantic value scores directly from RGB observations and text prompts, without generating any text from visual observations.

We address the task of ObjectNav [10], where a robot is tasked with searching for an instance of a target object category (e.g., ‘bed’) in a previously unseen environment. This semantic navigation task encourages the robot to understand and navigate the environment based on high-level semantic concepts, such as the type of room it’s in or the types of objects it sees, rather than relying solely on geometric cues. The robot only has access to an egocentric RGB-D camera and an odometry sensor that provides its current forward and horizontal distance and heading relative to its starting pose. The action space consists of the following: move_forward ($0.25m$), turn_left ($30^{\circ}$), turn_right ($30^{\circ}$), look_up ($30^{\circ}$), look_down ($30^{\circ}$), and stop. An episode is defined as successfully completed if stop is called within 1 m of any instance of the target object in 500 or fewer steps.

As depicted in Fig. 2, our approach is divided into three phases: initialization, exploration, and goal navigation. In the initialization phase, the robot rotates in place for a complete turn to set up its frontier and value maps, which are crucial for the subsequent exploration phase. During exploration, the robot persistently updates the frontier and value maps to create frontier waypoints and select the most valuable one for locating the specified target object category and navigating to it. Once it detects a target object instance, it transitions to the goal navigation phase. In the goal navigation phase, the robot simply navigates to the nearest point on the detected target object and triggers stop once it is within sufficient proximity.

Datasets. We evaluate our approach using the Habitat [5] simulator on the validation splits of three different datasets of 3D scans of real-world environments; Gibson [6], HM3D [8], and MP3D [7]. We use the ObjectNav validation split for Gibson developed in SemExp [16] which contains 1000 episodes across 5 scenes. HM3D’s validation split contains 2000 episodes across 20 scenes and 6 object categories. MP3D’s validation split contains 2195 episodes across 11 scenes and 21 object categories.

Metrics. For all approaches, we report success rate (SR) and Success weighted by inverse Path Length (SPL) [31]. SPL scores the efficiency of an agent’s path by comparing it to the length of the shortest path from the start position to the closest instance of the target object category. It is zero if the agent did not succeed; otherwise, it is the shortest path length divided by the agent’s path length (larger is better).

Baselines. We evaluate VLFM by comparing it to several state-of-the-art (SOTA) techniques for zero-shot object navigation: CLIP on Wheels (CoW) [1], ESC [2], SemUtil [3], and ZSON [32]. ZSON is an open vocabulary method that uses CLIP to transfer a method for a different navigation task (ImageNav) zero-shot to the ObjectNav task. It is trained on ImageNav in which images are used as goals instead of object names; at test-time, text-embeddings of object category names are used as goals instead. CoW explores the closest frontier until the target object is detected using either CLIP features or an open-vocabulary object detector, and then navigates directly to the detected target. Similarly to our approach, ESC and SemUtil both perform semantic frontier-based exploration, but frontiers are evaluated using nearby object detections that are converted to text and evaluated using a text-only model (e.g., an LLM, BERT) that also considers the target object category.

In addition to the above zero-shot SOTA methods, we also include a comparison with supervised methods: PONI [19], PIRLNav [15], RegQLearn [14], and SemExp [16]. SemExp and PONI both build maps during navigation and train task-specific policies to perform semantic inference to predict likely target locations to explore. PIRLNav is an end-to-end policy trained with behavioral cloning on 77k human demonstrations and fine-tuned with online deep reinforcement learning. RegQLearn is an end-to-end policy trained only with deep reinforcement learning.

In this section, we aim to address the following questions:

1. 1.

   How well can VLFM perform ObjectNav in various datasets in comparison to other trained or zero-shot methods?

2. 2.

   How do different methods of fusing current and previously seen values affect the performance of VLFM?

3. 3.

   Can VLFM be deployed successfully in the real world?

This paper presents VLFM, a zero-shot framework for ObjectNav in novel environments. Our key innovation is spatially grounding joint vision-language-based semantic reasoning with pre-trained models in a new approach to frontier waypoint selection in order to perform target-driven navigation in novel environments. VLFM uses a semantic prompt-based method to infer which frontier to navigate to using a pre-trained vision-language model, detects visible target objects with a pre-trained object detector, and navigates to these frontiers and target objects using a pre-trained policy. This modular nature of our method allows different components to be swapped in as improved models become available. Experiments in simulated 3D home environments demonstrate that VLFM achieves state-of-the-art zero-shot navigation performance on the Object Goal Navigation benchmark. Demonstrations in an office building on a Spot Arm platform prove the viability of our method in real-world scenarios.

VLFM has a number of limitations that could be addressed by future work. First, we assume target objects will be easily visible in the scene from the default height of the robot camera. Future work could investigate policies to increase interaction with the environment during the search process, including actively directing the robot camera to look in promising locations and using manipulation to execute search actions such as looking inside closed drawers. Furthermore, we build a map of semantic information and occupancy information with our frontier and value map, but the value map, which contains semantic information, provides only task-specific semantic information. So, we cannot leverage this map in sequentially executed semantic navigation tasks to different objects or in executing other navigation tasks requiring targets specified by language, such as vision-language navigation. Future work could explore alternate prompt formulations, value map designs, and methods of tracking semantic information during task execution to enable long-horizon planning and multitask execution.

Overall, our results highlight the promise of leveraging foundation models like BLIP-2 in a zero-shot manner within robotic systems to provide spatially grounded joint vision-language-based semantic reasoning without task-specific training.

The Georgia Tech effort was supported in part by ONR YIPs, ARO PECASE, and Korea Evaluation Institute of Industrial Technology (KEIT) funded by the Korea Government (MOTIE) under Grant No.20018216, Development of mobile intelligence SW for autonomous navigation of legged robots in dynamic and atypical environments for real application. The views and conclusions are those of the authors and should not be interpreted as representing the U.S. Government, or any sponsor.