CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) requires agents to follow human instructions in unseen environments. This is a challenging problem since instructions heavily rely on the contents of the scene, possibly unknown to a general agent. A common approach to VLN is to use supervised learning; we argue that this is not practical, given the drastic shift in semantics from scene to scene that impacts the performance of a trained model. We observe this phenomenon in SOTA supervised learning approaches for VLN, which have large performance drops on “unseen” environments that are absent from the training dataset.

In this work, we seek to address this issue of generalizing to new environments, and propose to solve VLN in a fully zero-shot manner. The agent is assumed to have no prior knowledge about the instructions or the environment. This setting is does not rely on a particular VLN dataset and is free from any environmental bias that datasets may inherently have.

A substantial body of prior work tackles VLN using supervised learning Gu et al. (2022); Guhur et al. (2021); Wang et al. (2019b); Hong et al. (2021). Models are first trained on a set of “seen” environments and instructions, before being evaluated on both “seen” and “unseen” test data. This paradigm often involves encoding instructions and learning topological relations during training Chen et al. (2021); Wang et al. (2021); Chen et al. (2022a), and imitating navigational decisions during inference. These approaches usually see a significant drop in performance on unseen data. Figure 1 compares results of popular VLN approaches on the coarse-grained instruction following task of REVERIE Qi et al. (2020). Observe the significant drop in performance on the “Unseen Val” and “Unseen Test” sets, when compared with “Seen Val” which validates on previously seen environments. This drop holds across all approaches. Performance in comparison with humans is also significantly worse, not just for REVERIE, but also for other popular instruction following tasks such as R2R Anderson et al. (2018) and SOON Zhu et al. (2021).

Current VLN models are also dataset specific; a trained model from one will not generalize to another. For instance, training on REVERIE and testing on SOON may not give comparable results, despite both involving the coarse-grained instruction following task. A critical factor affecting this generalization is the lack of sufficient training data, and a few articles address this issue by proposing data augmentation techniques Chen et al. (2022b); Li et al. (2022). However, data augmentation has only shown minor improvements in performance.

Household environments are visually diverse, each with unique layouts and objects. As such, instructions given by humans to describe target locations usually contain environment-specific cues. For example, the same target item “bottle” could be described as “next to the mirror” in one instruction and as “above the sink” in another. There could also be unique environment-specific identifiers such as “orange-striped bathroom” or “painting depicting crudely-drawn people”.

We hypothesize that the visual distinctiveness of environments causes people to use unique language involving those scenes. In our inspection of the REVERIE dataset we noticed a low pairwise average cosine similarity of $0.32$ between all seen and unseen instructions, substantiating this claim. As all environments are potentially distinct from one another, training datasets fail to capture all the visuo-linguistic diversity needed to generalize (Figure 2).

Our work makes the following contributions:

• •

  We present a novel approach to solve coarse-grained instruction following tasks: breaking down guidance language into keyphrases, visually grounding them, and using the grounding scores to drive CLIP-Nav, a novel “zero-shot” navigation scheme.

• •

  Further, we incorporate backtracking into our scheme, and present Seq CLIP-Nav which drastically improves CLIP-Nav’s results, showcasing the importance of backtracking in solving such tasks.

• •

  Our results establish a zero-shot baseline on the task of REVERIE Qi et al. (2020), surpassing the unseen supervised baseline without any form of dataset-specific finetuning in terms of SR and SPL. Our SPL results on this dataset also improves on the SOTA.

• •

  Finally, we establish a new metric to measure generalizability in VLN tasks — Relative Change in Success (RCS). This metric quantitatively showcases the improved performance of our CLIP-based approaches over other supervised methods.

Vision-and-Language Navigation (VLN): Coarse-grained Qi et al. (2020); Zhu et al. (2021) and fine-grained Anderson et al. (2018); Ku et al. (2020); Vasudevan et al. (2021) instruction following tasks have recently been of great interest. A majority of approaches attempting to solve these tasks use some form of supervision, employing behavior cloning Huang et al. (2019), reinforcement learning Wang et al. (2018), or even imitation learning Wang et al. (2019a). These approaches are limited to solving VLN either on a single or a specific set of datasets, and do little to analyze cross-dataset performance. Since VLN by definition is intended to be solved in arbitrary environments, we argue that methods should give consistent performance across all scenes, previously seen or not.

Zero-Shot VLN: A recent paradigm shift in machine learning has led to the emergence of large deep learning models that are pre-trained on vast amounts of unlabeled data, and finetuned on a variety of downstream tasks. Examples of this include BERT Devlin et al. (2018) and GPT-3 Brown et al. (2020), which have shown SOTA performance on various natural language tasks. For zero-shot VLN, we are particularly interested utilizing CLIP Radford et al. (2021), which has shown improved zero-shot performance on several downstream vision-language tasks Shen et al. (2021).

Gadre et al. (2022) in CLIP on Wheels (CoW) attempt to utilize CLIP to perform zero-shot object navigation. Object navigation involves exploring the environment to find a target object without any guidance instructions. CoW modularizes this task into exploration and object localization, and uses CLIP for the latter.

While underlying task of navigating to a target location is the same, VLN is very different from object navigation. VLN uses natural human instruction language as opposed to template language. For example, a coarse-grained VLN instruction would be “Go to the kitchen on your right and water the plant there.”, while object navigation uses only the target word “plant”. CoW augments these words with templates (a photo of a $<$object$>$ in a video game) and then grounds them with CLIP. We instead use CLIP not just to ground keyphrases extracted from the VLN instruction, but also to drive our exploration policy.

To the best of our knowledge, there are no works that attempt to solve coarse-grained instruction following in a generalized, zero-shot setting, which forms the basis of our research. We look at solving VLN without any form of dataset-specific finetuning, and seek to transfer CLIP’s powerful language grounding capabilities into a sequential decision making pipeline for navigation.

Performing zero-shot VLN in a household environment requires the agent to have a sense of structural priors to make sequential decisions about how to get to an unseen target location. Consider Figure 3 for instance. The command is for the robot to “Go to the kitchen”, but the panoramic view does not contain visual elements associated with a kitchen. In such a case, we require the agent to pick the best view (and consequently the best direction) that would potentially lead to a kitchen. In this work, we use CLIP to make sequential navigational decisions, asserting its capability in capturing structural priors of indoor hosuehold environments.

Our approach consists of three steps:

1. 1.

   Instruction Breakdown - Decompose coarse-grained instructions into keyphrases.

2. 2.

   Vision-Language Grounding - Ground keyphrases in the environment using CLIP.

3. 3.

   Zero-Shot Navigation - Utilize the CLIP scores to make navigational decisions.

Given that our approach is zero-shot and requires no finetuning, we could choose any fixed set of datapoints for evaluation. We opt to use the Seen and Unseen validation splits of the REVERIE Qi et al. (2020), a popular dataset to conduct our experiments. Cross-comparing results between these splits helps us infer the generalizability of our approach, while also enabling similar comparisons on other supervised learning approaches on this dataset.

REVERIE contains several human-annotated instructions for paths taken from the Matterport3D Chang et al. (2017) environment. The language used in the instructions uniquely capture various aspects of the environment. Each path is described as a discrete set of adjacent photorealistic panoramic images. They contain around 8 adjacent pano images (or hops) on average, which have been annotated with coarse-grained language guidance to reach target locations.

Our results establish a baseline for zero-shot VLN, on the task of REVERIE. We use the following metrics to evaluate agent performance.

1. 1.

   Success Rate (SR) - This is the fraction of episodes where the agent successfully reaches and stops at the target location.

2. 2.

   Success Weighted by Inverse Path Length (SPL) - We use the definition provided by Batra et.al. in Batra et al. (2020). This is a standard metric used in several VLN tasks, and tells us about the optimality of the agent’s success route, when compared with the ground-truth oracle path.

3. 3.

   Oracle Succss Rate (OSR) - This is the fraction of episodes where the agent passes through the target location in its path, but does not stop there. This accounts for paths that may have overshot the target, due to inaccurate grounding.

4. 4.

   Relative Change in Success (RCS) - We also compute the percentage relative change in performance between Seen and Unseen data, which gives us insight into the generalizability of each approach. This is defined as -

   $$RCS=\frac{\lvert Seen-Unseen\rvert}{\max\{Seen,Unseen\}}\times 100$$

   A lower score indicates that the agent is performing similarly across the splits, while a higher score indicates overfitting on the seen training environments.

Table 1 compares our Zero-Shot (ZS) results on the Seen and Unseen Val splits of REVERIE with SOTA fully supervised methods. We also compare with a Random Walk approach that chooses a random neighboring node for $8$ steps. This is the upper limit on the average path length of the dataset which is between $5$-$8$ steps. Observe the improvement in performance of Seq CLIP-Nav over the REVERIE Baseline approach (FAST-MATTN), in terms of SR and SPL on the Unseen split. The Unseen SPL even outperforms the SOTA supervised learning approaches - Airbert and DUET, showing that when our agent does take the right path in an unseen environment, it tends to do so in a more optimal manner. Also notice the significant improvement in the SR performance of Seq CLIP-Nav over CLIP-Nav, even while the OSRs are similar. This indicates the strong influence of backtracking, in preventing the agent from overshooting once it reaches a target location.

On the Seen split, the supervised approaches outperform our methods in terms of SR and SPL, and this is quite evident since they have been trained on these environments. The higher RCS scores on these methods however indicate that they perform much worse on unseen environments in comparison to seen ones. This shows poor generalizability, and that they might have overfit on the training dataset. In contrast, the lower RCS scores across SR, OSR and SPL on our CLIP-based approaches indicates a far more consistent performance. We can thus quantitatively infer that our approaches generalize better in new environments over supervised approaches, satisfying one of our primary objectives for zero-shot navigation. This is a promising result, as it shows that our CLIP-based models without any finetuning are able to make consistent embodied navigational decisions irrespective of the type of environment that the agent is placed in.

Additionally, we also obtain “Best OSR” results of $47.51\%$ on CLIP-Nav, and $48.41\%$ on Seq CLIP-Nav on the unseen split. These are the single best values across all the house scans. The higher values when compared to the overall OSR shows that our approach is able to perform particularly well in certain types of environments, and analyzing the influencing visuo-lingual factors to transfer knowledge is part of a future study.

We tackle the problem of Vision-and-Language Navigation (VLN) in a fully zero-shot manner to address the issue of generalizability in unseen environments. Our approaches CLIP-Nav, and Seq CLIP-Nav give a far more consistent performance over fully supervised SOTA approaches on the REVERIE dataset when assessed by our generalizability metric — Relative Change in Success (RCS). Seq CLIP-Nav in particular gives improved results over the REVERIE baseline on the unseen validation set in terms of SR, and also improves upon SOTA supervised learning approaches in terms of SPL. This showcases the capability of CLIP without any dataset-specific finetuning in being able to make accurate sequential navigational decisions necessary for zero-shot VLN.

In the future, we intend to study the cross-dataset performance of CLIP-Nav on other fine and coarse-grained instruction sets in indoor and outdoor settings as well as when there is a cooperative dialog for instruction following Padmakumar et al. (2022); Gao et al. (2022). Our current backtracking score while simplistic, still drastically influences performance; improving this via meta-learning is a future study. Yet another exciting direction is a Virtual Reality (VR) experiment to learn human patterns in zero-shot instruction following. Analyzing people’s zero-shot paths in VR environments could give us insight into their intuition for sequential decision making, and can be modeled to improve CLIP-Nav.