Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation

Visual Object Navigation (VON) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], which is considered fundamental for home service agents, refers to the task in which an agent is required to find a user-specified object within a provided scene. Previous VON studies can be broadly categorised into two groups. The first category, known as closed-vocabulary navigation [1, 2, 3, 4, 6, 7, 8, 9, 10], involves a predetermined set of target object categories so that the agent is restricted to selecting objects from this predefined set. On the other hand, open-vocabulary navigation [5, 11, 12] encompasses a more flexible approach where the target objects are described in natural language, meaning the agent is not limited to specific predefined categories.

In real-world applications of VON, users are typically required to provide the name of the object they want the agent to locate, regardless of whether it falls under the closed-vocabulary or open-vocabulary group. This requirement introduces two conditions that must be fulfilled: 1) the user must possess knowledge of the object’s name in the given environment; and 2) the specified object exists within the environment. These conditions are necessary for successful object navigation tasks in practical scenarios. In common simulators (such as AI2Thor [16] and Habitat [17, 18]), these two conditions can be easily satisfied. As direct access to the scene’s metadata is available in the simulators, the existence of target objects can be checked using the metadata, or the object names in the metadata can be directly used as target objects. However, in the real world, where the user may not be omniscient of the environment (e.g., at a friend’s house), the two conditions described above may not be met, which will result in failure to satisfy the user’s demand for an object. Additionally, it is possible that when the user specifies an object to search for the agent is able to find an object with a similar function within the scene, when the specified object does not exist. For example, an object such as a bottle of water may be specified which has a related demand, to quench thirst. In the case where a bottle of water is not present in the scene, this demand can be adequately met by a cup of tea or apple juice found by the agent, however this would fail the VON task. This challenge illustrates a key difficulty VON faces in real environments.

From a psychological point of view [19, 20, 21, 22], this paper considers the user’s motivation to make the agent search for an object. In the VON task the user initially has a certain demand, then thinks of an object to satisfy it, and finally asks the agent to search for that specific object. Then, would it not be more consistent with the constraints of the real world environment to omit the second step and directly use the demand as an instruction to ask the agent to find an object that satisfies the demand? This paper argues that utilising user demands as instructions has the following benefits: 1) Users no longer need to consider the objects in the environment, but only their own demands. This demand-driven description of the task is more in line with the psychological perspective of human motivations. 2) The mapping between demands and objects is many-to-many (i.e., a demand instruction can be satisfied by multiple objects, and an object can potentially satisfy multiple demand instructions), which can theoretically increase the probability of an agent satisfying user demands in comparison to VON. 3) The demands, when described in natural language have a wider descriptive space than just specifying an object, and the tasks are thus described more naturally and in accordance with the daily habits of humans.

Therefore, this paper proposes the task of Demand-Driven Navigation (DDN). In the DDN task, the agent receives a demand instruction described in natural language (e.g., I am thirsty), and then the agent must search for an object within the provided environment to satisfy this demand. Our task dataset is generated semi-automatically: it is initially generated by GPT-3 [23] (In the entire paper, we use gpt-3.5-turbo version), and then manually filtered and supplemented. The DDN task presents new challenges for embodied agents: 1) Different real-world environments contain different objects and, even within a fixed environment, the categories of objects can change over time due to human interaction with the environment, resulting in agents potentially looking for different objects when attempting to satisfy the same demand. 2) The many-to-many mapping between instructions and objects requires agents to reason depending on common sense knowledge, human preferences and scene grounding information: to identify what objects potentially exist in the scene to satisfy the user’s demand, and where these objects are most likely to be located. 3) The agent needs to determine whether the current objects within the field of view satisfy the user’s demand from the objects’ visual geometry. There exists the possibility that this reasoning is not only based on the visual features, but stems from the functionality of the object, necessitating the agent having common sense knowledge about the object. In summary, the core challenge of DDN is how to use common sense knowledge, human preferences and scene grounding information to interpret demand instructions and efficiently locate and identify objects that satisfy the user’s demands.

It is noted that the essence of an object satisfying a demand is that certain attributes of the object fulfill the demand instructions. For example, for the demand of “I am thirsty,” the object to be sought only needs to have attributes such as “potable” and “can quench thirst”. Both a bottle of water and a cup of tea are objects that have these attributes. To address the challenge of mapping attributes and demand instructions while considering their relevance to the current scene, this paper proposes a novel method that extracts the common sense knowledge from LLMs to learn the demand-conditioned textual attribute features of objects. Then the proposed method aligns these textual attribute features with visual attribute features using the multi-modal model CLIP [24]. The learned visual attribute features contain common sense knowledge and human preferences from LLMs, and also obtain scene grounding information via CLIP. This paper also trains a demand-based visual grounding model to output the bounding box of objects in the RGB input that match the demand when an episode ends. Experiments are conducted on the AI2Thor simulator [16] with the ProcThor dataset [25]. The proposed method is compared with closed-vocabulary object navigation, open-vocabulary object navigation, and their variants. This paper also evaluates the performance of GPT-3 and multi-modal large language model(MM-LLM), MiniGPT-4 [26], operated agents which accomplish this task via prompt engineering. The experiments demonstrate that the demand-conditioned attribute features extracted from the LLM and aligned with CLIP effectively assist in navigation and outperform baselines. In summary, this paper’s contributions are listed as follows:

• •

  This paper proposes the task of Demand-Driven Navigation (DDN), that requires the agent to find an object to meet the user’s demand. The DDN task relies heavily on common sense knowledge, human preferences, and scene grounding information.

• •

  This paper evaluates two different categories of VON algorithms and their variants combined with GPT-3, as well as the performance of large language models (e.g., GPT-3 and MiniGPT-4) on the DDN task. The results demonstrate that existing algorithms have difficulty solving the DDN task.

• •

  This paper provides a novel method to tackle the DDN task and a benchmark for this. The proposed method extracts the common sense knowledge from LLMs to learn textual attribute features and uses CLIP to align the textual and visual attribute features. The experiments demonstrate that the demand-conditioned attribute features extracted from the LLM and aligned with CLIP effectively assist in navigation and outperform the baselines.

In the DDN task, an agent is randomly initialised at a starting position and orientation in a mapless and unseen environment. The agent is required to find an object that meets a demand instruction described in natural language (e.g., “I am thirsty”). Formally, let $\mathcal{D}$ denote a set of demand instructions, $\mathcal{S}$ denote a set of navigable scenes, and $\mathcal{O}$ denote a set of object categories that exist in the real world. To determine whether a found object satisfies the demand, let $\mathcal{G}:D\times O\rightarrow\{0,1\}$ denote a discriminator, outputting 1 if the input object satisfies the demand, or 0 if it does not. In the real world, $\mathcal{G}$ is effectuated by the user, but to quantitatively evaluate the proposed method and the baselines, $\mathcal{G}$ is effectuated by the DDN dataset collected in Sec. 4.

At the beginning of each episode, the agent is initialised at a scene $s\in\mathcal{S}$ with a initial pose $p_{0}$ (i.e., position and orientation), and provided a natural language demand instruction $d=<d_{1},d_{2}\cdots d_{L}>\in\mathcal{D}$, where $L$ is the length of the instruction and $d_{i}$ is a single word token. In the demand instruction $d$, the task is made more challenging by not providing specific object names. This is to ensure the agent learns to reason in scenarios where these objects are not specified by the user. Then the agent is required to find an object $o\in\mathcal{O}$ to satisfy the demand instruction $d$ and only take RGB images as sensor inputs. The agent’s action space is $\mathrm{MoveAhead},\mathrm{RotateRight},\mathrm{RotateLeft},\mathrm{LookUp},\mathrm{LookDown}$, and $\mathrm{Done}$. When the agent selects the action $\mathrm{Done}$, the agent is also required to output a bounding box $b$ to indicate the object that satisfies the demand within the current field of view RGB image. Then it is determined whether the success criteria have been satisfied. The success criteria for the DDN task include two conditions. 1) The navigation success criterion: this requires there is an object in the field of view that satisfies the demand instruction and the horizontal distance between the agent and the object is less than a threshold $c_{navi}$. 2) The selection success criterion: this requires that given that the navigation is successful, the intersection over union (IoU) [56] between the output bounding box $b$ and the ground truth bounding box of an object that satisfies the demand instruction is greater than a threshold $c_{sele}$. There is a 100 step limit to succeed during the episode.

Although mappings between objects and instructions will vary with the environment, a fixed mapping between demand instructions and objects is required to check selection success for training the agent in a specific environment, referred to as world-grounding mappings (WG mappings). The modifier “world-grounding” emphasises the connection to the specific environment of the real world and will vary with the environment. For example, the ProcThor dataset  [25] and the Replica dataset [57] will have different WG mappings. To simulate real-world scenarios, it is assumed, unless otherwise stated, that the agent does not have access to the metadata of the object categories in the environment or the WG mappings that are used to determine selection success during training, validation and testing. The set of world-grounding mappings constructed in this paper form the demand-driven navigation dataset.

To construct the WG mappings, the metadata regarding object categories in the environment is obtained and GPT-3 is used to establish a fixed set of WG mappings between demand instructions and objects $\mathcal{F}:\mathcal{D}\rightarrow\mathcal{O_{s}}$, where $\mathcal{O_{s}}$ is a subset of $\mathcal{O}$. For example, $\mathcal{F}(``\text{I}\ \text{am}\ \text{thirsty}^{\prime\prime})=\{\text{Water},\text{Tea},\text{AppleJuice}\}$. Concretely, prompt engineering is used to inform GPT-3 of the object categories that may be present in the experimental environment. Then GPT-3 is used to determine which demands these objects can satisfy and returns this information in the form of a demand instruction and a statement of which objects present satisfy this demand instruction. It has been found that the WG mappings generated by GPT-3 are not absolutely accurate. Due to the presence of some errors in the generation, manual filtering and supplementation is then used to correct and enhance the dataset. These WG mappings $\mathcal{F}$ are used during training and testing, only to discriminate the selection success (i.e., to effectuate discriminator $\mathcal{G}$). In total, approximately 2600 WG mappings are generated. Please see supplementary material for details of the generation process, the prompts, and the statistical features of the dataset.

It is important to note that if an object can fulfill a demand, it is because one of its functions or attributes can satisfy that demand. In other words, if multiple objects can fulfill the same demand, the objects should have similar attributes. For example, if your demand is to heat food, the corresponding objects that can satisfy the demand are a stove, an oven or a microwave, which all have similar attributes such as heating and temperature control. The relationship between attributes and instructions is not specific to a particular environment, but instead is universal common sense knowledge. Thus, this common sense knowledge can be extracted from LLMs. This universality turns the multi-object-target search into single-attribute-target search: the similarity between attributes reduces the complexity of policy learning. Therefore, we expect to extract the attribute features of the objects conditioned on the given demand from the LLM, i.e.,, demand-conditioned textual attribute features.

In this section, how the textual attribute features are learned from LLMs (Sec. 5.1), how they are aligned from text to vision (Sec. 5.2), and how to train the model (Sec. 5.3).

This paper proposes Demand-driven Navigation which requires an agent to locate an object within the current environment to satisfy a user’s demand. The DDN task presents new challenges for embodied agents, which rely heavily on reasoning with common sense knowledge, human preferences and scene grounding information. The existing VON methods are modified to fit to the proposed DDN task and they have not been found to have satisfactory performance on the DDN task. The LLM GPT-3 and the open source MM-LLM, MiniGPT-4 also show some drawbacks on the DDN task, such as being unable to determine the distance between the object and the agent from visual inputs. This paper proposes to learn demand-conditioned object attribute features from LLMs and align them to visual navigation via CLIP. Such attribute features not only contain common sense knowledge and human preferences from LLMs, but also obtain the scene grounding information obtained from CLIP. Moreover, the attribute features are consistent across objects (as long as they satisfy the same demand), reducing the complexity of policy learning. Experimental results on AI2thor with the ProcThor dataset demonstrate that the proposed method is effective and outperforms the baselines.

This work was supported by National Natural Science Foundation of China - General Program (62376006) and The National Youth Talent Support Program (8200800081). We also thank Benjamin Redhead for correcting and polishing the grammar and wording of our paper.

This section will elaborate on some details. The table of contents is as follows.

• •

  Demand-Driven Navigation Dataset 8.1

  • –

    Generation Process 8.1.1

  • –

    Statistical Features 8.1.2

• •

  Demand-Driven Navigation Method 8.2

  • –

    Textual Attribute Feature Learning 8.2.1

    • *

      Language-grounding Mappings 8.2.1

    • *

      t-SNE of Demand-conditioned Attribute Feature 8.2.1

  • –

    Policy Learning 8.2.2

    • *

      Pipeline in Transformer 8.2.2

    • *

      Trajectory Collection 8.2.2

    • *

      Demand-based Visual Grounding Model 8.2.2

  • –

    Model Pre-training and Fine-tuning(8.2.3

    • *

      Image Encoder: Vision Transformer 8.2.3

    • *

      DETR 8.2.3

• •

  Experiments 8.3

  • –

    Metrics 8.3.1

  • –

    Baselines 8.3.2

    • *

      VTN-object, VTN-demand, VTN-GPT 8.3.2

    • *

      ZSON-object, ZSON-demand, ZSON-GPT 8.3.2

    • *

      GPT-3+Prompt, MiniGPT-3 8.3.2

• •

  Ethics 8.4