Multi-modal Situated Reasoning in 3D Scenes

We formulate a situation with four components: (i) the location represented by numerical coordinates $s^{loc}=(x,y,z)$, (ii) the orientation represented by a rotation angle $s^{rot}$ within XY plane, (iii) the action description of the presumed agent, and (iv) the location description associating with surrounding objects. We provide illustrative examples for this formulation in \creffig:situation_examples.

The first step to sample a situation involves the sampling of location and orientation. We establish sampling rules tailored to four interaction types:

1. [nolistsep,noitemsep,leftmargin=*]

2. -

   Standing. We evenly sample a point from the floor area as the location and an angle within $[0,2\pi)$ as the orientation.

3. -

   Sitting. We randomly sample a point from the sitable area, \eg, chair and couch. The orientation is calculated based on the normal direction of the backrest.

4. -

   Interacting with large objects. For large objects, \eg, cabinet and refrigerator, we first parse the interactive part such as the door. Then we sample a standing point from the nearby floor as the location and use the normal direction of the interactive part as the orientation.

5. -

   Interacting with small objects. For small objects, \eg, bag and trash can, we first sample a standing point from the nearby floor as the location and then use the direction from the standing point to the object center as the orientation.

With the selected location and orientation, we then generate the descriptions. The action description of standing and sitting simply follows I am standing/sitting on the floor/obj. The action description of interacting is generated by prompting GPT-3.5 [openai2022chatgpt]. For location description, we first calculate the spatial relationships between the presumed agent and each object, including distance, coarse direction (\eg, left, right), and fine-grained direction denoted in clock style (\eg, 2 o’clock). We then prompt GPT-3.5 [openai2022chatgpt] with these spatial relationships to derive location descriptions.

Similar to prior works [huang2023embodied, jia2024sceneverse], we use scene-graph-based prompts for LLM-based automatic data generation. Differently, we propose the situated scene graph, where the relations depend on specific situations. We define the situated scene graph as $\mathcal{G}=(\mathcal{V,\mathcal{E}})$, where $\mathcal{V}$ is the set of nodes $v_{i}$ (objects) and $\mathcal{E}$ is the set of edges $e_{jk}$ (relations). Each node $v_{i}$ contains information about the object location, size, and attributes, \ie, color, shape, material, usage, texture, structure, and state.

We follow prior works [achlioptas2020referit3d, wald2020learning, jia2024sceneverse] in terms of scene graph generation. Specifically, we first instantiate the nodes according to the annotations of each object instance, and then obtain the attributes by prompting GPT-4V [openai2023gpt4] with cropped object images. We provide the details of attribute generation in Appendix. For the relations, we perform pair-wise calculation among the initialized objects. We consider five categories of relations: in-contact vertical relations (\eg, support, inside), non-contact vertical relations (\eg, above, below), horizontal distance (\eg, far, near), horizontal proximity relations (\eg, right, behind) and multi-object relations (\eg, align, between). In particular, the horizontal proximity relations between objects are derived conditioned on the situation (\ie, location and orientation).

Prior works [huang2023embodied, jia2024sceneverse] provide valuable experiences in scene-graph-based prompting for collecting QA pairs as well as its advantages over other approaches. Resembling previous conventions, we design several system prompts and hand-crafted seed examples to prompt GPT-3.5 [openai2022chatgpt] (see details in \supp). We also employ an Object-centric Chain of Thought scheme to elevate the data quality. Additionally, we categorize the questions into different types such as counting, spatial relations, navigation, \etcWe instruct GPT-3.5 [openai2022chatgpt] to also output the question type. Please refer to \crefsec:data_statistics for a detailed view of the question types.

To augment diversity of the generated QA pairs, we adopt various combinations of the seed examples for each run. Moreover, we propose to sample various sub-graphs given different visible distances and rotations. This elicits diverse scene graph prompts to generate richer QA pairs while mirroring the partially observable scenarios.

Refinement Procedure. Finally, we conduct refinement procedures on the generated data to uphold its integrity and quality. The refinement process encompasses two aspects. For the situated scene graphs, we examine the distribution of attributes and relations to mitigate any potential bias that could lead to hallucination. On the other hand, we conduct manual reviews on the generated QA pairs to ensure their alignment with the situated scene graph and verify the accuracy of responses. Several automatic functions based on regular expression matching are designed to correct the responses according to the scene graphs. Detailed statistics and the examples of the refinement procedures are provided in \supp.

Data Balance. Different from prior works about image based QA datasets[liu2023mmbench], \datasetnamefocuses on multi-modal spatial reasoning and situational awareness in 3D scenes. To assess such capabilities, we mainly collect question-answer pairs about object existence, counting, spatial relationship, direction-based navigation and object refer. The dataset also involves QA pairs of room type and detailed object description. Some questions such as What is the usage of the table? can be directly inferred from the questions without the 3D scene and the situation. To reduce such QA pairs, we control the distribution of the question types. Just as the Figure 1 shows, questions such as counting, existence, spatial relationship and object referring dominate the dataset, which are requires the strong capabilities of grounding and situational awareness in 3D scenes.

Prior works[huang2023embodied, zhao2023svit] about 2D VLMs and 3D VLMs have demonstrated the importance of data balance for the question type of existence. Unbalanced distribution of yes/no will lead to dramatic biased responses. To avoid this risk, we also augment the data distribution of existence. Specifically, we augment the QA pairs with no by the template such as Is there a red jacket on my right?/Can I find a car in the scene(room)?. In-the-scene objects and in-the-wild objects are both used in the template. For in-the-wild objects, we prompt ChatGPT to randomly generate more than 600 objects in a variety of domains. More details of this procedure can be found in appendix.

Through above framework, we get \msqacountmulti-modal situated reasoning question-answer pairs across ScanNet, 3RScan and ARKitScenes. We show some examples of \datasetnamein \creffig:data_examples.

In the scenario of 3D scene reasoning, progress is significantly hindered by the scarcity of high-quality data. Not only we take the first step to develop a universal pipeline to construct large scale multi-modal situated 3D scene reasoning cross multiple data sources, but also we first design a detailed procedure to comprehensively assess the data quality. To this end, we develop web pages to visualize and score the data instance manually. We assess the data quality considering following principles:

Question. A high quality question should be situational, spatial aware and unambiguous. For example, questions like: How many brown wooden tables are near me on my right?/Is the door open on my left front me? are both perfect examples.

Situation. Ideally, the situation description can locate the agent in the 3D scene uniquely. For example, a situation description: There is a gray trash can on my right in a middle distance. I am sitting on a black chair. In this description, the spatial relationship, distance and activity are clearly presented.

Answer. The correctness is crucial to guide the model to reason in 3D scenes. The questions with accurate and unique answers such as existence and counting can be scored according to the 3D scene, situation and question. For the questions such as describing attributes for a queried object, correctness of description and richness of detail are both factors considered.

is constructed based on ScanNet[dai2017scannet], 3RScan[wald2019rio] and ARKitScenes[baruch2021arkitscenes]. We visualize the distribution of the QA types and the HOI types in \creffig:QA_type_distribution and \creffig:HOI_type_distribution. Our dataset also designs train/val/test set according to 3D scenes. For ScanNet, we follow SQA3D’s practice for dataset split, for ARKitScenes and 3RScan, we follow their official split. More statistic results can be found in the Appendix. We provide the examples of \datasetnamein \creffig:data_examples.

We provide the experimental results in \creftable:mm_eval. The results are averaged across 3 datasets. Since \datasetnameincorporates 3D scenes and interleaved image-text question, which is beyond most models’ capability, we replace the image placeholders with the labels and translate our dataset to 3D and text only. In this way, LEO can be fine-tuned based on our training set. For LLMs and 2D VLMs, the scene is presented by an object list with the ground truth attributes. The detailed prompting can be found in Appendix.

Since \modelnameis built based on LEO, the results 56.48 vs. 55.86 indicates the advantage of our situation modeling method. For \modelname, we also conduct blind test, denoted as ’FT+blind test’ in \creftable:mm_eval. During test stage, we replace the scene tokens with random numbers before they are passed into the tokenizer. The dramatic performance drop indicates the importance of 3D scene representation for the multi-modal situated reasoning.

To further evaluate the 3D scene understanding and situation awareness, we introduce a novel task named as \shortnavi. Differently, this task has no question in the input prompting. Besides, the responses are limited to an action space, with four direction: left, right, forward and backward.

For fine-tuned models, we consider two settings: training from scratch and pretraining. The results of \modelnameindicate the usefulness of \datasetnamefor the situated downstream task. The results are reported in \creftable:eval_one_step_nav.

The results(45.61 vs. 37.05) indicate that \modelnameoutperforms LEO by a large margin in this task, demonstrating the advantage of our situation modeling method since they share the same model architecture.

To verify the advantage of \datasetnameover existing dataset such as LEO-align dataset, we conduct SFT experiments based on LEO-align pre-training model and \datasetnamepre-training model. The results(37.05 vs. 31.44) demonstrate \datasetnameis much more useful than LEO-align data for \shortnavi, even the amount of data in the latter is about 4 times that of ours. It is intuitive as \datasetnameis a situated reasoning dataset that will improve the model’s situation awareness while LEO-align is a mixed dataset collected from several non-situated 3D scene understanding tasks. Besides, we also conduct a blind test to verify the usefulness of location and orientation for this task. In the test stage, we randomly replace the location and orientation before they are passed into the scene encoder. The significant performance drop indicates the importance of such information for \shortnavi.

The results indicate that zero-shot models such as GPT-3.5 and GPT-4o struggle in this task.

To verify if SFT models will benefit from data scaling, we consider 3 types of diversity: scene, QA pair and situation. \creffig:scaling_MSQA and \creffig:scaling_one_step denote the scaling effect on held-in task and downstream task respectively. Several pretrained models based on different sampled subset are used for scaling analysis. As the quantity pre-training data increases, the performance on \datasetnameand \shortnaviincreases gradually. The results indicate that both held-in and downstream task benefit from data scaling.

We collect \datasetnamedata from ScanNet [dai2017scannet], 3RScan [wald2019rio] and ARKitScenes [baruch2021arkitscenes] based on a universal pipeline. Thus it can evaluate the capability of cross domain transferring for end-to-end models.

Specifically, we train MSR3D on \datasetname-ScanNet, \datasetname-3RScan and \datasetname-ARKitScenes respectively and get 3 domain specific models. Then we test the performances on 3 test sets and report the SFT(supervised-fine-tune) results on in-domain and the zero-shot results on out-domain datasets. The \creftable:cross_domain_transfer shows that the model can generalize to the novel domain in some context. The zero-shot performances 2.8092 vs. 2.6015 and 2.8526 vs. 2.4796 indicate that more training data and complex source domain lead to better generalization capability. The results in \creftable:cross_domain_transfer shows a promising direction to generalize the situated reasoning capability from base domains to novel domains.