Instance-incremental Scene Graph Generation from Real-world Point Clouds via Normalizing Flows

The scene graph was first proposed in 2D computer vision by [28]. This work used a scene graph to represent the semantic label of objects and their relationships for image retrieval. Later, there are lines of works promoting the studies of image-based scene graphs [29, 14, 11, 30, 8, 9, 16, 17]. These works proposed different approaches to deal with the scene graph representation problem, such as bidirectional-LSTM-based MotifNet [29], a variant of GCN [11], a variant of Transformer [8, 9], and a combination of RNN and attention mechanism [16]. Most of these methods relied on an object detector to extract node and edge features to compute 2D scene graphs. However, only a few works explored the representation of scene graphs in 3D point clouds due to the lack of datasets. 3RScan [31] dataset promoted the studies of 3D scene graphs and was regarded as a new benchmark for 3D scene graph learning [18, 7]. [18] proposed a novel method based on PointNet and GCN to regress a scene graph from the point-cloud-based scene. [7] put forward an improved Edge-oriented Graph Convolutional Network (EdgeGCN) to exploit multi-dimensional edge features for explicit relationship modeling, and experiments showed promising results. All these works focused on learning the scene graph from 2D/3D data through recognizing objects and their relationships. Although different from the ultimate goal of our task, the graph representation methods inspire us a lot.

Similar tasks of graph generation from images have been explored before. [25] proposed a new task of generating unconditional scene graphs with a seed object as the input. The autoregressive method [25] and the VAE-based method [26] learned the inner distribution of the observed images’ scene graphs and generated novel scene graphs by distribution sampling. [27] proposed an improved task named ‘Scene Graph Expansion.’ This task enriched an input seed graph by adding new nodes and the corresponding relationships. The authors also proposed an autoregressive method with an improved sequencing strategy to tackle this task.

Things make a difference between ours and the above image-based task. Our task outputs a continuous graph to describe the continuous spatial relationships in a global 3D scene, and all the graphic elements are within a whole graph. However, the above image-based tasks describe the spatial relationships in different regions of 2D views—the output may be a discontinuous graph consisting of isolated nodes/triples or subgraphs.

Numerous works applied deep models of instance-incremental graph generation in other fields, especially in molecular design [32, 21, 33] and social network prediction [34, 35]. [32, 34] respectively proposed VAE-based frameworks to generate semantically valid graphs used in molecule generation and link prediction of social networks. [21] designed the first energy-based model GraphEBM for generating molecular graphs. It applied Langevin dynamics to train a graph’s energy function by maximizing the likelihood and generating samples with low energies. The state-of-the-art models are built based on autoregressive approaches. [33] formulated the graph generation problem as a sequential decision process and added new nodes and edges based on current subgraph structures. GraphRNN [36] tackled this sequential decision problem using RNN, and the improved MolecularRNN [37] added the judgment of semantic labels of edges.

Recently, autoregressive generative models with normalizing flows have achieved good experimental results in many tasks [23, 38]. This flow-based approach maps the graph data to a latent base distribution (e.g., Gaussian). The invertible transformation gives the model a high capacity to model high-dimensional data. However, these methods do not consider the constraints of real-world entities, leaving gaps to be used in graph generation based on real-world 3D scenes. Our study fills the gap and introduces the flow-based model to instance-incremental graph generation in 3D space.

We employ a two-stage framework to tackle this generation problem, as presented in Fig. 2. In stage (a), the graph representation module, learned from observation data, converts the rooms of 3D points into graphs. In stages (b) and (${\rm{\tilde{b}}}$), the distribution mapping module captures observations’ inner object layout experiences; The invertible process, graph generation, leverages the modeled experiences to create graphic elements under the conditions of the current subgraph. All the details will be discussed below.

The structural representation of the point cloud is the basis of the learning experiences and extracting conditions from scenes. To achieve the structural representation, we explore a way to convert the unordered points to a semantically meaningful graph denoting instances with relationships. As illustrated in Fig. 3, we propose a module to extract the features of 3D instances (wall, floor, etc.) and learn the node/edge representation.

Specifically, parameter-sharing multi-layer perceptron (MLP) captures the point-wise features of the input point cloud D. Then, the max-pooling operates on the unordered encoded point cloud, along with the class-agnostic point-to-instance indicator $M\in{\{1,\ldots,m\}^{n}}$ [39, 7], resulting in the order-invariant encoding of the included m instances. ${X_{v}}\in{\mathbb{R}^{m\times 256}}$ denotes the instance-wise feature with 256 channels.

${X_{v}}$ is further propagated to facilitate the modeling of the representation of m nodes and ${m^{2}}$ one-to-one edges in ${G_{m}}=(V,E)$, as $V\in{\mathbb{R}^{m\times{c_{n}}}}$ and $E\in{\mathbb{R}^{m\times m\times{c_{e}}}}$. ${c_{n}}$ and ${c_{e}}$ equal to the number of object classes and relationship classes, respectively. In node representation learning, MLP performs on ${X_{v}}$ for node feature evolution, converting the channel number from 256 to the final ${c_{n}}$. In edge representation learning, we introduce an edge feature matrix ${E_{init}}\in{\mathbb{R}^{m\times m\times 256}}$ to establish the initial relationships between instances, where ${E_{init}}[i,j,:]={X_{v}}[i,:]-{X_{v}}[j,:]$ and $i,j\in\{0,\ldots,m-1\}$. MLP performs on ${E_{init}}$ for edge feature evolution, resulting in the final edge representation.

One of the core problems in this novel task is how to generate graphic elements (nodes/edges) ${\mathcal{Z}_{t}}$ under the conditions of the current subgraph ${G_{t}}$. This autoregressive conditional probability can be denoted as $P({z_{t}}|{G_{t}})$, where ${z_{t}}$ is the representation of ${\mathcal{Z}_{t}}$ with the one-hot label, and t refers to the t-th iteration. However, the graphic elements are discrete and follow complex data distributions. It poses challenges to calculate $P({z_{t}}|{G_{t}})$.

Normalizing flows define an invertible mapping between samples with complex distributions and samples conforming to base distributions [19]. It can map a complex graphic element ${z_{t}}$ into a sample ${\varepsilon_{t}}$ of Gaussian, achieving the data distribution modeling of graphic elements in the training stage. Besides, it can map a sample ${\varepsilon_{t}}$ of the Gaussian back to a graphic element ${z_{t}}$ in the generation stage.

We introduce it into our task and denote $P({z_{t}}|{G_{t}})$ as a Gaussian process:

Where ${\mu_{t}}$ and ${\sigma_{t}}$ are the Gaussian parameters decided by the current subgraph ${G_{t}}$ (further discussed in the Condition Evaluation paragraph). Through an affine transformation [23], ${z_{t}}$ can be converted to ${\varepsilon_{t}}$ conforming to the standard Gaussian ${\cal N}\left({0,1}\right)$.

We explain how to make equation (1) work and the details of the invertible distribution mapping in Fig. 4. As normalizing flows work on continuous data, thus we first convert discrete ${z_{t}}$ into continuous data, where $\alpha$ is the weight coefficient of $\gamma\sim U[0,1)$ [40].

In the training phase, condition evaluation functions work on $G_{t}$, resulting in Gaussian parameters ($\mu_{t}$, $\sigma_{t}$). Through the affine function $f^{-1}(\cdot)$, $z_{t}$ is mapped to $\varepsilon_{t}$. The equation (1) is workable if all the $\varepsilon_{t}$ accord with the standard Gaussian. The normalizing flows method calculates the distribution density behind equation (1) as [23]:

Where ${p_{Z}}\left(\cdot\right)$ is the exact density of the graphic elements, ${p_{N}}\left(\cdot\right)$ is the density of standard Gaussian, $\det\frac{{\partial{f^{-1}}\left({{z_{t}}}\right)}}{{\partial{z_{t}}}}$ is the Jacobian determinant. The Gaussian transformation is achieved by minimizing the negative log-likelihoods of equation (4).

In the t-th iteration of generation (inference time), $\varepsilon_{t}$ is sampled from ${P_{N}}(\varepsilon)$. A novel node or edge $z_{t}$ is generated through ${f}\left({{\varepsilon_{t}}}\right)={z_{t}}={\mu_{t}}+{\sigma_{t}}\cdot{\varepsilon_{t}}$, where ($\mu_{t}$, $\sigma_{t}$) are calculated according to the current subgraph.

Conditional Evaluation aims at calculating Gaussian parameters ($\mu_{t}$, $\sigma_{t}$) from the current subgraph $G_{t}$, which should fully consider the characteristics of the subgraph and the relationships between graphic elements. To achieve this goal, as presented in Fig. 5, a 4-layer GCN learns the node embeddings ${H_{t}}\in{\mathbb{R}^{{m_{t}}\times 128}}$, cooperating with sum-pooing to obtain the subgraph embedding ${\tilde{h}_{t}}\in{\mathbb{R}^{128}}$, where ${m_{t}}$ indicates the number of nodes in ${G_{t}}$. Based on this, a branch projects the subgraph embedding to Gaussian parameters ($u_{t}^{i}$, $\sigma_{t}^{i}$) in the generation of node i. Another branch outputs the Gaussian parameters ($u_{t}^{ij}$, $\sigma_{t}^{ij}$) in the generation of the edge between nodes i and j, considering both the embedding of the subgraph ${\tilde{h}_{t}}$ and the embeddings of adjacent nodes ($H_{t}^{i}$, $H_{t}^{j}$) [38].

Graph representation loss. Two multi-class cross-entropy losses ${L_{n}}$ and ${L_{e}}$ guide the representation learning with label semantics. ${L_{n}}$ is used for node representation learning while ${L_{e}}$ for the edge. Note that we add an ‘empty’ class in the representation learning for the edge $E\in{\mathbb{R}^{m\times m\times{c_{e}}}}$, indicating no adjacency between two nodes. It was discovered that the empty edges generally account for the majority of edges in the training dataset, bringing a class-imbalanced problem. Inspired by [7], the adjacency matrix R is given in the dataset. It helps to ignore the empty edge in the learning process to solve the problem.

Distribution mapping loss. Equation (4) formulates the exact probability density of a graphic element based on an invertible distribution mapping. To train the normalizing-flow-based model for the distribution mapping, we compute the log-likelihoods of all the N graphic elements and update model parameters by minimizing the negative log-likelihoods [38, 41]:

Thus, the joint loss function to train the overall model is defined as:

This section discusses (1) How to make the graph more plausible? (2) When to generate a node & when to generate an edge? (3) When does the graph generation stop?

To make the graphs more plausible, we consider adding constraints to the generation. 1) Space constraints. The total volume of generated entities needs to be within a reasonable range so that the space does not appear too crowded or sparse. 2) Semantic constraints. The semantic labels of generated entities should be reasonable. For example, a floor should not be generated in an enclosed room, as well as a bed in the kitchen. 3) Anti-overlapping operations. If we can give the suggested object poses, it can help prevent the object from overlapping in the downstream applications. Section $V$-B shows the implementation details.

Taking the node/edge generations with semantic constraints as examples (Fig. 6). Our method starts by generating a new node and determines whether it satisfies the semantic constraint. If satisfied, the node is added to the graph; otherwise, it is abandoned, and a new node is generated. Then, our method establishes the edges between the new node and existing nodes. If all the edges are empty, the generation is stopped; otherwise, we keep the non-empty edges and repeat the above processes. Experiments will demonstrate the generation results with constraints. Supplementary material-1 describes more details through pseudocode.

There are not many datasets for the scene graph generation tasks from point clouds. Only 3DSSG-O27R16 [7], a dataset of indoor scenes, can be used as the benchmark. It combines 3DSSG [18] and 3RScan [31], containing 1318 non-empty rooms in point clouds and the corresponding scene graph. Each point cloud provides XYZ coordinates, RGB values, etc. The semantic annotations of instances, including 27 object classes (e.g., lamp and ceiling) and 16 non-empty relationship classes (e.g., hanging on and attached to), are recorded in the scene graph. We remove 64 scenes from the dataset. These scenes include more than 50 instances and cost excessive computing resources in the training process. We randomly choose about 255 scenes as the test data. We remove the semantic annotations and the entities inside the room from each test data, only retaining an empty room with XYZ coordinates and RGB values for the instance-incremental graph generation. Besides, to verify the effectiveness of this novel task and the performance of methods on outdoor scans, we have made a new dataset GPL3D, and the details are in Section $V$-H.

The proposed network is end-to-end trained using back-propagation and Adam optimizer with an initial learning rate of 0.001. The experiments are conducted by Pytorch and Dive into Graphs (DIG) [42] on a single GeForce GTX 1080 Ti and an Intel(R) Core(TM) i7-7800X CPU; The training batch size is set to 32. The weighting value $\alpha$ in Equation (3) is set to 0.9; a 4-layer GCN is used to learn the graph and node embeddings. All these details will be discussed in the ablation studies. The graph generation starts from a random value sampled from the Gaussian, bringing randomness to the results. Each test data generates five graphs for evaluation to reduce the impact of randomness on the results.

Some data processing and statistical analysis are performed to support the generation with constraints. 1) Space constraints. We take the following measures to introduce space constraints into the generation. Firstly, we calculate the volume of each room in the test data. Next, we summarize the volume data of each entity in the dataset and model the per-category volume distribution as Gaussian. In graph generation, the volume of each generated entity is calculated through sampling from the corresponding Gaussian. The generation stops when the total volume of the generated entities is beyond a certain proportion of the room’s volume. 2) Semantic constraints aim to restrict the categories of objects generated in different kinds of rooms. On the one hand, we manually label the room functions in the test data, such as the living room and office. On the other hand, we establish a dictionary to record the relationships between the room functions and the object categories in the real world. If the category of a generated instance breaks the rules recorded in the dictionary, it will be regarded as invalid and needs to be regenerated. 3) Anti-overlapping operations. We consider the corresponding 3D object as a bounding box and randomly give the pose information according to the spatial relationships in the graph. If another object has occupied the place, reproduce the object pose; If a suitable object location cannot be found after several iterations, the node is abandoned, and a node with relational semantics is newly generated.

Our instance-incremental task outputs the graphs representing the suitable object layout schemes for a real-world scene. The layout schemes can be various. Thus, our task does not have a unique solution. A reasonable layout should accord with real-world layout habits and be consistent with human perception:

From the perspective of each generated instance, the instance should conform to the objective reality. For a completely enclosed space, the generated objects should be furniture, such as tables and sofas, rather than the existing entities, such as walls and floors. Besides, ‘a chair standing on the ceiling.’ is obviously wrong. Thus, we judge whether each generation conforms to the objective reality and introduce Node/Edge Validity to quantify the proportion of correctly generated nodes/edges in all the generated ones [23].

From the global perspective, the gap between the generated layouts and the layouts of reality should be small. Maximum Mean Discrepancy (MMD) indicates the similarity between two data distributions [43, 36]. We introduce MMD based on degrees and clustering coefficients to quantify the similarity between the generated and real-world layout graphs. The real-world ones are obtained through the manual annotation of numerous non-empty rooms of reality. The smaller the MMD, the smaller the gap between the generated layouts and the layouts of reality.

Uniqueness and Diversity metrics are introduced to quantify the graph difference. Uniqueness describes the percentage of graphs different from others [23]. Diversity is a stricter metric. For each test scene, we exclude the graph that is a subgraph of the other graphs, and diversity equals the remaining percentage [27]. Besides, we evaluate the complexity of methods from two aspects: the Model Parameters (Params) and the Floating Point Operations (FLOPs)

Baselines. Existing instance-incremental graph generation methods cannot be directly used in this new task, as described above. Thus, we introduce the widely used methods in other fields, the energy-based GraphEBM [21] and the RNN-based GraphRNN [36, 37], to cooperate with our point cloud graph representation as baselines. Note the VAEs-based method, such as JT-VAE [44], also works well in the instance-incremental generation of other fields. However, this kind of method relies on prior knowledge to guide the generation, which is unfair for the comparison. Thus, we do not consider it as the baseline method. Besides, we introduce a method consisting of the graph representation network SGG_point [7] and the graph expansion network GEMS [27] as the baseline, verifying if the image-based graph expansion method can be used in our point-cloud-based task. We call this baseline method SGG-GEMS.

Node/Edge Validity. Quantitative results using different methods are illustrated in TABLE I. On the one hand, our method significantly improves the edge validity, 45.5% higher than the SGG-GEMS in the second place. On the other hand, there is little performance difference between these methods in terms of node validity.

The validity metrics indicate the understanding ability of real-world scenes, especially for the included semantic information. Unlike the graphs with simple semantics, such as molecules, the node/edge semantics in a point-cloud-based scene is complicated.

The 3DSSG-O27R16 dataset contains 27 node classes and 17 edge classes (including the empty one). In the real-world experience learning stage, every method learns the data distribution from every scene’s node features $V\in{\mathbb{R}^{m\times{c_{n}}}}$ and edge features $E\in{\mathbb{R}^{m\times m\times{c_{e}}}}$, where ${c_{n}}=27$, ${c_{e}}=17$, and m equals the max object number 50. The edge feature matrix is much more complex than the node one, containing about 30 times as many elements as the latter, and the values are more dispersed. Thus, the data distribution learning of edges is more challenging than that of nodes.

All the methods work well in node generation because the data distribution learning of nodes is relatively easy. Even though the misjudgment of the exited objects’ semantics may bring mistakes to the generation of novel nodes, the node validities of all methods remain high. Things make a difference in the edge generation. GraphEBM and GraphRNN model the observations as an energy function or a sequence, respectively. These methods cannot model the complex data distributions of edges well, resulting in poor validities of generated edges. The improved sequencing strategy of SGG-GEMS improves this situation but still retains a low level. Our 3D-ANF transforms the observations into Gaussians with parameters adjusted according to conditions. It has a strong ability to model the data distributions and promotes the validities of graphic elements.

MMD. TABLE I illustrates that our method obtains the best MMD scores for degrees and clustering coefficients. It indicates that our generated graphs are closer to the real-world scene data.

The quality of generated graphs depends on the real-world experiences of object layouts learned from observation data. GraphRNN represents graphs under different orderings as sequences, and the improved SGG-GEMS introduces a novel method and external knowledge to improve the layout experience learning. However, on the one hand, these sequence-based methods cannot model the in-depth features of complex object layouts well. On the other hand, SGG-GEMS, an image-based method, often outputs a graph containing several subgraphs or isolated nodes/triples to denote the layout of a 3D room. It is inconsistent with the logic of reality, leaving an obvious gap between the generations and the observations. The energy-based GraphEBM also works not well because of the poor ability of layout experience learning. Unlike the baselines, our method learns the node and graph embeddings through GCN. It aggregates the features of neighbors through message-passing layers, which well models the layout information represented in the graph. Thus, our method performs well in terms of MMD scores.

Uniqueness $\&$ Diversity. TABLE I illustrates that our method and GraphEBM achieve 100% scores on graph uniqueness. However, the graphs generated by GraphEBM exhibit greater diversity than ours. GraphRNN and SGG-GEMS achieve the lowest score of graph uniqueness and diversity.

The real-world layout experiences constrain the objects placed in the scene and their spatial relationships. The poor edge validity and MMD metrics indicate that GraphEBM performs poorly in learning the layout experiences, which results in weak constraints from reality in graph generation. Thus, the graphs may be unreasonable but diverse. Differently, our method achieves a good balance between graph rationality and diversity. The graphs accord with the real-world layout habits and also retain an acceptable score on graph diversity.

In summary, our proposed 3D-ANF achieves this novel instance-incremental graph generation and outputs reliable novel graphs; besides, our method outperforms other baselines in most evaluation metrics.

Method Complexity. TABLE II shows that our 3D-ANF has the second least number of model parameters and the fewest FLOPs. The sequence-based GraphRNN and SGG-GEMS suffer from relatively dense parameters and computations. They focus on modeling the relationships between elements in the sequence but cannot extract the layout habits behind the relationships well. The energy-based GraphEBM has the least number of model parameters. However, it needs a high computing cost to evaluate the energy value of the graph to be generated. By comparison, our efficient method achieves the best graph generation performance at a low cost.

Fig. 7 shows the visualization of the graph generation from an empty room in the point cloud. Our method extracts conditions from the current graphical scene, autoregressively generating novel nodes (e.g., plant) and edges (e.g., ‘a plant standing on the floor.’) with real-world layout experiences. As a result, a graph describing the proposed object layout is created.

Most generations conform to the layout habits in reality. However, some invalid generations have occurred, e.g., ‘clothes build in the wall.’ The reasons are as follows: each instance is generated considering both the real-world layout experiences and the conditions of the current scene. However, on the one hand, the layout experiences may not be fully understood and modeled; on the other hand, experiences come from observations, which have a distribution different from the data used for inference. The layout experiences may be unsuitable for some scenes, thus causing invalid generations. Fig. 7 also shows several generated graphs of the same scene. All these graphs have unique structures and semantics, providing a variety of schemes for object layouts.

We explore the advantages of several designs and quantify their influence on the generation results by individually changing or removing them.

Effect of the weight coefficient $\alpha$ in Equation (3). Equation (3) converts the discrete data into continuous data by adding noises to support the invertible distribution transformation in flow-based learning. Excessive noises destroy the graph characteristics, yet too small noises influence the graph data continuity. Firstly, TABLE III shows that validity metrics decline with the increase of $\alpha$. This experimental phenomenon is more evident in the edge validity, proving that excessive noise damages the graph information, especially the semantics. Secondly, setting $\alpha$ to 0.9 or 1 performs the best in the MMD metrics. It maintains a good balance between the graph data continuity and the graph characteristics. Lastly, our method achieves the best graph diversity score while setting $\alpha$ to 1, slightly better than setting $\alpha$ to 0.9. However, the graph diversity comes at the cost of reducing the edge validity by a large margin. All things considered, the weight coefficient $\alpha$ is set to 0.9.

Effect of the layers of GCN. GCN learns the node and graph embedding, which is the basis of the Gaussian parameter estimation. In this experiment, we explore the effectiveness of GCN layers on the final results. TABLE III shows that the 3-layer and 5-layer GCN have an evident decline in most metrics compared with the 4-layer one. We also use a 6-layer GCN for this ablation experiment. However, it cannot generate the graphs for many testing scenes, thus not listed in the table. It indicates that the embedding learning of observation data has key effects on the quality of generation results. A GCN with too few layers weakens the model’s learning ability and dramatically degrades the graph generation performance. A GCN with too many layers causes the output features over-smoothed [45, 46, 47, 48]. The indistinguishable features weaken the differences between objects and the differences between spatial relationships, leading to the learning of fuzzy layout experiences. It outputs graphs with low diversities and edge validities for scenes that have never been seen before, and there is a big gap between the generations and real-world layout graphs. In summary, a 4-layer GCN is the best choice.

Effect of the cross-entropy loss. The cross-entropy loss cooperates with the log-likelihood loss to guide the training of 3D-ANF. This experiment discusses whether we can remove the cross-entropy loss and only use the log-likelihood loss to guide the training process. TABLE III shows that removing the cross-entropy loss degrades the generation performance (-19% edge validity, +0.17 MMD for degrees, +0.11 MMD for clustering coefficients, and -8.8% for diversity). It proves that the semantic information of the existing instances and relationships in the scene is necessary for the instance-incremental generations of novel graphs.

We add constraints to the generation to make the generated graphs more plausible. We set the lower limit ${L_{l}}=\lambda\cdot{L_{m}}$ and upper limit ${L_{u}}=\beta\cdot{L_{m}}$ of the volume proportion, where ${L_{m}}$ is the mean value of volume proportion obtained from observation data statistics, $\lambda$ and $\beta$ are individually the weight coefficient for the lower and upper limit. For an empty room with a total volume of ${V_{t}}$, the total volume of generated objects should require: ${L_{l}}\cdot{V_{t}}\leq\sum\limits_{i=1}^{T_{v}}{{v_{i}}}\leq{L_{u}}\cdot{V_{t}}$, where ${T_{v}}$ is the total number of generated objects; and ${v_{i}}$ is the volume of an object (calculated as mentioned in Section $V$-B).

We also propose two types of semantic constraints. The first one limits the generated objects to furniture, such as tables and sofas, rather than the existing entities of the empty room, such as walls and floors. The second one requires the generated objects to be suitable for the room function. The generated edges both needed to conform to the objective reality. Besides, the anti-overlapping operations described in Section $V$-B are selectively used in graph generation with constraints.

TABLE IV illustrates the quantitative results with different constraints. By introducing constraints into the generation, the validity of generated nodes and edges all reach 100.0%. However, the increased values of MMD indicate that the constraints expand the difference between the structures of generated graphs and those of observations. Besides, the decreased graph uniqueness and diversity values prove that these constraints reduce the differences among the generated graphs.

Fig. 8, 9, and Fig. 4 of the supplementary material-2 show the generated graph with different constraints, considering different real-world rooms in the point cloud as the input. Taking the living room in Fig. 8 as an example, on the one hand, some unreasonable instances appear in Fig. 8 (b) without the constraint of the living room. For example, ‘a bed in the living room.’ The generated instances in Fig. 8 (c) are more suitable for a living room by strengthening the constraints on generated objects’ categories. On the other hand, space constraints in the generation help control the room’s crowding degree. The output layout of our method is very close to that in the real living room.

Fig.2 of the supplementary material-2 illustrates the generation visualization of baseline methods. GraphEBM generates some objects suitable for this room. However, the generated spatial information is insufficient to describe a reasonable layout, making the object arrangement random and messy. SGG-GEMS generated an isolated subgraph without any relationship with the objects in the room, which is inconsistent with the logic of reality.

We construct the corresponding 3D scenes to showcase the generated scene graphs (Supplementary material-2). Note that our generated scene graph provides the category of generated objects with approximate locations for a real-world scene. Lots of other prediction work should be done to achieve the final object arrangement in a real-world scene. They are not the focus of this paper. Thus, we replace them with manual operations to achieve the final object arrangements.

No graphical dataset of outdoor point-cloud-based scenes is available for this task, mainly due to the lack of graph annotations. We have made a new graphical dataset based on the Paris-Lille-3D [49] to address this problem. Paris-Lille-3D is a large-scale urban outdoor point cloud dataset acquired by mobile laser scanners in cities of France, covering more than 2 km of streets. This new dataset is a graphical version of Paris-Lille-3D, which we call Graphical-Paris-Lille-3D (GPL3D). Supplementary material-3 describes the GPL3D in detail. It contains 15 object classes (e.g., road and pedestrian) and 9 non-empty relationship classes (e.g., adjacent to and close to). The Lille scenes are for training and Paris for testing.

We conduct comparison experiments on the newly proposed graphical dataset GPL3D, and TABLE V illustrates the experiment results. Our method also achieves the best scores in most evaluation metrics.

The results prove that our method has a better generalization ability than other methods. Specifically, the layout graphs of GPL3D are different from the 3DSSG-O27R16. On the one hand, the graphs have lower feature dimensions due to the fewer label categories. On the other hand, the adjacency matrices are sparse due to the limited annotations of spatial relationships. Our method performs the best on all the datasets, indicating that it can learn the real-world layout experiences from the scenes with different layout graph characteristics. It verifies the generalization ability of our method.

The baseline methods lag behind ours in terms of validities and MMD. Even though the graph complexity of GPL3D decreases a lot, these sequence-based and energy-based methods cannot extract the layout experiences from reality as well as ours due to the limited capability in data distribution modeling from 3D scenes.

The graph diversities of all the methods generally become better. As the first version of GPL3D, the annotations of spatial relationships used for layout learning are not as many as those of 3DSSG-O27R16. A limited number of learning samples leads to weakened restrictions from real-world habits during the generation, resulting in a general increase in graph diversities. Fig. 10 illustrates the generated graphs with reasonable object layouts. The corresponding 3D scenes of these graphs are in Fig.5 and Fig.6 of supplementary material-2.