Structured Graph Variational Autoencoders for Indoor Furniture layout Generation

We represent an indoor scene as an attributed graph $G=(V,E,X)$. Here the nodes $V$ denote room layout components (floor, wall, windows) and furniture items (sofa, chair, bed), the edges $E\subseteq V\times V$ denote relationships between the nodes (e.g., relative orientation of sofa to wall), and the attributes $X$ denote features associated with the nodes and edges (e.g., location of furniture items or relative orientation between bed and wall). The graph has two node types (room nodes $V_{R}$, furniture nodes $V_{F}$), three edge types (room-room edges $E_{RR}$, room-furniture edges $E_{RF}$ and furniture-furniture edges $E_{FF}$), and five attribute types corresponding to these node and edge types ($X_{R}$, $X_{F}$, $X_{RR}$, $X_{RF}$ and $X_{FF}$). In this work, we consider the graph as complete, that is, $E_{RF}=V_{R}\times V_{F},E_{FF}=V_{F}\times V_{F}$ and $E_{RR}=V_{R}\times V_{R}$.

We will identify two main subgraphs of $G$. The room layout graph $G_{R}=(V_{R},E_{R},{X_{R},X_{RR}})$ consists of $n_{R}:=|V_{R}|$ nodes (or room elements) and $e_{R}:=|E_{R}|$ edges, where the node attributes $X_{R}\in{\mathbb{R}}^{n_{R}\times d_{R}}$ denote the class, location, orientation and size of the room element, and the edge attributes $X_{RR}\in{\mathbb{R}}^{e_{R}\times d_{RR}}$ encode geometric or functional relationships between two room elements (relative location, relative orientation etc.). The room type $T$ is also encoded as a categorical feature into $X_{R}$. Similarly, the furniture layout graph $G_{F}=(V_{F},E_{F},{X_{F},X_{FF}})$ consists of $n_{F}:=|V_{F}|$ nodes (or furniture items) and $e_{F}=|E_{F}|$ edges, where the node attributes $X_{F}\in{\mathbb{R}}^{n_{F}\times d_{F}}$ denote the class of the furniture item, its location, orientation, size, and its 3D shape descriptor, and its edge attributes $X_{FF}\in{\mathbb{R}}^{e_{F}\times d_{FF}}$ encode geometric or functional relationships between two furniture items. We obtain the shape descriptors of each furniture item by processinga a 3D point cloud of the item through PointNet Qi u. a. (2017) pretrained on the Stanford ShapeNet dataset Chang u. a. (2015).

We would like to design and learn a probabilistic model $p(G_{F}\mid G_{R},T,n_{F})$ that generates a furniture layout $G_{F}$ given the room layout $G_{R}$, type $T$ (say, bedroom or library room) and number of furniture items $n_{F}$ to place in the room. We assume there exists latent $Z$ such that   $p(G_{F}\mid G_{R},T,n_{F})~{}=$

$$\!\!\!\int\!\!p(G_{F}\!\!\mid\!\!Z,n_{F},G_{R},T)p(Z\!\!\mid\!\!n_{F},G_{R},T)dZ.$$

(1)

Our proposed model consists of three main ingredients:

1. 1.

   An encoder, $q_{\phi}(Z\mid G,T,n_{F})$, which maps both room and furniture layouts ($G_{R}$ and $G_{F}$ resp.) as well as room type and number of furniture items to a latent variable $Z$, which captures the diversity of room-aware furniture layouts. The parameters of the encoder are denoted as $\phi$.

2. 2.

   A decoder, $p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)$, which maps the number of furniture items, the latent variable as well as the room layout and type to a furniture layout. The parameters of the decoder are denoted as $\theta^{\prime}$.

3. 3.

   A prior model $p_{\theta^{\prime\prime}}(Z\mid n_{F},G_{R},T)$. The difference between the prior model and the encoder is that the prior model only considers the room layout $G_{R}$ and not $G$. The parameters of the prior model are denoted as $\theta^{\prime\prime}$.

The encoder, decoder and prior model are parameterized with GNNs which we will describe in the next subsection.

Given a training set, consisting of indoor scenes in the form of attributed graphs ${\mathcal{G}}=\{G_{1},G_{2},...,G_{n}\}$, we learn the parameters $\{\theta^{\prime},\theta^{\prime\prime},\phi\}$ by optimizing the following empirical average over the training set:

$\displaystyle{\mathcal{L}}(\theta^{\prime},\theta^{\prime\prime},\phi)=$

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\mathscr{L}(G_{i},\theta^{\prime},\theta^{\prime\prime},\phi),$

(2)

where, for any $G\in{\mathcal{G}}$, $L(G,\theta^{\prime},\theta^{\prime\prime},\phi)$ denotes the Evidence Lower Bound (ELBO) defined as:

		$\displaystyle\mathscr{L}(G,\theta^{\prime},\theta^{\prime\prime},\phi)$		(3)
		$\displaystyle=\mathbb{E}_{Z\sim q_{\phi}(Z\mid G,T,n_{F})}[\log p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)]$
		$\displaystyle-KL(q_{\phi}(Z\mid G,T,n_{F})\mid\mid p_{\theta^{\prime\prime}}(Z\mid n_{F},G_{R},T)).$

As mentioned in §1, we optimize (2) under constraints to speed up convergence, as we will describe in subsection 3.3.

Graph Encoder. The encoder models the approximate posterior of a latent variable $Z$ given $(G,T,n_{F})$. We assume that the distribution of $Z$, $q_{\phi}(Z\mid G,T,n_{F})$, is Gaussian with mean $\mu_{\phi}(G,T,n_{F})$ and a diagonal covariance matrix with diagonal entries $\sigma_{\phi}(G,T,n_{F})$. The distribution parameters $(\mu_{\phi},\sigma_{\phi})$ are modeled as the output of an attention-based message passing graph neural network (MP-GNN) with weights $\phi$. The design of the MP-GNN is inspired by Mavroudi u. a. (2020), where each layer $l=1,\ldots,L$ of the MP-GNN maps a graph $G^{l-1}$ to another graph $G^{l}$ by updating the graph’s node and edge features. Specifically, let $h_{i}^{l}$ and $h_{ij}^{l}$ denote the features of node $i$ and edge $(i,j)$ of graph $G^{l}$, respectively. Let the input to the network be the graph $G^{0}=G$, so that $h_{i}^{0}$ and $h_{ij}^{0}$ denote the node features (rows of $X_{R}$ and $X_{F}$) and edge features (rows of $X_{RR}$, $X_{FF}$ and $X_{RF}$), respectively. At each iteration of node and edge refinement, the MP-GNN: (1) adapts the scalar edge weights by using an attention mechanism; (2) updates the edge attributes depending on the edge type, the attention-based edge weights, the attributes of the connected nodes and the previous edge attribute; and (3) updates the node attribute by aggregating attributes from incoming edges.

After $L$ layers of refinement, we obtain a graph $G^{L}$, whose node features are mapped via a linear layer with weights $W_{\mu},W_{\sigma}$ to obtain the parameters of the Gaussian model as $\mu_{\phi}^{i}(G,T)=W_{\mu}h_{i}^{L},\;\sigma^{i}_{\phi}(G,T)=\exp(W_{\sigma}h_{i}^{L})$ where $i=1,\dots,n_{F}$. Note that there is a different Gaussian for each node of the graph. Therefore, the output of the encoder will be two matrices $\mu_{\phi}(G,T,S)\in\mathbb{R}^{n_{F}\times d_{F}}$ and $\sigma_{\phi}(G,T,S)\in\mathbb{R}^{n_{F}\times d_{F}}$ corresponding to the mean and standard deviation vectors of the latent variable matrix $Z\in\mathbb{R}^{n_{F}\times d_{F}}$.

Graph Decoder. The decoder maps $(n_{F},Z)$ and the room layout, type $(G_{R},T)$ to a desired furniture layout via the distribution $p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)$. The generative process proceeds as follows,

An initial fully connected furniture graph $G_{F}^{0}$ is instantiated. Each node of $G_{F}^{0}$ is associated with a feature of dimension $d_{F}$ corresponding to one of the rows of $Z$. Each edge $(i,j)$ of $G_{F}^{0}$ of type $\epsilon\in\{RF,FF\}$ is associated with a feature $Z_{ij}=(Z_{i},Z_{j})$ as the concatenation of the node features. As a result, we obtain an initial graph $G_{0}$ that includes both the initial furniture graph $G_{F}^{0}$ as well as the given room graph $G_{R}$ as subgraphs. The initial graph $G_{0}$ is passed to a MP-GNN, which follows the same operations as the encoder MP-GNN.

The output of the MP-GNN is the furniture subgraph $G_{F}^{L}$ of the final graph $G^{L}$. Each furniture node of $G^{L}$ is then individually processed through a MLP to produce parameters for the furniture layout graph distribution $p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)$, which can be factorized in the following way:

		$\displaystyle p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)$		(4)
	$\displaystyle=$	$\displaystyle\prod_{i=1}^{n_{F}}p_{\theta^{\prime}}(shape_{i}\mid Z,G_{R},T)p_{\theta^{\prime}}(orien_{i}\mid Z,G_{R},T)$
		$\displaystyle p_{\theta^{\prime}}(loc_{i}\mid Z,G_{R},T)p_{\theta^{\prime}}(size_{i}\mid shape_{i})p_{\theta^{\prime}}(cat_{i}\mid shape_{i}).$

More specifically, we assume that given latent $Z$, room layout $G_{R}$ and type $T$, the furniture features are independent of each other (a standard assumption in the VAE literature). For a furniture, we further assume that shape, orientation and location features are independent given $Z,G_{F}$ and $T$. However, since the PointNet shape features implicitly capture the configuration of the furniture item in 3D space we condition the size and category distributions on the shape feature.

We parameterize the shape and location features as a normal distribution, the size feature as a lognormal distribution whose support is restricted to be positive valued, category and orientation features as categorical distribtions. Since both the Encoder and Decoder graphs have $n_{F}$ nodes that are in one-to-one correspondence, we can define our reconstruction loss (first term in (3)) by simply comparing their node features without the need for an explicit matching procedure.

Graph Prior. Recall our latent space $Z$ is modelled such that there is a latent variable corresponding to each furniture node in the graph. Many popular graph VAE models assume an i.i.d. normal prior for each node Kipf und Welling (2016b); Luo u. a. (2020). However, such a model is restrictive for our purposes. MP-GNNs achieve permutation equivariance by sharing the weight matrices across every node in the graph. When the graph is complete, as is the case here, the marginal distribution of every output node after $L$ GNN layers will be identical if they are initialized as i.i.d. Gaussian at the input layer. Since, the output nodes of the decoder GNN after $L$ layers correspond to different furniture features, having identical marginals is detrimental. This claim is supported by experiments (Figure 2) where the i.i.d. prior baseline models struggle to learn proper furniture placements. To remedy this, we propose to parameterize prior distribution as an autoregressive model based on linear gaussian models Bishop (2006). More specifically,

	$\displaystyle p(Z^{0}\mid G_{R},T)\!$	$\displaystyle=\!{\mathcal{N}}(\mu_{\theta^{\prime\prime}}(G_{R},T),\sigma^{0}_{\theta^{\prime\prime}}(G_{R},T)),$		(5)
	$\displaystyle p(Z^{i}|Z^{k<i},G_{R},T)\!$	$\displaystyle=\!{\mathcal{N}}(\sum_{k<i}A^{k}_{\theta^{\prime\prime}}(G_{R},T)Z^{k},\sigma^{i}_{\theta^{\prime\prime}}(G_{R},T)).$

where $Z^{i}$ refers to the latent corresponding to the $i^{th}$ furniture node. Thus, the $i^{th}$ furniture node latent is given by a Gaussian whose mean is a linear function of all the latents $k<i$. Such a structure ensures that all the latent variables are jointly Gaussian. This allows us to analytically compute the KL divergence term and thus was favoured over more expressive probabilistic models which would introduce more stochasticity in the objective due to the need of estimating the KL divergence term via sampling. We implement (5) (see also Figure 1) with two networks:

• •

  Room Aggregator for $\bm{p(Z^{0}\mid G_{R},T)}$: The room aggregator is an MP-GNN with the same architecture as the Graph Encoder. except that the input to the network is just $(G_{R},T)$ with the node and edge features initialized to $X_{R}$ and $X_{RF}$, respectively. After $L$ GNN layers, all the room node features are aggregated by a mean pooling operation to obtain a global representation of the room layout plan $X_{R}^{agg}$. This $X_{R}^{agg}$ is then passed through an MLP to compute $\mu_{\theta^{\prime\prime}}(G_{R},T)$ and $\sigma^{0}_{\theta^{\prime\prime}}(G_{R},T)$.

• •

  RNN Prior for $\bm{p(Z^{i}|Z^{k<i},G_{R},T)}$: We use a recurrent neural network to predict the matrix $A^{k}_{\theta^{\prime\prime}}(G_{R},T)$ and the variance $\sigma^{i}_{\theta^{\prime\prime}}(G_{R},T)$ at each node index. The RNN is initialized with $X_{R}^{agg}$. We need additional constraints on each $A^{k}_{\theta^{\prime\prime}}(G_{R},T)$ to prevent the dynamics model in (5) from diverging to infinity. This is typically done by controlling the spectral radius or its proxy, the spectral norm Lacy und Bernstein (2003), of the matrices $\{A^{k}_{\theta^{\prime\prime}}(G_{R},T):k\in[1,2,...,n_{F}]\}$. Thus, the predicted matrix is taken to be $\frac{A^{k}_{\theta^{\prime\prime}}(G_{R},T)}{||A^{k}_{\theta^{\prime\prime}}(G_{R},T)||_{2}}$, where $||A||_{2}$ is the spectral norm of some matrix $A$.

Note that the proposed autoregressive prior could in principle be reexpressed as a more traditional i.i.d. Gaussian prior, which is then passed through an additional non-equivariant transformation layer that can be absorbed into the decoder. But a significant difference emerges in practice when facing the key challenge of incorporating this non-equivariant factor into subsequent model training, given that there is no longer a canonical ordering between the different nodes in a graph. When the proposed autoregressive prior formulation is adopted, such an ordering is only required to evaluate $p_{\theta^{\prime\prime}}(Z\mid G_{R},T,n_{F})$ for any $Z$ sampled from the posterior $q_{\phi}(Z\mid G,T,n_{F})$ to compute the KL divergence term in the ELBO (3). However, by design this term can be expressed analytically, and as we will soon demonstrate, an efficient compensatory permutation can be efficiently computed. In contrast, with an alternative autoregressive decoder formulation, the search for an appropriate ordering is instead needed for computing the VAE reconstruction term (i.e., evaluating the decoder $p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)$ for any $Z$ sampled from the posterior), and hence becomes entangled with the non-analytic stochastic sampling required for obtaining approximate reconstructions.

Computing the KL divergence term. Let $Z=\{Z^{1},Z^{2},...,Z^{n_{F}}\}$ be the set of latent variables correponding to $n_{F}$ furniture items to be placed in the room. Let $\pi$ denote the ordering among these variables. Given $\pi$, the likelihood of observing $Z$ under our proposed prior is defined as

$$p_{\theta^{\prime\prime}}(Z\mid G_{R},\pi,n_{F},T)=\prod_{i=1}^{n_{F}}p_{\theta^{\prime\prime}}(Z^{\pi(i)}\mid Z^{\pi(j<i)},G_{R},T),$$

(7)

However, for $Z\sim q_{\phi}(Z\mid G,T,n_{F})$ (the approximate posterior) we do not know this ordering $\pi$. Thus, given $Z$, we define the optimal order $\pi^{*}$ to be

$$\pi^{*}\!=\!\operatorname*{arg\,min}_{\pi}KL(q_{\phi}(Z\!\mid\!G,T,n_{F})\!\mid\mid\!p_{\theta^{\prime\prime}}(Z\!\mid\!G_{R},\pi,n_{F},T)).$$

(8)

Recall $q_{\phi}(Z\mid G,T,n_{F})=\prod_{i=1}^{n_{F}}{\mathcal{N}}(Z^{i};\mu_{\phi}^{i}(G),\sigma_{\phi}^{i}(G))$ Since both the prior and posterior are jointly Gaussian, computing (8) reduces to solving a Quadratic Assignment Problem (QAP). For simplicity let us denote the distributions as

	$\displaystyle q_{\phi}(Z\mid G,T,n_{F})$	$\displaystyle={\mathcal{N}}(\mu_{0},\Sigma_{0})$		(9)
	$\displaystyle p_{\theta^{\prime\prime}}(Z\mid G_{R},\pi,n_{F},T)$	$\displaystyle={\mathcal{N}}(\tilde{\pi}\mu_{1},\tilde{\pi}\Sigma_{1}\tilde{\pi}^{T}),$

where $\tilde{\pi}=\pi\otimes I_{d_{F}\times d_{F}}$. Note $\pi\in{\mathbb{R}}^{n_{F}\times n_{F}}$ and the kronecker product comes from the fact that we are only allowed to permute blocks of $\mu_{1}$ and $\Sigma_{1}$, of size $d_{F}$ and $d_{F}\times d_{F}$ respectively, where $d_{F}$ is the dimension of the latent variable used for each furniture node. In other words, we can only permute latent vectors corresponding to furniture nodes as a whole and not the intra dimensions of $Z$ within any furniture node. Here $\mu_{0},\mu_{1}\in{\mathbb{R}}^{n_{F}d_{F}}$. Similarly, $\Sigma_{0},\Sigma_{1}\in{\mathbb{R}}^{n_{F}d_{F}\times n_{F}d_{F}}$. Note that due to the autoregressive structure $\Sigma_{1}$ will not be a diagonal matrix.

We show in the Appendix (§6.4.1) that (8) is equivalent to

$$\min_{\pi}Tr\left(\Sigma_{1}^{-1}\tilde{\pi}^{T}\left[\Sigma_{0}+\mu_{0}\mu_{0}^{T}\right]\tilde{\pi}\right)-2Tr\left(\tilde{\pi}^{T}\mu_{0}\mu_{1}^{T}\Sigma_{1}^{-1}\right).$$

(10)

Notice that (10) is a Quadratic Assignment Problem (QAP) which is known to be NP-Hard. For this we propose to use a fast approximation algorithm, called FAQ, introduced in Vogelstein u. a. (2015). FAQ first relaxes the optimization problem from set of permutation matrices to the set of all doubly stochastic matrices. It then iteratively proceeds by solving linearizations of the objective (10) using the Franke-Wolfe method. These linearizations reduce the QAP to just a linear assignment problem (LAP), which can be efficiently solved by the Hungarian algorithm. After Franke-Wolfe terminates, we project the doubly stochastic solution back to the set of permutations by solving another LAP. FAQ has a runtime complexity that is cubic in the number of nodes per iteration which is faster than the quartic complexity of the matching procedure used in Simonovsky und Komodakis (2018). More details in Appendix §6.4.2.

We need to compute the optimal $\pi^{*}$ for each graph in a mini-batch, and then compute the KL divergence term in (3) analytically given this ordering. We observe no significant gains in performance in running the FAQ algorithm more than $1$ step per graph, which further speeds our method.

Learning under constraints: To facilitate faster convergence and also ensure fidelity of the learned solution, we enforce certain constraints on the reconstructed room. These constraints are derived from training data and do not require external annotations. Given input furniture graph $G_{F}$ to the encoder and the reconstructed graph $\tilde{G}_{F}$ by the decoder, we enforce the relative positions of predicted furnitures in $\tilde{G}_{F}$ to be “close" to the ground truth relative positions in $G_{F}$. Similarly, we apply constraints on the relative position of the predicted furniture items with the room walls, windows and doors. Finally we apply a constraint penalizing the relative orientations between different furniture items from being too “far" away from the relative orientations in $G$. We explain these constraints more clearly in Appendix 6.5.

Having described all the ingredients in our model, we finally present the complete optimization objective as

$\displaystyle\max_{\theta^{\prime},\theta^{\prime\prime},\phi}{\mathcal{L}}(\theta^{\prime},\theta^{\prime\prime},\phi)\quad s.t.\qquad\frac{1}{n}\sum_{i=1}^{n}\textrm{ Constr}(G_{i})\leq\epsilon.$

(11)

Here $\epsilon$ is a user-defined hyperparameter that determines the strictness of enforcing these constraints. ${\mathcal{L}}(\theta^{\prime},\theta^{\prime\prime},\phi)$ is as defined in (2) and $i$ is in iterator over the scene graphs in the training set. We employ the learning under constraints framework introduced by Chamon und Ribeiro (2020) which results in a primal-dual saddle point optimization problem. We give exact algorithmic details in the Appendix (Algorithm 1).

For inference, we start with a room layout graph $G_{R}$ and type $T$ along with the number of furnitures to be placed in the room $n_{F}$. We then use the learnt autoregressive prior to sample $n_{F}$ latent variables recursively. This latent $Z$ along with $(n_{F},G_{R},T)$ is processed by the graph decoder to generate the funiture layout subgraph $G_{F}$. For scene rendering, we use the predicted shape descriptor for each furniture item and perform a nearest neighbour lookup using the $\ell_{2}$ distance against a database of 3D furniture mesh objects indexed by there respective PointNet feature. We then use the predicted size and orientation to place furnitures in the scene. Note the inclusion of shape descriptors allows the model to reason about different appearance of furniture items that is more room aware as opposed to retrieval based on just furniture category & size as in Wang u. a. (2020); Paschalidou u. a. (2021).

As explained in §3.1, our indoor scene representation is an attributed graph. We now describe these attributes in detail.

• •

  Room node features $\bm{X_{R}}$,

  1. 1.

     Node Type: A 4D 1-hot embedding of the node type. The four categories are wall, door, floor, window.

  2. 2.

     Room Type: A 4D 1-hot embedding of the room type. The four categories are living room, bedroom, dining room, library.

  3. 3.

     Location: A 6D vector representing the minimum and maximum vertex positions of the 3D bounding box corresponding to the room node.

  4. 4.

     Normal: A 3D vector representing the direction of the normal vector corresponding to the room node.

• •

  Furniture node features $\bm{X_{F}}$,

  1. 1.

     Super category: A 7D 1-hot embedding of the node type. The seven categories are Cabinet/Shelf, Bed, Chair, Table, Sofa, Pier/Stool, Lighting.

  2. 2.

     Shape: A 1024D embedding of the 3D shape descriptor obtained by processing a 3D point cloud of the furniture item through PointNet.

  3. 3.

     Location: A 3D vector representing the centroid of the furniture item.

  4. 4.

     Orientation: A 3D vector representing the direction the “front" side of the furniture is facing.

  5. 5.

     Size: A 3D vector representing the dimensions of the furniture along each axis.

• •

  Furniture-furniture edge features $\bm{X_{FF}}$, let us arbitrarily chose a furniture-furniture edge and label its source node as $s$ with receiver node as $r$. We will describe the features for this edge.

  1. 1.

     Center-center distance: A scalar representing the distance between the centroids of furniture $s$ and $r$.

  2. 2.

     Relative orientation: A scalar representing the signed dot product between the orientation feature of $s$ and $r$.

  3. 3.

     Center-center orientation: A 3D vector representing the unit vector connecting the centroids of $r$ and $s$.

  4. 4.

     Orientation: A 3D vector representing the direction the “front" side of the furniture is facing.

  5. 5.

     Bbox-bbox distance: The shortest distance between the corners of the bounding box of $s$ and $r$.

• •

  Room-furniture edge features $\bm{X_{RF}}$, let us arbitrarily chose a room-furniture edge and label its source node as $s$ with receiver node as $r$. We will describe the features for this edge.

  1. 1.

     Center-room distance: A scalar representing the distance between the centroid of furniture $r$ and room node $s$. This is computed in 2D as a point to line distance. Every wall/window/door can be treated as a line segment in 2D.

  2. 2.

     Center-room center: A scalar representing the distance between the centroids of $r$ and $s$.

  3. 3.

     Bbox-room dist A scalar representing the shortest distance between corners of the bounding box of furniture item $r$ to the room node $s$. This is computed in 2D as a point-to-line distance.

  4. 4.

     Bbox-room center A scalar representing the shortest distance between corners of the bounding box of furniture item $r$ to the centroid of the room node $s$.

  5. 5.

     Relative orientation: A signed dot product between the normal of $s$ with the orientation attribute of $r$.

• •

  Room-room edge features $\bm{X_{RR}}$, let us arbitrarily chose a room-room edge and label its source node as $s$ with receiver node as $r$. We will describe the features for this edge.

  1. 1.

     Center-center distance: A scalar representing the distance between the centroids of room nodes $s$ and $r$.

  2. 2.

     Relative orientation: A scalar representing the signed dot product between the normal vectors of $s$ and $r$.

  3. 3.

     Longest distance: A scalar representing the longest distance between the corners of the bounding boxes of $r$ and $s$.

  4. 4.

     Shortest distance: A scalar representing the shortest distance between the corners of the bounding boxes of $r$ and $s$.

The reconstruction loss is the first term in the ELBO (3).

$$\mathbb{E}_{Z\sim q_{\phi}(Z\mid G,T,n_{F})}[\log p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)]$$

In subsection 3.2 we described the distribution $p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)$ as

		$\displaystyle p_{\theta^{\prime}}(G_{F}\mid Z,n_{F},G_{R},T)$
	$\displaystyle=$	$\displaystyle\prod_{i=1}^{n_{F}}p_{\theta^{\prime}}(shape_{i}\mid Z,G_{R},T)p_{\theta^{\prime}}(orien_{i}\mid Z,G_{R},T)$
		$\displaystyle p_{\theta^{\prime}}(loc_{i}\mid Z,G_{R},T)p_{\theta^{\prime}}(size_{i}\mid shape_{i})p_{\theta^{\prime}}(cat_{i}\mid shape_{i}).$

Here $shape_{i},orien_{i},loc_{i},size_{i}$ and $cat_{i}$ refer to the ground truth values of the 3D shape descriptor (PointNet features), orientation, location, size and category respectively for furniture node $i$ in $G_{F}$.

We will now describe each of these distributions in detail along with the corresponding loss term. For the normal and lognormal distributions used we assume the variance parameter is a fixed constant equal to $1$ and do not learn them.

	$\displaystyle p_{\theta^{\prime}}(shape_{i}\mid Z,G_{R},T)$	$\displaystyle={\mathcal{N}}(\mu^{shape}_{\theta^{\prime}}(Z,G_{R},T),const.)$
	$\displaystyle\log p_{\theta^{\prime}}(shape_{i}\mid Z,G_{R},T)$	$\displaystyle=-\left(\mu^{shape}_{\theta^{\prime}}(Z,G_{R},T)-shape_{i}\right)^{2}$

	$\displaystyle p_{\theta^{\prime}}(orient_{i}\mid Z,G_{R},T)$	$\displaystyle=Categorical(\Theta^{orient}_{\theta^{\prime}}(Z,G_{R},T))$
	$\displaystyle\log p_{\theta^{\prime}}(orient_{i}\mid Z,G_{R},T)$	$\displaystyle=-H(orient_{i},\Theta^{orient}_{\theta^{\prime}}(Z,G_{R},T))$

$orient_{i}$ is a $4$D $1$-hot embedding of the ground truth orientation and $\Theta$ is the parameter of the categorical distribution. $H(.,.)$ denotes the cross entropy between two distributions.

	$\displaystyle p_{\theta^{\prime}}(loc_{i}\mid Z,G_{R},T)$	$\displaystyle={\mathcal{N}}(\mu^{loc}_{\theta^{\prime}}(Z,G_{R},T),const.)$
	$\displaystyle\log p_{\theta^{\prime}}(loc_{i}\mid Z,G_{R},T)$	$\displaystyle=-\left(\mu^{loc}_{\theta^{\prime}}(Z,G_{R},T)-loc_{i}\right)^{2}$

	$\displaystyle p_{\theta^{\prime}}(size_{i}\mid shape_{i})$	$\displaystyle=LogNormal(\mu^{size}_{\theta^{\prime}}(\mu^{shape}_{\theta^{\prime}}(Z,G_{R},T)),const.)$
	$\displaystyle\log p_{\theta^{\prime}}(size_{i}\mid shape_{i})$	$\displaystyle=-\left(\mu^{size}_{\theta^{\prime}}(\mu^{shape}_{\theta^{\prime}}(Z,G_{R},T))-\ln size_{i}\right)^{2}$

	$\displaystyle p_{\theta^{\prime}}(cat_{i}\mid shape_{i})$	$\displaystyle=Categorical(\Theta^{cat}_{\theta^{\prime}}(\mu^{shape}_{\theta^{\prime}}(Z,G_{R},T)))$
	$\displaystyle\log p_{\theta^{\prime}}(cat_{i}\mid shape_{i})$	$\displaystyle=-H(cat_{i},\Theta^{cat}_{\theta^{\prime}}(\mu^{shape}_{\theta^{\prime}}(Z,G_{R},T)))$

Each furniture node feature in the output of the MP-GNN Decoder is individually processed using a multi-layer Perceptron (MLP). Specifically, let $\hat{h}_{i}^{L}$ be the output feature of the decoder for furniture node $i$. In our experiments $\hat{h}_{i}^{L}\in\mathbb{R}^{1034}$.

$$\mu^{shape}_{\theta^{\prime}}(Z,G_{R},T)=\textrm{Linear}_{\theta^{\prime}}(1034,1024)$$

$$\Theta^{orient}_{\theta^{\prime}}(Z,G_{R},T))=\textrm{SoftMax}(\textrm{Linear}_{\theta^{\prime}}(1034,4))$$

$$\mu^{loc}_{\theta^{\prime}}(Z,G_{R},T)=\textrm{Linear}_{\theta^{\prime}}(1034,3)$$

$$\mu^{size}_{\theta^{\prime}}=\textrm{Linear}_{\theta^{\prime}}(1024,512)\rightarrow\textrm{ ReLU }\rightarrow\textrm{Linear}_{\theta^{\prime}}(512,3)$$

This is a three-layer MLP with ReLU activations and $1024$ input neurons since the input is the mean of the shape features $\mu^{shape}_{\theta^{\prime}}(Z,G_{R},T))$.