Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images

Early methods synthesize a specific type of 3D scene with manually defined rules [12, 2], simple statistic models learned from 3D scene instances [27, 5, 4, 11], or And-Or Graph (AoG)[17] of all valid object distributions in a scene.

Learning-based methods model the object properties and layouts in 3D scenes with a deep neural network learned from an annotated 3D scene dataset. All these methods abstract the objects and their properties (e.g. position, orientation, type, and size) in a scene as nodes with attributes and represent the 3D scenes as top-view 2D images [25, 18], node graphs [24, 10, 29] or trees [8] in scene layout generation. All these methods require a collection of 3D annotated scenes for training. Moreover, most of these methods except [29] represent the object geometry as its bounding box and thus fail to model the object shape and detailed object layouts. Different from these methods, we learn a GAN model from a collection of semantic-segmented depth images captured from the scenes. our method models the scene as a semantic scene volume and can generate scenes with more concrete object shapes and their detailed layouts.

A set of methods have been developed to complete or reconstruct 3D scenes from single [20, 7, 21, 28] or multiple RGB and depth images [3]. Some object layout reconstruction methods [15, 22, 16] recover the object pose, bounding box, and object layout from a single-view RGB image. All these methods are designed for reconstructing the geometry and semantic structure of a specific 3D scene and require a collection of complete 3D scenes for training. On the contrary, our method aims for generating different 3D scene layouts and is learned from a collection of semantic-segmented depth images.

A set of methods [6, 14, 9] have been proposed for learning the 3D GAN model of the objects in one category from 2D image collections. Different from these methods that focus on generating geometry or appearance of 3D objects from outside-looking-in images, our method focuses on generating indoor scene layout from inside-looking-out images.

Although the volumetric GAN in our method is adapted from HoloGAN [14], these two approaches are different. By representing objects with feature volumes, The HoloGAN does not disentangle objects’ shape and appearance and thus fails to generate a consistent projection of 3D objects under different views. Instead, our method models the scene geometry and layouts as semantic volume, which guarantees consistent projection from different views and is critical for learning 3D scene layouts from images captured from different unknown scenes. Also, different from HoloGAN that applies a single discriminator in training, we proposed a multiple-view discriminator for our task.

The input of our method is a collection of semantic-segmented depth images ${I^{d,s}_{i},i=1,2,\ldots,N}$ captured from different unknown rooms in a specific scene category, where the $I^{d}$ and $I^{s}$ refers to a depth image and its semantic label image. Each pixel in the semantic label image $I^{s}$ records the probabilities of the pixels’ visible surface point belonging to each object class $c_{j}$ in ${c_{0},c_{1},c_{j},\ldots,c_{C},c_{e}}$, where $c_{e}$ is the label of empty space and $C$ is the number of all object classes in a specific scene category. For input semantic-segmented images, the semantic label vector in each pixel is a binary vector of $1$ for the ground truth object class of the pixel and $0$ for other object classes. For all input images, we assume that their camera’s intrinsic parameters are known and each image includes a wall region of the scene so that we can estimate the camera’s pose and distance for the visible part of the scene in each image. Since we have no information about the underlying 3D rooms of the input images, we have no idea whether two images are captured from the same room.

We represent a generated 3D indoor scene with a semantic scene volume $S_{V}$ with fixed spatial resolution $w\times{h}\times{d}$ ($32\times 32\times 16$ in our implementation), each voxel of which stores a probability vector of its semantic label ${p_{0},p_{1},\ldots,p_{C},p_{e}}$. We align the floor of a scene with the XY plane of the volume and set the floor center to the center of the bottom volume layer $(h/2,w/2,0)$ ($(16,16,0)$ in our implementation). We predefine and fix the voxel’s physical stride $\gamma$ for each scene category so that the maximal room size $(d\gamma,h\gamma,w\gamma)$ that can be modeled by the semantic scene volume is determined. Given a room with size $\psi=(R_{x},R_{y},R_{z})$, we represent the layout, types, and shapes of the objects in the room with the semantic volume $S_{V}$, where all voxels out of the room boundary are labeled as empty.

Our volumetric GAN consists of three main components: a generator $G$, a discriminator $D$, and a differentiable projection layer that connects the generator and the discriminator. As shown in Figure. 1, the generator $G$ takes a latent vector $z_{s}$ and room size $\psi$ as input and outputs a semantic scene volume $S_{V}$ of the generated 3D scene. An encoder network $E_{\psi}$ encodes the room size $\psi$ into a set of conditional features of the generator. In the training stage, the projection layer renders the generated semantic scene volume $S_{V}$ from different views and feeds the rendered semantic-depth images to the discriminator $D$ to distinguish them from the real ones sampled from the training dataset. In the inference stage, we extract the object instances from the generated semantic scene volume in a post-processing step and then generate the final 3D scene by replacing all object instances with CAD models that are retrieved from an object database and best match the shapes and orientations of the object instances.

To acquire the object instances from the semantic scene volume, we first set the semantic label of each voxel as the one with the maximal probability. We randomly pick a voxel and iteratively group its neighboring voxels with the same label as an object instance via the flood filling algorithm. After that, we mark all voxels in the object instance is processed. We repeat this process until all voxels in the volume are processed. To remove the outliers, we discard the object instances the sizes of which are smaller than the minimal size of objects in the same object class of the 3D object database. Our experiments show that only around $1\%$ of object instances are removed in this step.

For each object instance $\hat{M}$ extracted from the semantic scene volume, we search the 3D object database $E(M)$ to find a 3D object $M_{i}$ and its rotation $\phi$ along $Z$ axis so that $M_{i}$ and $\hat{M}$ belong to the same object class and the rotated $M_{i}^{\theta}$ best matches the shape of the $\hat{M}$ and has minimal collision with the surrounding objects in the volume:

where $CD$ is the Chamfer distance between the candidate instance $\hat{M}$ and the rotated object $M_{i}^{\phi}$. $w_{c}$ is the penalty term of spatial collision, defined by the IoU between the rotated model $M_{i}^{\phi}$ placed in the scene and the surrounding voxels with other object class ID. $\lambda$ is a scalar ($1.0$ in our implementation) to balance the distance and collision terms.

Figure. 4 illustrates the results generated by our object retrieval and placement algorithm. Note that the volumetric representation offers the rough shape and orientation of the objects in the scene and thus result in detailed object layouts (e.g. TV and TV stands) that are difficult to be modeled by the object’s bounding box.

We implement our algorithm with Tensorflow and train our GAN model on a machine with 4 TESLA V100 GPUs. We train the network via the Adam optimizer. The sizes of semantic scene volumes and images are $32\times 32\times 16$ and $32\times 18$ respectively. The learning rate is $2e^{-4}$ and the batch size is 128. The training converges after 2,000 epochs.

We test our method on semantic-segmented depth images datasets in Structured3D [29] and Matterport3D [1] scene collections. We also apply our method to semantic-depth images inferred RGB images of NYUv2 [19] dataset. In all experiments, we use the ShapeNet dataset[26] for retrieving and placing CAD models to generate final 3D scenes.

For Structured3D, we train our model on three scene categories (Bedroom, Living room, and Kitchen) that exhibit rich layout variations. For each scene category, we collect the image by checking the angles between the ray to the room center and the optical axis of the images and choosing all images with angles smaller than $\pm{45}^{\circ}$. After that, we downsample the images to $32\times 18$. For this purpose, we first project the semantic-segmented depth images back into the 3D space and then voxelize the 3D space into a $32\times 32\times 16$ volume. Finally, we render the volume to the $32\times 18$ semantic-segmented depth images from the original viewpoint. For each scene category, we also merge the classes of objects with similar functionalities into one and remove the classes of objects that appear infrequently in scenes. The class ID of the pixels rendered from the removed objects is set to empty. For Matterport3D, we follow the same procedure to collect images from the bedroom category in our experiment.

For the NYUv2 dataset, we generate depth and semantic-segmented maps for all unlabeled RGB images in the bedroom scene category with the method in [13]. We then manually remove noisy results and select 1459 images with relatively large view coverage. After that, we follow the procedure described above to generate the training image set from the inferred semantic-segmented depth images. The statistics of the training dataset used in our experiments are listed in Table. 1.

We would like to thank anonymous reviewers for their constructive comments and suggestions, and Yue Dong and Nathan Holdstein for their help to proofread the paper.