RIM-Net: Recursive Implicit Fields for
Unsupervised Learning of Hierarchical Shape Structures

Shape parts obtained at each level of the hierarchy should together, via a union, reconstruct the input shape. We define $f_{i,j}({p})$ as the predicted probability of point ${p}$ being inside the part corresponding to the $i$th node at the $j$th level. Then the maximum of the set $\{f_{i,j}({p}),i=1,...2^{j}\}$ is the probability of point ${p}$ being inside the union of these parts. Therefore, the reconstruction loss of point ${p}$ at level $j$ is defined as

$\displaystyle L_{recon_{j}}({p})=(y_{gt}-\max(\{f_{i,j}({p})\}))^{2},i=1,\ldots,2^{j}$

(2)

where $y_{gt}\in\{0,1\}$ is the ground-truth value for point ${p}$ in the training set. We sum up the losses of all levels to form the total reconstruction loss:

$$L_{recon}^{(N)}({p})=\sum_{j=1}^{N}L_{recon_{j}}({p}).$$

(3)

In addition to shape reconstruction, we explicitly enforce a decomposition relation between an internal node and its two children. Geometrically, the part represented by an internal node should be the union of its two child parts. This constraint is imposed by a decomposition loss. Specifically, given a point ${p}$, its probability of belonging the parent node, $f_{i,j}({p})$, should be equal to the maximum probability of belonging one of the two child nodes, i.e., $f_{2i-1,j+1}({p})$ and $f_{2i,j+1}({p})$. So the hierarchy loss for the $i$th node at level $j$ is

$$L_{hie_{i,j}}({p})=(f_{i,j}({p})-\max(f_{2i-1,j+1}({p}),f_{2i,j+1}({p})))^{2}.$$

(4)

The total decomposition loss at a point ${p}$ is the sum of the $L_{hie_{i,j}}({p})$ for all internal nodes. For a tree with $N$ levels, we have

$$L_{hie}^{(N)}({p})=\sum_{j=1}^{N-1}(\sum_{i=1}^{2^{j}}L_{hie_{i,j}}({p})).$$

(5)

Finally, our full loss function is formulated as follows:

$$L^{(N)}=\sum_{{p}\in P}(\alpha L_{recon}^{(N)}({p})+\beta L_{hie}^{(N)}({p})),$$

(6)

where $P$ is the set of all the sample points in the training set and $N$ the number of levels of the structure tree. We set $\alpha=1$ and $\beta=10$ during the training process.

We conduct experiments to evaluate the use of per-point Gaussians by RIM-Net for implicit part reconstruction. In Fig. 4(a), we show a structure hierarchy predicted by the same network architecture as RIM-Net, except that the part decoder only outputs a scalar probability value per point. This is contrasted to the result in (d) by RIM-Net, demonstrating that the per-point Gaussians help obtain finer structures and better visual quality. For example, the decomposition of chair back and seat cannot be split in (a) and there is no more segmentation at level 2 and 3.

For a further demonstration to isolate advantages of using the Gaussians, we remove the hierarchy and only keep the first-level part decoder in RIM-Net, operating on the root feature vector output from the 3D conv-net encoder. The compared architectures are the same except they output different local point distributions for shape reconstruction: points (i.e., single value per point) [4] and spheres (consisting of a scaling factor, a center point, and a radii). We tried different numbers of branches in the part decoder, two and up, to provide a more general picture. Fig. 3 shows that with higher degrees of freedom provided by per-point Gaussians, the network tends to improve on both reconstruction and part inference: it can obtain fine-grained segmentation of the shape, the more part branches, the finer the division, while the method with points fails to segment shape finer.

In learning hierarchical structures via reconstruction, we use a decomposition loss to constrain the parent-children decomposition relation in the hierarchy. We train an ablated version of our RIM-Net without decomposition loss, keeping the other settings unchanged. In Fig. 4(b), we see the ablated model fails to maintain the parent-children relation in various levels and the decomposition cannot go finer compared with our full method. This verifies that the reconstruction loss only is not enough to learn a good hierarchical structure.

Our method progressively trains each level and then finetunes the full network. Here, we come up with a baseline without the progressive training strategy. It trains the full network jointly from scratch. The number of iterations is the same as that for training one level in our progressive method. As shown in Fig. 4(c), our method is unable to achieve a fine-grained and reasonable segmentation without the progressive training.

HA [22] is a representative work of learning hierarchical shape structures in a self-supervised manner. Since there is no well-established evaluation metric or protocol available, we provide a qualitative comparison of the learned hierarchies in Fig. 5. HA decomposes a part into smaller ones into deep levels and then selects a set of reasonable parts across various levels as the final output. In our method, the decomposition at each level is always a full reconstruction of the input model. As shown in Fig. 6, our decomposition results look more consistent and meaningful. HA learns shape hierarchies through abstracting shape parts as cuboids, which may overly decompose a geometrically complex part into many small cuboids. In contrast, our method models the geometric variations of shape parts with implicit fields, leading to more consistent structures.

We compare with more structure learning methods. Since most of the structure learning methods do not produce hierarchy, we compare them with the finest level of our recursive implicit fields. In the quantitative comparison in Tab. 2, our method outperforms all alternatives in CD, IoU and LFD, especially on tables and chairs with rich structures. Fig. 6 provides a visual comparison. VP [24] and SQ [19] abstract an object into sets of cuboids or superquadrics, respectively, causing the loss of fine details and the meaningfulness of the parts, especially for chairs. HA [22] learns better structures with its hierarchical cuboid abstraction, but is still unable to handle the shape variations on, e.g., the round table and the four-legged table. CA [26] can generate shape structures with more details, but shares the same drawback as HA. BAE-Net [4] preserves fine details of the objects but only captures coarse parts. In contrast, RIM-Net is able to learn fine-grained parts with accurate reconstruction, under the same setting of maximum part count. For example, BAE-Net fails to segment the individual legs and cannot separate the back from the seat of chairs while our method succeeds.

Next, we evaluate structure learning from the segmentation point of view, measuring the consistency and meaningfulness of the segments. In particular, we transfer the segmentations onto point clouds as per-point part label and compute the mIoU to evaluate co-segmentation quality, following [4]. Since different methods output different numbers of parts for the same category, we take the ground-truth dataset of [27] and conduct voting-based label snapping as in [4], for a fair comparison. The “leg” and “support” labels are merged as “support” for the table category. From Tab. 3, our method is superior to other unsupervised structure learning methods.

On unsupervised single-view reconstruction of hierarchical structures, the work of Paschalidou et al. [18] is the most relevant. Their work predicts structure hierarchies based on spatial positions: A part is decomposed in three ways (up-down, front-back, and left-right) without accounting for the meaningfulness of parts. We visually compare the hierarchies of the two methods in Fig. 7. The shape expression ability of superquadrics of [18] is limited. Moreover, their decomposition could cause over-segmentation. For example, the back of the chair is segmented at each level, which is obviously unnecessary. In our hierarchy, each level constitutes a good approximation of the input, and the segments at each level are semantically meaningful. More importantly, their method cannot produce consistent segmentation for shapes of the same category, due to spatial position based decomposition.

We compare to alternative methods, IM-Net [5] and BSP-Net [3], which also employ implicit fields, to demonstrate how our key designs, the hierarchy and the per-point Gaussians, improve reconstruction quality. Different from our method, IM-Net outputs a single implicit field for the holistic shape while BSP-Net outputs a set of convexes together forming the field.

For this experiment, we use the same dataset as IM-Net. It contains five representative categories from ShapeNet [2] with rendered views by 3D-R2N2 [6], i.e., airplanes (4,045), cars (7,497), chairs (6,778), rifles (2,373) and tables (8,509). For all methods, we train a separate model for each category. The train/test split we use is the same as [3]. For the training of implicit field SVR, we adopt the training scheme in [5, 3]. We first pretrain a 3D auto-encoder. We then supervise the feature reconstruction quality of the image encoder module through measuring the mean square error (MSE) between the features extracted for the input image and the pre-trained features of the 3D auto-encoder.

As reported in Tab. 4, our method outperforms the two methods in CD and LFD. Fig. 8 shows qualitative results. As demonstrated in the visual results, our method achieves comparable reconstruction quality with the state of the arts, while additionally producing hierarchical structures without any supervision. The inferred segmentations by our method are shown in colors in the row of “Ours*”.