Unsupervised Learning of 3D Semantic Keypoints with Mutual Reconstruction

Implicit methods employ self-related metrics to measure the quality of keypoints. A typical implicit method is skeleton merger [18], whose key idea is to reconstruct skeleton-liked objects based on its keypoints through an encoder-decoder architecture. Another implicit way [5] utilizes a convex combination of local points to generate local semantic keypoints, which are then measured by how close they are to the origin point cloud. Unsupervised stable interest point (USIP) [12] predicts keypoints with a Siamese architecture, and the two inputs are two partial views from a 3D object. Implicit methods can achieve good spatial consistency and are relatively light-weight. However, they generally fail to mine semantic consistency information.

Explicit methods cope with category-level information directly. Keypoint deformer [10] employs a Siamese architecture for shape deformation to detect shape control keypoints; the difference between two input shapes is analysed by comparing their keypoint sets. The cage [31] method is crucial to keypoint deformer [10]; to deform a point cloud, cage [31] takes the origin point cloud, shape control points on point cloud, and target cage as input, the output of cage consists of a deformed cage and a deformed point cloud under the constraint of cage. Another explicit method [9] learns both category-specific shape basis and instance-specific parameters such as rotations and coefficients during training; however, the method requires a symmetric hypothesis. Different from the two previous works, our method evaluates keypoints from the self and mutual reconstruction quality by estimated keypoints and do not require additional hypotheses on inputs.

A number of neural networks have been proposed, e.g., PointNet [16], PointNet++ [17], and PointConv [28], which directly consume 3D point clouds. PointNet [16] is a pioneering work, which extracts features from point clouds with point-wise MLPs and permutation-invariant functions. Based on PointNet, PointNet++ [17] introduces a hierarchical structure to consider both local and global features; PointNet is applied after several sampling and grouping layers; PointNet++ is also employed as an encoder by several unsupervised 3D semantic keypoint detection methods [18, 5]. More recent point cloud encoders include [20, 28, 27]. These encoders have achieved success in tasks like registration [26] and reconstruction [15]. Several keypoint detection methods[10, 18, 5] also employ PointNet++ [17] as the encoder.

In previous point cloud learning works [1, 33], MLP is frequently leveraged to generate point clouds from the encoded features. Specifically, FoldingNet [30] proposes a folding-based decoder to deform 2D grid onto 3D object surface of a point cloud. Many works [7, 8, 29] follow FoldingNet [30] and decode the features based on structure deformation. In [19], tree structure is used to decode structured point clouds. From the functional view, most decoders leveraged by 3D semantic keypoint detection methods [18, 12, 5, 9] focus on reconstructing the original shape of the input. An exception is keypoint deformer [10], whose decoder tries to deform the source shape into the target shape through cage-based deformation.

Our mutual reconstruction module is depicted in Fig. 3. The mutual reconstruction process utilizes several outputs from self reconstruction, including keypoint sets $KP_{1},KP_{2}$ and the global feature $GF$.

Mutual reconstruction is supposed to be able to extract category-level semantic consistent keypoints as illustrated in Fig. 4. The figure illustrates the semantic ambiguity of self-reconstruction, which can be resolved by the mutual reconstruction module. When the method with only self-related tasks (e.g., self reconstruction) predicts object-wise keypoints which are not semantically consistent, it may fail to notice the inconsistency as the topology information is inconsistent as well (we visualize the topology information as a sequence of connection, while some methods employ topology information implicitly); however, the mutual reconstruction model is sensitive when either the topology or semantic label prediction is not correct, as additional shapes are considered in mutual reconstruction and the constraint on keypoint consistency is much tighter.

The self reconstruction module is presented in Fig. 5. Specifically, the point-wise feature can also be considered as point-wise score, because the keypoints are actually generated by linear combination of origin points. In other words, for the point with a higher score (feature value), it contributes more to the keypoint prediction. We simply define the point-wise score to be the sum of the $k$-dim feature, as visualized in Fig. 3 (the airplane in red).

Self reconstruction is also a critical component for mining the semantic information from an instance [18, 5]. To ensure category-level semantic consistent information, instance and cross-instance information should be mined, such that self reconstruction is utilized as complementary to mutual reconstruction.

The designed encoder is supposed to generate keypoints proposals $K_{1},K_{2}$ from input point clouds $P_{1},P_{2}$. First, we employ the PointNet++ [17] encoder and it offers a $K$-dimension point-wise feature for every point in the origin point cloud, thus the shape of point feature matrix $F$ is $K\times N$. Keypoints are calculated by:

After keypoints proposals $KP$ are generated by the encoder, they are reshaped into new keypoint sets $KP^{\prime}$, which are utilized by the decoder for mutual reconstruction. We reshape source keypoint set $KP$ with a point-wise offsets $O_{kp}$ as:

and

where $O_{kp}$ is calculated by feeding offsets of origin point clouds $O_{p}$ into a 3-layers MLP, as in the following:

and

The reshaped source keypoints are fed to the decoder for reconstruction.

We build our decoder following skeleton merger [18]. The decoder takes keypoint sets $KP,KP^{\prime}$ and global feature (activation strengths and trainable offsets) as input. It first generates $n(n-1)/2$ line-like skeletons, each of them is composed of a series of points with fixed intervals. Second, trainable offsets are added to every point on the skeleton-like point cloud. Finally, $n(n-1)/2$ activation strengths are applied to the $n(n-1)/2$ skeletons for reconstruction; only skeletons with high activation strengths contribute to the reconstruction process. As such, shapes are reconstructed by decoders.

We calculate reconstruction loss with Composite Chamfer Distance (CCD) [18]. CCD is a modified Chamfer Distance which takes the activation strengths into consideration. For fidelity loss, the CCD between $\hat{X}$ and $X$ is given as:

where $\hat{X}_{i}$ is the $i$-th skeleton of point cloud $\hat{X}$, and $a_{i}$ is the activation strength of $\hat{X}_{i}$. For the coverage loss, there is a change from the fidelity loss that more than one skeleton are considered in the order of how close they are to the given point, until the sum of their activation strengths exceeds $1$ [18].

We apply the CCD loss in both self and mutual reconstruction tasks. For self reconstruction, we calculate the CCD between the input target shape $P_{t}$ and output target shape $P_{t}^{\prime}$:

For mutual reconstruction, we calculate the CCD between the input target shape $P_{t}$ and output source shape $P_{s}^{\prime}$:

The eventual reconstruction loss is a combination of the two losses:

where $\lambda_{s}$ and $\lambda_{m}$ are weights to control the contributions of self and mutual reconstructions.

The trainable offsets in our decoder are calculated by multiple MLPs. To keep the locality of every points on the skeleton, we apply an $L_{2}$ regularization on them. $L_{2}$ regularization is also imposed on the keypoint offset $O_{K}$, in order to reduce the geometric changes of keypoints when reconstructing the other shape.

We first evaluate the semantic consistency on KeypointNet[32] with DAS, which is introduced by [18]. Given the estimated keypoints on a source point cloud, DAS employs a reference point cloud for keypoint quality evaluation. We predict keypoints with our model on both source and reference point clouds, and use the human annotation on the reference point cloud to align our keypoints with annotated keypoints. The closet keypoint to a human annotation point is considered to be aligned with the annotation. DAS then calculates the ratio of of aligned keypoints between the source and reference point clouds.

The results are shown in Table 1. It can be found that ISS is significantly inferior to others, because it tries to find distinctive and repeatable points rather than points with semantic information. Compared with two recent unsupervised learning methods, our method also surpasses them in most categories.

We report the mIoU of predicted keypoints and ground truth ones. The results are shown in Table 2.

As witnessed by the table, our method achieves the best performance on four categories. Noth that Fernandez et al. [9] only reported results on the ‘Airplane’ and ‘Chair’ categories, while our method still outperforms it on these two categories.

We also test the mean part correspondence ratio on ShapeNet Part dataset. This metric is not as strict as DAS and mIoU, because it defines two semantic keypoints are corresponding if they are in the same semantic part of objects in a category. The comparative results are shown Table 3.

Due the that the part correspondence ratio is a loose metric, the gaps among tested methods are not as dramatic as those in Tables 1 and 2. Remarkably, our method also achieves the best performance under this metric.

We first visualize the 3D keypoint distribution and the keypoint features based on t-SNE in Fig. 6, where different colors indicate different semantic labels. Here, we take the skeleton merger method as a comparison.

It can bee seen that our method ensures more consistent alignment of semantic keypoints, as keypoints of the same semantic label tend to be close to each other in the 3D space. Besides, the t-SNE results suggest that our encoder learns more distinctive category-level information from point clouds. Finally, we give a comparative semantic keypoint detetcion results in Fig. 7.