Unsupervised Learning on 3D Point Clouds by Clustering and Contrasting

Recent years have witnessed significant progress in feature learning for 3D shapes due to the ability of deep learning techniques to consume 3D point clouds directly. PointNet [8], and DeepSets [28] are the pioneering architectures that can handle unordered and unstructured 3D points by independently learning each point of the point cloud and fusing point features with invariant permutation operations. Though efficient, PointNet and DeepSets ignore the local structures that are indispensable to describe the 3D shape. PointNet++ [9] was then proposed to mitigate this issue by developing sampling and grouping operations to extract features from point clusters hierarchically. Similarly, various recent studies such as PointCNN [29], PointConv [30] DGCNN [10] and Relation-Shape CNN [31] also focus on extracting more semantic features from the local region by separating points into scales or bins, and then, aggregating these features by concatenation [21] or with an RNN. PAConv [32] uses a plug-and-play convolutional operation for deep representation learning on 3D point clouds. AdaptConv [33] proposed to adaptively establish the relationship between a pair of points according to their learned features, effectively and precisely capturing the diverse relations between points from different semantic parts. Sun et al. [34] proposed a point-to-surface representation for 3D point cloud learning considering both the point and the geometric surface simultaneously. [35] developed a perturbation learning-based point cloud upsampling method to generate uniform, clean, and dense point clouds. LGA [36] presented a layer-wise geometry aggregation framework for lossless LiDAR point cloud geometry compression. Transformer3D-Det [37] proposed to solve the 3D object detection task by using the attention mechanism to model the relationship from neighboring clusters to produce more accurate voting centers. Although these methods achieve remarkable success, they require supervised information during the feature learning process. The dependence on annotation impedes the deployment of point cloud models into new real-world settings where labeled data is scarce. Therefore, it is important to develop methods to reduce the number of annotated samples that are required to achieve the required performance of deep learning-based point cloud understanding tasks.

Current 3D sensing modalities have enabled the generation of extensive unlabeled 3D point cloud data [13]. This has boosted a recent line of works on learning discriminative representations of 3D objects using unsupervised approaches. Tasks such as semantic segmentation, registration, object classification, and part segmentation combined with unsupervised pre-training can outperform the traditional fully supervised training pipelines [38, 39]. Unsupervised representation learning approaches could be roughly classified into two categories: generative and discriminative models. Generative models are performed by conducting self-reconstruction that first encodes a point cloud into a feature or distribution and then decodes it back to a point cloud [21]. For example, FoldingNet [16] leverages a graph-based encoder and a folding-based decoder to deform a canonical 2D grid onto the surface of a point cloud. Liu et al. [40] proposed a local-to-global auto-encoder to simultaneously learn the local and global structure of point clouds by local to global reconstruction. [41] focused on designing a graph-based decoder by leveraging a learnable graph topology to push the codeword to preserve representative features. With the help of GAN, Panos et al. [42] trained the network to generate plausible point clouds by combining hierarchical Bayesian and generative models. However, generative models are sensitive to transformation and tend to learn different representations if the point clouds are rotated or translated. This weakens the network’s ability for point cloud understanding tasks. Moreover, it is not always feasible to reconstruct back the shape from pose-invariant feature representations [22].

As for discriminative methods, some of them rely on using auxiliary handcrafted prediction tasks to learn their representation. For instance, Gao et al. [43] self-train a feature encoder to capture the graph structures by reconstructing these node-wise transformations from the representations of the original and transformed graphs. Following the impressive results achieved with Jigsaw puzzles-based methods in the image domain, Sauder et al. [44] introduced a self-supervised learning task to reconstruct a point cloud from its randomly rearranged parts. However, these two methods are still sensitive to rotation. In contrast, contrastive approaches [18, 22, 25, 45], which are robust to transformation, currently achieve state-of-the-art performance in unsupervised learning. For example, in order to learn representations, Info3D [22] maximizes the mutual information between the 3D shape and a geometric transformed version of the 3D shape. PointContrast [45] was the first to research a unified framework of the contrastive paradigm for 3D representation learning. However, at first, contrastive methods depend on customized strategies to mine and store negative samples, since they often require comparing each example with a large number of negative samples to work well. Apart from that, because of the lack of adequate and semantic local structure supervision, most previous unsupervised approaches are prone to error accumulation during the local structure learning process [18, 21], which weakens the network’s ability to learn the local geometry.

To mitigate these issues, we propose an unsupervised learning method to train a point cloud feature encoder that jointly leverages instance-level contrasting and point-level clustering. The proposed instance-level contrasting representation learning method is inspired by SimSiam [25], and extends its simplicity to the learning of 3D point cloud global representation. Note that instance-level contrasting only depends on the positive pairs. The point-level clustering, which is formulated by implementing an EM-like algorithm, provides local supervision to extract discriminative local features. Our method remains agnostic to the specific choice of the 3D representation or the underlying neural network architecture.

From the human perspective, both the global and local shape information play vital roles in 3D point cloud understanding. The global geometry depicts the overall shape, and the local geometry depicts the detailed shape, which inspires us to learn distinctive representations that retain the global and local geometry. To achieve the goal, this paper proposes an unsupervised method by jointly learning global (instance-level contrasting) and local (point-level clustering) shape information. Our instance-level contrasting-based unsupervised feature learning approach is based on a Siamese network structure [46]. Siamese networks, which have been widely used for 2D unsupervised representation learning tasks, such as BYOL [15] and SimSiam [25], are weight-sharing neural networks applied to two or more inputs. For point-level clustering, we provide an EM-like soft clustering algorithm.

In this paper, we consider a 3D point cloud $\bm{\mathcal{P}}=\{\bm{p}_{i}\in\mathbb{R}^{3}|i=1,2,...,N\}$ with $N$ elements, in which each point $\bm{p}_{i}$ is represented by a 3D coordinate. $\bm{\mathcal{P}}$ is processed by an encoder backbone $f_{\varphi}$ that yields a point-wise feature matrix $\bm{\mathcal{F}}=\{\bm{f}_{p_{i}}\}_{i=1}^{N}$. $\bm{f}_{p_{i}}$ is a feature vector. Our goal is to train a feature encoder $f_{\varphi}$ ($e.g.$, PointNet) with parameters $\varphi$ in an unsupervised fashion. The pipeline of our framework is illustrated in Fig. 1, which includes two modules: instance-level contrasting for the global feature learning, and point-level clustering for the local feature learning. Here is a summary of the two modules before we get into details. The two randomly augmented views $\bm{\mathcal{P}}^{a}$ and $\bm{\mathcal{P}}^{b}$ of $\bm{\mathcal{P}}$ are processed by $f_{\varphi}$ that yields features $\bm{\mathcal{F}}^{a}$ and $\bm{\mathcal{F}}^{b}$. The encoder $f_{\varphi}$ shares weights between the two views. The instance-level contrasting takes as inputs $\bm{\mathcal{F}}^{a}$ and $\bm{\mathcal{F}}^{b}$, and the point-level clustering takes as inputs $\bm{\mathcal{P}}^{a}$ and $\bm{\mathcal{F}}^{a}$. $f_{\varphi}$ is then trained by jointly minimizing the instance-level contrasting loss and the point-level clustering loss. The encoder only receives gradient from the top branch. After training, both the instance-level contrasting and the point-level clustering module are discarded, and $f_{\varphi}$ can be transferred to downstream tasks.

For all of our algorithmic discussions, $\|\cdot\|_{2}$ is $l_{2}$-norm, and the Frobenius dot product of two matrices $\bm{A}\in\mathbb{R}^{N\times J}$ and $\bm{B}\in\mathbb{R}^{N\times J}$ is denoted by

As shown in Fig. 2, the core of our end-to-end point-level clustering-based unsupervised feature learning framework consists of two components: a) class probability $\bm{S}$ prediction, and b) label $\bm{\gamma}$ reassigning. It can be interpreted as a semantic segmentation task in which each point $\bm{p}_{i}$ in a point cloud $\bm{\mathcal{P}}$ is assigned to one of $J$ possible semantic categories or partitions. Specifically, a point cloud $\bm{\mathcal{P}}$ is processed by a neural network consisting of a backbone $f_{\varphi}$ and a class probability prediction operator that outputs a class probability matrix $\bm{S}=\{s_{ij}\in[0,1]\}_{i,j}^{N,J}$. The label reassigning operator, which takes as inputs $\bm{S}$ and $\bm{\mathcal{P}}$, yields a pseudo-label matrix $\bm{\gamma}=\{\gamma_{ij}\in\{0,1\}\}_{i,j}^{N,J}$. The network parameters of $f_{\varphi}$ are then learned by minimizing the average cross-entropy loss between the pseudo-label $\bm{\gamma}$ and the predicted class probability $\bm{S}$ [47]. The average cross-entropy loss is written as

Training with the objective in Eq. (2) requires a labeled dataset. Since the point-wise labels are unavailable, we require a mechanism to assign the label $\bm{\gamma}$ automatically. Next, we will give a detailed explanation of the class probability prediction and the label reassigning.

The instance-level contrasting is proposed to learn the global representations of the 3D point clouds. As shown is Fig. 5, our architecture takes as input two randomly augmented views $\bm{\mathcal{P}}^{a},\bm{\mathcal{P}}^{b}$ from a point cloud $\bm{\mathcal{P}}$. The two views are processed by a neural network consisting of an encoder backbone $f_{\varphi}$, Maxpooling $\rho$ and a projection MLP $g$ head [25]. The encoder $f_{\varphi}$ and projection $g$ share weights between the two views. A prediction MLP head [15], denoted as $q$, transforms the output of one view and matches it to the other view. In particular, the predictor is only applied to the one branch, making the architecture asymmetric [15]. Denoting the two output vectors as $\bm{q}^{a}\triangleq q(g(\rho(\bm{\mathcal{F}}^{a})))$ and $\bm{z}^{b}\triangleq g(\rho(\bm{\mathcal{F}}^{b}))$ with $\bm{\mathcal{F}}^{a}=f_{\varphi}(\bm{\mathcal{P}}^{a})$ and $\bm{\mathcal{F}}^{b}=f_{\varphi}(\bm{\mathcal{P}}^{b})$. Following [25], a stop-gradient (stopgrad) operation is applied on $\bm{z}^{b}$ to prevent the model from collapsing to a constant mapping in the absence of using negative samples. The model is formulated by maximizing the agreement between $\bm{q}^{a}$ and $\bm{z}^{b}$. Specifically, we define the following mean squared error between the $l_{2}$-normalized prediction $\bm{q}^{a}$ and projection $\bm{z}^{b}$ to measure their agreement.

This is equivalent to the negative cosine similarity, up to a scale of 2. When stop-gradient is applied on $\bm{z}^{b}$, we measure the similarity between $\bm{q}^{a}$ and $\bm{z}^{b}$ by modifying Eq. (9) as:

This means that $\bm{z}^{b}$ is treated as a constant vector in this term. Following [25], we define a symmetrized loss as:

The minimum possible value of Eq. ( 11) is 0. The encoder on $\bm{q}^{a}$ receives no gradient from $\bm{z}^{b}$ in the first term, but it receives gradients from $\bm{q}^{b}$ in the second term (and vice versa for $\bm{q}^{b}$). If without adopting the stop-gradient, despite a zero loss during the training process, the representations learned are useless, as all point clouds tend to get the same representation [25], i.e., the model collapses to a constant mapping. We use the pseudocode in Algorithm 2 to illustrate the instance-level contrasting-based unsupervised feature learning.

By traversing two randomly augmented views $\bm{\mathcal{P}^{a}},\bm{\mathcal{P}^{b}}$ from a point cloud $\bm{\mathcal{P}}$, the cluster-level loss is finally computed by

where $\eta>0$ is a regularization parameter. We set it as $2\times 10^{-3}$ in this paper. $\bm{\gamma}^{a}$, $\bm{S}^{a}$ and $\bm{C}^{a}$ represent the posterior probability, the predicted probability and the normalized centers of $\bm{\mathcal{P}^{a}}$, respectively. Similarly, $\bm{\gamma}^{b}$, $S^{b}$ and $\bm{C}_{*}^{b}$ are related to $\bm{\mathcal{P}^{b}}$.

Finally, the total loss is defined as a combination of $\mathcal{L}_{global}$ and $\mathcal{L}_{local}$:

Our implementation is built on the Pytorch [51] library. We used AdamW [52] as the default optimizer with the base learning rate of 0.001. The batch size was set to 32, and the learning rate was delayed by 0.7 in every 20 epochs. We trained our model 200th epoch. As shown in Figure 6, the loss function decreased and stabilized at around the 200th epoch. All of our models were trained on two Tesla V100-PCI-E-32G GPUs.