PointCG: Self-supervised Point Cloud Learning via Joint Completion and Generation

Self-supervised representation learning aims to derive robust and general representations from unlabeled datasets, which can be broadly classified into two categories based on the types of pretext tasks: contrastive learning and generative learning.

Contrastive learning-based methods (e.g., BYOL [18], SimSiam [6], DINO [19], STRL [9], CrossPoint [20]) define the augmented views of a sample as positive samples, while considering other instances as negative samples, thereby constructing discriminative tasks. Generative learning-based methods (e.g., GPT [21], Point-BERT [7], Point-MAE [3], OcCo [10], MaskFeat3D [22]) are based on the intuition that effective feature abstractions contain sufficient information to reconstruct the original geometric structures [11]. In the point cloud processing community, where 3D assets are relatively scarce, generative learning has garnered widespread attention due to its data efficiency [4, 3, 13]. Among them, MAE stands out as one of the representative paradigms. It involves masking a substantial portion of input data, followed by the use of an encoder to extract informative representations and a decoder to reconstruct explicit features (e.g., pixels or points) or implicit features (e.g., discrete tokens). Taking Point-BERT [7], MaskFeat3D [22], and IAE [23] as examples, each of these methods utilizes the visible groups as input after masking and reconstructs the positions of masked points, surface normals, and surface variations, as well as the implicit features of the masked points. However, after random masking [3] or partial occlusion [13], the visible groups often retain the overall structure of the object (Fig. 7), and there are substantial overlap regions between input and target patches (Fig. 2). The leakage of overall structure and point location information will reduce the difficulty in reconstructing masked patches, thus limiting the learning and inference capabilities of the encoder.

To avoid the leakage of the object’s overall shape and minimize overlap, we simulate scanners to capture visible points from arbitrary views as input. Our approach is conceptually aligned with OcCo [10], which employs the Z-Buffer algorithm [24] to select visible points from multiple views, subsequently completing the original point clouds with an encoder-decoder architecture. The Z-Buffer algorithm addressed within rendering relies on two assumptions: the points satisfy sampling criteria (e.g., Nyquist condition) and the points are associated with normals (or the normals can be estimated) [25]. However, our method seeks rigorous theoretical support for visibility computation without requiring normal estimation, point rendering, or surface reconstruction. Therefore, we employ the Hidden Point Removal (HPR) operator to compute visibility in a more robust manner.

Recently, cross-modal learning has been a popular research topic, aiming at extracting informative representations from multiple modalities such as images, audio, and point clouds. It has the potential to enhance the performance of various tasks, including visual recognition, speech recognition, and point cloud analysis.

In point cloud analysis, a variety of methods have been proposed for cross-modal learning, such as CrossPoint [20], PointMCD [26], TAP [14], and PointVST [11]. CrossPoint [20] establishes cross-modal contrastive learning between images and point clouds, demonstrating that the correspondence between images and points can enhance 3D object understanding. PointMCD [26] obtains a powerful point encoder by aligning the multi-view visual and geometric descriptors generated by a pretrained image encoder and a learnable point encoder. Both CrossPoint [20] and PointMCD [26] are based on the contrastive paradigm and rely heavily on extensive 3D-2D paired data. Generative methods, such as TAP [14] and PointVST [11], generate images from specific views based on the input point clouds. These methods use regular 2D images as generation objectives to provide precise supervision.

In this paper, we follow the generative learning paradigm and propose a unified pre-training framework with two complementary pretext tasks: hidden point completion and arbitrary-view image generation. The spatial awareness provided by 3D completion addresses the geometric insensitivity inherent in image supervision, as shown in the second line of column one in Fig. 1. Additionally, we demonstrate the mutual enhancement between the two pretext tasks through various experiments.

Given a complete point cloud $P=\{p_{i}|{1}\leq{i}\leq{N}\}\in\mathbb{R}^{3}$, we randomly select the camera position $C=[azimuth,elevation,distance]$, where $distance$ is fixed at 1.0. $azimuth$ and $elevation$ are randomly chosen within the range of $[0,2\pi]$. The HPR operator [25] is employed to determine whether ${p_{i}}$ is visible from $C$. It mainly consists of two steps: inversion transformation and convex hull construction.

Inversion transformation. We employ spherical flip [27] to reflect each point $p_{i}\in{P}$ to the spherical surface (denoted as $\hat{p_{i}}\in\hat{P}$) along the ray from $C$ through $p_{i}$ to the spherical surface, as illustrated in Fig. 4 (a).

Convex hull construction. The visible points from $C$ inverted on the spherical surface are situated on the convex hull of $\hat{P}\cup{C}$. Therefore, we need to compute the collection of triangular planes, which make up the convex hull. Then we extract all vertices (magenta points in $\hat{P}$ in Fig. 4 (b)) of the convex hull and project them back onto the original point cloud to obtain the visible points $P_{v}$ (magenta points in $P$ in Fig. 4 (b)). The remaining points of the original point cloud are hidden from $C$, denoted as $P_{h}$.

For each input, we employ the FPS and KNN to divide the visible points $P_{v}$ into patches with $v$ centers. Simultaneously, we extract $h$ central points from the hidden points $P_{h}$ and retrieve $k$ nearest neighbor points from the complete point cloud as the target patches $P_{GT}$. Then, the visible patches are projected into tokens $T^{v}\in\mathbb{R}^{v\times{d}}$ with a lightweight PointNet [28], where $d$ is the dimension of features. Subsequently, a learnable Multi-Layer Perceptron (MLP) is adopted to embed the visible and hidden centers into positional tokens denoted as $T_{L}^{v}$ and $T_{L}^{h}$, respectively. Finally, we extract representations $T_{E}$ by an encoder and capture the tokens $T_{D}$ with a decoder for completing the original point clouds:

where $T_{H}$ represents the hidden tokens, which is initialized by duplicating a learnable token of dimension $d$. We concatenate the visible points’ features $T_{E}$ and $T_{H}$, as well as the positional tokens $T_{L}^{v}$ and $T_{L}^{h}$ as the inputs of the decoder. Based on the outputs $T_{D}$ of the decoder, we will reconstruct the $k$ nearest neighbors of $h$ center points by a reconstruction head of a fully connected (FC) layer:

where $P_{pre}$ denotes as the predicted hidden point patches.

Loss function. The Chamfer distance [29] is employed as the reconstruction loss:

where $P_{GT}\in\mathbb{R}^{h\times{k}\times{3}}$ denotes the reconstruction targets.

Completion based on visible points from random views. To assess the effectiveness of our self-supervised model initialized with pre-trained weights, we randomly select an instance from synthetic dataset ModelNet40 [36] and real-world dataset ScanObjectNN [37] separately and reconstruct the original point clouds. The visualization results of Point-MAE [3] and our model are shown in Fig. 6.

Compared to Point-MAE, our method not only completes the chair’s pivot axis and the five-pronged base with greater fidelity but also obtains smoother surface structures. Our method achieves remarkable performance in reconstructing both synthetic and real-world data with visible points from arbitrary views.

Completion based on grouped patches and partial points. To demonstrate the robustness and generalizability of our method in the 3D completion task, we devise two methods to obtain grouped patches and partial points as inputs. The reconstruction results are visualized in Fig. 7.

In the first method, we obtain $2$, $8$, and $26$ central points via FPS, and then acquire $32$ neighboring points with KNN to form groups [3, 8] denoted by ‘G2’, ‘G8’, and ‘G26’. In the second method, we randomly select one group, consisting of $64$, $256$, and $832$ points, denoted as ‘P64’, ‘P256’, and ‘P832’, respectively.

For ‘G2’ and ‘G8’, both Point-MAE and our method can complete the overall structure, but our method excels in recovering finer geometric details and sharper edges. The ‘G26’ column retains sufficient information, and the results are satisfactory for both methods. For ‘P64’ and ‘P256’, our method successfully completes entire structures and captures many local details. In contrast, the results of Point-MAE appear quite blurry. Although the ‘G26’ and ‘P832’ have the same number of input points, the reconstruction results of Point-MAE based on ‘P832’ are significantly lower than ‘G26’.

As shown, the inputs obtained by random masking retain objects’ structural information while exposing the coordinates of target points. These MAE methods employing random masking strategy, such as Point-MAE, exhibit poorer reconstruction performance when the inputs are partial and lack of complete structural integrity. Our method, however, excels in extracting informative representations and demonstrates strong inference capability, leading to superior reconstruction performance, even from highly partial point data.

To assess the discrimination of the representations extracted by the pre-trained encoder, we validate the encoder on the shape classification task using the ModelNet40 [36] and ScanObjectNN [37] datasets.

Shape classification on synthetic data. ModelNet40 [36] contains $12,311$ clean 3D CAD models, covering $40$ object categories. We fine-tune the pre-trained encoder, and the results are presented in Tab. I. Our method achieves 94.03% with global fine-tuning, surpassing the reproduced version of Point-MAE (Rep.) (93.21%) by 0.82% and the publicly released accuracy by 0.23%. To validate the effectiveness of our architecture, we incorporate Point-M2AE as the backbone and evaluate its classification performance on ModelNet40. The classification results outperform the outcomes of the reproduced Point-M2AE model.

Besides, we also attempt to freeze the parameters of our pre-trained model, and validate it with a Linear-SVM classifier in Tab. II.

Our method outperforms Point-MAE [3] and Point-M2AE [8] by margins of +1.46% and +0.29%, respectively. The results highlight the superior quality of the 3D representation learned by our method.

Shape classification on the real-world data. Evaluating a pre-trained model’s performance on real-world datasets is crucial, as real-world scenes tend to be more complex than synthetic ones. We follow the common practice to evaluate our model on three variants: ‘OBJ-BG’, ‘OBJ-ONLY’, and ‘PB-T50-RS’ of ScanObjectNN [37].

To further validate the effectiveness of our design, we follow PointVST [11] with DGCNN as the encoder to construct a pre-training network (DGCNN+PointCG) and evaluate it on the ‘PB-T50-RS’ variant. Additionally, we reproduce TAP [14] and PointVST [11] using their official pretrained models.

As presented in Tab. III, our method outperforms CrossPoint [20], as well as the reproduced TAP [14] and PointVST [11], all using DGCNN [38] as the encoder. When utilizing Point-MAE or Point-M2AE as the backbone, the classification results exhibit a significant improvement over the reproduced Point-MAE and Point-M2AE. These results underscore the discriminative power of the representations extracted by the encoder, even in complex real-world scenes.

Few-shot Learning. Following previous works [45, 7, 10, 3], we conduct few-shot learning experiments using the pre-trained model on ModelNet40 [36]. We adopt $n$-way, $m$-shot setting, where $n$ denotes the number of classes randomly selected from the dataset, and $m$ represents the number of objects randomly sampled for each class. This yields ${n}\times{m}$ objects for training. For evaluation, we randomly select $20$ unseen objects from each of $n$ classes.

The results with settings of $n\in\{5,10\}$ and $m\in\{10,20\}$ are presented in Tab. IV. As shown, our method consistently outperforms the baselines in nearly all few-shot settings, with minimal deviation. This highlights the robustness and generalization of the representations extracted by the PointCG encoder, even in data-limited scenarios.

The task of part segmentation aims to predict more fine-grained class labels for every instance. We conduct part segmentation on ShapeNetPart [47], which comprises $16,881$ samples shared by $16$ categories, annotated with $50$ parts in total. As illustrated in Tab. V, our method achieves competitive results and outperforms others in eleven categories.

Visualization of part segmentation. Fine-grained part segmentation holds immense practical value. To highlight the clear advantage of our method in this task, we visualize the results and compare them with Point-MAE [3] in Fig. 8.

As depicted in the third line, our method accurately segments the fuselage and tail fin of the airplane, along with the earphone and headband. This reveals the capability of our method to capture discriminative features of points belonging to distinct sections within the same instance.

Large-scale indoor datasets introduce more complexities as they cover larger scenes in real-world environments with noise and outliers. We evaluate the performance of our pre-trained model on the 3D semantic segmentation task using the Stanford large-scale 3D Indoor Spaces (S3DIS) [48] dataset. S3DIS includes data from $6$ indoor areas, comprising a total of $272$ rooms. We fine-tune the pre-trained model with Area 1-5 and evaluate it with Area 6. The results for each category are shown in Tab. VI. Our method outperforms Point-MAE [3] across all categories except ‘beam’ and ‘board’. The results underscore our model’s capability to extract contextual and semantic information, which is crucial for producing fine-grained segmentation outcomes.

In reference to the semantic segmentation experiments of STRL [9], we fine-tune our pre-trained model on one area in Area 1-5, followed by evaluation on Area 6. We extend the experiments of Point-MAE [3] based on the pre-trained model released in the official code and present the mean $IoU$ across all class categories $mIoU(\%)$ and the classification accuracy Acc (%) in Tab. VII. Our model exhibits a significant improvement in accuracy and $mIoU$ compared to STRL [9] and Point-MAE [3]. These results demonstrate the capability of our model to extract contextual and semantic information, leading to fine-grained segmentation results.

To validate the effectiveness of our method in scene-level prediction tasks, we conduct object detection experiments on ScanNetV2 [49], a 3D indoor scene dataset with rich annotations, including 1,513 scenes across 18 object classes. The dataset includes semantic labels, per-point instances, and both 2D and 3D bounding boxes. Following TAP [14], we adopt 3DETR [50] as a baseline, construct the PointCG network, and pre-train on the object-level dataset ShapeNet55 [33].

As shown in Tab. VIII, our method demonstrates superior performance compared to both the baseline 3DETR [50] and TAP [14] in both $AP_{0.25}$ and $AP_{0.5}$ metrics. This improvement indicates that the encoder trained by PointCG effectively captures discriminative information and generalizes well to complex scenes even when pre-trained with object-level datasets.

To investigate the architectural designs of our method, we conduct comprehensive ablation studies with Point-MAE as the backbone model and elucidate the individual contribution of each module.

Effectiveness of the components. As shown in Tab. IX, we validate the effectiveness of each module by enhancing and replacing modules on the baseline.We adopt Point-MAE [3] for comparison and utilize hidden point completion (HPC) as the baseline (a). In (b), we add the feature alignment module with pre-trained Vit-B/16 [30]. Based on (b), we incorporate the arbitrary-view image generation (AIG) module, constituting our PointCG, denoted as (c). As shown in Tab. IX, while the inclusion of the feature alignment module based on HPC does not substantially improve classification accuracy, the exclusion of this module from PointCG yields a diminished Linear-SVM accuracy of $88.41\%$.

We further replace Vit-B/16 with Vit-B/32 (d) and ResNet50 [51] (e). While Vit-B/32 outperforms Vit-B/16 in the image domain, it does not improve the accuracy of classification. The model with ResNet50 [51] exhibits comparatively poorer performance.

To assess the impact of color in rendered images, we extract grayscale images from the rendered ones and pre-train with them in (f). This operation leads to a decrease in shape classification. Grayscale images may potentially lose structural or finer details inherent in the original images. In experiment (g), we extract depth maps from point clouds following PointCLIP [52] and pre-train with them instead of rendered images. The classification result shows poor performance.

Training and inference time. Tab. X presents pre-training and inference times for the classification task of each module. The results demonstrate that the pre-training time is notably longer than that of the baseline, whereas the inference time for classification remains comparable.

The pre-training of HPC requires approximately $397s$ per epoch. Compared to the baseline (Point-MAE), the primary time consumption arises from the data organization module, which is responsible for obtaining visible points from arbitrary views as inputs. Among the components, the 3D completion module has the shortest pre-training duration, while the feature alignment and AIG modules demand considerably more time.

Visualization of 3D Completion. To validate the positive impact of AIG on the 3D completion task, we exclude AIG from PointCG (W/O AIG) and pre-train the network. The 3D completion results of this variant and PointCG are shown in Fig. 9. As depicted, the edges of the aircraft wings are sharpened with our method. Without AIG, the completion edges exhibit point groups, attributed to solely relying on 3D completion, while the completion targets are point clusters.

Visualization of the generated images. To validate the provision of geometric structural information by 3D completion, we showcase the results of image generation after excluding the completion module and compare them with PointCG’s results in Fig. 10. As depicted in line (c), outcomes exhibit significant artifacts and lack local structural information. Specifically, in the case of the aircraft tail wing, only the wing’s shape is generated, omitting volumetric structural details. However, our method in line (d) successfully predicts the accurate shapes of the objects.

Camera perspectives. To identify more suitable inputs and predicted images, we design experiments with different camera positions as inputs, and the results are reported in Tab. XI. (a) takes the left-view image as input and the front-view as target, and (b) takes the left-view image as input and an arbitrary-view image as target. In (c), we use images from three different views (front, left, and top views) as inputs and an arbitrary-view image as target. Both the input and target of (d) are from arbitrary views, yielding the optimal results. Therefore, we utilize two arbitrary-view images as the input and target, respectively.

Pre-training with more complete inputs. We posit that if the inputs of 3D completion contain more structural information and more overlap areas with the targets, the completion task will be accomplished more easily. This leads to a reduced training intensity for the backbone, thereby diminishing the backbone’s perception of 3D objects. We design this study based on different inputs. The inputs in (a) are derived from a single arbitrary view. The inputs for (b) and (c) involve the addition of two and eight extra patches, respectively, to the input of (a). The points from two arbitrary views serve as the inputs for (d). Results are reported in Tab. XII.

In cases (b) and (c), the accuracy of the Linear-SVM during pre-training decreases as more patches are included in the inputs. The classification results via fine-tuning also show a decline. In case (d), inputs from two views offer more structural information about the input objects, which leads to a significant decrease in shape classification. This experiment reveals that as additional structural information is progressively included in the input, shape classification accuracy steadily decreases. This indicates that excessive exposure to object structure within the inputs hinders the model’s learning ability.

Image generation losses. AIG significantly impacts the backbone’s perception of 3D objects by precise supervision between the ground truth and the generated images. It always affects the quality of the generated images (as shown in Fig. 11). We conduct experiments to examine the performance of various loss functions for supervision, as outlined in Tab. XIII.

The $\mathcal{L}_{2}$ loss penalizes large errors more heavily and is more tolerant of small errors. In contrast, the $\mathcal{L}_{1}$ loss does not excessively penalize large errors. The classification results of the model with the $\mathcal{L}_{1}$ loss yield superior results compared to the $\mathcal{L}_{2}$ loss.

The multi-scale structural similarity (MS_SSIM) index preserves the contrast in high-frequency regions. While $\mathcal{L}_{1}$ is effective in preserving colors and luminance [32], but does not produce quite the same contrast as MS_SSIM. To leverage the benefits of both loss functions, we utilize the combination of them: $\mathcal{L}_{G}=\alpha*{\mathcal{L}_{1}}+\beta*\mathcal{L}_{MS\_SSIM}$. However, the classification quality is slightly lower than others. As shown in this table, the model with $\mathcal{L}_{G}=1.0*\mathcal{L}_{1}+0.2*\mathcal{L}_{MSFR}$ achieves the best classification results

Visualization of the generated images. To assess a more suitable image generation loss for our model, we visualize the generated images with different losses in Fig. 11. Despite minor artifacts near the edges or corners, our model consistently produces complete and structurally clear images in line (d). This indicates that our model possesses the capability to capture geometric structures and stereoscopic knowledge about 3D objects, as well as the ability to infer the occluded points from arbitrary views.

Qualitative results of 2D image generation. While Figs. 10 and 11 provide visual examples of the generated images, they lack rigorous qualitative results to substantiate any significant improvements over baseline methods in terms of preserving 3D structure in 2D images. To address this, we present Tab. XIV, which provides quantitative evaluations of the generated images under various architectures and image generation losses.

We employ Mean Squared Error (MSE), Structural Similarity index (SSIM), Peak Signal to Noise Ratio (PSNR), and Normalized Mutual Information (NMI) as our primary image evaluation metrics. Clearly, the configuration of PointCG with $1.0*\mathcal{L}_{1}+0.2*\mathcal{L}_{MSFR}$ (d) achieves the best performance across all metrics. By contrast, PointCG without 3D Completion (a) exhibits the poorest results. This indicates that the 3D completion module significantly enhances the quality of the generated images.