Let Images Give You More:
Point Cloud Cross-Modal Training for Shape Analysis

Let $\mathcal{P}\in\mathbb{R}^{N\times 3}$ and $y\in\mathbb{R}^{1}$ be the point cloud and ground-truth label of the 3D object. Its corresponding view-image counterparts can be denoted as $\mathcal{I}\in\mathbb{R}^{V\times H\times W\times 3}$, where $N$, $V$ and $(H,W)$ are the number of points, number of view-images and image size, respectively. View images can be gained by rendering the 3D CAD model [38] or perspective projecting the raw point cloud [12]. We denote $T$ and $S$ as the image and point cloud analysis networks, respectively, and regard them as the teacher and student in traditional knowledge distillation (KD). For these networks, we split each of them into two parts: (i) Encoders (i.e., feature extractors $\mathrm{Enc}^{img}(\cdot)$ and $\mathrm{Enc}^{pts}(\cdot)$), the output of which at the last layer are global feature representations $\mathcal{F}^{img}\in\mathbb{R}^{D}$ and $\mathcal{F}^{pts}\in\mathbb{R}^{D}$. (ii) Classifiers, which project the feature representation $\mathcal{F}^{img}$ and $\mathcal{F}^{pts}$ into class logits through $\mathrm{Cls}^{img}(\mathcal{F}^{img})$ and $\mathrm{Cls}^{pts}(\mathcal{F}^{pts})$, where $\mathrm{Cls}(\cdot)$ denotes the classifier.

Since we formulate the cross-modal learning as a KD problem, its goal in the learning process is to distill the prior knowledge from the image into point cloud features, obtaining an image-enhanced ideal feature $\mathcal{F}^{KD}$. During the knowledge distillation, we parameterize the teacher and student networks with $\theta_{T}$ and $\theta_{S}$, and denote the knowledge of the teacher network as $\mathcal{K}_{T}$. From the Bayesian perspective, a neural network can be viewed as a probability model, e.g., $P(y|\mathcal{P},\theta_{S})$ for point cloud analysis model as an example: given an input point cloud $\mathcal{P}$, the network assigns an output probability with the parameters $\theta_{S}$. Therefore, if we want the student network guided by image knowledge on the input sample, our goal can be further reformulated as maximizing the probability $\mathcal{P}(\mathcal{F}^{KD}|\mathcal{P},y;\theta_{S},\mathcal{K}_{T})$, which can also be used to measure the ability of student network extracting the feature with cross-modal information. To find the lower bound of the above probability, we define the discrepancy $g$ between theoretically discriminative features $\mathcal{F}^{img}_{*}$ and $\mathcal{F}^{pts}_{*}$ as

where $\mathcal{F}^{img}_{*}$ and $\mathcal{F}^{pts}_{*}$ are ideal features in specific modalities. In Lemma 1, we give the lower bound of cross-modal learning as a KD problem. The proof is provided in the supplementary material.

Lemma 1: By the definition above, $\mathcal{P}(\mathcal{F}^{KD}|\mathcal{P},y;\theta_{S},\mathcal{K}_{T})$ is bounded below by $\mathcal{P}(\mathcal{F}^{KD}|\mathcal{I},y;\theta_{T})+\lambda-g$, where $\lambda$ is

In Lemma 1, $\lambda$ measures the compatibility of knowledge distillation of the image networks, and can be viewed as a constant when the architectures are determined. $\mathcal{P}(\mathcal{F}^{KD}|\mathcal{I},y;\theta_{T})$ is also a constant when the parameters $\theta_{T}$ and the architecture of the point cloud analysis model are fixed, e.g., using a pre-trained image network. Therefore, the Lemma ensures that during the knowledge distillation, one can maximize $\mathcal{P}(\mathcal{F}^{KD}|\mathcal{P},y;\theta_{S},\mathcal{K}_{T})$ through minimizing $g$, which gives a theoretical guarantee for the KD problem.

If we adopt previous KD studies in cross-modal scenario, they minimize $g$ through making student directly approximate the teacher’s features [23, 1, 5] or predict logits [17]. However, in a common KD problem, the teacher and student are generally trained on the same dataset with an identical distribution [17]. Moreover, the teacher generally achieves better performance than the student. In contrast, the image and point cloud analysis models tend to learn different feature representations and logits distribution which are generally complementary. Direct alignment may cause a limited gain or even negative transfer. Moreover, previous KD approaches treat encoders and classifiers as a whole architecture since the teacher and student networks generally have the same components. However, the point cloud convolution in encoder is significantly different from the 2D CNN, but they have the same classifier design.

To solve the cross-modal KD problem, we propose PointCMT, which effectively solves the cross-modal learning problem. The workflow of PointCMT is demonstrated in Figure 2 (a) and can be summarized as Algorithm 1. Specifically, there are three stages exist in PointCMT. In Stage I (Section 3.2), we train the image encoder and classifier using view-images and ground-truth labels. In Stage II (Section 3.3), we train the cross-modal point generator (CMPG) through image features, and we independently align features and logits of two modalities in stage III (Section 3.4), where the encoder and classifier of point cloud analysis network are both enhanced.

For each 3D object, we use multiple view-images (i.e., rendered color images or projected images from raw point cloud) as additional data, and the whole process can be described as:

Inspired by view-based learning approaches [38], $V$ view-images from $\mathcal{I}$ flow into a shared-weights image feature extractor $\mathrm{CNN}(\cdot)$ to obtain a series of vectors. By aggregating the view-level vectors via an aggregation function $\mathcal{A}\{\cdot\}$, we obtain a global feature representation $\mathcal{F}^{img}$ from all images, which integrates shape information from multiple views. Finally, an image classifier maps the above global feature to gain a prediction logits through $\mathrm{Cls}^{img}(\mathcal{F}^{img})$, which is supervised by the ground truth label $y$ through cross-entropy loss $\mathcal{L}_{CE}$.

A cross-modal point generator (CMPG) can be seen as a nonlinear transformation $\mathbb{R}^{D}\rightarrow\mathbb{R}^{N\times 3}$ that maps the global feature representation $\mathcal{F}^{img}$ acquired from images into the Euclidean space. Thus, it can avoid potential negative transfer effectively by directly aligning cross-modal features from different distributions. In order to better learn image priors in the point cloud analysis network, we pre-train the CMPG through the image feature $\mathcal{F}^{img}\in\mathbb{R}^{D}$, and fix it in the distillation stage. Figure 2 (b) illustrates the pre-training stage of CMPG. It takes the image feature as input and reconstruct a point cloud $\hat{\mathcal{P}}^{img}\in\mathbb{R}^{N\times 3}$, which is supervised by the original point cloud ${\mathcal{P}}$ through Earth Mover’s distance (EMD) [37]:

where $|\mathcal{P}|=|\mathcal{P}^{img}|$ and $\phi:{\mathcal{P}}\rightarrow\hat{\mathcal{P}}^{img}$ is a bijection, i.e., for each point $p\in{\mathcal{P}}$, $\phi(\cdot)$ finds a sole point correspondence in $\hat{\mathcal{P}}^{img}$. After pre-training, CMPG reconstructs a point cloud through an image-relative feature representation.

During the training stage, three objectiveness should be optimized:

We firstly evaluate our PointCMT on synthetic dataset ModelNet40 [48], which is a large-scale 3D CAD model dataset.

Though ModelNet40 is the widely used benchmark for point cloud analysis, it may not meet the realistic requirement due to its synthetic nature. To this end, we also conduct experiments on the ScanObjectNN benchmark [41], which is a real-world dataset.

We evaluate our approach under limited data scenarios in Table 3. Here, we only sample a small amount of training data in each category on ModelNet40, and evaluate the entire testing data. Our PointCMT shows an even more significant gap when using a small subset of the training data, again compared to PointNet++ which trained from scratch. Especially when facing only 2% and 10% of the training data, we achieve about 1.9% and 2.8% improvements, respectively. This result illustrates that PointCMT provides more vital guidance for point cloud models in the low data regime.

The ablation results on three datasets are summarized in Table 4, in which we use PointNet++ as our baselines. We first test the effectiveness of feature enhancement (FE) and classifier enhancement (CE) in PointCMT. The results demonstrate that only using FE already significantly boosts the performance on both datasets, i.e., increases the overall accuracy by 0.4% and 3.1% on ModelNet40 and ScanObjetNN. Only using classifier enhancement (CE) improves the accuracy by around 0.6% and 2.9%. Finally, it is surprising that when we use both FE and CE during the training phase, it achieves the best result of 94.4% and 83.3%, respectively.

To further verify the effectiveness of our proposed method compared with typical teach-student architecture and other knowledge distillation manners, we compare PointCMT with typical approaches of knowledge transfer in Table 5. Among all the methods, Hinton et al. [17] is a pioneer study for knowledge distillation, while Huang et al. [21] and Yang et al. [55] are recent works. As shown in the table, directly aligning features as [17] between two modalities will cause a negative transfer on ModelNet40. This phenomenon does not appear on ScanObjectNN, since view images projected via point clouds may have a smaller gap than rendered images of CAD models. Nevertheless, other KD techniques only achieve marginal improvement compared with PointCMT.

In this section, we compare results with different view-image generation strategies. As shown in Figure 3, multiple view-image types can be applied in our framework, and we compare the results in Table 6. As illustrated in the table, images rendered from the CAD model improve more compared with only using projection since the former provides additional shade and texture information. In contrast, we find out that using additional colors in the OBJ_ONLY dataset cannot boost performance. The reason is that OBJ_ONLY dataset only contains 2,902 objects, and image networks are easier to overfit when using the color information.

As we mentioned in the manuscript, the discrepancy between two modalities make the KD problem challenging, which is the initial motivation of our PointCMT. In this section, we analyze different strategies that are used to eliminate the discrepancy. Specifically, we design two normalization:

Normalize-I: We assume the features from image and point cloud networks follow two Gaussian distributions. We normalize the mean and standard deviation of the features from image network and make it closer to the features from point cloud network. For every batch of paired image and point cloud data, denote $N$ as batch size, let $\{(\mathcal{F}^{pts}_{i},\mathcal{F}^{img}_{i})\}_{i=1}^{N}$ be feature pairs from point cloud and image networks. Let mean($\mathcal{F}^{pts}$), std($\mathcal{F}^{pts}$) be the mean and standard deviation of $\{\mathcal{F}^{pts}_{i}\}_{i=1}^{N}$, and mean($\mathcal{F}^{img}$) and std($\mathcal{F}^{img}$) be the mean and standard deviation of $\{\mathcal{F}^{img}_{i}\}_{i=1}^{N}$. We normalize image features through:

Normalize-II: We regard features as vectors in a feature space, and normalize their norms into the same scale. Let norm($\mathcal{F}^{pts}$) and norm($\mathcal{F}^{img}$) be the mean of $\{||\mathcal{F}^{pts}_{i}||_{2}\}_{i=1}^{n}$ and $\{||\mathcal{F}^{img}_{i}||_{2}\}_{i=1}^{n}$, where $||\cdot||_{2}$ denote 2-norm, we normalize image features through:

During the experiment, we train PointNet++ with above normalization through KD loss:

where $\text{Norm}(\cdot)$ denotes the normalization operations mentioned above. As shown in Table 7, exploiting normalization strategies can slightly improve the performance on ModelNet40 and eliminate the negative transfer. Nevertheless, our PointCMT still achieves superior performance upon such naive normalization.

In Table 8, we illustrate the results on ModelNet40 without pre-trained image feature extractor. As shown in the table, when we train image feature extractor from scratch, PointCMT still gains 94.2% overall accuracy, i.e., with only 0.2% performance drop. Moreover, the results on ScanObjectNN are obtained without pre-trained image feature extractor, since the view-images used on this dataset are from perspective projection.

In this section, we demonstrate the performance of image networks pre-trained in the Stage I. As shown in Table 9, through exploiting rendered view-image from CAD models, the image network can achieve very high overall accuracy of 97.0% on ModelNet40, boosting the PointNet++ via PointCMT by a 1% performance gain. Only using projection in the image network can only obtain 93.8% on ModelNet40. Nevertheless, it still increases the performance of PointNet++ by 0.6%. Moreover, we find out that in the cross-modal KD scenario, the performance of the teacher will not always better than the student. Still, PointCMT effectively learn the complementary information from the teacher, and improve the performance of point cloud analysis approaches. For the ScanObjectNN dataset, using additional color information makes image networks overfit on the OBJ_ONLY sub-set, and thus hampers the performance.

In this section, we demonstrate the training speed of our PointCMT. As shown in the Table 10, the additional training stage of I (image encoder and image classifier) and II (CMPG) actually introduce little extra cost in the entire training phrase since the small epoch numbers for stage I and few parameters of CMPG. For the speed evaluation of the Stage III, since the image network has been trained, we fixed pre-trained network and generate objects’ features offline, which can be directly exploited in the Stage II and III without repeatedly forwarding the image network.

To illustrate the superiority of our PointCMT, we set two experiments for part segmentation on ShapeNetPart dataset, as shown in Table 11. (a) PointNet++ with pre-trained encoder (trained from scratch on ModelNet40); (b) PointNet++ with pre-trained encoder (trained with PointCMT on ModelNet40). As shown in Table, utilizing pre-trained encoder trained with PointCMT effectively improve the performance, especially for the more challenging metric of Class avg IoU. Here, Instance avg IoU and Class avg IoU denote the IoU averaged by all instances and each class, respectively.