Masked Spatio-Temporal Structure Prediction for Self-supervised Learning
on Point Cloud Videos

Mask prediction has been proven to be an excellent self-supervised task for visual representation learning [8, 45, 15]. By reconstructing target signals from the masked input, mask prediction enables the network to learn rich representations and boosts self-supervised learning [4, 2, 39, 41]. Chen et al. [4] extended GPT [3] to operate the pixel sequence for prediction. Bao et al. [2] and Wang et al. [39] introduced another successful framework, BERT [19], to predict the identities of masked tokens.

Then, He et al. [15] proposed MAE as a scalable vision learner to predict the pixels of masked patches. Feichtenhofer et al. [13] and Tong et al. [37] extended MAE to video representation learning by masking patch tubes. Wei et al. [41] developed MaskFeat to predict the HOG features of masked spatio-temporal tubes for self-supervised video pre-training. The regular structure of images and videos makes it easy to obtain patches or patch tubes, which facilitates the design of masking strategies. Pang et al. [29] extended MAE to unstructured static point clouds and designed a masking strategy based on the local spatial neighborhoods. However, point cloud videos are not only spatially irregular but also temporally misaligned across frames [9, 11]. To remedy this, we design a point-tube masking strategy.

Contrastive learning has made significant progress on static point clouds [44, 17, 49, 32]. Xie et al. [44] proposed PointContrast to discriminate two geometric views of matched points using contrastive loss. Hou et al. [17] introduced contextual contrastive learning to PointContrast for data-efficient point pre-training. Rao et al. [32] mapped the local and global features to shared representation space and applied a contrastive loss on them. Zhang et al. [49] used the instance discrimination task on two augmented versions of a point cloud, while Huang et al. [18] pre-trained static point clouds using spatio-temporal augmentations. They transformed different views at point, region, object, and scene levels, and then used contrastive learning to judge their semantic consistency. However, there are limited augmentation methods that can guarantee the semantic consistency of point clouds, let alone point cloud videos.

Prediction-based methods on point clouds also attract a lot of attention. Yang et al. [46] proposed a folding-based autoencoder that deforms a 2D grid to reconstruct the target 3D point cloud. Recently, mask prediction has been extended to static point clouds. Liu et al. [22] designed a point discrimination task for masked patches. Yu et al. [47] proposed PointBERT with an offline point tokenizer. Pang et al. [29] used a simpler task, reconstructing masked point coordinates, for point pre-training in an end-to-end manner. However, these methods solely focus on the geometric representation learning of static point clouds. In this paper, on top of learning appearance information, an explicit motion information learning method is designed for point cloud videos.

Supervised learning dominates point cloud video research. [11, 12] and [9, 10, 42, 43] use convolution-based methods and attention-based methods to implicitly learn the long-term features of point cloud videos, respectively. In addition, [40, 50, 24] apply empirical-based dynamics methods modeling point cloud videos. Wang et al. [40] introduced temporal rank pooling [14] to capture the frame-level dynamics. Zhong et al. [50] proposed a two-stream framework and used feature-level ST-surfaces [30] in the dynamics learning branch. Liu et al. [24] used a scene flow estimator [23] or an alternative grouping method to do point tracking for dynamics modeling. Although these methods are effective, they require complex calculations or additional modules.

Self-supervised learning on point cloud videos is understudied [34, 35]. Wang et al. [38] pre-trained the encoder by predicting the temporal order of shuffled segments to learn the dynamics of point cloud videos. Zhang et al. [48] developed complete-to-partial 4D distillation to predict the representations of point cloud frames within a short time window. However, these methods learn motion information using clip-level or frame-level pretext tasks. In this paper, we propose the temporal cardinality difference prediction for fine-grained dynamics learning on point cloud videos.

The masking strategy includes three steps: input division, embedding, and masking operation.

Division. Point tubes are introduced as division units of the input point cloud video. Specifically, given a point cloud video $\boldsymbol{P}$, Farthest Point Sampling is used to select $N$ key points $\boldsymbol{\hat{p}}$ from the input. Next, we construct one point tube for each key point. The point tube centered at $i$-th key point $\hat{p}_{i}$ is denoted as $\boldsymbol{Tube}_{\hat{p}_{i}}\!=\!\{p\ |\ p\!\in\!\boldsymbol{P},\ \mathcal{D}_{s}(p,{\hat{p}_{i}})\!<\!r,\ \mathcal{D}_{t}(p,{\hat{p}_{i}})\!<\!\frac{l}{2}\}$, where $p$ is one of the input points, $\mathcal{D}_{s}$ is the Euclidean distance, $\mathcal{D}_{t}$ is the difference in frame timestamps of two points, $r$ is the radius of a spatial neighborhood and $l$ is the number of frames in a point tube. Then, we use random sampling to select $n$ points in each spatial neighborhood.

In this way, a point cloud video is divided into $N$ point tubes, and each point tube contains $l\times n$ points. To ensure all points are covered by point tubes, $r$ and $l$ are set to maintain a minor overlap between adjacent point tubes.

Embedding. Following our baseline [9], each point tube is encoded into an embedding:

where $\boldsymbol{Tube}_{\hat{p}_{i}}^{t}$ is the $t$-th frame of the point tube centered at $\hat{p}_{i}$ and $f(\cdot)$ is an MLP-based feature extractor. More details can be found in [9].

Masking. The design of the masking operation is related to the information redundancy of input [15, 13, 37]. Because of spatio-temporal coherence, the redundancy of the point cloud video is higher than low-dimensional data. Therefore, our MaST-Pre uses a high masking ratio on point cloud videos and the empirical result is 75%. A high ratio is helpful to alleviate information leakage and make reconstruction a meaningful self-supervised task. In addition, our MaST-Pre uses random masking of point tubes. As a spacetime-agnostic method, random masking is more effective than structure-aware strategies [13].

Our autoencoder is based on vanilla Transformers of point cloud videos [9]. An asymmetric encoder-decoder design is introduced to MaST-Pre.

Encoder. To better capture the dynamics of point cloud videos, joint spatio-temporal attention is adopted [1, 25]. In addition, only the visible point tubes with spatio-temporal positional embeddings are fed into the encoder during pre-training.

Decoder. The decoder is similar to our encoder but a lightweight vanilla Transformer [9]. It takes both encoded point tubes and masked ones as input. By adding a full set of spatio-temporal positional embeddings to all tokens, location clues are provided for self-supervised learning. After decoding, only the embeddings of masked point tubes are fed to the following prediction heads.

It is demonstrated in [31, 36] that effective video representation integrates appearance and motion information. Therefore, two-stream self-supervised tasks are proposed to explicitly predict motions and reconstruct the appearance of masked point cloud videos.

Appearance Stream. The reconstruction objectives are the point coordinates of masked point tubes. The $l_{2}$ Chamfer Distance loss is used between predictions $\boldsymbol{P}_{pre}\in\mathbb{R}^{{l}\times{n}\times{3}}$ and the ground truth $\boldsymbol{P}_{gt}\in\mathbb{R}^{{l}\times{n}\times{3}}$:

Motion Stream. We propose the temporal cardinality difference as the target of the motion prediction stream. As shown in Fig. 3(a), the spherical support domain of the key point is divided into 8 octants. We follow the conventional rule that the area where the $xyz$-coordinates are greater than 0 is the first octant I and then increases counterclockwise.

Then, count the cardinality of each octant into the corresponding bin of a histogram. To alleviate the noise in real-world point clouds, a probabilistic approach is employed. Specifically, calculate the angular distance $d$ of each point to the central line of its current octant, and divide it by $90^{\circ}$ to normalize. For example, the current octant of Point2 (the green one in Fig. 3(a)) is octant III. The angular difference $d2$ between Point2 and the central dashed line of octant III is $30^{\circ}$. Consequently, the probabilities of Point2 belonging to octant IV and III are $\frac{1}{3}$ (i.e., $\frac{30^{\circ}}{90^{\circ}}$) and $\frac{2}{3}$ (i.e., $\frac{60^{\circ}}{90^{\circ}}$), respectively. In particular, when a point falls on an axis (e.g., the $-y$ axis), its probabilities belonging to octant IV and III are both 0.5.

Next, as shown in Fig. 3(b), the cardinality histograms $\boldsymbol{C}\!\in\!\mathbb{R}^{8}$ between adjacent frames of a point tube are subtracted to obtain its temporal cardinality difference $\boldsymbol{C\!D}\!\in\!\mathbb{R}^{8}$, which constitutes the ground truth $\boldsymbol{M}_{gt}\!\in\!\mathbb{R}^{{(l\!-\!1)}\times 8}$ of the motion stream. The decoded embeddings of masked point tubes are passed through a linear layer to obtain the motion prediction $\boldsymbol{M}_{pre}\!\in\!\mathbb{R}^{{(l\!-\!1)}\times 8}$. The smooth $l_{1}$ loss of $i$-th $\boldsymbol{C\!D}$ between prediction and ground truth is denoted as $\mathcal{L}_{m}^{i}$. The loss of our motion stream is denoted as $\mathcal{L}_{motion}$:

Overall, the total loss of our MaST-Pre is defined as:

With both loss terms, our MaST-Pre can simultaneously learn the geometry and dynamics of point cloud videos.

In this paper, to demonstrate the effectiveness of our spatio-temporal representations, we focus on long-term point cloud video tasks, including 4D action recognition and 4D gesture recognition.

MSRAction-3D [20] and NTU-RGBD [33] are utilized for the action recognition task. (1) MSRAction-3D is comprised of 567 videos in 20 daily actions. The average frame number contained in each video is about 40. Following [9, 10], 270 videos are used as the training set and 297 videos are adopted as the test data. (2) NTU-RGBD consists of 56,880 videos with 60 fine-grained action categories. The frame number of each video is about 30 to 300. Under the cross-subject setting [33], 40,320 training videos and 16,560 test videos are used.

SHREC’17 [7] and NvGesture [28] are utilized for the gesture recognition task. (1) SHREC’17 is comprised of 2800 videos in 28 gestures. Following [7], this dataset is split into 1960 training videos and 840 test videos. (2) NvGesture consists of 1532 videos with 25 gesture classes. Following [27], 1050 videos are assigned to the training set and the remaining 482 videos to the test set.

During pre-training, given a point cloud video, 24 frames are densely sampled and 1024 points are selected for each frame. Following [9, 10], the frame sampling stride is set to 2/1 on NTU-RGBD/MSRAction-3D and only random scaling is employed for data augmentation. For division and embedding, the temporal downsampling rate is set to 2 and the temporal kernel size $l$ of each point tube is set to 3. Meanwhile, the spatial downsampling rate is set to 32. The radius of each support domain $r$ is set to 0.1/0.3 on NTU-RGBD/MSRAction-3D and the number of neighbor points $n$ within the spherical query is set to 32. The masking ratio is set to 75% unless otherwise specified.

P4Transformer [9] is utilized as our encoder, which consists of 10/5 layers of vanilla Transformers on NTU-RGBD/MSRAction-3D. The decoder is a 4-layer transformer. To verify the extensibility, PST-Transformer [10] is also used as an encoder on MSRAction-3D. Our MaST-Pre is pre-trained for 200 epochs and linear warmup is utilized for the first 10 epochs. The AdamW optimizer is used with a batch size of 128, and the initial learning rate is set to 0.001 with a cosine decay strategy.

We first evaluate our MaST-Pre by fine-tuning the pre-trained encoder with a new classifier in a supervised manner. End-to-end fine-tuning experiments are conducted on NTU-RGBD and MSRAction-3D, respectively. In each experiment, the same dataset is used for pre-training and fine-tuning.

MSRAction-3D. During fine-tuning, 24 frames are densely sampled and 2048 points are selected in each frame. Following [11], the spatial search radius is set to 0.7. The model is trained for 50 epochs with a batch size of 12 on 4 GPUs. We use the AdamW optimizer and the initial learning rate is set to 0.001 with a cosine decay strategy. As shown in Table 1, compared with baselines trained in a fully supervised manner, our MaST-Pre introduces accuracy improvements for both P4Transformer and PST-Transformer. According to prior experience, the masked autoencoder needs to be fed a considerable amount of data in the pre-training stage to learn useful knowledge [15, 13]. However, the MSRAction-3D dataset is too small to bring significant improvement.

NTU-RGBD. The setup of fine-tuning is the same as pre-training, except that the pre-trained model is fine-tuned for 20 epochs with a batch size of 48 on 8 GPUs and the initial learning rate is set to 0.0005. From the end-to-end fine-tuning in Table 2, we can see that our pre-training method introduces an accuracy improvement compared with the baseline. By predicting the spatio-temporal structure, our MaST-Pre learns appearance and motion information during pre-training.

We also evaluate the learned representations using a semi-supervised learning experiment. Specifically, the cross-subject training set of NTU-RGBD is used for pre-training, and then only a 50% training set is used for fine-tuning in a supervised manner. The setup of our semi-supervised learning experiment is the same as end-to-end fine-tuning on NTU-RGBD (Sec. 4.3).

From Table 2, we can see that the 50% semi-supervised result produced by our MaST-Pre achieves comparable performance to the fully supervised baseline even with only limited annotated data. This clearly demonstrates that MaST-Pre learns high-quality representations.

To evaluate the generalization ability of the representations learned by MaST-Pre, we conduct experiments by transferring the pre-trained encoder to other datasets. Specifically, the encoder is first pre-trained on NTU-RGBD following the setup in Sec. 4.2, and then fine-tuned with a new classifier on NvGesture and SHREC’17, respectively. Our transfer experiments are not only cross-dataset but also cross-task, i.e., from action recognition to gesture recognition. We compare our fine-tuned results to the fully supervised baseline in Table 3.

During fine-tuning, an AdamW optimizer with a batch size of 24 is used, and the initial learning rate is set to 0.002 with a cosine decay strategy. The pre-trained model is fine-tuned for 50 epochs on NvGesture and SHREC’17. As shown in Table 3, our MaST-Pre pre-training facilitates the P4Transformer to produce superior accuracy compared to the fully supervised baseline. Moreover, our MaST-Pre also performs faster convergence. Compared with the baseline without pre-training, significant improvements are achieved after fine-tuning for only 30 epochs (e.g., 84.8% $\rightarrow$ 87.6% on NvGesture and 87.5% $\rightarrow$ 90.2% on SHREC’17). This demonstrates that our MaST-Pre has excellent generalization ability across different tasks, facilitating the accuracy improvement of downstream tasks.

In order to balance authority and efficiency, the experiments of ablation studies are conducted on 10% NTU-RGBD, which contains 4032 training videos and 1656 test videos with category balance.

Architecture Design. Our MaST-Pre utilizes a two-stream prediction to jointly learn both appearance and motion information. To demonstrate the effectiveness of this architecture, we first present the performance of model A0 as the 10% NTU-RGBD baseline in a fully supervised manner (65.85%). Then, model A1 is developed by removing the motion prediction stream. Quantitative results are presented in Table 4. It shows that with solely the appearance stream, the performance gain introduced by model A1 pre-training is limited. When these two streams are combined, comprehensive information can be learned and superior accuracy is achieved by our model A2 (65.85% $\rightarrow$ 78.25%). This demonstrates the effectiveness of our mask-based pre-training on point cloud videos and the necessity of the task of explicitly predicting motions.

Masking Ratio. The masking operation plays a critical role in our MaST-Pre. Therefore, we conduct experiments to study different masking ratios, and the results are presented in Table 5. It shows that a high masking ratio is beneficial to our MaST-Pre and the highest accuracy is achieved at 75% masking ratio (model B2).

Appearance Stream. Right reconstruction targets in the appearance stream contribute to the performance of our MaST-Pre. For model D3, in each point tube, the reconstruction loss (Eq. 2) is first calculated in each frame separately and then aggregated over $l$ frames, which is a decoupled manner. In contrast, model D2 calculates the reconstruction loss in a coupled manner by considering all points together. We also develop model D1 to reconstruct only the middle frame of each point tube.

As shown in Table 6, model D2 brings no accuracy gain and is even worse than the baseline model A0 (58.60% vs. 65.85%). In addition, D3 outperforms D1 and D2. This is because D3 implicitly learns spatio-temporal information during the process of reconstructing decoupled point tubes. We further visualize the reconstruction results in Fig. 4.

Temporal Cardinality Difference. In order to investigate the temporal cardinality difference, we present the results under different sections, interpolations, and strides in Table 7. We develop the model E1/E3 to divide the support domain into 4/16 sections, while their accuracy is lower than E2 with 8 sections. This is because small space resolution will introduce noises and large resolution makes temporal cardinality difference insensitive to motions.

Next, we develop model F1 by removing interpolation. While F2 outperforms F1 because interpolation improves its robustness. Finally, we develop model G2 to calculate the cardinality difference with temporal stride 2, but its performance is worse than G1 with stride 1. This is because a large temporal stride cannot capture fine-grained motions.

We further visualize multiple samples of temporal cardinality difference in Fig. 5 to demonstrate its effectiveness in modeling motions. We present three typical actions, each consisting of two samples. As shown in Fig. 5(a), the temporal cardinality differences of the two raising hands actions project extremely similar motion patterns. Points in the first and seventh octants flow out heavily over time. Meanwhile, temporal cardinality differences within putting hands down (Fig. 5(b)) also display similarities, as well as in kicking forward (Fig. 5(c)). In particular, the temporal cardinality differences between raising hands and putting hands down are approximately reversible, which reflects its effectiveness in modeling dynamics.

Computational Complexity. Table 8 shows the pre-training complexity and corresponding fine-tuning accuracy of the two models. After adding the motion prediction stream, model H2 achieves much higher accuracy than H1 with only a minor increase in pre-training complexities (70.13% $\rightarrow$ 78.25%).