Fast Point Cloud Geometry Compression
with Context-based Residual Coding
and INR-based Refinement

Recently, deep learning has significantly influenced the research on point cloud compression. One facet of this research endeavor involves utilizing deep conditional entropy models to losslessly encode the octree-structured point cloud geometry into a compact bitstream [24, 45, 14, 49]. Moreover, parallel to the advancements in deep lossy image compression [1, 2, 35, 8, 18, 63, 19, 28, 64, 32], some researchers have shifted to using a combination of autoencoder architecture and deep entropy models to optimize the rate-distortion trade-off in an end-to-end manner. The techniques can be further categorized into voxel-based [53, 52, 51, 37, 62] and point-based methods [56, 25, 55, 21, 59, 60, 26]. The former needs to voxelize the input before feeding it into stacked 3D convolutional layers, and the voxelization process results in a critical artifact of missing points in the reconstructed point clouds. In contrast, the latter preserves the original density and details through direct processing of raw point clouds.

Point-based methods struggle to efficiently reduce redundancy due to the irregular and unordered nature of raw point clouds. To reduce the bitrate, two straightforward methods are proposed: global maximum pooling [56, 25] and downsampling [59, 55, 60, 21, 26]. Global maximum pooling, although simple to implement, is only suitable for sparse point clouds, as it faces challenges in reconstructing a dense point cloud from a single feature vector. On the other hand, the second strategy is more effective for dense point clouds because a feature vector is only responsible for reconstructing a specific patch of points. Among them, 3QNet [26] utilizes a learned codebook for vector quantization of latent features, while others [59, 60, 21] use scalar quantization.

Point cloud upsampling aims to transform a sparse point cloud into a dense and uniformly distributed set of points, similar to how image super-resolution increases image resolution [10, 48, 23, 47, 17, 33, 34]. Previous methods typically expand the number of points by converting expanded features to coordinates or offsets [61, 58, 30, 43, 31], necessitating a model trained for a specific upsampling rate. Recently, magnification-flexible point cloud upsampling methods have become more popular due to the increased flexibility  [57, 44, 13, 65, 22]. Among them, INR has been leveraged for its ability to naturally support arbitrary-scale upsampling [13, 65, 22]. However, such methods suffer from computational inefficiency. Specifically, Grad-PU [22] must iteratively refine initialized points, requiring both forward and compute-intensive backward propagation at each iteration to determine the refinement step and direction respectively.

The overall architecture of the proposed CRCIR compression system is presented in Fig. 1. It consists of a low complexity base layer for reconstructing main structures and an INR-based refinement layer for further recovering finer details. Given a dense 3D point cloud $\mathcal{P}\in\mathbb{R}^{n\times 3}$ to be compressed, the compression workflow involves the following steps:

1. 1.

   $\mathcal{P}\in\mathbb{R}^{n\times 3}$ is first downsampled to $\mathcal{P}_{s}\in\mathbb{R}^{m\times 3}$ using the farthest point sampling (FPS) method [42] and then compressed by an efficient traditional coordinate codec Google Draco [15]. The decompressed point cloud is denoted as $\hat{\mathcal{P}}_{s}$ and then fed into a non-learning upsampling module to obtain a dense yet coarse point cloud $\hat{\mathcal{P}}_{u}$ as the base layer point cloud.

2. 2.

   The residuals $\mathcal{E}$ between $\mathcal{P}$ and $\hat{\mathcal{P}}_{u}$ are transformed and quantized to latent features $\hat{y}$ and then compressed using a learned NTC method as the refinement layer. Specifically, to improve the coding efficiency, we learn a conditional entropy model based on the downsampled point cloud $\hat{\mathcal{P}}_{s}$ for compressing the residuals $\mathcal{E}$.

3. 3.

   In the decoder, the decompressed residuals $\hat{\mathcal{E}}$ are predicted based on the compressed features $\hat{y}$ and the base layer point cloud $\hat{\mathcal{P}}_{u}$. After that, the base layer point cloud $\hat{\mathcal{P}}_{u}$ is refined by adding the decompressed residuals $\hat{\mathcal{E}}$, resulting in a finer reconstruction $\hat{\mathcal{P}}$ with higher fidelity.

It’s noteworthy that an INR is incorporated into the decoder of the refinement layer. This INR-based refinement layer can take as input the coordinates of arbitrary density and refine the input points by predicting the residuals between the input and the underlying surface represented by a dense point set. That is to say, by adjusting the decoder-end upsampling rate $r^{\prime}$ to an arbitrary value (same with or different from the encoder-end upsampling rate $r$), we can control the density of the reconstructed point cloud to any desired level, not necessarily the same as the original point cloud. This design enables our compression system to function as an arbitrary-scale point cloud reconstruction network.

In point clouds, most points belong to the smooth regions, also known as low-frequency components. These areas make up the main structures of point cloud and generally are easier to reconstruct. On the other hand, sharp regions, characterized by a minority of points and known as high-frequency components, present challenges in achieving accurate reconstruction. Existing methods for compressing point clouds often rely on complex neural networks to handle all points, including those in both smooth and sharp regions. However, this approach leads to inefficient use of computational resources, especially when reconstructing smooth regions. To address this issue, we design a base layer dedicated to reconstructing main structures with very low complexity. Specifically, the downsampling and compression module in the base layer both adopts the efficient off-the-shelf algorithms FPS [42] and Google Draco [15], respectively. To obtain the base layer point cloud $\hat{\mathcal{P}}_{u}$, we design a simple but effective upsampling method to predict a dense point cloud $\hat{\mathcal{P}}_{u}$ from the downsampled and decompressed point cloud $\hat{\mathcal{P}}_{s}$, which is defined as:

Here $\hat{p}_{s}^{(i)}$ represents the $i$-th point in $\hat{\mathcal{P}}_{s}$, the set $\{\hat{p}_{s}^{(ij)}|1\leq j\leq r\}$ comprises the nearest $R$ neighbors of $\hat{p}_{s}^{(i)}$ in $\hat{\mathcal{P}}_{s}$, and $u_{ij}$ denotes the associated interpolation weights. The selection of the interpolation weights are provided in the supplementary material.

Point cloud reconstruction can be regarded as sampling points on the underlying surface. However, existing point-based compression methods [56, 25, 55, 21, 59, 26] often neglect the modeling of the underlying surface and simply learn a mapping from latent features to the coordinates at a predefined density. The absence of surface modeling prevents these methods from flexibly controlling the density of the reconstructed point cloud.

To address this issue, we propose an INR-based decoder for the refinement layer. This INR-based decoder take the coordinates of any points as input and predicts the residuals between these points and the underlying surface. In this context, the set of points whose residuals are $\mathop{0}\limits^{\rightarrow}$ implicitly represent the underlying surface. Sampling points on this learned implicit surface involves three steps: 1. sampling initialized points within the bounding box; 2. predicting residuals for the initialized points; 3. adding the predicted residuals to the initialized points. Specifically, the step 1 is achieved by the base layer and the sampling density is determined by the decoder-end base layer upsampling rate $r^{\prime}$. By setting $r^{\prime}$ to a desired value, it becomes possible to sample a set of points on the underlying surface with the intended target cardinality. Besides, $r^{\prime}$ can exceed the encoder-end base layer upsampling rate $r$. In this case, a denser point cloud can be reconstructed compared to the original one.

Furthermore, to eliminate the need for retraining INR on a per-object basis in decoder-only architecture, the proposed method pairs the INR-based decoder $g_{s}(\cdot)$ with an encoder $g_{a}(\cdot)$. The decoder $g_{s}(\cdot)$ employs a similar architecture to the decoder in [40] and the architecture of the encoder is shown in Fig. 2. Instance-specific information is captured by the encoder $g_{a}(\cdot)$ and encoded into learned latent features, which then condition the shared decoder $g_{s}(\cdot)$. This design allows the proposed CRCIR method to streamline the compression process, significantly reducing coding latency.

The overall refinement layer operates by fusing the residuals $\mathcal{E}$ of the base layer point cloud $\hat{\mathcal{P}}_{u}$ into point-wise latent features with the encoder $g_{a}(\cdot)$ and predicting residuals with the decoder $g_{s}(\cdot)$ based on these features. To permit parallel computation, we partition $\hat{\mathcal{P}}_{u}$ into several clusters and let the encoder $g_{a}(\cdot)$ and the decoder $g_{s}(\cdot)$ work on each cluster independently. A cluster contains points in $\hat{\mathcal{P}}_{u}$ that is upsampled from the same point in decompressed sparse point cloud $\hat{\mathcal{P}}_{s}$ and the cluster centroid is set to this corresponding point in $\hat{\mathcal{P}}_{s}$. The workflow of the refinement layer involves the following two stages:

1. 1.

   In the compression phase, the relative positions between $r$ points within a cluster and the cluster centroid, along with the corresponding residuals of these $r$ points, are concatenated and then fed into the encoder $g_{a}(\cdot)$. For each cluster, the input is first mapped to features with two stacked ResNet blocks [20], then aggregated with an attention-based weighting block [6] and finally re-encoded into a latent feature vector. Subsequently, the latent features $y$ are quantized to $\hat{y}$ and entropy coded.

2. 2.

   In the decompression phase, the base layer reconstructs $\hat{\mathcal{P}}_{u}$ with the upsampling rate that is either aligned with the downsampling rate in the compression phase or flexibly configured with a desired number. Next, the INR-based decoder $g_{s}(\cdot)$ predicts residuals for each point in $\hat{\mathcal{P}}_{u}$ to yield a finer reconstruction $\hat{\mathcal{P}}$. To achieve this, for each query point in $\hat{\mathcal{P}}_{u}$, its associated latent feature is determined by using the row of $\hat{y}$ corresponding to the cluster centroid. Subsequently, the relative position between the query point and its corresponding cluster centroid, along with the associated latent feature, are fed into the decoder $g_{s}(\cdot)$ to estimate the residual. By adding the predicted residuals, points in $\hat{\mathcal{P}}_{u}$ are projected onto the learned implicit surface.

Unlike the common 1-D sequences with natural order such as audio or text, the latent features $\hat{y}$ generated by the encoder $g_{a}(\cdot)$ are arranged in an irregular and unordered layout. The lack of regularity and order poses challenges in forming local contexts, a crucial factor for coding efficiency. To our best knowledge, none of the existing point-based compression methods have effectively exploited local contexts for point cloud coding. The SOTA methods such as DPCC [21] often rely on an unconditional fully factorized entropy model [1], resulting in sub-optimal coding performance.

In contrast to the existing methods, we propose a novel graph-based content-adaptive conditional entropy model that removes statistical redundancies among neighboring features against the irregularity of raw 3D point clouds. Specifically, given the unordered latent features $\hat{y}$ (extracted from residuals) to be compressed, we first use KNN to characterize the neighboring relations of $\hat{y}$ based on the positions provided by the sparse point cloud $\hat{\mathcal{P}}_{s}$. As such, latent features $\hat{y}$ can be interpreted as a graph signal supported on a KNN graph with $m$ nodes where each node corresponds to a point in $\hat{\mathcal{P}}_{s}$. By employing graph structure, the CRCIR method can exploit the statistical correlation between latent features residing in neighboring nodes. This statistical dependency in a locality is captured by introducing a paired hyperencoder $h_{a}(\cdot)$ and hyperdecoder $h_{s}(\cdot)$ to learn a hyperprior [2] on the latent representation $\hat{y}$. Specifically, the probability models of $\hat{y}$ is assumed to be Gaussian distributions whose means and scales are conditioned on local content. The hyperencoder $h_{a}(\cdot)$ learns hyperprior $\hat{z}$ from latent features $\hat{y}$ and $\hat{\mathcal{P}}_{s}$. The hyperdecoder $h_{s}(\cdot)$ predicts the means and scales of $\hat{y}$ based on the learned hyperprior $\hat{z}$.

The detailed architectures of the proposed entropy model are as follows:

1. 1.

   As illustrated in Fig. 3, the hyperencoder $h_{a}(\cdot)$ progressively downsamples the graph-organized latent features $y$ to $z$ using stacked three graph coarsening blocks. At the same time, $\hat{\mathcal{P}}_{s}$ is also downsampled to $\mathcal{P}_{c}$ with the same length of $z$. Each feature of $z$ corresponds to a distinct point in $\mathcal{P}_{c}$, while the features in $\hat{y}$ are aggregated and then quantized to $\hat{z}$ followed by being entropy coded with an unconditional factorized entropy model. Specifically, each row of $\hat{z}$ corresponds to a point in $\mathcal{P}_{c}$ and exhibits strong correlation with a set of neighboring features in $\hat{y}$.

2. 2.

   As depicted in Fig. 4, in the hyperdecoder $h_{s}(\cdot)$, $\hat{z}$ is re-encoded into $z_{1}$ and the positional information of $\hat{z}$ is derived by using three deterministic FPS processes to replicate $\hat{\mathcal{P}}_{c}$ where the downsampling rates match the values used in the compression phase. Next, for each point in $\hat{\mathcal{P}}_{s}$, its $k$ nearest neighbors in $\hat{\mathcal{P}}_{c}$ are characterized and the distances are recorded as $\{d_{1},d_{2},\cdots,d_{k}\}$ and $k$ features in $z_{1}$ are gathered. Subsequently, these $k$ features are combined using a weighted sum, where the weights are determined by $w_{j}=\frac{\exp(-\alpha d_{j})}{\sum_{i=1}^{k}\exp(-\alpha d_{i})},~{}j=1,\cdots,k$. Finally, the means and scales are predicted from $z_{3}$ with a ResNet block.

One intriguing feature of the proposed CRCIR method is its ability to function as an arbitrary-scale upsampling network. To evaluate the upsampling performance, we compare it with two baselines: DPCC [21]+Grad-PU [22] and 3QNet [26]+Grad-PU [22], where Grad-PU is the SOTA arbitrary-scale upsampling method. We conduct comparative evaluations on the PU-GAN dataset [30] using point clouds progressively downsampled from $120$k points to $15$k points. Specifically, each downsampling process removes $15$k points from the dense point cloud. The point cloud with $15$k points is compressed and the decompressed point cloud is fed into the upsampling network, which performs upsampling at rates ranging from $\times 2$ to $\times 8$. The remaining point clouds serve as the ground truth for corresponding upsampling rate. Tab. 2 presents the distortion and the sum of decompression and upsampling runtime under different upsampling rates. The CRCIR method notably and consistently surpasses the competing baselines in both distortion and efficiency metrics. Besides, the runtime of our methods remains unchanged despite increasing the upsampling rate, whereas competing baselines suffer from a significant increase in runtime. This discrepancy in upsampling efficiency stems from the technical difference between Grad-PU and ours in the method to refine the initialized point cloud. Grad-PU requires iterative refinement and compute-intensive backward propagation, whereas the CRCIR method circumvent them by directly estimating the residual between the initialized point and the underlying surface. This substantially alleviate the increased workload associated with refining more initialized points.

In Fig. 8, we present the effectiveness and necessity of exploiting content-aware conditional entropy model, using experimental results from the ShapeNet dataset. RD-curves of DPCC [21] and 3QNet [26] are included for reference. Utilizing the proposed conditional entropy model drastically shifts R-D curves towards lower bitrates and enables our method to surpass 3QNet [26] whose complexity is two orders of magnitude higher. This implies that employing a conditional entropy model rather than a simple unconditional fully factorized entropy model is a key factor in achieving outstanding results.

By incorporating INR, we enable our decoder to sample points on the underlying surface at arbitrary densities. This allows our method to function as both a compression and arbitrary-scale upsampling network for 3D point clouds. However, when dealing with highly sparse input point clouds like LiDAR data, our method remains effective for compression but less so for arbitrary-scale upsampling. This is primarily because of the challenges involved in inferring plausible underlying surface from sparse and unevenly distributed LiDAR point clouds. To our knowledge, no existing INR-based upsampling methods exhibits remarkable capacity in effectively upsampling LiDAR point clouds at arbitrary scales.