Learning to Predict on Octree for Scalable Point Cloud Geometry Coding

To achieve scalability, we adopt the octree representation. A point cloud is represented in a sequence of $L$ occupancy levels: $\mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{L}$. The octree is coded from the first level occupancy $\mathbf{X}_{1}$ to the last level $\mathbf{X}_{L}$, sequentially and losslessly. The point cloud can be reconstructed to level $l$ if the coded bit-streams for $\mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{l}$ are given. When the coded bit-streams for all occupancy levels are given, we can losslessly reconstruct the original point cloud.

Each non-empty node $x_{l-1,i}$ at level $l-1$, location $i$ has 8 children at level $l$, represented in occupancy $\tilde{v}_{i}^{l}=\{v_{j}\in\{0,1\}:j=1,2,\cdots,8\}$. With context-based entropy coding, we code $\tilde{v}_{i}^{l}$ into a bit-stream based on the conditional probability mass function $p(\tilde{v}_{i}^{l}|{\tilde{C}}_{i}^{l})$, where ${\tilde{C}}_{i}^{l}$ represents the context. In general ${\tilde{C}}_{i}^{l}$ may include nodes in $\mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{l-1}$, as well as the adjacent nodes in $\mathbf{X}_{l}$ that have been coded. With the same context available at the decoder, the decoding process is to map the bit-stream back to $\tilde{v}_{i}^{l}$.

The main challenge in context-based entropy coding is how to form the context ${\tilde{C}}_{i}^{l})$ and how to estimate the probability distribution $p(\tilde{v}_{i}^{l}|{\tilde{C}}_{i}^{l})$. To accurately estimate the probability distribution, it is important to design a context that fully utilizes all the information from previously decoded nodes. When coding an octree node at the current level $\mathbf{X}_{l}$, the context should take into account of both the information at the upper level $\mathbf{X}_{l-1}$ and the current level. However, since not all the information at the current level is known when the current node is being coded, we need to treat the known and the unknown part separately to ensure that the context for probability estimation used during the encoding is available during the decoding process.

In the proposed probability estimation method, we code the nodes $\tilde{v}_{i}^{l}$ following a fixed spatial order. We ensure that when a node at the spatial position $(x_{i},y_{i},z_{i})$ is being coded, nodes at the same level with coordinates $c\in\{(x,y,z):z<z_{i}\}\cup\{(x,y,z):y<y_{i},z=z_{i}\}\cup\{(x,y,z):x<x_{i},y=y_{i},z=z_{i}\}$ have already been coded. When coding $\tilde{v}_{i}^{l}$, we form a context cube ${\tilde{C}}_{i}^{l}$ of size $2k\times 2k\times 2k$ centered at $\tilde{v}_{i}^{l}$, corresponding to a cube of size $k\times k\times k$ at the level $l-1$. The known half of the context cube is filled with the true occupancy, i.e. 0 for empty voxels and 1 for occupied ones. For the other half of ${\tilde{C}}_{i}^{l}$ that has not been coded, we fill them with 0 if their parent nodes indicate that these children nodes are empty, and 0.5 if their parent nodes indicate that the unknown voxel can be potentially non-empty. Since this context involves uncertainty in signal values in the uncoded voxels, we call it “noisy” context. We propose to use a convolution neural network to “denoise” this context, so that the output of the network represents $p(\tilde{v}_{i}^{l}|{\tilde{C}}_{i}^{l})$, the predicted probabilities that the center $2\times 2\times 2$ voxels are non-empty. These predicted probabilities will then be used by an entropy coder.

To reduce the complexity, we train one neural network for probability estimation at all octree levels. To make use of the level information, along with the original noisy context we add another 3D input channel with the same size of $2k\times 2k\times 2k$, and all elements in this channel are set to the level index $l$. The resulting 3D tensor provides the information about the neighboring voxels’ occupancy (which is noisy for the unknown half) and the level in the octree. Inspired by the architecture of DnCNN for image denoising [28], we design the network architecture shown in Fig. 1(a). The network takes the context cube and the octree level channel as input. A sigmoid activation is used at the final layer to generate the probability distribution $\mathbf{p}\in\mathbb{R}^{8}$, with each element $p_{j}\in[0,1],j=1,2,\cdots,8$, where $p_{j}:=Pr\{v_{j}=1\}$.

Compared to the network used in VoxelContext-Net [23], this network does not have fully connected layers at the end in order to preserve more spatial information. We also develop another probability estimation network with fully connected layers, similar to the one used in the VoxelContext-Net [23], shown in Fig. 1(b). However, the context in [23] only uses the occupancy information in the parent level. We will provide performance comparison of these two network architectures in Sec. 4.3.

When the bit-rate is restricted due to the network throughput constraint, the point cloud cannot be transmitted in full precision. With the proposed scalable coding method, the bitstream may only contain information up to level $l$ of the octree. To further enhance the quality of the reconstructed point cloud, we propose to estimate a finer resolution representation of the point cloud at the decoder side. After losslessly decoding the bitstream to the $l$-th level of the octree, we upsample each octree node at the $l$-th level, to a $4\times 4\times 4$ voxel grid, to generate a lossy reconstruction of the original octree up to level $l+2$ without using additional bits.

We upsample a node based on its neighboring context. For a node at location $i$ from the decoded $l$-th level point cloud, a local voxel context $C_{i}^{l}$ centered at this node is formed. The network takes $C_{i}^{l}$ as the input and maps it to $\tilde{v}_{i}^{l+2}$ that is 4 times larger along each dimension than the input. The center $4\times 4\times 4$ voxels of the $C_{i}^{l+2}$ are then binarized and used to form the upsampled point cloud at level $l+2$. This model is repeatedly applied to each node at the $l$-th level without conditional dependency. Thus, the upsampling of all nodes can run simultaneously on a multi-threaded processing unit (e.g., a GPU) to vastly reduce the processing time.

The network architecture is shown in Fig. 2 (a). The network takes an input tensor with size of $k\times k\times k$, and maps it to a tensor with size $4k\times 4k\times 4k$. From the output tensor, the center $4\times 4\times 4$ cube is cropped, and binarized with a threshold $t$ to the final predicted occupancy voxel grid.

Since the density and pattern of points at different levels on the octree are diverse, we train the upsampling network separately for every depth level. Note that the model upsamples the octree by two levels only when the decoded point cloud level $l\leq L-2$, where $L$ is the maximal level of the original point cloud. When $l=L-1$, a similar model with only one upsampling layer is used to estimate the decoded point cloud from the $l$-th level to the full levels. When $l=L$ (lossless coding), no post-processing model is applied.

For comparison, we also develop an alternative architecture with fully connected layers at the end to predict the upsampled points, similar to the one used in the VoxelContext-Net [23], as shown in Fig. 2 (b). The performance comparison is provided in Sec. 4.3.

We train the probability estimation network to directly minimize the expected bits needed to code the occupancy. This is equal to the binary cross entropy (BCE) loss function over the ground truth occupancy and the predicted probability. The loss for each training sample (corresponding to one non-empty parent node) is

$$\mathcal{L}_{e}(\mathbf{x},\mathbf{q})=-\sum_{j=1}^{8}x_{j}\log q_{j}+(1-x_{j})\log(1-q_{j}),$$

(1)

where $x_{j}\in\{0,1\}$ denotes the occupancy ground truth and $q_{j}$ is the estimated probability, $q_{j}=Pr\{v_{j}=1\}$.

We adopt the same loss function for the training of the upsampling network, which upsamples the decoded point cloud from level $l$ to level $l+2$ (when $l<L-1$). The loss function is calculated between the ground truth and the predicted probabilities both at the target upsampled level.

For lossless compression, we calculate the bpp for G-PCC, VoxContext-net and the proposed method, on the test point clouds, the results are shown in Table 1. For the proposed method, we show the results using both Model A, shown in Fig. 1(a), and Model B, shown in Fig. 1(b). The context size is $k=5$. Note that the upsampling model is not used for the lossless compression. As shown, Model A and Model B both outperform the baseline methods. Compared to G-PCC, Model A and Model B reduce the number of bits by 32.1% and 27.0%, respectively, on average. For the VoxContext-net model, we observe that it requires more bits than G-PCC to compress the dense 8iVSLF point cloud. This differs from the performance gain over G-PCC reported in [23]. This could be caused by several reasons: Firstly, we train the model on a subset of the ShapeNet dataset and test on the 8iVSLF dataset for a fair comparison, whereas the original paper [23] reports the testing results on the ShapeNet dataset; Secondly, we used only a subset of the ShapeNet dataset for training whereas the work in [23] used the entire training set of the ShapeNet; Finally, the handcrafted G-PCC explores the previously coded neighbors in the current coding level while VoxContext-net does not, and such information could be especially useful for the dense point cloud data.

The rate-distortion (RD) curves of lossy compression by these methods are shown in Fig. 4. Compared with G-PCC and VoxContextNet, both our Model A and Model B save around 80% of bit rate. Model A uses the fully convolutional network in Fig. 1(a) for probability prediction, and the fully convolutional network in Fig. 2(a) for upsampling. Model B uses the architectures in Fig. 1(b) and Fig. 2(b), for probability prediction and upsampling, respectively. Visualization results of longdress_vox10_1300 for our proposed method and baseline methods are shown in Fig. 3.