PR-GCN: A Deep Graph Convolutional Network with Point Refinement
for 6D Pose Estimation

RGB-D based 6D pose estimation recovers 6D poses of objects in RGB-D images, where 6D pose is usually represented by a rotation matrix $\bm{R}\in SO(3)$ and a translation vector $\bm{t}\in\mathbb{R}^{3}$. For this issue, we propose the PR-GCN approach. As Fig. 2 depicts, it consists of four steps: object localization and 3D points generation, 3D points refinement, GCN-based multi-modal fusion, and 6D pose prediction.

Object Localization and 3D Points Generation. Given an RGB-D image $I=(I_{rgb},I_{d})$, we firstly locate objects on $I_{rgb}$ using the off-the-shelf Faster R-CNN [33] detector, where $I_{rgb}$ and $I_{d}$ denote the RGB and depth channels of $I$. According to the detected bounding boxes, we crop the sub-images $\{I_{o}\}=\{(I_{o,rgb},I_{o,d})\}$, each of which contains an instance $o$. $I_{o,rgb}$ and $I_{o,d}$ are the RGB and depth channels of $I_{o}$. As in PoseCNN [40], we add a segmentation head to remove background of $I_{o,rgb}$. With $M_{o}$ (foreground mask) and $I_{o,d}$, raw 3D points $\bm{P}_{o}=[\bm{p}^{(1)}_{o};\cdots;\bm{p}^{(i)}_{o};\cdots;\bm{p}^{(N)}_{o}]\in\mathbb{R}^{N\times 3}$ are rendered by Point-Cloud Transform (PCT) [8], where $N$ is the number of points and $\bm{p}^{(i)}_{o}\in\mathbb{R}^{3}$ is the 3D coordinate of the $i$-th point. It is worth noting that $\bm{P}_{o}$ could be severely incomplete and noisy due to external occlusions and sensor noise (see Fig. 1).

3D Points Refinement. To polish the quality of the generated raw 3D points $\bm{P}_{o}$, we propose the PRN module. As in Fig. 2, it is composed of an MLP-based encoder and a multi-resolution decoder to recover the complete and accurate 3D point cloud $\hat{\bm{P}}_{o}=[\hat{\bm{p}}^{(1)}_{o};\cdots;\hat{\bm{p}}^{(m)}_{o};\cdots;\hat{\bm{p}}^{(M)}_{o}]\in\mathbb{R}^{M\times 3}$, where $M$ is the number of refined points. A regularized multi-resolution regression loss is formulated in training, enhancing its ability of filtering out noise in $\bm{P}_{o}$.

GCN-based Multi-Modal Fusion. For more sufficient RGB-D fusion, we propose the MMF-GCN module. As in Fig. 2, it extracts texture and geometry features from $I_{o,rgb}$ and $\bm{P}_{o}$, respectively, and a graph is built based on geometry distribution. Accordingly, $I_{o,rgb}$ and $\bm{P}_{o}$ are initially integrated by applying a GCN $GCN_{f}(\cdot)$ on the previously built graph through local information propagation. The geometry clues from the refined 3D points are encoded by introducing an extra GCN $GCN_{ref}(\cdot)$ and then incorporated into the initially fused features, which are fed into several stacked fully-connected layers $T(\cdot)$ for further fusion. The resulting feature $\bm{G}_{o}=[\bm{g}_{o}^{(k)}]_{k=1,\cdots,K}\in\mathbb{R}^{K\times d}$ is therefore the multi-modal representation for successive 6D pose estimation, where $K$ and $d$ refer to the number of points and the feature dimension, respectively. Since MMF-GCN captures local geometry-aware inter-modality correlation and leverages refined 3D point clouds, it is expected to deliver more discriminative and robust features.

6D Pose Prediction. $[\bm{g}_{o}^{(k)}]_{k=1,\cdots,K}$ is finally fed into three regression branches: $REG_{r}(\cdot)$, $REG_{t}(\cdot)$, $REG_{c}(\cdot)$, for rotations $\{\hat{\bm{R}}_{o}^{(k)}=REG_{r}(\bm{g}_{o}^{(k)})\}$, translations $\{\hat{\bm{t}}_{o}^{(k)}=REG_{t}(\bm{g}_{o}^{(k)})\}$, confidence scores $\{s_{o}^{(k)}=REG_{c}(\bm{g}_{o}^{(k)})\}$, respectively. Each branch has four fully-connected layers. Similar to [9, 29, 38], we select the candidate with the highest confidence score as the estimated pose, formulated as:

$$(\hat{\bm{R}}_{o},\hat{\bm{t}}_{o})=\mathop{\textrm{argmax}}\limits_{\left\{(\hat{\bm{R}}_{o}^{(k)},\hat{\bm{t}}_{o}^{(k)})|k=1,\cdots,K\right\}}~{}~{}s_{o}^{(k)}~{}.$$

(1)

Recall that PRN aims to generate the refined 3D point cloud $\hat{\bm{P}}_{o}$ from the raw one of a low quality $\bm{P}_{o}$. As in Fig. 3, PRN is endowed with an encoder-decoder architecture. To deal with the change in point density (resolution), we downsample $\bm{P}_{o}$ with $1/2$ and $1/4$ scales, resulting in two extra point clouds: $\bm{P}_{o,1/2}\in\mathbb{R}^{\frac{N}{2}\times 3}$ and $\bm{P}_{o,1/4}\in\mathbb{R}^{\frac{N}{4}\times 3}$.

Accordingly, the encoder $E(\cdot)$ has three branches $E_{1}(\cdot)$, $E_{1/2}(\cdot)$, $E_{1/4}(\cdot)$, whose inputs are $\bm{P}_{o}$, $\bm{P}_{o,1/2}$, $\bm{P}_{o,1/4}$ and outputs are the three representations $\bm{v}_{1}=E_{1}(\bm{P}_{o})$, $\bm{v}_{1/2}=E_{1/2}(\bm{P}_{o,1/2})$, $\bm{v}_{1/4}=E_{1/4}(\bm{P}_{o,1/4})$. Each branch is a stack of six MLP layers. The concatenation of $\bm{v}_{1}$, $\bm{v}_{1/2}$ and $\bm{v}_{1/4}$ followed by an MLP layer forms the intermediate latent representation $\bm{v}$.

The decoder $DEC(\cdot)$ employs a multi-resolution structure as [43] does. The first branch with four fully-connected (FC) layers as well as one reshape (RS) operation is built to obtain the coarse point cloud of a low-resolution $\hat{\bm{P}}_{o,1/8}=RS_{5}(FC_{4}(FC_{3}(FC_{2}(FC_{1}(\bm{v})))))\in\mathbb{R}^{\frac{M}{8}\times 3}$ (or $\mathbb{R}^{\frac{M}{8}\times 3\times 1}$ equivalently). The second branch consists of the first two shared FC layers, one additional FC layer, one MLP layer and one RS operation, generating a mediate-resolution point cloud $\hat{\bm{P}}_{o,1/4}=RS_{2,3}(MLP_{2,2}(FC_{2,1}(FC_{2}(FC_{1}(\bm{v}))))).$ Then, $\hat{\bm{P}}_{o,1/8}$ is integrated into $\hat{\bm{P}}_{o,1/4}$ by a broadcasting additive operation $\hat{\bm{P}}_{o,1/4}:=\hat{\bm{P}}_{o,1/4}\bigoplus\hat{\bm{P}}_{o,1/8}$, and $\hat{\bm{P}}_{o,1/4}$ is reshaped into $\hat{\bm{P}}_{o,1/4}:=RS_{2,4}(\hat{\bm{P}}_{o,1/4})\in\mathbb{R}^{\frac{M}{4}\times 3}$. Similarly, the third branch finally renders a high-resolution point cloud $\hat{\bm{P}}_{o}\in\mathbb{R}^{M\times 3}$ by sharing the first FC layer, adding one FC layer, three MLP layers and one RS operation, and incorporating the mediate-resolution information as $\hat{\bm{P}}_{o}=RS_{1,7}(RS_{1,6}(MLP_{1,5}(MLP_{1,4}(MLP_{1,3}(FC_{1,2}(FC_{1}(\bm{v})\\ )))))\bigoplus\hat{\bm{P}}_{o,1/4}).$

For training PRN, we develop a multi-resolution regression loss formulated as follows:

$$\begin{array}[]{rl}\mathcal{L}_{\rm{mr}}=d_{\rm C}(\hat{\bm{P}}_{o},\bm{P}_{o,GT})+\sum\limits_{r\in\{\frac{1}{8},\frac{1}{4}\}}d_{\rm C}(\hat{\bm{P}}_{o,r},\bm{P}_{o,GT})\end{array}$$

(2)

where $\bm{P}_{o,GT}=[\bm{p}_{o,GT}^{(1)};\cdots;\bm{p}_{o,GT}^{(H)}]\in\mathbb{R}^{H\times 3}$ is ground-truth point cloud of object $o$. $d_{\rm C}(\cdot,\cdot)$ is the Chamfer distance defined as $d_{\rm C}(\bm{P},\bm{Q})=\frac{1}{M}\sum\limits_{i}\underset{j}{\rm min}\|\bm{p}^{(i)}-\bm{q}^{(j)}\|_{2}^{2}+\frac{1}{N}\sum\limits_{j}\underset{i}{\rm min}\|\bm{q}^{(j)}-\bm{p}^{(i)}\|_{2}^{2}$, given $\bm{P}=[\bm{p}^{(m)}]_{m=1,\cdots,M}\in\mathbb{R}^{M\times 3}$ and $\bm{Q}=[\bm{q}^{(n)}]_{n=1,\cdots,N}\in\mathbb{R}^{N\times 3}$.

In Eq. (2), $\hat{\bm{P}}_{o}$ is forced to fit the ground-truth in low-to-high resolutions when minimizing $\mathcal{L}_{\rm{mr}}$. In other words, $DEC(\cdot)$ is forced to predict high-quality points in multiple resolutions by a unified structure, and thus optimized with more supervision than the single-resolution case. Moreover, as shown in Fig. 3, the high-resolution output $\hat{\bm{P}}_{o}$ integrates multi-resolution information from $\hat{\bm{P}}_{o,1/4}$ and $\hat{\bm{P}}_{o,1/8}$. As a consequence, PRN mitigates the incompleteness and decreases the noise of the raw 3D points.

Despite the aforementioned advantages of $\mathcal{L}_{\rm{mr}}$, it fails to perceive the global point distribution of $\bm{P}_{o,GT}$. We handle this problem by introducing the adversarial loss:

$$\small\mathcal{L}_{\rm adv}=\sum\limits_{h=1}^{H}\log(D(\bm{p}_{o,GT}^{(h)}))+\sum\limits_{m=1}^{M}\log(1-D(\hat{\bm{p}}_{o}^{(m)})),$$

(3)

where $D(\cdot)$ is the discriminator to classify whether a point belongs to $\bm{P}_{o,GT}$ (“real”) or not (“fake”). By minimizing $\mathcal{L}_{\rm{adv}}$, PRN is expected to generate $\hat{\bm{P}}_{o}$ that captures holistic point distribution of $\bm{P}_{o,GT}$, benefiting the quality of $\bm{P}_{o}$.

The regularized multi-resolution regression loss is thus formulated as:

$$\small\mathcal{L}_{\rm prn}=\sum\limits_{o}\left(\lambda\cdot\mathcal{L}_{\rm{adv}}+\beta\cdot\mathcal{L}_{\rm{mr}}\right),$$

(4)

where $\lambda$ and $\beta$ are the trade-off hyper-parameters.

As mentioned before, given the RGB ($I_{o,rgb}$) and point cloud ($\bm{P}_{o}$ and $\hat{\bm{P}}_{o}$) data of object $o$, MMF-GCN integrates multi-modal information into more effective representation ($\bm{G}_{o}$) for accurate 6D pose estimation.

Specifically, MMF-GCN first extracts the geometry feature $\bm{f}_{o,d}^{(i)}$ from $\bm{P}_{o}$ and the texture feature $\bm{f}_{o,rgb}^{(i)}$ from $I_{o,rgb}$ for the $i$-th point $\bm{p}^{(i)}_{o}\in\bm{P}_{o}$. The normalized coordinate of $\bm{p}_{o}^{(i)}$ is directly used as the geometry feature, and by mapping this coordinate to the pixel on $I_{o,rgb}$, PSPNet [46] with the ResNet-18 backbone is adopted to compute pixel-wise representation as the texture feature.

When $\{\bm{f}_{o,rgb}^{(i)}\}$ and $\{\bm{f}_{o,d}^{(i)}\}$ are ready, a $k$-Nearest Neighbor ($k-$NN) graph $\mathcal{G}_{f}=(\mathcal{V}_{f},\mathcal{E}_{f})$ is constructed. $\mathcal{V}_{f}=\{\bm{p}_{o}^{(1)},\cdots,\bm{p}_{o}^{(N)}\}$ and $\mathcal{E}_{f}=\{(\bm{p}_{o}^{(i)},\bm{p}_{o}^{(j)})|\bm{p}_{o}^{(j)}\in\mathcal{N}_{k}(\bm{p}_{o}^{(i)})\}$ denote the vertices and the edges, and $\mathcal{N}_{k}(\bm{p}_{o}^{(i)})$ indicates the $k$ nearest neighbors of $\bm{p}_{o}^{(i)}$. The edge feature is defined as $\bm{e}^{(i,j)}=h_{\bm{\theta}}(\bm{f}_{o}^{(i)}-\bm{f}_{o}^{(j)},\bm{f}_{o}^{(i)})$ with $\bm{f}_{o}^{(i)}=[\bm{f}_{o,rgb}^{(i)},\bm{f}_{o,d}^{(i)}]$, where $h_{\bm{\theta}}(\cdot,\cdot)$ is a nonlinear function parameterized by $\bm{\theta}$.

Afterwards, a graph convolution network $GCN_{f}(\cdot)$ is employed to capture local inter-modality correlations, with EdgeConv [39] for graph convolutions. The basic updating scheme is formulated as:

$$\bm{g}_{o,f}^{(i,l)}=MP\left(h_{\bm{\theta}^{(l-1)}}\left(\bm{g}_{o,f}^{(i,l-1)}-\bm{g}_{o,f}^{(j,l-1)},\bm{g}_{o,f}^{(i,l-1)}\right)\right),$$

where $\bm{g}_{o,f}^{(i,l)}$ denotes the $i$-th edge feature in the $l$-th layer, $h_{\bm{\theta}^{(l-1)}}(\cdot,\cdot)$ is a nonlinear function in the $(l-1)$-th layer, and $MP(\cdot)$ refers to max pooling. The representation $\bm{G}_{o,f}=[\bm{g}_{o,f}^{(j)}]_{j=1,\cdots,J}\in\mathbb{R}^{J\times d_{f}}$ is then obtained, where $\bm{G}_{o,f}=GCN_{f}(\{\bm{f}_{o,rgb}^{(i)},\bm{f}_{o,d}^{(i)}\})$; $J$ and $d_{f}$ are the point number and the feature dimension, respectively.

As $\bm{P}_{o}$ is usually incomplete and noisy, MMF-GCN encodes the geometry attribute of $\hat{\bm{P}}_{o}$ and incorporates it into $\bm{G}_{o,f}$ as a complement. Concretely, similar to $\mathcal{G}_{f}$, another $k$-NN graph $\mathcal{G}_{ref}$ is built based on $\hat{\bm{P}}_{o}$, and an extra GCN $GCN_{ref}(\cdot)$ is employed. The refined geometry feature is calculated using EdgeConv: $\bm{G}_{o,ref}=[\bm{g}_{o,ref}^{(j)}]_{k=1,\cdots,J}\in\mathbb{R}^{J\times d_{ref}}$, where $\bm{G}_{o,ref}=GCN_{ref}(\hat{\bm{P}}_{o})$ and $d_{ref}$ is the feature dimension. $\bm{G}_{o,ref}$ is subsequently combined with $\bm{G}_{o,f}$ through simple concatenation, which is further integrated by a few stacked FC layers $T(\cdot)$. At last, multi-modal representation is formed as $\bm{G}_{o}=T([\bm{G}_{o,r},\bm{G}_{o,ref}])$ for 6D pose estimation.

The objective function for training PR-GCN consists of two parts: the pose estimation loss $\mathcal{L}_{\rm{pose}}$ and the regularized multi-resolution regression loss $\mathcal{L}_{\rm{prn}}$ as depicted in Eq. (4).

Given the ground-truth 6D pose $(\bm{R}_{o},\bm{t}_{o})$ and the predictions $\{(\hat{\bm{R}}_{o}^{(k)},\hat{\bm{t}}_{o}^{(k)},s^{(k)}_{o})\}$ at $K$ points $\{\bm{x}_{o}^{(k)}\}$, the pose estimation error of the $i$-th prediction $(\hat{\bm{R}}_{o}^{(i)},\hat{\bm{t}}_{o}^{(i)})$ is defined as $e_{o}^{(i)}=\frac{1}{K}\sum_{j=1}^{K}\min_{k}\|(\bm{R}_{o}\bm{x}_{o}^{(j)}+\bm{t}_{o})-(\hat{\bm{R}}_{o}^{(i)}\bm{x}^{(k)}_{o}+\hat{\bm{t}}_{o}^{(i)})\|^{2}_{2}$. Based on $e_{o}^{(i)}$, we adopt an extra regularization term on the prediction scores $\{s^{(i)}_{o}\}$ as in [38] and formulate the pose estimation loss as:

$$\mathcal{L}_{\rm{pose}}=\frac{1}{K}\sum_{o}\sum_{i}e^{(i)}_{o}\cdot\left(s^{(i)}_{o}-\log(s^{(i)}_{o})\right).$$

(5)

By combining Eq. (5) and Eq. (4), the overall training objective function is written as:

$$\mathcal{L}=\mathcal{L}_{\rm{pose}}+\mu\cdot\mathcal{L}_{\rm{prn}},$$

(6)

where $\mu$ is the trade-off hyper-parameter.

Extensive evaluation is made on three datasets: Linemod [11], Occlusion Linemod [2] and YCB-Video [40].

Linemod [11] is composed of 15 RGB-D videos of 15 low-textured objects. Following [32], 13 objects are considered and the standard training/testing split is adopted as in [38, 40]. Occlusion Linemod is collected by annotating a subset of Linemod (8 out of 15 objects), where each image has multiple occluded objects, making it more challenging. YCB-Video [40] includes 21 objects with various textures and sizes. It provides RGB-D images and detailed pose annotations. There are 130K real images from 92 videos and 80K synthetically rendered ones, and 16,189 real and all the synthesized images are used in training, according to [38].

As in the literature, two main metrics are employed for evaluation, i.e., Average Distance (ADD) [40] and ADD-Symmetric (ADD-S) [40], designed for general objects and symmetric objects, respectively. DenseFusion [38] gives the ADD-S smaller than 2 centimeters (ADD-S$<$2cm) for real applications e.g. robotic manipulation, and PoseCNN [40] and DenseFusion [38] report the Area Under the ADD-S Curve (AUC) with the maximum threshold at 0.1m. We also show them for comparison.

We fix the size of RGB images as 480$\times$640. The numbers of raw/refined 3D points, i.e., $N$/$M$, are set to 100/512 and 100/1,024 on Linemod (Occlusion Linemod) and YCB-Video, respectively. In MMF-GCN, refined point clouds are down-sampled to 100 points by FPS. When building the graphs $\mathcal{G}_{f}$ and $\mathcal{G}_{ref}$, we utilize 30 nearest neighbors, i.e., $k=30$. In PRN, the hyper-parameters $\lambda$, $\beta$ and $\mu$ in the overall training loss $\mathcal{L}$ are set to 0.05, 0.95 and 1.0. To train PR-GCN in a more stable way, PRN and MMF-GCN are progressively optimized. For instance, on YCB-Video, PRN and MMF-GCN are first alternatively trained for 15 epochs and then jointly optimized for 30 epochs.

In PRN training, we adopt the ADAM optimizer with the learning rate of 0.0001 and the batch size of 48. The rest parts of PR-GCN are trained for 40, 20 and 60 epochs on Linemod, Occlusion Linemod and YCB-Video, respectively, where the learning rate is initially set to 0.0001 and decayed by a factor of 0.3 after half of the maximal epochs.

Results on Linemod. We first compare PR-GCN to the state-of-the-art methods on Linemod, including the RGB based models: PoseCNN (+DeepIM) [40, 18], PVNet [29], CDPN [19], DPOD [44] and DPVL [42] and the RGB-D based ones: Point Fusion [41], SSD6D (+ICP) [14], Dense Fusion [38], PVN3D [9] and G2L[4]. Several approaches, denoted by ‘$*$’ or ‘${\dagger}$’ in Table 1, only adopt real or synthetic training data, whist the others use both. In our work, we mainly consider the setting with the two types of training data, and also report the performance with real data only.

Table 1 summarizes the ADD(-S) of different methods on Linemod, and we can see that the RGB-D based deep models (e.g., PVN3D and G2L) remarkably outperform the RGB based ones by a large margin due to additional geometry information given by the depth channel. Regarding the RGB-D counterparts, the proposed PR-GCN achieves better performance, which improves PointFusion and DenseFusion by 25.2% and 4.6%, respectively. Our approach also boosts the performance of keypoint-based methods, including PVN3D and G2L. It is worth noting that the second best method, i.e. PVN3D, uses 70,000 synthetic training data, and needs to train different models for distinct object categories. In contrast, our method trains a universal model for all object categories, and merely utilizes 3,500 synthetic training data, which is much more efficient than PVN3D.

Results on Occlusion Linemod. To evaluate the robustness of PR-GCN to inter-object occlusions, we display detailed results on Occlusion Linemod, in comparison with PoseCNN [40], DeepHeat [26], SS [12], Pix2Pose [28], PVNet [29], HybridPose [34] and PVN3D [9]. As shown in Table 2, our method consistently reaches the top ADD(-S) AUC and achieves the best mean ADD(-S) AUC, highlighting its superiority in the presence of heavy occlusions.

Results on YCB-Video. We then extend our analysis on this database and compare PR-GCN with PoseCNN (+ICP) [40], DenseFusion [38], PVN3D [9], CF [5] and G2L [4]. Table 3 shows the AUC and ADD-S<2cm for various methods. It can be observed that our method achieves the highest performance on both the metrics. For instance, compared to PVN3D and DenseFusion, PR-GCN improves the ADD-S$<$2cm by 0.9% and 1.7%, respectively.

Qualitative results. We additionally provide qualitative results in Fig. 4, comparing to DenseFusion and PVN3D. Due to cluttered backgrounds and severe occlusions, DenseFusion and PVN3D predict inaccurate poses in many cases, while our PR-GCN performs more robustly with much better results. We also demonstrate failure cases in Fig. 4, revealing that PR-GCN fails when dealing with extremely occluded objects and some symmetric ones.

Inference efficiency. Besides the accuracy, we evaluate the efficiency of our method on Linemod. As shown in Table 4, each key component infers fast, and the full pipeline takes 68ms on an Nvidia 1080Ti GPU, which is acceptable in downstream tasks such as robotic grasping.

We comprehensively validate individual components of PR-GCN in the following.

The impact of PRN and MMF-GCN. The baseline method removes PRN and replaces MMF-GCN by DGCNN [39] which adopts the same basic point cloud aggregator as our PR-GCN. As Table 5 displays, PRN boosts the baseline by 2.0%, indicating that refined point clouds contribute to pose estimation, while MMF-GCN achieves an improvement of 2.1%, demonstrating its advantage in integrating multi-modal features. The combination of PRN and MMF-GCN further enhances the performance.

The generalizability of PRN and MMF-GCN. We generalize the PRN and MMF-GCN modules to two state-of-the-art frameworks including PVN3D [9] and DenseFusion [38], and evaluate their performance on YCB-Video. Note that PVN3D cannot utilize PRN directly, since segmentation is required on the whole scene while PRN focuses on specific objects. We thus only evaluate the effect of MMF-GCN on PVN3D. As shown in Table 6, PRN promotes the ADD-S of DenseFusion by 1%, and a similar improvement can be observed when applying MMF-GCN. The results indicate that PRN and MMF-GCN benefit other frameworks for 6D pose estimation.

The influence of segmentation. As in Fig. 2, our framework introduces RGB-based segmentation to extract foreground objects, while PoseCNN [40], DenseFusion [38] and PVN3D [9] adopt different instance segmentation models. To validate the effect of segmentation, we replace the segmentation model in our framework by the counterparts used in PoseCNN and PVN3D as well as the ground-truth, and report their AUC and ADD-S$<$2cm metrics on YCB-Video. Similarly, we evaluate this factor on other frameworks, including PoseCNN, DenseFusion and PVN3D. As reported in Table 7, all these frameworks achieve the highest AUC and ADD-S$<$2cm using ground-truth, indicating that better segmentation boosts the estimation accuracy. Meanwhile, with segmentation alternatives, our framework consistently outperforms the others, showing that PR-GCN is superior, regardless of which segmentation model is used.

The effect of the multi-resolution regression loss on PRN. We finally validate the credit of the regularized multi-resolution regression loss $\mathcal{L}_{\rm{prn}}$. For comparison, we apply $\mathcal{L}_{\rm{prn}}$ on $\hat{\bm{P}}_{o}$ only, denoted by PRN-SR, while the multi-resolution case is denoted by PRN-MR. We also report the result without using PRN (WO-PRN). As summarized in Table 8, adopting the single-resolution loss promotes the performance of our method. When the multi-resolution loss is added, ADD-S is further boosted to 95.8%, demonstrating its effectiveness. Moreover, we visualize the generated refined 3D points in Fig. 5, and the results clearly show the advantage of PRN in dealing with the incompleteness and noise, after adding the loss $\mathcal{L}_{\rm{prn}}$.