Loop closure detection using local 3D deep descriptors

Given a 3D point and a set of its neighbouring points, namely a patch, a local 3D descriptor is a compact numerical representation of these points [35, 36, 14, 13].

Let $\mathcal{P}_{t}=\{\mathbf{p}_{t}\}\subset\mathbb{R}^{3}$ be the point cloud of the current keyframe at frame $t$ that is defined as an unordered set of 3D points, and $\Pi_{t}=[\mathcal{P}_{t-1},...,\mathcal{P}_{0}]$ be an ordered collection of point clouds stored up to frame $t-1$, with $t>0$. The size of this collection and the number of points of each point may vary over time and across scenes. At each $t$, we randomly choose a set of $n_{t}=n\cdot|\mathcal{P}_{t}|$ points as centres for our 3D descriptors, where $n\in\mathbb{R}_{(0,1]}$ and $|\cdot|$ is the cardinality of a set. The subsampled point cloud is defined as $\tilde{\mathcal{P}}_{t}\subset\mathcal{P}_{t}$ where $|\tilde{\mathcal{P}}_{t}|=n_{t}$. Let $\mathcal{D}_{t}=\{\mathbf{d}_{t}\}\subset\mathbb{R}^{d}$ be the set of $d$-dimensional descriptors where $|\mathcal{D}_{t}|=|\tilde{\mathcal{P}}_{t}|$.

Given a point in $\tilde{\mathcal{P}}_{t}$, we can extract a patch composed of neighbouring points within a spherical region with a certain radius, and use a deep neural network to encode this patch into a compact descriptor. Let $\mathcal{X}_{t}=\{\mathbf{x}_{t}\}\subset\mathcal{P}_{t}$ be the patch extracted from $\mathcal{P}_{t}$ and $\mathbf{d}_{t}=\Phi_{\Theta}(\mathcal{X}_{t})$, where $\Phi_{\Theta}$ is the deep neural network with parameters $\Theta$ that encodes $\mathcal{X}_{t}$ into the descriptor $\mathbf{d}_{t}$, such that $\lVert\mathbf{d}_{t}\rVert=1$. We experimentally select the best-performing deep neural network for the descriptor encoding, more details can be found in Sec. IV-C.

When a loop closure occurs, it is highly likely that a portion of the current keyframe’s point cloud overlaps with a portion of another point cloud in previous keyframes. Except for some occlusions, there should exist corresponding surfaces between overlapping point cloud regions, thus corresponding descriptors. In general, the higher this overlap is, the higher the likelihood of detecting a loop. We detect candidate loops through the estimation of the overlap between point cloud pairs in the descriptor space, and confirm the occurrence of a loop by computing (i) the transformation to register a candidate point cloud pair and (ii) the novel overlap measure RON that measures the ratio of nearest points in both the descriptor and metric space.

Specifically, we determine the overlap region between the two point clouds $\tilde{\mathcal{P}}_{t}$ and $\tilde{\mathcal{P}}_{t^{\prime}}$, where $t^{\prime}<t$, by selecting points whose descriptors, $\mathcal{D}_{t}$ and $\mathcal{D}_{t^{\prime}}$, are mutually nearest neighbours (MNN). The computation of MNN descriptors is efficient and does not require the estimation of the transformation matrix between two viewpoints. Let $\mathcal{C}_{t,t^{\prime}}$ be the set of corresponding points defined as

where $\mathrm{NN}(\cdot)$ is the nearest neighbour search based on the L2 norm. We compute the MNN overlap between the two point clouds as

where $o_{t,t^{\prime}}\in\mathbb{R}_{[0,1]}$. If the overlap $o_{t,t^{\prime}}$ is greater than a threshold $\tau_{o}$, then we deem these keyframes to form a candidate loop.

Points with the corresponding descriptors being mutually neighbours does not necessarily guarantee the points in the metric space to be also close to each other, for example flat regions may be ambiguous. To verifying the loop, we further register the candidate pairs by estimating the relative 6DoF transformation and estimate RON for the loop confirmation.

Different methods can be used to register two point clouds [15, 37]. Without loss of generality, we use RANSAC [15] as we found that it performs well in practice in terms of computational efficiency and robustness to noise. Let $\mathbf{T}_{t,t^{\prime}}\in\mathbb{R}^{4\times 4}$ be the transformation estimated with RANSAC between $\tilde{\mathcal{P}}_{t}$ and $\tilde{\mathcal{P}}_{t^{\prime}}$ using $\mathcal{D}_{t}$ and $\mathcal{D}_{t^{\prime}}$, respectively.

Lastly, with the registered point clouds, we compute RON as the ratio of MNN points in the descriptor space that are close in the metric space, i.e., whose distance in the metric space is below a certain error. Let RON be defined as

where $\lVert\cdot\rVert_{2}$ is the L2 norm, $\circ$ is the operator that applies $\mathbf{T}_{t,t^{\prime}}$ to a 3D point and $\tau_{e}$ is the maximum error between two corresponding points that takes into account the reconstruction noise. If $\mathtt{RON}_{t,t^{\prime}}$ is greater than a threshold $\tau_{\rho}$, then we confirm the occurrence of a loop. The transformation $\mathbf{T}_{t,t^{\prime}}$ can be fed to the module in charge of solving the pose graph problem to optimise the camera poses in any SLAM system. The pseudocode for our loop verification approach is shown in Algorithm 1.

Fig. 1 contains two examples of loops, and illustrates the estimated overlap regions of point clouds reconstructed using RGBD SLAM [3] and captured with LiDAR. Although the L3Ds that we use are trained on a different domain than that of the examples, we can observe how our approach can successfully determine mutually-nearest neighbour points in the overlap region (purple) and how the estimated transformation can be used to register each point cloud pair. In (d) we can observe that there are no mutually-nearest neighbours in the centre of the point clouds, this is due do the flat regions that carry little geometric information and the relative descriptors lack distinctiveness, so they cannot be matched reliably.

We compare our approach against two recent LiDAR-based methods [8, 9] for loop closure detection.

OverlapNet [8] estimates the overlap with a deep neural network with the inputs extracted from the LiDAR scans containing both the geometric and semantic information. OverlapNet defines the overlap between a scan pair as the ratio of the projected range pixels within 1m over all valid range pixels, where the candidate scan pairs are projected into range images with a common coordinate system. For fair evaluation, we reproduce the results using the provided model with inputs reflecting only geometrical information, i.e., depths, normals and intensities.

LiDAR Iris [9] uses a global descriptor for LiDAR scans. Each scan is converted into a binary signature image using a series of LoG-Gabor filtering and thresholding operations. LiDAR Iris defines the overlap as the Hamming distance of a pair of the binary signature images. We reproduce the results using the provided code to generate the signature images and compute the Hamming distance.

Within this comparative evaluation, we also assess three variants of our L3D-based loop detection approach, namely L3D-based Overlap that quantifies the overlap following the ground-truth overlap computation as in OverlapNet [8], L3D-based MNN that quantifies the overlap only as the ratio of MNN (Eq. 2), and L3D-based RON that quantifies the overlap based on the proposed RON (Eq. 3).

Fig. 3 reports the Precision-Recall (PR) curves under the Setting 2, where each frame is queried amongst all the previously seen frames to find the loop. Without the positional prior as a search constraint, all methods achieve a lower Precision values along the PR curves due to the large amount of False Positives as the threshold increases. Precision is generally low for all the methods because querying a point cloud against all the others at previous time steps leads to a higher likelihood of producing false positive loop detections due to similar neighbourhoods along the trajectory. In terms of the Area Under Curve (AUC) performance, OverlapNet is better than LiDAR IRIS, while our proposed method L3D-based RON is still the besting-performing method amongst all the other ones under this more challenging setup.

For the evaluation of visual RGBD SLAM, we use nine sequences from TUM-RGBD and seven sequences from ICL. TUM-RGBD contains real scenes recorded with a Kinect at 30Hz and its ground truth is captured using a motion capture system working at 100Hz. ICL contains photo-realistic scenes rendered with a simulated Kinect at 20-30Hz, featuring real-world trajectories captured using a motion capture system working at 100Hz.

We use the recent RESLAM [3] to serve as the overall SLAM system where we embed our proposed approach. The original RESLAM performs the selection of candidate keyframe pairs based on the similarity between the current and the previous keyframes using 2D visual cues, i.e. Fern. If none of the previous keyframes yields a sufficient similarity with the current one, the loop closure is terminated, and a global and compact representation of the new keyframe is added to a database. Otherwise a geometric-based assessment is further carried out using pose graph optimisation.

We use the original parameters of RESLAM and set $n=0.4$, $\tau_{o}=0.13$, $\tau_{e}=10$cm and $\tau_{\rho}=0.2$ (see notations in Sec. III-B). We set the max number of RANSAC iterations to 500K and the termination policy to 1K iterations for an average inlier error of 3cm. We report the results of RESLAM from their paper [3] and also include our reproduced results as the authors of [3] stated that different hardware may provide different results.

We use RESLAM [repro.1] and RESLAM [repro.2] to refer to the results reproduced by us using two configurations of RESLAM: we use the original parameters in the former, while we set the parameters to detect loop closures more frequently in the latter. The evaluation of the latter was performed to show the behaviour of RESLAM with a number of detected loops comparable to those of our approach. We also evaluate an ablated version of our approach, i.e. only the module to estimate the transformation matrix between the keyframe and loop frame based on L3D, i.e. DIP [13] in our experiments. This ablation experiment is designed to show the benefit of using L3D for pose estimation after the loop closures detected with RESLAM’s visual cues. We name this version RESLAM [w. L3D].

We show in Fig. 4 some qualitative results on the RGBD-TUM and ICL datasets. These sequences include a variety of trajectory patterns, i.e. from the handheld camera of fr1/room to the micro aerial vehicle mounted camera of deer/run. We can visually appreciate how the trajectory estimated using our proposed L3D-based loop closure approach is much closer to the ground-truth one. Some part of these trajectories are marked and zoomed-in with a purple bounding box to highlight the improvements.

Fig. 5 shows some registration results on the keyframes corresponding to detected loops when the 6DoF transformation is estimated using the RESLAM’s loop closure approach and ours. We can observe that the registration that is obtained with RESLAM provides a systematic misalignment. We noticed this also on other sequences. At times, the transformation estimated with RESLAM is incorrect, see c) and d), while our loop closure approach can often register the keyframes correctly. Lastly, Fig. 6 and Fig. 7 show qualitative localisation and mapping results using our loop closure approach. From these examples we can observe the full structure of the environment in the form of coloured point cloud.

We provide a quantitative analysis of traditional and recent 3D deep descriptors using the TUM-RGBD dataset. This experiment motivates the choice of DIP descriptor by comparing it with descriptors in the literature. We adopt the registration recall as the performance measure [36, 13], which measures the ratio of point cloud pairs whose average distance error between the corresponding points after being registered by the estimated transformation is below 0.2m.

We use the edge-based point clouds produced by RESLAM from four sequences of TUM-RGBD. For each sequence we randomly select 1000 point cloud pairs with at least 30% overlap. We compare four descriptors, one handcrafted approach, namely FPFH [27], and three deep learning based approaches: one global approach, namely FCGF [35], and two local approaches, namely SpinNet [14] and DIP [13]. For FPFH we used its Open3D implementation, whereas for the others, we used the code and models provided by the authors. For all the deep learning based descriptors, we use their models trained on the 3DMatch dataset [39].

Tab. III shows the results of this experiment. FPFH and FCGF are the fastest descriptors to compute, however at a large cost of the registration error. DIP achieves the best registration recall with a better compromise between the computation efficiency and the registration performance compared to SpinNet.