Representing 3D sparse map points and lines for camera relocalization

Image retrieval-based methods relocalize by simply comparing a query image with posed images in the database [18] and approximating the pose of the query with the highest-ranking retrieved images [19]. Pose regression employs a neural network to predict the absolute camera pose from a query image or the relative pose between a query and retrieved images [17, 20, 13, 19, 21]. However, both image retrieval and pose regression-based methods exhibit lower accuracy than the image structure-based methods discussed subsequently.

Feature-matching-based (FM) approaches are typically divided into direct [22, 23] and indirect methods [6, 2, 24]. Direct methods employ a strategy of matching 2D-3D correspondences directly from query image features to 3D points in SfM models. Although direct approaches can yield accurate camera poses, they are limited in scalability to larger scenes because of memory consumption and ambiguities [6]. Conversely, indirect approaches begin with an image retrieval step against images in the database [6, 2] and then match the features from the query image to those in the retrieved images. This creates a very robust pipeline under challenging conditions [7, 14].

Traditionally, FM-based methods rely solely on point features for relocalization. However, recent advancements suggest that incorporating line features can significantly enhance this pipeline [24, 2, 1]. However, the introduction of additional features increases the complexity and cost of FM-based pipelines owing to the additional matching steps required for line features [25, 8, 9], and additional memory is required for storing line descriptors [2].

Another prominent category within relocalization methodologies is scene coordinate regression (SCR) [16, 10, 13, 11]. These techniques infer 3D coordinates within the scene space from dense 2D pixel positions in the images, effectively embedding map representations within the neural network’s weights. This approach offers notable benefits, including minimal storage requirements [11, 26] and enhanced privacy due to the implicit nature of the map representation [27]. Nevertheless, recent learning-based approaches [28, 29, 30] have demonstrated that focusing on key landmarks can improve relocalization accuracy and robustness to environmental changes [28, 29]. Specifically, the sparse map learning approach detailed in [28] has shown superior performance compared with dense SCR [13, 10], particularly in scenarios characterized by significant domain shifts or limited training data availability.

Our method aligns closely with this innovative trend as we endeavor to create neural-based surrogate maps that incorporate both point and line features. This, results in significant enhancement of the camera relocalization task.

Recent advancements in SfM and visual SLAM have been explored with many successful mapping elements, such as points, lines, edges, planes, and objects [1, 5, 31, 2, 3, 32, 33]. With the demand for additional mapping elements, there is a clear need for a more efficient mapping representation strategy that extends beyond the basic storage of descriptor vectors [7, 6, 2]. To address this issue, specifically for point and line maps, we introduce a neural-based surrogate model capable of representing both 3D points and lines through their descriptors. This simplifies the matching process for multiple mapping elements.

Assume that we have a set of 2D keypoints $\{\mathbf{p}_{i}\}^{N}$ and a set of 2D line segments $\{\mathbf{l}_{i}\}^{M}$ extracted from image $\mathbf{I}^{r}$, each associated with visual descriptors $\{\mathbf{d}_{i}^{p}\}^{N}$ and $\{\mathbf{d}_{i}^{l}\}^{M}$ respectively. Here, $r$ denotes the image sourced from the reference database used to construct the 3D points and line map. We aim to develop a learning function $\mathcal{F}(.)$ that inputs the two sets of visual descriptors $\{\mathbf{d}_{i}^{p}\}^{N}$ and $\{\mathbf{d}_{i}^{l}\}^{M}$, and outputs the corresponding sets of 3D points $\{\mathbf{P}_{i}\in\mathbb{R}^{3}\}^{N}$ and lines $\{\mathbf{L}_{i}\in\mathbb{R}^{6}\}^{M}$ sets in the world coordinates system. The ultimate goal is to estimate a six degrees of freedom (6 DOF) camera pose $\mathbf{T}\in\mathbb{R}^{4\times 4}$ for any new query image $\mathbf{I}$ from the same environment.

This section presents in detail the PL2Map model designed to learn the representation of the sparse 2D-3D correspondences for both points and lines. Both points and lines possess interchangeable features, such as line endpoints and adjacent points, which can be integrated to enhance the development and accuracy of the final 3D map [25]. Fig. 2 shows the proposed architecture including three sub-blocks.

• •

  Front-End: We use the available 2D detectors to extract both the point and line positions and their descriptors from the input image.

• •

  Attentional Refinement: Points and line features/descriptors are boosted by a subsequent attentional refinement module, targeting the awareness of the surrounding point and line features [28].

• •

  Mapping Regressors: The last module consists of two MLPs that are used to regress the 3D map lines and points.

The predicted $\hat{\mathbf{P}_{i}}$ and $\hat{\mathbf{L}}_{i}$ are optimized using their pseudo ground truths $\mathbf{P}_{i}$ and $\mathbf{L}_{i}$ from 3D models for each image as follows:

where $r\in\{0,1\}$, $\gamma$ is a robust norm. Because the loss Eq. 7 only focuses on regressing valid points and lines, we optimize the reliability prediction for non-robust descriptors as follows:

where $\hat{r}_{i}$ is calculated using Eq. 6 for both points and lines. In Fig. 5, we show an example of reliability prediction results. Furthermore, we optimize the model using the available camera poses by reprojecting the predicted 3D points and lines onto the image plane:

where $\mathbf{T}$ is the ground truth pose, $\pi(.)$ is the reprojection function, $\mathbf{u}_{i}^{p}\in\mathbb{R}^{2}$ and $\mathbf{u}_{i}^{l}\in\mathbb{R}^{4}$ are the 2D positions of the point and line endpoints on the image, $\psi(.)$ is a function that calculates the distance between reprojected 3D lines and its ground truth coordinates $\mathbf{u}_{i}^{l}$, as illustrated in Fig. 4, and $v_{i}\in\{0,1\}$, $v_{i}=1$ when a prediction is between 10 cm and 1000 m in front of the camera, and has projection error lower than 1000 px, and otherwise $v_{i}=0$.

However, the reprojection loss, as illustrated in Eq. 9, is still highly non-convex and difficult to train at the beginning stage. Thus, we adapt the robust projection error defined in [11] as follows:

where $\tau(.)$ is the threshold used to dynamically rescale $tanh(.)$, and varies throughout the training:

where $t\in(0,1)$ denotes the relative training progress. This forces the threshold $\tau$ to have a circular schedule that remains close $\tau_{max}$ at the beginning and reaches $\tau_{min}$ at the end of the training.

Finally, we integrate all loss functions to optimize the surrogate model as follows:

where $\delta$ is the hyperparameter coefficient used to balance three loss functions.

We conducted our experiments on two standard camera re-localization datasets, both indoor and outdoor:

7Scenes [16]: An RGB-D dataset consists of seven different small environments, scaled up to 18$m^{3}$. For each scene, the authors provided several thousand images for training and testing. This dataset features challenges such as repeating structures of the Stair scene or motion blurs that occur throughout the dataset. Images were recorded using KinectFusion [40] which also provides ground-truth poses and depth channels.

Cambridge Landmarks [17]: An RGB outdoor dataset scaling from 875$m^{2}$ to 8000$m^{2}$. The dataset consists of five scenes, including several hundred frames for the training and test sets, split by the authors. Ground-truth camera poses were obtained by reconstructing the SfM models.

SfM ground truth models. The main purpose of our method is to represent multiple 3D sparse models as a neural network for later localization with high-accuracy constraints and low storage demand. We leveraged two SfM tools, point-based SfM Hloc [6, 7] and line-based SfM Limap [2] to obtain these models. Hloc and Limap are currently the most robust localization methods that utilize points and line maps. However, they are burdened by large memory requirements and complex feature-matching processes, posing challenges for real-time applications and lower-end consumer devices. This creates a large gap between real-time and low-grade consumer application. The proposed method addresses this burden. For all experiments, we used Superpoint [35] and DeepLSD [37] to extract the 2D points and lines.

In this section, we report our localization results for learning indoor sparse point and line maps representation. We compare our results with those obtained from state-of-the-art baseline methods, as detailed in Table I. In Fig. 6, we show some example results of 3D points and lines mapped by PL2Map.

We compare our indoor localization results against three established baselines: Hloc [6], PtLine [24], and Limap [2]. These were chosen because of their reliance on similar mapping components, specifically, points and lines. Regarding the use of only point maps, our approach demonstrated superior performance over Hloc in six out of seven evaluated scenes. For instance, within the Pumpkin scene, our method achieved localization errors of 3.5 cm in translation, 0.96^∘ in rotation, and attained an accuracy of 63.1 % (using a threshold of 5 cm / 5 deg). In contrast, Hloc exhibited less favorable results, with localization errors of 5.0 cm in translation, 1.32^∘ of localization errors, and 50.4% accuracy. These findings underscore the efficacy of our surrogate mapping model, which yields a performance improvement of 12.7% over that of Hloc’s database-based mapping approach.

Line-assisted localization. In Table I, we present our localization results using both the predicted points and line map. As can be seen, we achieved the best results when localizing with lines-assisted. These results confirm the efficiency of lines combined with points for visual localization, where an improvement can be observed for all seven scenes. Interestingly, our method learned using points and lines SfM models produced by Hloc and Limap, but the localization results obtained by our re-mapping method can even reach a large improvement margin.

Additionally, we provide localization results with available depth to refine the SfM models in both point and line maps. The results are listed in Table II. The original PL2Map still struggles with the repeated structure of the Stairs scene, but with SfM models refined using depth data, PL2Map shows a significant improvement in Stairs scene accuracy from 33.3% to 53.0%. The overall improvement was also observed for all scenes, evidenced in average median errors. Note that depth is used solely for training and inference relies solely on RGB images.

In this section, we present the localization results of the proposed method using the Cambridge outdoor dataset [17]. Unlike small-scale indoor environments, learning and SCR-based methods [10, 28, 11] still struggle to close the gap with FM-based methods in large-scale outdoor scenarios. Thus, we compared our method with four additional SCR-based baselines: SANet [12], DSAC^∗ [10], ACE [11], and D2S [28]. We present the results in Table III. Among the learning-based approaches, our localization results exhibited the lowest errors when utilizing only the predicted point maps. For instance, in the King’s College scene, our method outperforms [28], which also employs keypoint descriptors, by achieving a 51% reduction in translation error. This substantial improvement narrows the accuracy gap with FM-based methods.

Line-assisted localization. Table III also includes the results of our method when localization is achieved through the integration of both points and line maps. Across all five scenes, PL2Map demonstrated an enhancement in localization accuracy compared with scenarios where only the predicted points map was utilized.

Similar to [28], our method demonstrated the ability to disregard outlier features associated with dynamic elements or those unsuitable for localization through a single feedforward pass. This result is shown in Fig, 7, where the PnP RANSAC step can benefit from focusing on a high-quality set of correspondences [28].

Table IV compares our approach with three primary baselines in terms of localization requirements, highlighting the efficiency of the proposed method by eliminating the need for a matching step and storing 3D maps as descriptors. Consequently, our approach requires significantly less memory, requiring approximately 25 MB for the network weight, in stark contrast to the several GBs required by Hloc [6] and Limap [2].