D2S: Representing sparse descriptors and 3D coordinates for camera relocalization

Image-based visual re-localization methods[12] were arguably the simplest solution, which constructing maps just by storing a set of images and their camera pose in a database. Given a query image, we seek for the most similar images in the database by comparing their global descriptor [4, 13]. With the matched images representing a location in a scene that is close to the query image [13, 4], the pose of the query image can be derived from the top retrieved images pose. Despite demonstrating scalability in large environments, these approaches have limited accuracy in determining camera poses. Thus they are often used as initialization for later pose refinements.

6DoF absolute camera pose regression approaches (APR) have been studied extensively to overcome the drawbacks of image retrieval. Notably, PoseNet[14] was first proposed, along with subsequent developments [14, 15, 16]. APR learns to directly encode the 6DoF camera pose from the image, resulting in a lightweight system where memory demand and estimation time are independent of the reference data size. However, theoretical conclusions [17] suggest that APR behaviors are more closely related to image retrieval than methods that estimate poses by 3D geometry. Thus APR could hardly outperform a handcraft retrieval baseline.

Features matching-based (FM) re-localizers compute the camera pose from 2D-3D correspondences between 2D pixels and 3D scene maps, constructed using Structure-from-Motion (SfM) [18, 2]. Early FM-based solutions match local descriptors in 2D images to 3D points in the SfM model [19, 20, 21, 22]. Despite achieving accurate pose estimations, exhaustive matching creates a substantial computational bottleneck. Additionally, as SfM models grow in size, direct matching becomes ambiguous and difficult to establish under significant appearance changes. Leveraging the robustness of image retrieval in scalability and handling appearance changes, [2, 23] proposed a hierarchical re-localization approach that uses image retrieval as an initialization step before performing 2D-3D direct matching. Instead of performing complex 2D-3D matching, [24] proposed deep feature alignment, reducing system complexity while preserving accurate estimation.

Compared to the FM-based pipeline, scene coordinate regression (SCR) approaches are more straightforward. By employing a single regression network, 3D coordinates can be directly extracted from 2D pixels [5, 9, 6, 7, 11]. These correspondences then serve as the input for RANSAC-based pose optimization. SCR methods have demonstrated superior accuracy in camera pose estimation in small-scale and stationary environments. However, scaling these methods to larger, dynamic environments remains challenging. They struggle with adapting to novel viewpoints and handling the increased complexity that comes with larger-scale environments [10]. Our proposed model, adopting similar idea to SCR but aim to address the aforementioned problem. By utilizing the robustness of local keypoint descriptors, our model seeks to enhance performance in high viewpoint changes and dynamic scenarios.

Visual SLAM systems [25] allow continuous map updates, but most visual re-localization techniques, including FM [2, 26] and SCR-based methods [9, 6, 7, 11], lack such adaptability, requiring reconstruction and labeling for updates. APR-based approaches have pioneered self-supervised updates with unlabeled data, utilizing sequential frames [27] or feature matching [16]. ACE0 [28] represents the first SCR-based method to introduce this capability but remains limited under highly dynamic conditions [10]. In this study, we demonstrate that the proposed sparse regression-based D2S adapts naturally to such conditions through self-supervision without needing reconstruction or labeling.

Inspired by the universal continuous set function approximator [29], we first propose a simple element set learning function $\mathfrak{F}(.)$. It receives a set of descriptors $\{\mathbf{d}^{i}\}_{i=0}^{k}$ as input and outputs corresponding scene coordinates $\{\mathbf{w}^{i}\}_{i=0}^{k}$. Overall, the proposed function can be described as follows:

where $\mathfrak{F}:\mathbb{R}^{k\times D}\to\mathbb{R}^{k\times 4}$ and $f:\mathbb{R}^{D}\to\mathbb{R}^{4}$. $k$ is the number of descriptors, and $D$ is descriptor dimension. The function $f(.)$ is a shared nonlinear function. It receives a descriptor vector $\mathbf{d}\in\mathbb{R}^{D}$ and outputs a scene coordinate vector $\mathbf{w}=(x,y,z,p)^{T}$. We introduce an additional dimension $p$ for the scene coordinate, which represents the reliability probability for localization of the input descriptor $\mathbf{d}$. To ensure that the range of the reliability probability output lies within $[0,1]$ while enabling function $f(.)$ to produce $p\in\mathbb{R}$, we compute the final reliability prediction using the following equation:

where $\beta$ is a scale factor chosen to make the expected value of reliability prediction $\hat{z}$ easy to reach a small value when the input descriptors belong to high uncertainty regions.

The proposed module is theoretically simple. As illustrated in Eq. 1, it requires only a non-linear function $f$ to compute the scene coordinates of given descriptors. In practice, we approximate $f$ using a multi-layer perceptron (MLP) network, which is similar to [11].

Solely relying on independent descriptors to regress scene coordinates may lead to an insufficient understanding of the global context and the distinctiveness of feature clusters. To address this issue, we draw inspiration from recent works on feature matching [23], which leverage attention aggregation [30] to learn both local and global attributes of sparse features. We employ a simplified version for the scenes regression task, which can be described as follows:

where $\mathcal{A}:\mathbb{R}^{k\times D}\to\mathbb{R}^{k\times D}$, and $\mathcal{A}^{l}\circ...\mathcal{A}^{1}(.)$ represents a nested residual attention update for each descriptor $\mathbf{d}^{i}$ across $l$ graph layers. Given a coarse local descriptor $\mathbf{d}^{i}$, $\mathcal{A}^{l}(.)$ is used to compute its fine regression descriptor $\prescript{(l)}{}{\mathbf{d}^{i}}$ by letting it communicate with each other across $l$ attention layers. This implicitly incorporates visual relationships and contextual cues with other descriptors in the same image, enabling it to extract robust features automatically through an end-to-end learning pipeline from scene coordinates. In addition, the proposed reliability-aware detection, as described in Eq. 2, can also encourage our attention module to favor features from reliable areas (e.g. buildings, stable objects) and suppress those from unreliable regions (e.g. trees, sky, human).

As $\prescript{(l)}{}{\mathbf{d}^{i}}$ is the representation for the output regression descriptor of element $i$ at layer $l$, Eq. 3 can be simplified as follows:

which now has a similar form to the original Eq. 1, where the final regression layer $f$ is simply a shared MLP.

Similar to a previous study [23], we also consider the attentional module as a multi-layer graph neural network. However, we only leverage the self-edge, which is based on self-attention [30], to connect the descriptor $i$ to all others in the same image. The message passing formulation for element $i$ at the layer $(l+1)$ is described as

where $\mathbf{m}_{\mathfrak{I}\rightarrow i}$ denotes the aggregated message result from all descriptors $\{\prescript{(l)}{}{\mathbf{d}^{i}}\}_{i=1}^{k}$, with $\mathfrak{I}=\{1,...,k\}$ indicating the set of descriptor indices, and $[.||.]$ denoting concatenation. This module consists of $L$ chained layers with different parameters. Therefore, the message $\mathbf{m}_{\mathfrak{I}\rightarrow i}$ is different in each layer.

The self-attention mechanism performs an interaction to map the query vector $\mathbf{q}_{i}$ against a set of key vectors $\{\mathbf{k}_{j}\}_{j\in\mathfrak{I}}$, associated with candidate descriptors, to compute the attention score $\alpha_{ij}=$ Softmax${}_{j}(\mathbf{q}_{i}^{T}\mathbf{k}_{j})$. It then presents the best-matched descriptors with their value vectors $\{\mathbf{v}_{j}\}_{j\in\mathfrak{I}}$ represented by a scored average of values, which in practice is the computation of message $\mathbf{m}_{\mathfrak{I}\rightarrow i}$:

The query, key, and value are derived from linear projections between the descriptor $\prescript{(l)}{}{\mathbf{d}}$ with three different weight matrices $\mathbf{W}_{q}$, $\mathbf{W}_{k}$, and $\mathbf{W}_{v}$. This linear projection for all descriptors is expressed as

Note that the weight parameters are different in each graph layer. In practice, we also applied multi-head attention [30] to improve the expressivity from different representation subspaces of different weight parameters. Fig. 3 presents example results of attention on robust descriptors.

Due to the significant changes that can occur in environments over time since map creation, the ability to update from unlabeled data is crucial for most localization systems. We thus introduce a simple algorithm to update the proposed D2S with additional unlabeled data in a self-supervised learning manner.

Given a set of training images $\mathcal{T}$ and its labeled database of correspondent 2D–3D descriptors $\mathcal{D}_{\mathcal{T}}$ and $\mathcal{W}_{\mathcal{T}}$, we aim to find a set of pseudo-labels for new observation images $\mathcal{U}$. The pseudo-labels denote the newly generated 2D–3D correspondences of the unlabeled data $\mathcal{U}$. In detail, we extract the local descriptors of all images in $\mathcal{U}$, resulting in $\mathcal{D}_{\mathcal{U}}$. For each unlabeled image $\mathbf{I}_{\mathcal{U}}^{i}$, we then retrieve 10 nearest training images in the training database $\mathcal{T}$. Subsequently, we match two descriptor sources using a specific matching algorithm such as SuperGlue [23]. Finally, we merge the top 10 matching pairs by copying their ground truth world coordinates from $\mathcal{T}$ to $\mathcal{U}$. This step is shown as line 6 of the algorithm 1, where the $\mathbf{W}^{i}_{\mathcal{U}}$ is the obtained pseudo scene coordinates of descriptor matrix $\mathbf{D}^{i}_{\mathcal{U}}$ and $s$ is the number of valid world coordinates. This process is summarized in algorithm 1.

According to Eq. 1, the output of sparse scene coordinates $\mathbf{w}$, predicted by D2S, includes four variables $(x,y,z,p)^{T}$. This output is rewritten as $(\hat{\mathbf{y}},p)^{T}$, where $\hat{\mathbf{y}}$ is the abbreviation of the estimated world coordinates vector and $p$ is the reliability probability. The loss function to minimize the estimated scene coordinates can be defined as

where $N$ is the number of mini-batch sizes, $K$ is the number of descriptors in the current frame, and $\mathbf{y}_{i}$ is the ground truth scene coordinate. We multiply the difference between estimated and ground truth world coordinates by $z_{i}\in\{0,1\}$, which is the ground truth reliability of descriptors. The goal is to ensure that the optimization focuses on robust descriptors while learning to ignore unreliable descriptors.

Because the loss Eq. 8 is used to optimize only the robust features, we propose an additional loss function to simultaneously learn to be aware of this assumption, as follows:

As described in Section III-A, $\beta$ is a scale factor chosen to make the expected reliability prediction easy to reach a small value when the input descriptors belong to high uncertainty regions.

In addition, we also apply the re-projection error loss for each 2D descriptor’s position. The loss function aims to align the general scene coordinates following the correct camera rays, which is described as follows:

where $\mathbf{R}_{j}$ is the rotation matrix and $\mathbf{t}_{j}$ is the translation vector of camera pose $j$ obtained from SfM models. $\pi(.)$ is a function to convert the estimated 3D camera coordinates to 2D pixel position, with $u_{i}$ as its ground truth pixel coordinate. $z_{i}\in\{0,1\}$, where $z_{i}=1$ if keypoint has 3D coordinate available in SfM models, and $z_{i}=0$ otherwise.

Finally, these loss functions are combined with the scale factor as follows:

where $\tau(t)=\omega(t)\tau_{max}+\tau_{min},\text{ with }\omega(t)=\sqrt{1-t^{2}}$, $t\in(0,1)$ denotes the relative training progress [11]. The $\alpha_{r}=1$ if $\frac{c_{i}}{c_{total}}>0.8$, otherwise $\alpha_{r}=0$, where $c_{i}$ and $c_{total}$ are current and total training iterations respectively.

We estimate the camera pose using the most robust 3D coordinates based on their predicted reliability $\hat{z}$. We visualize examples of predicted robust coordinates in Fig. 4. Finally, a robust minimal solver (PnP+RANSAC[31]) is used, followed by a Levenberg–Marquardt-based nonlinear refinement, to compute the camera pose.

In contrast to DSAC^∗ [7] and several SCR-based works [5, 9, 6], which only focus on estimating dense 3D scene coordinates of the input image, our D2S learn to detect of salient keypoints and its coordinates. We believe that the SCR network should learn to pay more attention to salient features; this can lead to a better localization performance [10]. This is also supported by empirical evidence of D2S’s results.

The Indoor-6 dataset [10], composed of images captured at different times and under varying lighting conditions, poses a considerable challenge. Table I shows the results of our method. D2S consistently surpasses PoseNet [14], DSAC* [7], ACE [11], and NBE+SLD [10] in localization accuracy across all six scenes.

Specifically, D2S achieves a 28.4% improvement in Scene 2a compared to NBE+SLD, lowering the error from 7.2cm/0.68° to 4.0cm/0.41°. Averaged across the six scenes, D2S records an error of 3.38cm/0.62° and an improvement of 16.2% in accuracy.

In terms of storage, NBE+SLD requires two separate models, consuming 132 MB, while our method reduces this to a single network requiring only 22 MB, achieving six times the storage efficiency. Although ACE [11] uses the least memory, its performance is 28.3% lower than that of D2S, highlighting the robustness of our approach in diverse environmental conditions.

To evaluate the quality of re-localization performance on the Cambridge Landmarks[14] dataset, we report the median pose error for each scene, and the results are displayed in Table II. In comparison, APR-based approaches including PoseNet [14], FeatLoc [15], and DFNet [16], achieved the lowest performance with low memory demand for all scenes. In contrast, the FM-based Hloc [23, 2] pipeline achieved the highest re-localization accuracy but requires a high memory footprint. We also report the results of the compressed version SceneSqueezer (SS) [26] on this dataset, which has a significantly low memory footprint. However, its accuracy is much worse compared to Hloc.

Importantly, we report the results of SRC-based approaches as well as the storage demand of each scene. The proposed D2S achieved the highest performance in all scenes. In detail, for example, in the King’s College scene, our method shows 53% position improvement compared to the second-best method of DSAC*[7]. The reduction of error is from 0.15m/0.3^∘ to 0.07m/0.12^∘. On average, our method marks the top-1 accuracy among learning-based localization methods, which improved by 31.1% in positional error compared to the second-best method of SCoCR [6].

This section reports the performance of the proposed method and other baselines on the proposed Ritsumeikan BKC dataset, illustrated in Fig. 5. This dataset has not been evaluated by previous localization methods; therefore, we selected three state-of-the-art baselines, FM-based Hloc [2, 23], SCR-based DSAC* [7] and ACE[11], to draw a comparison with the proposed D2S. We used the same training and localization configurations stated in the original papers[2, 7, 11].

Table III reports the median localization errors of ACE, DSAC*, Hloc, and the proposed D2S on different test sequences of the BKC dataset. The BKC dataset features several challenges for visual localization methods on their capacity to generalize beyond the training data (high domain shifts and transition from day to night). Thus, it is particularly challenging for SCR approaches because significant domain shifts can result in substantial differences between training and test images. This is evident in the DSAC* and ACE’s worse results on testing sequences. On the other hand, the proposed D2S outperforms DSAC* and ACE with a remarkable improvement in localization accuracy of 32.1%.

Self-update with unlabeled data (D2S+). Importantly, the BKC dataset is proposed to evaluate localization methods in the capability of updating with unlabeled observations. Note that the previous datasets [14, 10] do not offer any available new observations for evaluating self-supervision. In the BKC dataset, each test sequence in has a corresponding unlabeled sequence recorded at different times. For self-supervision, we update the model with an additional 50k iterations using data generated by Algorithm 1.

We also report the results of D2S+ when updated with unlabeled data in Table III. Notably, the improvement in accuracy obtained through D2S, is from 56.8% to 79.9%, within the threshold of 0.5m/10^∘. D2S+ even surpasses Hloc and ACE0 [28] accuracy by 8.3% and 44.3% respectively.

System efficiency. The proposed method achieves an average processing speed of 21 frames per second (FPS) on the Cambridge dataset and 55 FPS on the Indoor-6 dataset. When compared to NBE+SLD [10] on the Indoor-6 dataset, D2S demonstrates a significant advantage, being 4.8X faster in inference speed and even more accurate (63.1% .vs 46.9%). NBE+SLD requires approximately 132MB, whereas the proposed method is 6X smaller by using only 22 MB for storing D2S parameters. In the details of the D2S pipeline, the SuperPoint feedforward process takes 10 ms for an Indoor-6 image with a resolution of 640 pixels, and 19 ms for a Cambridge image with a resolution of 1024 pixels. Additionally, the computation of 3D coordinates in D2S requires 4.0 ms. The PnP RANSAC process takes an average of 24.5 ms on the Cambridge dataset and 4.0 ms on the Indoor-6 dataset.

Reliability effects. The proposed method effectively eliminates low-reliability descriptors, achieved through the use of the simple loss function presented in Equation 2 in Section III-A. We guide this loss function using the pre-built SfM model for each scene. An ablation study demonstrating this capability is provided in Fig. 6, which shows a significant improvement in localization performance upon removing the uncertain descriptors identified by D2S.

Different feature extractors (FEs). We present a brief comparison between the proposed D2S and FM-based Hloc when using different FEs, as shown in Table IV and Fig. 7. The results demonstrate that the proposed D2S can significantly reduce memory requirements for localization using different FEs while maintaining comparable accuracies.

Graph Attention. We show that graph attention (GA) layers are critical for achieving the best performance of D2S, shown in Fig. 8. We evaluate our method by varying the number of GA layers from 0 to 9 on the Indoor-6 dataset. As depicted in Fig. 8(a), without GA layers, D2S reaches only 16.5% accuracy, while the introduction of a single GA layer raises accuracy to 40.52%. The highest accuracy of 66.3% is observed with 9 GA layers. However, to ensure a lightweight model, we utilize only 5 GA layers in this study. Additionally, Fig. 8(b) shows that the inclusion of GA layers improves pose estimation time, likely due to the enhanced quality of 3D point predictions with more GA layers.

Loss functions. We highlight the importance of the loss components $\mathcal{L}_{u}$ and $\mathcal{L}_{r}$ in training D2S by reporting the average median errors and accuracies at thresholds of 5cm/5% on the Cambridge Landmarks dataset. When both $\mathcal{L}_{u}$ and $\mathcal{L}_{r}$ are utilized, D2S achieves its best performance with an average median error of 11.2cm/0.19^∘, and an accuracy of 30.5%. Dropping $\mathcal{L}_{u}$ leads to a performance decline, increasing the average median error to 14.1cm/0.23^∘, while reducing the accuracy to 24.3%. Similarly, excluding $\mathcal{L}_{r}$ results in an average median error of 11.6cm/0.20^∘, with an accuracy of 25.9%.

Limitations. Since the proposed method relies on sparse keypoints, it inherits the typical challenges faced by sparse feature-based methods, particularly in low-texture or repetitive-structure regions. Additionally, when applied to large-scale environments, it is recommended to train multiple D2S models for each sub-region to ensure optimal performance. However, this approach may result in significantly longer training times.