FocusTune: Tuning Visual Localization through Focus-Guided Sampling

Feature detection serves as a foundational element in numerous geometric computer vision applications, such as Simultaneous Localization and Mapping (SLAM)[10] and Structure from Motion (SfM)[46]. These applications rely on the robust identification of pixels, or features, that exhibit consistent characteristics across varying lighting conditions and viewpoints. Such features typically originate from 2D pixels with distinct textural properties, as opposed to texture-deficient uniform areas. The conventional approach for feature detection employs the Difference-of-Gaussian technique [35]. This method involves iteratively smoothing the input image across multiple scales and computing the differences between successive scales. Pixels that remain sharp through this process are designated as features. Owing to its computational efficiency, this technique has been widely adopted in prevalent feature description algorithms [35, 2, 36].

More recent developments [18, 38, 21] have explored the use of neural networks for feature detection. For example, the method described in [38] utilizes an attention layer to aggregate the semantic significance of local features, thereby facilitating the identification of the most relevant features while also providing reliable confidence scores. Conversely, [18] presents a learning-based corner detection model trained on a large synthetic dataset comprising basic shapes like triangles and cubes.

Our work builds upon this expansive body of literature, demonstrating that learning-based localization systems can also achieve enhanced performance by focusing on textured regions during training.

Classical structured solutions [25, 33, 44, 43, 45] for visual localization tasks establish 2D-3D correspondences between query images and environment maps. The nearest-neighbor search is typically done by comparing the feature descriptors [35] of the query image and the descriptors of the 3D points. Therefore, structured methods require access to the descriptor database of the map at run time, making them unsuitable for memory-constrained devices. While GoMatch [55] eliminates the need for storing descriptors, this comes at the expense of reduced accuracy.

The advent of deep learning has enabled absolute pose regression (APR) models to directly regress camera poses from query images [27, 47, 52, 31, 55]. APR models output the rotation (e.g. represented by a quaternion) and the translation vectors of the corresponding camera pose to the input image. While earlier methods [27] relied on a pre-trained GoogLeNet [49] as the backbone, more recent network architectures increase the performance of APR models with spatial LSTMs [52] or transformers [47, 31, 50]. However, despite being fast and requiring no access to maps at query time, the accuracy of APR models is still far behind classical structured methods.

Scene coordinate regression models predict the scene coordinates for the pixels in the query images [48, 51, 8, 14, 32]. SCR models can be implemented by random forests [48, 51] or convolutional neural networks [8, 3, 5, 6, 7, 19, 23]. SCR models are often optimized via the re-projection error, which works well in practice, thus leading to a much higher accuracy than APR models that are competitive with structured methods in certain settings. Up until recently, the mapping time of SCR models was problematic, often requiring hours of mapping time [8, 7] for convergence.

Multiple approaches were proposed in the literature to reduce the mapping time of SCR models [3, 20, 48, 15, 54]. SCoRF [48] showed a random forest can localize well in just 10 minutes. [15] leverages depth information to further increase the accuracy. SAnet [54] requires access to database scene images to retrieve and interpolate database scene coordinates. Despite being scene-agnostic and having a short mapping time, its weaknesses include higher memory requirement and lower accuracy compared to classical structured methods.

[3, 20] recently proposed to divide the SCR network into two parts: scene-agnostic pre-trained backbone and scene-specific regression heads. The backbone is trained once on a large dataset [16], whereas the regression heads are trained specifically for a given scene. [20] uses another convolutional layer to implement the regression head. This design choice cuts the mapping time to only 2 minutes, however, increases the memory footprint and decreases the accuracy. On the other hand, ACE [3] uses multi-layer perceptron layers to realize the regression heads, leading to state-of-the-art results on both indoor and outdoor settings with only RGB mapping images.

In this paper, we show that it is possible to further extend the accuracy of ACE models with a small overhead at training time by heuristic sampling of the training buffer.

Given a dataset of $N$ grayscale images $\mathcal{D}=\{\mathit{I}_{1},\mathit{I}_{2},\ldots,\mathit{I}_{N}\}$ and an associated ground-truth map produced via a Structure-from-Motion (SfM) algorithm [53, 46], the problem is to estimate the 6-DoF camera pose $H_{i}\in\text{SE}(3)$ for a given image $\mathit{I}_{i}\in\mathcal{D}$. Here, the camera pose $H_{i}$ is an element of $\text{SE}(3)$ which refers to the special Euclidean group that represents rigid body transformations in 3D space, including both rotation and translation.

To accomplish this, we approximate a function $\tilde{f}_{(\mathit{I},W)}=~{}X$, where $X\in\mathbb{R}^{3\times\frac{w}{8}\times\frac{h}{8}}$ contains the scene coordinates $\mathbf{x}_{ij}\in\mathbb{R}^{3}$ for every $j$-th pixel in downsampled image $\mathit{I}_{i}$, using a neural network parameterized by weights $W$. Therefore, $f$ implements a mapping from grayscale images to 3D coordinates, $\tilde{f}_{(\mathit{I},W)}:\mathbb{R}^{w\times h}\rightarrow\mathbb{R}^{3\times\frac{w}{8}\times\frac{h}{8}}$. The network is trained on images $\mathit{I}_{i}\in\mathcal{D}$ and the corresponding ground-truth poses $H_{i}$ by minimizing a re-projection objective given by:

where $K_{i}\in\mathbb{R}^{3\times 3}$ is the camera calibration matrix, $\mathbf{y}_{ij}\in\mathbb{R}^{2}$ denotes the $j$-th pixel coordinate in the pixel grid of $\mathit{I}_{i}$ and $\mathbf{x}_{ij}\in\mathbb{R}^{3}$ represents the $j$-th predicted scene coordinate. The hat operator $\hat{\cdot}$ denotes the homogeneous representation and $||\cdot||_{1}$ the L1-norm. While the fundamental objective remains consistent with Eq. 1, more complex objectives are used in practice; we refer the reader to [8, 3] for more details.

Finally, the camera pose can be computed given pairs of 2D-3D correspondences $\{(\mathbf{y}_{ij},\mathbf{x}_{ij})\}$ using an off-the-shelf Perspective-n-Point solver [29, 39].

While FocusTune is a generic method that is in principle compatible with a range of SCR methods, we have chosen to demonstrate our sampling technique within the Accelerated Coordinate Encoding (ACE) [3] framework that we briefly describe in this section. ACE partitions the neural network function $\tilde{f}_{(\mathit{I},W)}$ into a convolutional backbone $f_{\text{B}}$ (which only needs to be trained once and remains the same across all scenes) and a multi-layer perceptron (MLP) head $f_{\text{H}}$ (which is scene specific). Formally, the convolutional backbone $f_{\text{B}}:\mathbb{R}^{h\times w}\rightarrow\mathbb{R}^{512\times\frac{h}{8}\times\frac{w}{8}}$, pre-trained on ScanNet [16], generates dense local feature descriptors $\mathbf{d}_{ij}\in\mathbb{R}^{512}$ for a query image $\mathit{I}_{i}$. Subsequently, each descriptor $\mathbf{d}_{ij}$ is passed to the regression head $f_{\text{H}}:\mathbb{R}^{512\times\frac{h}{8}\times\frac{w}{8}}\rightarrow\mathbb{R}^{3\times\frac{h}{8}\times\frac{w}{8}}$ to generate scene coordinates $\mathbf{x}_{ij}\in\mathbb{R}^{3}$. Every scene coordinate $\mathbf{x}_{ij}$ corresponds to the pixel coordinate $\mathbf{y}_{ij}\in\mathbb{R}^{2}$ of the descriptor $\mathbf{d}_{ij}$ to form a pair of 2D-3D correspondences $\{(\mathbf{y}_{ij},\mathbf{x}_{ij})\}$ to re-localize the image $\mathit{I}_{i}$.

For training, ACE employs a buffer $\mathbf{B}$ containing sampled feature descriptors along with their geometric constraints. Each instance $\mathcal{G}_{k}\in\mathbf{B}$ is expressed as $(\mathbf{d}_{k},\mathbf{y}_{k},K_{k},H_{k})$, where $\mathbf{d}_{k}$ is the descriptor vector at pixel coordinate $\mathbf{y}_{k}$ in an image $\mathit{I}$ with associated intrinsic matrix $K_{k}$ and extrinsic matrix $H_{k}$. Note that the index notation changes from $ij$ at the image level (Eq. 1) to $k$ at the buffer level because a fixed amount of pixels is sampled from a particular training image for the buffer.

For larger scenes like those in the Cambridge Landmarks dataset [27], it is beneficial to divide the scene into multiple smaller sub-scenes $m$ and train a separate $f_{\text{H}_{m}}$ for each of them. At run time, all $f_{\text{H}_{m}}$ models are queried and the prediction with the highest inlier count is selected [3].

The effectiveness of Accelerated Coordinate Encoding (ACE) is inherently linked to its ability to infer ground-truth scene geometry with precision (see Fig. 2). While the re-projection error in Eq. 1 does not explicitly address the depth attributes of the descriptors, the network function $f_{\text{H}}$ is conditioned to perform implicit triangulation through the training buffer $\mathbf{B}$. Specifically, $f_{\text{H}}$ is required to produce consistent scene coordinates for similar descriptors $(\mathbf{d}_{1},\mathbf{d}_{2},\mathbf{d}_{3},\dots)$ by leveraging the geometric constraints such as $\{(\mathbf{y}_{1},K_{1},H_{1}),(\mathbf{y}_{2},K_{2},H_{2}),(\mathbf{y}_{3},K_{3},H_{3}),\dots\}$.

Triangulation becomes feasible only when the 3D lines parameterized by the geometrical constraints $\{(\mathbf{y}_{k},K_{k},H_{k})\}$ intersect at a common 3D point. However, tracking the feasibility of the geometrical constraints is intractable since there is no easy way to know exactly which constraints take part in the triangulation of a particular point.

We made a key observation that triangulation is often possible for 3D points with unique repeatable appearances rather than texture-less or ambiguous regions (refer to Fig. 3). To identify such regions from the training images, a straightforward strategy might involve off-the-shelf feature detectors [18, 37] or foreground/background segmentation models [30, 28]. However, during our experiments, these methods did not consistently yield effective results. On the other hand, the SfM program already employed robust local feature methods [35, 2, 18, 37, 21] to identify these points (Fig. 3, top row) during the reconstruction phase. Therefore, instead of employing a third-party system to identify reliable features for the training buffer $\mathbf{B}$, we rely on the SfM map to reduce the likelihood of sampling infeasible triangulation (Fig. 3, bottom row).

To alleviate the issues associated with non-distinctive areas of the scene, we introduce a simple sampling strategy for populating the buffer $\mathbf{B}$. For each query image $\mathit{I}_{i}$, we employ a pre-defined set of seed keypoints $\mathcal{Y}_{i}$ that are generated by re-projecting the corresponding 3D points from the SfM map into the 2D image plane using the ground-truth camera poses (see Fig. 1).

To generate the set of seed keypoints $\mathcal{Y}_{i}$ for each image $\mathit{I}_{i}$, we initially retrieve the 3D coordinates $\mathcal{X}_{i}$ of the 3D points that are visible in the image $\mathit{I}_{i}$ from the SfM map. These coordinates are re-projected into the 2D image plane using the ground-truth intrinsic and extrinsic matrices, resulting in $\mathcal{Y}_{i}=K_{i}H_{i}^{-1}\mathcal{X}_{i}$.

Due to random rotations and resizing applied to the camera matrices due to the data augmentation process during training time, some of the re-projected pixel coordinates may fall outside the image frame. Such invalid coordinates are then discarded, retaining only the valid set as the seed keypoints. Each seed keypoint forms a circle of radius $\rho$ pixels around it. Within these circles, keypoints are sampled uniformly, while areas outside the circles are excluded from sampling. This targeted sampling technique allows the network to concentrate on semantically relevant portions of the image, such as architectural features, rather than non-distinctive, texture-less zones (e.g., sky). We refer the reader to Fig. 4 for a comparison of both sampling strategies.

We incorporate our FocusTune sampling approach into the ACE learning pipeline as described in Brachmann et al. [3]. Our training buffer consists of 8 million instances. Keeping consistent with ACE’s original settings, we optimise $f_{\text{H}}$ over 16 complete iterations using a batch size of 5120. For comprehensive details on other hyper-parameters, the reader is referred to [3]. Unless otherwise mentioned, we set the sampling radius $\rho=5$ which is the only hyperparameter of our method.

FocusTune requires ground-truth 3D models for training; therefore, our evaluations are based on two datasets that provide such ground-truth information:

The 7-scenes dataset [48] provides ground-truth camera poses for small-scale indoor scenes using a Kinect RGB-D camera with $640\times 480$ resolution. Even though the depth information is provided for each image, we only use RGB frames to train $f_{\text{H}}$. Seven sequences were captured for each scene by different users and then were split into separate sets for training and testing purposes. Ambiguities, such as repeated steps in the Stairs scene, as well as specularities like reflective cupboards in the RedKitchen scene, are present in both RGB and depth images. Other challenges include motion blur, different lighting conditions, flat surfaces, and sensor noise. The ground-truth camera poses and 3D point cloud for each sequence were obtained using COLMAP [46] and provided by [4]. With the sampling radius $\rho=5$, our sampling method makes use of approx. $15.4\%$ of the pixels in the training images on average.

The Cambridge Landmarks Dataset [27] was collected at 5 different sites at the University of Cambridge in a large-scale outdoor urban setting with a Google LG Nexus 5 smartphone into high-definition videos. These videos were subsampled at a 2 Hz rate to obtain RGB frames. Substantial urban obstacles like pedestrians and vehicles were evident, with data being gathered across various timeframes that reflected diverse lighting and weather situations. Training and testing sets came from separate walking trajectories rather than being selected from a single trajectory, thereby rendering it more challenging for the localization task. The ground-truth training and testing camera poses as well as the 3D models were reconstructed using VisualSfM [53] and made available with the dataset. With sampling radius $\rho=5$, our sampling method makes use of approximately $16\%$ of the pixels in the training images on average.

In this section, we assess the sensitivity of our method to variations in the sampling radius $\rho$. The bottom plot in Fig. 5 illustrates the median translation error for the sequences from the Cambridge Landmarks dataset [27].

To determine the operational radius for our experiments, we aggregate the results from the five sequences using the following approach: we individually normalize the results of each sequence with its corresponding minimum error. For each radius value, we calculate the mean of the normalized errors across the different sequences and add one standard deviation. The upper plot in Fig. 5 presents the resulting normalized error, which combines the outcomes from all five sequences.

For the remainder of our experiments, we select a sampling radius of $\rho=5$, which corresponds to the point where our approach demonstrates optimal performance with the lowest error. We observe that larger values result in decreased performance because the buffer becomes populated with keypoints from non-textured areas. Note that $\rho\to\infty$ falls back to the default ACE random sampling. Conversely, smaller values lead to reduced performance due to a decrease in the number of keypoints.

To elucidate the efficiency of our sampling method, we compared two training buffers: $\mathbf{B}_{\text{ACE}}$ using the random sampling strategy employed by ACE and $\mathbf{B}_{\text{FocusTune}}$ using our sampling strategy. Both mean and median re-projection errors dropped markedly across all scenes when employing our sampling strategy, suggesting more effective dataset utilization (Fig. 6).