LDL: Line Distance Functions for Panoramic Localization

LDL operates using a 3D map consisting of line segments and local features. We build such a map starting from a colored point cloud $P=\{X{,}C\}$. To obtain the 3D line segments we use the line extraction method from Xiaohu et al. [54], which can quickly process point clouds containing millions of points within a few seconds. We further remove short, noisy line segments from the raw detection with a simple filtering step: given the point cloud bounding box of size $b_{x}\times b_{y}\times b_{z}$, we filter out 3D line segments shorter than $\lambda(b_{x}+b_{y}+b_{z})/3$ with $\lambda=0.1$ in all our experiments. The 2D line segments are then filtered with an adaptive length threshold to match the filtering rate of 3D line segments. Specifically, we choose the threshold value such that the ratio of lines filtered in 2D equals that in 3D.

To obtain local features embedded in the 3D map, we first render synthetic views at various locations using the point cloud color values. Specifically, we project the input point cloud $P{=}\{X,C\}$ at a virtual camera and assign the measured color $Y(u,v){=}C_{n}$ at the projected location of the corresponding 3D coordinate $(u,v){=}\Pi(RX_{n}+t)$ to create the synthetic view $Y$. We then extract local features for each synthetic view $Y$ using SuperPoint [12], and back-project the local features to their 3D locations, which in turn results in keypoint descriptors embedded in 3D space. Note that while we illustrate map building using a colored point cloud, our setup can also work with line-based SfM maps [40, 39, 32] since the input to our pipeline is lines and associated local features.

Similar to map building, we extract line segments and local features from the query panorama image. We use LSD [18] to acquire line segments, which is a robust line detection algorithm that can stably extract lines even under motion blur or lighting changes. To remove noisy line detections as in the 3D case, we filter 2D line segments with an adaptive length threshold to match the filtering rate of 3D line segments. Specifically, for each scene we choose the threshold value such that the ratio of lines filtered in 2D equals that in 3D. Then, we extract local feature descriptors using SuperPoint [12], where the results will later be used for pose refinement in Section 3.4.

Distance functions are designed to compare the holistic spatial context captured from the large field of view in panorama images. They are defined for every point including void regions without any lines and can quickly rank poses. Compared to Chamfer distance or learned line embeddings used in prior work [33, 57], LDL does not attempt pairwise matching between lines, which is often costly and can incur failure modes. For example, it is ambiguous to correctly match between densely packed lines as shown in Figure 2a.

A line distance function is a dense field of distance values to detect lines in the 2D query image or the spherical projection at an arbitrary pose in 3D. For a point $x$ on the unit sphere $\mathbb{S}^{2}$, the 2D line distance function is given as

Here $D(x,l)$ is the spherical distance from $x$ to line segment $l=(s,e)$, namely

where $\mathcal{Q}(s,e)$ is the spherical quadrilateral formed from $\{s,e,\pm(s\times e)/\|s\times e\|\}$.

Similarly, the 3D line distance function is defined for each candidate rotation $R\in SO(3)$ and translation $t\in\mathbb{R}^{3}$. Using the spherical projection function $\Pi(\cdot):\mathbb{R}^{3}\rightarrow\mathbb{S}^{2}$ that maps a point in 3D to a point on the unit sphere, the 3D line segment $\tilde{l}=(\tilde{s},\tilde{e})$ is projected to 2D under the candidate transformation as $l=(\Pi(R\tilde{s}+t),\Pi(R\tilde{e}+t))$. For simplicity, let $\Pi_{L}(\tilde{l};R,t)$ denote the projection of a line segment in 3D to the spherical surface. Then the 3D line distance function is defined as follows,

As shown in Figure 3, one can expect poses closer to the ground truth to have similar 2D and 3D line distance functions. Therefore, we evaluate $N_{t}\times N_{r}$ poses according to the similarity of line distance functions.

We apply a robust loss function that measures inlier counts to quantify the affinity of the line distance functions. For each candidate pose $\{R,t\}$ we count the number of points whose distance function differs below a threshold $\tau$,

where $\mathbbm{1}\{\cdot\}$ is the indicator function and $Q\subset\mathbb{S}^{2}$ is a set of query points uniformly sampled from a sphere. The loss function only considers inlier counts, and thus is robust to outliers from scene changes or line misdetections. We validate the efficacy of the robust loss function in Section 4.2.

To further enhance pose search using line distance functions, we propose to decompose the distance functions along three principal directions. While line distance functions provide useful evidence for line-based localization, they lack a sense of direction as in Figure 2b, where the distance functions alone cannot effectively distinguish rotated views at a fixed translation.

We split line segments along the principal directions used for rotation estimation and define separate line distance functions for each group of lines, as shown in Figure 3. Recall from Section 3.2 that each candidate rotation $R$ is obtained from a pair of triplets in 2D and 3D principal directions denoted as $\hat{P}_{2D}^{R}{=}\{p_{1},p_{2},p_{3}\}$ and $\hat{P}_{3D}^{R}{=}\{\tilde{p}_{1},\tilde{p}_{2},\tilde{p}_{3}\}$. We associate line segments that are parallel to directions in $\hat{P}_{2D}^{R},\hat{P}_{3D}^{R}$, leading to three groups of line segments $L_{2D}^{R}{=}\{L_{2D}^{1},L_{2D}^{2},L_{2D}^{3}\}$ and $L_{3D}^{R}{=}\{L_{3D}^{1},L_{3D}^{2},L_{3D}^{3}\}$ in 2D and 3D, respectively. We separately define line distance functions for the three groups using Equation 2, namely $f_{2D}(x;L_{2D}^{i})$ and $f_{3D}(x;L_{3D}^{i},R,t)$ for $i=1,2,3$. Then the robust loss function in Equation 4 can be modified to accommodate the decomposed distance functions,

We validate the importance of distance function decomposition in Section 4.2.

We first assess the localization performance of LDL against the baselines in the Stanford 2D-3D-S dataset, as shown in Table B.1. LDL performs competitively against the strong baselines (Structure-based and Line Transformer) that apply powerful neural networks for candidate pose search. While the dataset contains hallways and auditoriums with large featureless regions or repetitive structure, LDL leverages the holistic distribution of lines using distance functions and shows stable performance without resorting to costly neural network computations. Further, LDL shows superior performance when compared against the Chamfer distance-based method, which indicates that solely focusing on line matches for ranking candidate poses can lead to suboptimal performance.

We additionally compare LDL against baselines in the OmniScenes dataset, as shown in Table 3. Unlike the Stanford 2D-3D-S dataset, all images exhibit blur from camera motion and approximately half of the images contain changes in object layout. In splits not containing changes, LDL performs competitively against the baselines, which supports our claim that line distance functions enable effective pose search without using neural networks. Further, LDL attains the highest accuracy in splits containing scene changes and notably in the extreme split that contains the largest amount of motion blur. This is due to the stable line extraction [18, 19, 55, 59] that enables resilience against motion blur, and the robust distance function comparison (Equation 4) that rejects outliers for handling scene changes. We further verify the importance of each components in LDL in Section 4.2.

To validate the illumination robustness of LDL, we measure localization performance after applying synthetic color variations. We select Room 3 from the Extreme split in OmniScenes for evaluation. As shown in Figure 4, the image gamma, white balance, and average intensity are modified to an arbitrary value, where further details are deferred to the supplementary material. We report the results of LDL along with the baselines in Table 2. CPO, PICCOLO, and the structure-based baseline all suffer from performance degradation, as the color values are directly utilized for finding initial poses. Notably, Yoon et al. [57] also shows performance drop, as Transformer line features are affected by the illumination changes of the image. As LDL relies on the spatial structure of line segments for candidate pose search, it is robust to illumination variations, leading to stable performance across all color variations. Further, note that while all the methods excluding PICCOLO [26] and CPO [27] use local feature matching for pose refinement, there is a large performance gap between LDL and the other methods. This validates our focus on designing a stable candidate pose selection method, as modern feature descriptors and matching algorithms [43, 44, 12, 13] are fairly robust against adversaries such as illumination changes.

To evaluate the efficacy of line distance functions for candidate pose search, we compare the retrieval accuracy of LDL against NetVLAD [3], which is a widely used global feature extractor [43, 57, 23]. Note that NetVLAD is used as the candidate pose selection module in the structure-based baseline. We use the Extreme split from OmniScenes for evaluation, where the translation and rotation error recall curve along with the runtime for processing a single candidate pose is reported in Figure 5. For fair comparison we use the identical pool of translations for both methods as $N_{t}=50$ and assign a large number of candidate rotations for NetVLAD with $N_{r}=216$. Additional setup details are reported in the supplementary material. While neural network-based pose search methods can perform city-scale search [3, 20, 17], the line distance functions in LDL exhibit competitive performance to NetVLAD in indoor environments. The distance functions provide highly discriminative spatial context, which enables effective pose search. Furthermore, the runtime for pose search in LDL is much shorter than NetVLAD, due to the highly efficient computation of distance functions only conducted on sparse sphere points. This is in contrast to NetVLAD where visual features are computed with a neural network for each view. The line distance functions enable quick and effective pose initialization, which in turn allow LDL to be usable in various practical localization scenarios.

We analyze the runtime of LDL in Table 4b where we decompose the runtime for localizing a single query image from OmniScenes [26]. We assume that 3D scanning along with map building is done offline and only consider the computation time for online operations, namely 2D line segment extraction, candidate pose selection and refinement. Overall, the pose selection process including rotation estimation and distance function computation exhibits a small runtime for both CPU and GPU, which validates the efficiency of our proposed line-based pose search. Nevertheless, the pose refinement exhibits a relatively larger runtime, which is mainly due to the large number of features in panoramas compared to normal images with a smaller field of view. While we attained our focus in pose search and used the off-the-shelf local feature matching algorithms for pose refinement [44, 12], devising highly efficient feature matching algorithms tailored specifically for panoramas is left as future work.

We assess the scalability of LDL to large-scale indoor scenes using the OmniScenes [26] dataset. While the previous set of experiments assume room-scale localization scenarios, here we test LDL using the entire OmniScenes dataset as the 3D map. Table 4a shows the localization results, where LDL is compared against the structure-based method at various number of candidate poses ($K$). LDL exhibits performance on a par with the structure-based method, which indicates that line distance functions can scalably handle large scenes consisting of multiple rooms. Nevertheless, scaling LDL to even larger scale scenes (e.g. building-scale scenes as in InLoc [51]) is left as future work.

While the main goal of LDL is to offer fast and robust localization based on lines, we find that with a small modification our method can offer light-weight privacy protection in client-server localization scenarios [14, 11, 6, 50, 49]. Following prior works [34, 50], we consider the case where a client using an edge device wants to localize oneself against a 3D line map stored in the cloud. Privacy breaches occur if the service provider maliciously tries to view the visual data captured by the client. This is possible even when only the local feature descriptors are shared between the client and server, by using feature inversion methods [37] that reconstruct the original image from a sparse set of local features as shown in Figure 6.

By changing LDL to only exploit local features near lines during refinement, we can prevent privacy breaches including feature inversion attacks without largely sacrificing localization performance. First, as LDL uses line segments for candidate pose selection the clients only need to share the extracted line segments with the service providers for initial pose search, instead of the entire view that would be needed for global feature-based methods. Second, as local features near line segments are shared with the service provider for pose refinement, feature inversion methods cannot faithfully recover the original visual content. We validate this claim with a small set of experiments performed in the Stanford 2D-3D-S dataset [4], where we filter descriptors whose spherical distances to the nearest line segment are over 0.05 rad. As shown in Figure 6, this line-based filtering degrades the quality of feature inversion attacks by hiding objects that potentially contain sensitive information while only incurring small drops in localization accuracy. We report additional details and results regarding the potential of LDL for privacy preservation in the supplementary material.