SGL: Structure Guidance Learning for Camera Localization

In this section, we introduce the proposed method in detail. Multiple researches [35, 36] have revealed that structure-based approaches and the scene coordinates prediction branch of the end-to-end ones yield higher localization accuracy than the metrics regression branch of the end-to-end ones. On the other hand, although a variety of learning-based components (e.g., local feature matching, image retrieval) are introduced into structure-based methods, the main calculation burden still lies on the classical epipolar geometry computing, especially on the mapping stage. Instead, our proposed method relies on the high-performance DNNs to predict the 3D points instead of geometry computing. Namely, we substitute the SFM model with the DNN to represent the scene structure. Fig. 2 indicates the whole architecture of our model together with the training procedure. Table I details the layer construction of both branches.

Inspired by the previous solid works on the other computer vision topic [37, 38], we also split the network into the parallel format. Receptive branch $Recep(X)$ is supposed to deal with the deeper features including the context information of the target scene and the global information with the large receptive area obtained by the cascaded multiple convolution layers.

In this branch, we employ a Fully Convolution Network (FCN) style structure which is thoroughly testified by [33, 39] on the camera localization tasks. After the common pre-processing (e.g., normalization and dimensional reduction) of input image $X\in\mathbb{R}^{h\times w\times 3}$ with height $h$ and width $w$, the shallow layers are applied to encode the visual clues into the 1/8 feature map. The following Convolution Neural Networks (CNNs) are responsible for recovering the deeper context information within the feature map. We also create residual shortcuts [40] to ensure the model convergence during the back-propagation process of the training stage. On the back-end of the Receptive Branch, there is the spatial block, which consists of four $1\times 1$ kernel convolution layers $Conv$. Sufficient ablation studies in Sec. IV-D implicate that its complexity is highly related to the scene content and scale. Hence, we surmise that this block is capable of embedding the spatial information of the scene into the feature map and name it with spatial block.

Compared to the Receptive Branch, Structure Branch is responsible for extracting the low-level visual features. For that reason, the shallower network structure is more appropriate. Consequently, the 3-stacked convolution layers are adopted as the feature extractor. Meanwhile, this block is also occupied to form the confidence map $Conf_{u,v}\in\mathbb{R}^{64\times h/8\times w/8}$ for the further key-point refinement, therefore we mark this block as the confidence block. The channel dimension of 64 is used to remap the $8\times 8$ down-sampled window-size pixels. From the confidence map, we define the pixel-wise bias $B_{i,j}\in\mathbb{R}^{2}$ by

$$B_{u,v}=\mathop{\arg\max}\limits_{i,j}(Conf_{u,v}),$$

(1)

where $i\in[0,8)$ and $j\in[0,8)$ are the index of the $8\times 8$ patch, $u\in[0,w/8)$ and $v\in[0,h/8)$ are the path index within the image plane. Moreover, the confidence map also produces the absolute value of every sampled pixel, representing the reliability of the prediction. Considering the value disparity, we use the dynamic threshold (changing depending on the confidence outputs) for denying low-quality predictions in contrary to a fixed-threshold filter.

$$Thresh=\alpha\max_{u,v}(Conf_{u,v})+(1-\alpha)\min_{u,v}(Conf_{u,v}),$$

(2)

where $\alpha\in[0,1]$ is a tunable parameter, that adjusts the domination difference between datasets.

The output feature maps of the confidence block and the whole receptive branch are fused by the $concat$ operator. The following cascaded $Conv$ layers help to converge the channel-wise information and produce the grid-sampled 3-channel feature map $Coord\in\mathbb{R}^{3\times h/8\times w/8}$ representing the corresponding 3-axis global coordinate of the down-sampled pixels. However, the direct output of $Coord$ is corresponding to the sparse 2D pixel positions. Due to the resolution limits, the original output will face the wrong 2D position prediction problems. Consequently, we add an additional operator that helps us refine the 2D position on the image plane. The following equation depicts the details of the key-point position refinement and selection strategy, where $Coarse_{i,j}$ and $Fine_{i,j}$ are the original coarse 2D pixel positions and the refined positions. By using $Thresh$ we are able to filter the unreliable predictions.

$$Fine_{u,v}=\left\{\begin{aligned} Coarse_{u,v}+B_{u,v},\quad Conf_{u,v}\geq Thresh.\\ Null,\quad Conf_{u,v}<Thresh.\end{aligned}\right.$$

(3)

With predicted 2D points and corresponding 3D predictions, we are capable of conducting PnP + RANSAC to calculate the final camera poses.

Inspired by the traditional BA process in the SFM pipeline, we transfer BA into network training and further experiments also show its practicality. Fig. 3 thoroughly compare the algorithm components between SFM and the SGL training procedure. In the SFM pipeline, image retrieval model and local feature matching model are occupied to achieve the mapping database. With the following geometric methods, including triangulation, PnP, RANSAC and BA, we are able to produce the high accurate sparse point cloud.

Whereas, in the SGL training pipeline, triangulation is substituted by SGL model inference. SGL achieves to get sampled 3D positions in the global frame in the meantime produce the 2D-3D pixel-level pairs. Afterwards, we take the advantage of PnP + RANSAC to calculate the camera poses. BA trainer is the training strategy that fully uses the BA idea into model training and helps the model better fit the scene. Finally, we fuse the BA loss and the metric loss and back propagate the gradients into the model weights.

Given an image whose predicted pose is denoted by the rotation matrix $R$ and the translation vector $t$, we can calculate the pose by PnP + RANSAC solver with 2D-3D matches. Relatively, the ground truth poses are rotation matrix $\overline{R}$ and translation vector $\overline{t}$. The metric loss including angle loss $\mathcal{L}_{angle}$ and translation loss $\mathcal{L}_{trans}$ can be obtained by following two equations.

$$\mathcal{L}_{angle}=\frac{1}{\pi}\arccos{(\frac{1}{2}trace(\overline{R}^{T}\cdot R-1))}.$$

(4)

$$\mathcal{L}_{trans}=\|\overline{t}-t\|_{2}.$$

(5)

Subsequently, we define the BA loss $\mathcal{L}_{BA}$ by the following equation.

$$\mathcal{L}_{BA}=\sum\limits_{i=0}^{n}\left[{\|p_{i}-K\cdot\left(R_{i}\cdot y_{i}+t_{i}\right)\|_{2}+(\frac{1}{n}\sum\limits_{j=0}^{n}{y}_{j}-y_{i})}\right],$$

(6)

where $i$ is the index number in the total $n$ bundle images. $p_{i}$ represents the 2D positions on the image plane of the sampled pixels, while $y_{i}$ is the predicted corresponding 3D position vector in the global frame. $K$ is the $3\times 3$ camera intrinsic matrix. By applying the SGL predicted rotation $R_{i}$ and translation $t_{i}$, we are able to calculate the reprojection errors of every image in the bundle set. Considering the instability of the SGL inference, we use the mean coordinates to compute the variance into the loss, in order to stabilize the network outputs. Consequently, we synthesize all the loss elements together, illustrated by the following equation.

$$\mathcal{L}_{all}=\mathcal{L}_{angle}+\beta\mathcal{L}_{trans}+\gamma\mathcal{L}_{BA},$$

(7)

where $\beta$ and $\gamma$ weigh the contributions of the different loss components.

We quantify the visual localization performance of SGL on two public benchmark datasets Microsoft 7-Scenes [14] (indoor) and Cambridge Landmarks Dataset [15] (outdoor). Both datasets contain challenging conditions, including motion blur, large illumination change, dynamic obstacles and etc. Ground truth poses are provided by the higher accurate equipments, which are the Kinect RGB-D camera in 7-Scenes and the abundant SFM data in Cambridge Landmarks. Besides, both benchmark datasets are widely occupied for visual localization evaluation. Many state-of-the-art localization researches have reported their statistic results, which makes them the validate baselines.

Hyperparameters $\beta=100$ and $\gamma=0.25$ are set fixed, while $\alpha$ varies from $0.70$ to $0.95$ on different subsets, which will be discussed in Sec. IV-D. The following experiments are conducted on the NVIDIA Tesla V100 GPU with the CUDA 10.2 and Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz.

In quantitative analysis, we select two of the representative visual localization methods among each branch discussed before. Statistics show that structure-based and coordinates prediction methods perform better than metrics regression ones. Because the coordinates prediction methods benefit from the high performance of the DNN and achieve the better performance than structure-based ones. Furthermore, our proposed SGL network outperforms most methods on both indoor and outdoor benchmark datasets, with higher translation and rotation accuracy. In Fig. 4 we also demonstrate the localization trajectory of those datasets to prove the accuracy.

Confidence Block. To illustrate the validity of the confidence block, we visualize the output of the confidence block in Fig. 5. The high values of the down-sampled map represent the high confidence, and vice versa. Eq. 2 ensures us to select the high-confidence points with high-quality observation. Qualitatively, with the help of key-point filter and refinement, the green points focus mostly on the objects with meaningful textures, such as architectures, cabinets and posters. Points on the objects with less texture information, like the sky, black screen and windows are neglected by our strategy. Moreover, although green points are formatted in a grid-like way because of the down-sample effect by the CNNs, some points that lie near the object edges are tunned to fit the contours, because of Eq. 1.

Correspondingly, due to the instability and inexplicability of the DNNs, the outputs of the confidence block vary greatly under different circumstances on a single sequence. Therefore instead of a fixed threshold, a dynamic threshold would avoid uneven remaining points circumstances caused by the unreasonable threshold. Fig. 6 provides the ablation study of the localization accuracy with respect to the tunable confidence ratio $\alpha$. As $\alpha$ balances the number of SGL predictions, therefore medium number for preserved key-points shows better performance on localization precision. Correspondingly, Fig. 6 shows $\alpha$ that lies within 0.70 0.95 holds lower rotation and translation errors. Because too few observations will result in noise in the PnP process, and too large $\alpha$ leads to failures in excluding those low-quality predictions of SGL.

Bundle Adjustment Trainer. We occupy HOW [46] to achieve the images corresponding graph and SuperGlue [24] to acquire the local feature matches. Nevertheless, the original output of the local feature networks do not always share the same key-points with the SGL. As a result, we have to adjust the matching pairs to fit the SGL key-point predictions. In the BA training process, we fully trust the matches produced by SuperGlue. As shown in Fig 7, we downsample the original local feature matches produced by SuperGlue by $8\times 8$ pixels per patch. Namely, every patch only preserves a single match. On the other hand, SGL provides the key-point pixel-wise position in every patch. Fig. 7 shows the circumstances when the key-point is tunned, modified and even filtered during the training process.

Fig. 8 illustrates the training status of the SGL network. With retrieved $k$ candidates and local feature matches, we are able to establish the key-point level correspondences in every bundle of training images. SGL’s confidence module predicts the high-confidence patches and eliminates the low-confidence ones, which are represented by the dark mosaics in the mid and bottom rows of the Fig. 8. As the training carries forward, the confidence module will gradually fit the scene and locate the low texture patches to exclude them from the later training stage. Table. III interprets that with the BA trainer’s help the localization accuracy rises on both datasets.

Spatial Block. In the Sec. III we introduce the spatial block in the receptive branch, because the depth of this block is sensitive to the final localization performance in different scenes, according to the statistics in Table. III. By increasing the $Conv$ layer number of the spatial block, we can improve the precision of SGL on the most subsets of 7-Scenes (except Stairs), while on the Cambridge dataset such phenomenon goes oppositely. Typically, when we substitute the spatial block with a much more complicated block, Transformer [47], the accuracy polarized greatly on these scenes. But the trend of accuracy transfers synchronizes with the ablation study on the changes of Conv layer number. In some subsets of both datasets, the higher complexity of spatial block has great effects on precision, while its performance is not stable especially on Cambridge datasets. Through all the ablation experiments we attribute the sensitivity of the spatial block to the diversity of the scenes on different subsets.