Discriminative and Semantic Feature Selection for Place Recognition towards Dynamic Environments

In our proposed method, we apply a hybrid supervision to train our proposed feature selection network, which includes strong supervised regression from semantic segmentation tasks and self-supervised learning from triplets optimization tasks. The construction of triplets need matching relationship between images. Therefore, it is necessary to prepare the semantic information and matching information of the public dataset.

Matching information in our work means the triplets composited of a query image ($I_{q}$), a positively matching image ($I_{p}$), and a negatively matching image ($I_{n}$). Matched images are caught from the same place, that is, a loop closure in the SLAM system. To obtain accurate matches, we refer to the method [29] to detect loops. We extract SuperPoint [16] features from the images and apply BoW-based loop closure algorithm to coarsely detect loop candidates. Then topological graph is constructed to verify the candidates. For better accuracy, we manually check the obtained matches to make sure all the matches are definitely from the same scene. These matched loops describe the same place under different visual conditions, as shown in Fig. 1(a), which includes pedestrians, varying illuminations, and moving cars, etc. In addition, we also need mismatches to construct triplets. For each query image, an image taken in different places, or not passing the graph verification, is considered as the mismatch, i.e. negatively matching image $I_{n}$.

For semantic information, we simply apply HRNet [30], a panoramic segmentation network with high performance, to obtain pixel-level semantic labels of images, as shown in Fig.1(b). We do not consider any improvement about segmentation and it is beyond this work. Then, we manually designate some static categories, like walls and buildings, and the others are defined as dynamics, following the reference in [31]. The label of static items is set to 1 and dynamics is 0 to fit the strategy that higher activation means higher probability of semantic stability. After that, we obtain a binary semantic stability map $S\in\mathcal{R}^{H\times W}$, where H and W is the height and width of original frame.

Our approach obtain an one-dimensional activation map with value between 0 and 1, and its resolution is the same as the original image. It is theoretically similar to panoramic segmentation. Thus, we decide to use panoramic segmentation model as our basic backbone. Additionally, the trade-off between computational efficiency and accuracy is necessary for real-time SLAM, so we experimentally compare the comprehensive performance of some famous panoramic segmentation network [22, 21, 23], as shown later in Section III-A, and finally choose U-Net [21] as our backbone.

For achieving better computational efficiency, we improve the vanilla U-net network to adapt to our method. Firstly, the number of down-sampling is reduced to improve the speed of network proceeding. Secondly, we increase the depth of the network to improve accuracy. Finally, we use a convolution layer and Sigmoid function as the header to get the one-dimension activation map $A\in\mathcal{R}^{H\times W}$. The network structure is shown in Fig. 2

To train our proposed feature selection model, we use a hybrid loss function (see Section II-C5) to optimize the parameters, including semantic loss (Section II-C1) based on semantic stability and matching loss (Section II-C2) based on matching information of triplets. Additionally, we design two different distance measurement between images based on various characters of dense features and sparse features (Section II-C3 and II-C4). Detailed descriptions are listed in following paragraphs.

To select a backbone that balances speed and accuracy, we compare the performance of SegNet[22], U-net[21], and DeepLab[23] in multiple panoramic segmentation datasets. The result is shown in Table I. For higher processing speed, we finally choose U-net as the backbone for our method.

In the training process, we set the maximal training epoches as 50 to let the network be fully trained. The initial learning rate is set to 1e-3, which will multiply 0.1 in the epoch 20, 30 and 40. In addition, the hyper-parameter $\alpha$ in loss function is initially set as 1 and will multiply 0.9 in every epoch to ensure that semantic loss mainly affects the initial supervision of the network and then matching loss gradually leads the optimization.

We evaluate our method on typical outdoor datasets since dynamic occlusions are frequently occurred in such datasets. CityCentre [33] contains 2474 images and it is caught during 2.0 km exploration in a campus with many similar buildings and shrubs. KITTI odometry [34] dataset is one of the most famous binocular benchmark and it contains 22 sequences. 11 sequences (00-10) of them have ground-truth trajectories which can be used for visual odometry or SLAM evaluation.

For place recognition, We use CityCentre and 12 sequences from KITTI to evaluate our methods. Generally used Precision and Recall metrics are measured. Precision refers to as the ratio between correctly detected loops and all the detection while Recall is the ratio between correct detections and all the loop events existing in the current scene. We use ground-truth loops from original publication for CityCentre and manually annotated detections in [35] are used as a reference on KITTI dataset.

For SLAM evaluation, we use all of the 11 sequences (00-10) with ground-truth trajectories. CityCentre is not used as we cannot find the public ground-truth poses. We calculate the average error of rotation and the standard deviation of offset between estimation and ground-truth for fair and quantitative comparison.

In this experiment, we plug our feature selection method into BoW-based place recognition framework [9] and then evaluate the effectiveness on KITTI and CityCentre. We test our approach with popularly used sparse and dense local feature, ORB [8] and SuperPoint [16], for verifying the versatility and generalization of our method. To comprehensively analyze the effectiveness of our method, we compare with original features and features selected by semantic labels, dubbed as “trad.” and “seman.”.

For original features, we directly use the response value of the characteristic point to select Top-500 features with highest responses. And for the semantic selection methods, we discard features locating on the dynamic objects to avoid the interference from dynamics. Then we also select the Top-500 feature points with the highest response value from remaining features based on the responses of original features. As for our method, the feature points are selected by estimated activation value. 500 features with highest activation are obtained to be further processed. We use RANSAC to estimate the transform matrix between query and loop after place recognition, and we calculate the average re-projection error of inliers as a score to obtain precision-recall curves, as shown in Fig. 3. For quantitative comparison, we also provide area under curve (AUC) in Table II.

According to the results, we can see that using semantic static/dynamic labels, which is manually defined based on the segmentation results, can sometimes help to discard confusing features so that the performance of place recognition has been improved. However, it sometimes negatively affect the system, such as in KITTI-07 and KITTI-08. It is because that the definition of dynamics and statics is unreasonable and unstable. Moreoever, it cannot focus on distinctive regions based on the texture of images. On the contrary, our proposed DSFeat works well on all the experiments and outperform compared baselines with a lot margins. It concludes that our method has a more stable estimation of dynamic and static attributes and it can detect distinctive regions which helps detection a lot. Additionally, our method can achieve better performance when incorporating with both sparse features and dense features, which is the first feature selection model achieving such goals to the best of our knowledges.

For a more comprehensive evaluation, in this experiment, we integrate the proposed method into an practical SLAM system to verify the improvement. We use the ORB-SLAM2 system [14] as the framework in such experiment and it use ORB as the feature extraction method. Our method is used to select stable features in the feature extraction step of SLAM system. The selected features are then used to track, map, and detect loops, thus affects the performance of overall system. Besides, our provided activation values are also used as a weights in RANSAC procedure during feature matching and pose estimation to improve the accuracy.

We compare the original SLAM system with the one incorporating with our proposed DSFeat. Fig. 4 shows an example of trajectories estimated by the slam system and ground-truth in KITTI-02 sequence. Clearly, our estimated trajectory is more accurate than the estimation of original systems which qualitatively concludes the effectiveness of feature selection procedure. Besides, Table III records the quantitative results of the average error of rotation and the standard deviation of offset on 11 sequence of KITTI. According to the results, after applying our method to select stable features, the accuracy of the entire SLAM system are improved on almost all the scenes, which concludes the effectiveness and generalization of our proposed approach.

In order to intuitively visualize the improvement of our proposed DSFeat, we show the semantic labels, semantic static/dynamic map, and our estimated heatmap (activation map) in Fig. 5. Using manually-defined semantic labels to judge the dynamic/static regions is unreasonable and cannot detect distinctive regions. In the figure, all the car is designated to be dynamic according to the semantic segmentation. But cars parked at the roadsides (Fig. 5(b) and Fig. 5(c)) should be static and running ones (Fig. 5(a)) are dynamic. Our method can reliably distinguish them and help to accurately track, map, and detect loops. The selected feature keypoints can be seen in Fig. LABEL:fig:feature.