CORAL: Colored structural representation for
bi-modal place recognition

We introduce the elevation map representation defined on a grid map. Each 2D grid indexes an elevation to describe the environment structure. Originated in the 2D occupancy grid map, the elevation map replaces the occupancy information with the elevation, which is capable of representing the 2.5D ground surface.

First, the 6 degree-of-freedom (DOF) pose of the sensor denoted as $T$ is calculated by LiDAR inertial odometry algorithm, with respect to the global frame $G$. Given a 3D point $p_{i}$ of a new measurement in the sensor frame, we transform it into the global frame by $Tp_{i}$ and calculate the elevation:

$$e_{p}=PTp_{i}$$

(1)

the projection matrix $P=[0,0,1]$ retrieves 3rd entry of the transformed point as the elevation $e_{p}$, while the first two entries are rounded and then converted to the corresponding index of the grid map. For processing multiple observations of the same grid, the variance $\sigma_{p}$ of elevation is introduced to describe the uncertainty of elevation measurement according to range sensor models [5]. In this way, each grid data $({e_{g}},\sigma_{g}^{2})$ is updated according to the corresponding measurement data $({e_{p}},\sigma_{p}^{2})$. Note that only the measurement within the Mahalanobis distance threshold of the grid is fused by variance weighted strategy to obtain an updated grid data as follows:

$${e_{g}}=\frac{\sigma_{p}^{2}{e_{g}}+\sigma_{g}^{2}e_{p}}{\sigma_{p}^{2}+\sigma_{g}^{2}}\qquad\sigma_{g}^{2}=\frac{\sigma_{p}^{2}\sigma_{g}^{2}}{\sigma_{p}^{2}+\sigma_{g}^{2}}$$

(2)

Furthermore, to clear the dynamic objects due to the use of the accumulation mechanism, ray tracing is utilized to check whether the grid is crossed by a ray, and if the grid is occupied by a dynamic object, the elevation and variance of this grid are initialized. The detailed generation process can be found in [20].

Elevation image To utilize 2D convolution neural networks directly, the elevation map is converted into a one-channel grayscale image with the same size as the grid map, namely elevation image. As shown in Fig. 3, we scale the elevation value from $0$ to $255$ as the grayscale value of the elevation image. Additionally, a mean filter is applied to fill up some invalid pixels which have not been observed on the elevation image using a $3\times 3$ filter template. Compared with the single scan based BEV representation, the elevation image is much denser to describe the structure, also leading to more accurate point-to-pixel correspondences.

The architecture of the proposed compound network is shown in Fig. 2. The whole network can be divided into three main components: the feature extraction module, the fusion layer, and the local feature aggregation module.

Feature extraction Our multi-sensor network has two streams for feature extraction. The visual stream employs the lightweight ResNet18 [8] as the backbone to efficiently extract the visual features. Due to the utilize of several convolutional layers with stride 2, the output feature map suffers a dramatic reduction in size which is not suitable for fusion. So we use feature pyramid network (FPN) to combine four feature maps from the residual block group to recover the final feature map with the size same as the first residual block and exploit multi-scale feature information.

The structural stream comprises a group of convolutional layers to capture structural features, and four groups of residual blocks to extract fusion features. Except for the first group, all groups start with the convolution layer with stride 2 and all other convolutions are with stride 1. The number of $3\times 3$ kernel convolutions in each group is 2, 4, 4, 6, 6, and each group outputs the feature vector with the corresponding dimensions of 64, 64, 128, 192, and 256 respectively. The outputs of the last three residual blocks also exploit multi-scale information as the visual stream does.

Fusion Layer The fusion layer comprises two components: the projection layer and the fusion module. The goal of the projection layer is to convert the front-view visual feature map into the BEV visual feature map through sparse matrix multiplication. The grid index of the elevation image and the corresponding elevation value can be converted into a 3D position in the global frame. So each grid can be denoted as a 3D point and transformed into the LiDAR frame according to the estimated pose. With the intrinsic parameters of the camera and extrinsic parameters between the LiDAR and the camera, each 3D point can be projected onto the 2D camera image plane which helps retrieve corresponding visual image features. Following this idea, we implement a parameterless projection layer to find the correspondences. Considering that the coordinates of 2D projected points are often non-integers, we combine the visual features from adjacent discrete pixels by bilinear interpolating. After that visual feature map is generated in the BEV frame consistent with the elevation image, thus achieving appropriate features corresponding to later fusion.

Then, we arrive at the generation of CORAL representation in the fusion module. Denote the $i$th layer of structural feature map as $S_{i}$, and the corresponding BEV visual feature map as $V_{i}$. To maintain the same shape of $S_{i}$ and $V_{i}$, pooling operation $Pool(\centerdot)$ is used to adjust the output size of the raw projection layer $V$ and $1\times 1$ kernel convolution $Conv(\centerdot)$ is applied to keep the same channel size. We try two different methods for aggregating features. The first one uses element-wise concatenation to obtain CORAL $F_{i}$ defined by

$$F_{i}=\left[Conv(Pool(V)),S_{i}\right]$$

(3)

In the second method, feature maps are combined by element-wise summation given by

$$F_{i}=Conv(Pool(V))+S_{i}$$

(4)

We also propose two fusion strategies that combine the structural feature map and the visual feature map from the first residual layer, or all four blocks, which are evaluated in Section IV.

Local feature aggregation The local feature aggregation module learns to further extract a global descriptor. Considering the capacity of the NetVLAD on aggregating features, we feed CORAL to a NetVLAD layer and generate a global descriptor. Furthermore, we utilize a multi-layer perception (MLP) to process the raw output of NetVLAD for learning a dimension reduction mapping to decrease the size of the global descriptor which accelerates the nearest neighbor search.

We train our compound network in an end-to-end fashion to yield a bi-modal fused global descriptor. A margin-based loss is adopted to train pairs of samples labeled with positive or negative based on corresponding GPS position.

Loss function The training data is constructed as sets of tuples $\mathcal{T}=(P_{a},\left\{P_{pos}\right\},\left\{P_{neg}\right\})$. $P_{a}$ denotes an anchor with a pair of visual image and elevation image, $\left\{P_{pos}\right\},\left\{P_{neg}\right\}$ represent the set of anchor’s positive matches and negative matches determined by the their relative distances. We apply the squared Euclidean distance $\delta$ to evaluate the similarity of two global descriptors. Margin-based loss is used to minimize the distance $\delta_{a,pos}$ between the global descriptors of matching samples $(P_{a},P_{pos})$ while pushing apart the dissimilar matches $(P_{a},P_{neg})$. To achieve faster convergence and better discrimination, we only use the closest/hardest negative $P^{-}_{neg}$ in $\left\{P_{neg}\right\}$ and the most dissimilar positive $P^{+}_{pos}$ in $\left\{P_{pos}\right\}$ during back propagation. Comparing with various retrival loss functions, we utilize the lazy quadruplet [4] as the training loss:

$$\begin{split}\mathcal{L}(\mathcal{T},P_{neg*})=max([\alpha+\delta_{a,pos}-\delta_{a,neg})]_{+})\\ +max([\beta+\delta_{a,pos}-\delta_{a,neg*}]_{+})\end{split}$$

(5)

where $[\centerdot]_{+}$ is the hinge loss, $\alpha,\beta$ are constant margin parameters, and $P_{neg*}$ is randomly sampled from training data which is disimilar to all observations of $\mathcal{T}$.

Data sampling strategy The original hard-negative training strategy usually offers faster convergence but it may lead to a collapsed model. To alleviate this problem, we divide the training process into two stages. In the first stage, negative matches $\left\{P_{neg}\right\}$ are sampled randomly from all negative samples. This is followed by a second training stage, negative matches $\left\{P_{neg}\right\}$ are generated by looking for the hardest pairs from all global descriptors of negatives calculated by the latest model. This hard-negative mining strategy ensures negative samples become progressively harder, which avoids prolonged convergence and boosts the performance of the converged model.

We choose the Oxford Robotcar dataset [15] for training and testing. The car equipped with LiDAR sensors and cameras repeatedly drives through the same regions in one year and collects data in large-scale and long-term conditions across weather, season and illumination. To create training tuples, we define the similar observations with a relative distance less than $10m$ and heading difference less than 30 degrees as positive pairs, and those at least $50m$ apart as negative pairs. Referring to the data splitting rules of [1], we obtain 21636 training tuples from 44 sequences of the original dataset and 3011 testing samples from 23 sequences. To keep large overlapping regions between the camera image and the elevation image, the elevation map is set with the size of $80\times 80$ and the resolution of $0.5m$. The input of the visual image is downscaled into $112\times 112$ for extracting visual features efficiently. Furthermore, as scans are accumulated by 2D LiDAR scans at fixed intervals, there are some incomplete samples on the dataset. We discard them when generating the elevation image.

Generalization settings We choose KITTI dataset [7] to evaluate the generalization performance across city and sensor configurations. Only Sequence 00 which visits the same places repeatedly is used. The first $170s$ of Sequence 00 are used to construct the reference map and the rest are used as localization queries. The second dataset for generalization evaluation is collected by ourselves in different seasons, namely YQ. Data in overcast scene and snowy scene are used for mapping and query respectively. For a fair comparison, the same criteria on Oxford Robotcar dataset are applied.

We present qualitative results to demonstrate the feasibility of our CORAL-VLAD on the Oxford dataset under changing scene conditions. The common assessment indicators of place recognition - average recall@1 and the average recall@1% are used to evaluate the network performance. For fairness, the final global descriptor dimensions of all networks are set to 256.

Ablation study on fusion strategy We test our compound network with four feature fusion strategies operating on the intermediate visual feature map and the structural feature map, including (a) element-wise summarization in the first residual block (Sum-First), (b) element-wise concatenation in the first residual block (Con-First), (c) element-wise summarization in four residual blocks (Sum-Four), (d) element-wise concatenation in four residual blocks (Con-Four). The results are shown in Fig. 4 and Tab. I, the multi-scale fusion strategies outperform the single-scale due to the adequate information. And the better performance of concatenation than summation contributes to the intact information. In the following experiments, we use our network with the Con-Four fusion strategy as the final version, denoted as CORAL-VLAD.

Ablation study on sensor modal Furthermore, to investigate the contribution of the fusion step, we separate our fusion architecture into an independent visual stream and structural stream with a shared NetVLAD module to generate the global descriptor, called Vision NetVLAD(Vis-VLAD) and Elevation image NetVLAD(Ele-VLAD). The results are also shown in Fig. 4 and Tab. I. Except for the fusion strategy of Sum-First, results have been significantly improved using composite features. Furthermore, the results for single modal place recognition are shown, validating the correct design and implementation of our method.

Comparison with the-state-of-art methods We compare our method with the state-of-the-art single modal place recognition methods, PointNetVLAD (PN-VLAD) [1], LPD-Net [13] and Img-VLAD [2], as well as bi-modal place recognition method, compound network (Aug-Net)[19].

Using only structural data input, the performance of our Ele-VLAD is better than PN-VLAD, yet slightly worse than LPD-Net. Inputs of LPD-Net and PN-VLAD are 4096 filtered points with detailed 3D structural information of the environment while the elevation image only has $40\times 40$ grids with one-channel elevation. It also proves effectiveness of the elevation image for representing 3D geometric data. Although providing a rich appearance context, Vis-VLAD and Img-VLAD cannot outperform Ele-VLAD, which proves that the elevation image representation is more robust under environment variations. Results are shown in Fig. 1. Meanwhile, there are many over-exposed images on the Oxford dataset, causing loss of visual features.

Using composite descriptors, the performance of CORAL-VLAD exceeds all other methods, including Aug-Net. Note that there is a difference in the Aug-Net from [19], since we employ the PN-VLAD splitting rules clarified in [1] to ensure consistency among all methods. All comparison network training sessions last no longer than $24h$. We suppose that the main reason for the slightly worse results of Aug-Net is the incompleteness of the 3D volume based representation.

Comparison under different conditions We compare the performance of LPD-Net, Img-VLAD, CORAL-VLAD, which use inputs in different representation modalities under changing scene conditions. Fig. 6 shows the results when queries are taken from three different scene conditions against the testing database on the Oxford database (except query run). It can be seen that the huge variants of environmental conditions have little impact on retrieval performance using structural cues. As almost all corresponding visual features are lost at night scene, the visual-based approach obtains lots of incorrect matches. Accordingly, CORAL-VLAD has minimal improvement over LPD-Net in such condition, showing the limitation of additional visual modal.

Cases study Fig. 1 shows some of the successfully matched results in changing environments. We can observe that our network has learned robust features and alleviated the negative effect brought by dynamic obstacles and variations in illumination. Fig. 5 shows three wrong cases. We can see that the network is confused in the same scenes with opposite viewpoints (top row) and a few overlapping areas (middle row and bottom row).

Comparison on efficiency We further evaluate the running time of the network. Due to the use of lightweight network backbones and compact structural representation, our network implementation takes about $11ms$ which is faster than all methods except PN-VLAD shown in Tab. II. For generating the elevation map, we have a GPU-based implementation at almost 30Hz as shown in [21], achieving real-time performance for robotics applications.

To analyze the generalization of our network, we evaluate our network on the YQ and KITTI datasets using the model trained on the Oxford Robotcar dataset. These cross-city datasets consist of unobserved conditions with different sensor configurations, including sensor types and extrinsic parameters. Specifically, KITTI_laser uses the point cloud collected by LiDAR while KITTI_stereo uses the point cloud generated from stereo images.

In the case of KITTI_laser dataset, the elevation image is generated in real-time using Velodyne 64 HDL LiDAR shown in Fig. 7 and our network outperforms other methods in Tab. III. Although there is noise involved during the calculation of the disparity map on the KITTI_stereo dataset, generated elevation maps become blurred, which loses some structural information and introduces inaccurate matches between pixels and 3D points. CORAL-VLAD still achieves the best results, demonstrating the validity of our network for pure vision-based applications.

Additionally, we conduct experiments under different weather conditions on our campus. Unlike evaluating with similar conditions at a short period of time on the KITTI dataset, the unmanned ground vehicle equipped with a LiDAR and an RGB camera collects sensor data on sunny days in spring and snowy days in winter denoted as YQ dataset. The data collection platform is shown in Fig. 8. In addition, DGPS provides the ground-truth position to check the correctness of the matching results. The summer-overcast scene of the YQ dataset is utilized as the reference database, and the winter-snow scene as the query dataset. Matching examples are shown in Fig. 9, formulating a generalization evaluation on both conditions and sensor configurations. CORAL-VLAD still achieves the best performance in this scenario as shown in Tab. III, because of the elevation image based structural representation, transforming the texture into a geometric sensible representation.