Point Cloud based Hierarchical Deep Odometry Estimation

The KITTI dataset employs global pose coordinates on the local frame. Pose information is provided from the view of the first frame as the center of the coordinate system. These, however, are not suitable labels for the frame-to-frame odometry estimation task. Frame-to-frame pose transformations are achieved through following formula:

$T_{i,i+1}$ is the local transformation between two coordinate centers $X_{i}$ and $X_{i+1}$. $G_{0,i}$ and $G_{0,i+1}$ are the global transformation from the first frame (center of the global coordinate frame) to the frames $i$ and $i+1$.

The point-clouds in the KITTI dataset consist of $100k$ points per frame. This is a huge set of points. Given that lidar observations become less reliable at longer distances, we remove any point farther than $50m$ from the center in the $x$ and $z$ dimensions. There is still a large number of points remaining after this step. To reduce these points, we use the farthest point sampling strategy [Eldar et al., 1997] to sample only $12k$ and $6k$ points that are later used in our experiments.

Point-clouds collected in driving environments usually contain a large amount of ground points, which are not useful to extract motion information and are usually discarded as a pre-processing step [Ushani et al., 2017][Liu et al., 2019]. However, ground as a flat surface can provide cues regarding the pitch and roll orientations of the vehicle, which is valuable for the estimation of 6-DoF odometry. We choose a $75\%-25\%$ sampling ratio for non-ground and ground points, in an attempt to balance the size of the data and the accuracy of the system.

Another major issue with using deep learning models for odometry results from the highly imbalanced nature of the labels: the majority of roads are designed as straight lines, with only a few turns made in long drives. Deep learning models are highly dependent on datasets being as complete as possible, which is not the case in these scenarios.

On the dataset side, we employ over-sampling of the minority labels to address this challenge. As a result, models can learn the dynamics of the environment while assigning the required attention to minority transformations.

In 6-DoF motion, there are three rotation and three translation parameters that need to be estimated. Rotation components include $\alpha$, $\beta$, and $\gamma$ as the pitch, yaw, and roll, and translation components consist of motion in $x$, $y$, and $z$ axes. There are various methods that perform over-sampling - e.g. repetition, or Synthetic Minority Over-Sampling Technique [Chawla et al., 2002].

In order to achieve a more balanced dataset, we use the repetition method based on the amount of divergence from the average odometry value. To augment the dataset, we first calculate the average and standard deviation for the rotation and translation component norms individually. To decide on which samples to use, we follow 3 rules:

• •

  Translation: if $x_{t}<|\mu_{t}-a\sigma_{t}|$ then augment with $T$.

• •

  Rotation: if $x_{r}<|\mu_{r}-a\sigma_{r}|$ then augment with $T$.

• •

  If both translation and rotation rules are satisfied perform another augmentation run with $x$.

Subscripts $r$ and $t$ represent the rotation and translation components, respectively. $x$ is the input norm, $\mu$ is the average value, and $\sigma$ is the standard deviation for the corresponding input $T$. $a$ is the ratio that is used for the range interval.

Once a point pair is chosen to be repeated, number of copies $N_{x}$ is calculated using the following formula.

$D$ is a divisor value that explicitly controls the magnitude of the repetitions and is set to $4$.

In this way, we add approximately $10000$ samples for rotation, $6000$ samples for translation, and $2000$ samples for both.

In the majority of driving scenarios, the cars are in motion and are seldom stopped. To address this, we repeat the identity transformation with a random probability of $10\%$ on the dataset. This results in the addition of approximately $2000$ samples to the training set.

Figure 1 compares the distribution of data points before and after augmentation. The imbalance in specific components is not completely removed, but is less than in the original dataset. This is especially the case for yaw $(b)$ and $z$ components that have a major effect on the accuracy of odometry. It is worth mentioning that, as all these components are correlated with each other, completely removing the imbalance only using this data is an impossibly challenging task.

We rely on PointNet++ layers [Qi et al., 2017b] to build our model. Similar to their model, we propose a Siamese network that is able to regress the transformation between two point-clouds. Instead of passing the whole point-cloud to a PointNet feature extraction layer, we divide the inputs into two groups; one for ground points, and one for non-ground points. For ground points, we only use a single PointNet layer with a grouping distance threshold of $4$ that outputs $400$ points along with their descriptors. Descriptors are generated using 3 consecutive multi-layer perceptrons (MLP) of size $\left(64,96,128\right)$. For non-ground points, there are two layers that subsequently use grouping distances of $0.5$ and $1$. $1500$ points are produced in the first layer and they are summarized to $800$ points in the second layer. Both of these layers use 3 MLPs where the first one consists of $(64,80,96)$ and the second one has layers of size $(112,128,128)$. The distance metric used to group ground points is larger than the non-ground ones. This is due to the harsher sampling performed on the ground points that has resulted in larger distances between the points.

As the PointNet++ layers use farthest point sampling internally, we keep the ground and non-ground features separate for the feature extraction layer. The final outputs of ground and non-ground segments both have a dimensionality of $128$. Both feature maps are concatenated along the points dimension building a feature map of size $1200\times 128$. At this stage, features from each frame are passed to the flow embedding layer of [Liu et al., 2019]. We use the cosine distance metric to correlate the features of each frame to each other. For further future extraction in this layer, we use the MLP with $(128,128,128)$ width. Through some experimentation, we found using nearest neighbor with $k=10$ gave the best results at this step. The final output shape of this layer is $1200\times 128$.

Once the embedding between two frames is calculated, we run another feature extraction layer with radius of $1$ and MLPs of size $128,96,64$. This layer is also responsible for reducing the number of feature points that results in a feature map of shape $300\times 64$. All of the layers up to this stage include batch normalization [Ioffe and Szegedy, 2015] after each convolution.

The resulting 2D feature map is flattened and a drop-out layer with keep probability of $0.6$ is applied on it. In order to extract the rotation and translation parameters, we use two independent fully connected layers of width $128$ and $3$.

The final outputs of size $3$ are concatenated to build the 6-DoF Euler transformation parameters. Figure 2 shows the architecture of this network.

In traditional literature (optimization or RANSAC models) [Kitt et al., 2010][Badino et al., 2013], an odometry prediction is refined quickly through multiple iterations. We argue that the same could be applied in the case of deep learning-based odometry estimation models. Inspired by a hierarchical homography network [Nowruzi et al., 2017], we use multiple layers of the same network to train on the residuals of previous predictions. The new ground truth is calculated by multiplying the ground truth from the previous iteration by the inverse of its prediction. In this way, each network reduces the dynamic range of error from the previous models and successively achieves a better result. Figure 3 shows this process.

The higher accuracy comes with the cost of increased train and test times. The key element in this approach is to keep the computational complexity as low as possible for each module in order to satisfy real-time processing requirements for the odometry task.

The goal of the odometry model is to smooth the effect of errors caused by the registration network. Our proposed model requires a large amount of memory due to its mid-level feature representation. Adding the memory requirements that the temporal model imposes, training quickly becomes a challenge for the system. Furthermore, training the core registration network is already a difficult task. Extra parametrization from the LSTM model makes an already difficult task an even more challenging one.

To alleviate these issues we propose a two stage training approach that breaks the initial feature extractor and the temporal filter into two disjoint models. Once the core registration network is trained, mid-level features are extracted and used as inputs to train the temporal model.

The base of our proposed temporal model consists of two fully connected layers of width $64$ and $128$. It is followed by two bidirectional LSTM with $64$ hidden layers with a soft_sign activation function. The output of the LSTM is fed to a fully connected layer with a size of 64. Finally, in the output layer, we have two fully connected networks with 3 nodes each that estimate the rotation and translation, individually. Figure 4 shows this model.

We employ drop-out and batch-normalization after each layer. For the LSTM model, a temporal window of $5$ frames is utilized.

As explained in Section 3, the odometry labels suffer from data imbalance problem. The majority of the instances in the dataset follow a straight line, with only a few of them constituting the turns. To tackle these challenges, we incorporate measures in our loss function.

Using the naive L2-norm diminishes the effect of instances with larger errors, as most of the batches result in smaller errors. It is required to make the model more sensitive towards larger errors. This is achieved through the use of online-hard-example-mining (OHEM) loss [Shrivastava et al., 2016]. In OHEM, only the top $k$ samples with highest loss values are used to calculate the final loss. However, once this value is set it does not change during the train. This could result in a training session that has high fluctuations in loss values. To address this, we implement an adaptive version of OHEM loss, that increases the number of top $k$ after certain epochs. The model initially focuses on the hardest examples, before its attention shifts towards all of the examples. This provides the hardest examples that are less frequent with a chance to drive the network towards the global minimum.

Another aspect of the loss function is to analyze the effects of errors in each component of the label. Rotation and translation components are separately extracted from the model. To enable the model’s capability of adapting to each component, instead of using the naive approach, we employ a weighting mechanism. To introduce uncertainty in our loss function, we consider the $log$ of normal distribution and phrase it as a minimization problem. In odometry, there are 6 parameters to be learnt that represent two components. We use the same weight for parameters related to each component. In order to practically implement this loss function, we replace the standard deviation $\sigma$ of normal distribution with $\exp(w)$ that ensures the learned uncertainty is represented by a positive value. This results in the following final loss function.

$x_{t}$ and $x_{r}$ are predicted translation and rotation variables. $\hat{x_{t}}$ and $\hat{x_{r}}$ define the ground-truth. $w_{r}$ and $w_{t}$ are the trainable weighting parameters, and $l_{w}$ defines the final loss.

In this section we evaluate the performance of the hierarchical model acronymed as 8k4k_comp. Figure 5 shows the trajectories of various training iterations and Figure 6 presents their error metrics. As expected, estimates of the second iteration are far superior to the first iteration. The third iteration reduces the error further but the reduction rate is smaller than the previous iteration. This is due to the fact that after certain point, the learning process will plateau as the amount of information to learn is much smaller. The trend is clearly visible in RTE and RRE metrics.

Another way to increase the accuracy of odometry is by employing temporal features. However, training the frame-to-frame estimation model requires a significant amount of system resources. Adding LSTM on top of that model will dramatically increase these requirements. In such cases, it is common to use a pre-trained feature extraction network and provide mid-level features as inputs to an LSTM model. Following this idea, we evaluate three features maps taken from various layers of our trained model.

• •

  Feature maps immediately before the flow embedding layer (PREF).

• •

  Feature maps immediately after the flow embedding layer (POSF).

• •

  Feature maps prior to the fully connected layers in the feature extraction network (FCLF).

PREF is the earliest level of features among the three. This model requires the largest temporal model as the flow embedding and the feature extraction layers from the original model are retrained in temporal model.

To reduce the complexity further, POSF features are used. This is achieved by employing features after the flow embedding layer. However, a point-cloud feature extractor needs to be present in the LSTM.

Finally, FCLF is the smallest feature map. Most of the flow matching and feature extraction work is completed and only the fully connected model is left for the LSTM model.

FCLF features are outperforming both PREF and POSF features. This is the objective that is used to train the frame-to-frame registration model. FCLF mimics the registration model rather than learning temporal features as the provided features are much less informative compared to PREF and POSF.

Figure 7 shows the results of these comparisons. It is seen that the usage of mid-level features is not as fruitful as using a second iteration. In some cases the results are worse than the original feature extraction model that is all attributed to the dividing the training process into two parts.

In this group of experiments, we compare the proposed model to the state-of-the-art. Table 1 compares various models in terms of sequence translation drift percentage and mean sequence rotation error for lengths of $[100,800]\mathrm{m}$. Please note that the results for LOAM [Zhang and Singh, 2014], $\mathrm{ICP}_{\mathrm{reported}}$ are taken from [Li et al., 2019].

LOAM [Zhang and Singh, 2014] is one of the benchmarks in this field. It clearly outperforms all of the deep methods in the comparison. One main reason for that is the feature extraction back-bone network. In our work, we relied on PointNet++ [Qi et al., 2017b] features. Both LO-Net [Li et al., 2019] and DeepLO [Cho et al., 2019] use 2D depth and surface models such as vertex and normal representations. This way, they completely avoid the usage of 3D data for feature extraction. This trend is also visible across the field as 2D feature extraction models are more advanced than their 3D counterparts. However, there is a growing interest regarding the 3D feature extraction methods [Wang et al., 2019][Wu et al., 2019] that can enhance the performance of deep odometry estimation with 3D point clouds.

It is clearly seen that DeepLO [Cho et al., 2019] is over-fitting to the training set. Our model is also suffering from such a phenomenon, but the scale of over-fitting is much lower. One reason for this is the size of our model compared to the size of the input dataset. Using PointNet++ layers results in large networks that require a large amount of data. LO-Net [Li et al., 2019] addresses this problem by utilizing a 2D-based feature extraction network [Zhou et al., 2017]. We use an implementation of the ICP algorithm from the Open3D library¹¹1http://www.open3d.org/. In our implementation we use the sub-sampled point-clouds. This results in a significant drop in performance in comparison to the results of $\mathrm{ICP}_{\mathrm{reported}}$ [Li et al., 2019] that use the full point cloud. However, using the full point cloud data entails a large computational complexity burden. The sub-sampling stage that is used to reduce the computational complexity is another aspect that affects the estimation performance of our model. The same points are not always chosen to represent the same static objects. This inherently adds noise to our dataset. We further explore using ICP as a final step on our estimates that significantly improves the performance. This is an expected outcome, as the complexity of the residual problem to solve for ICP at this stage is less than the original one. Hence, it can easily find the corresponding points and estimate better registration parameters.

We evaluate the performance of the models with $12k$ and $6k$ input points. 8k4k represents the 8k and 4k division between non-ground and ground points. Similarly, 4k2k corresponds to 4k ground and 2k non-ground point sampling. Extra points only help in providing better descriptors at the first layer where the first sampling function in the network is called. This results in better performance of the model, especially in the first iteration where the disparity between matching points in two frames is much larger. However, the difference diminishes in the second and third iterations. This entails that by employing a hierarchical model we could reduce the complexity of the input point cloud by using coarser 3D point clouds. Results of this comparison are shown in Figure 8.

To better understand the importance of separately sampling ground and non-ground points, we train the same model (6k_comp) with $6k$ globally sampled points from the point cloud. We train this model in hierarchical manner and compare it to the 4k2k_comp model that employs 4k non-ground and 2k ground input points. As it is shown in Figure 9, sampling without distinction between ground and non-ground points results in far worse performance. This is due the large number of ground points in the point cloud that provide much less information regarding translation and orientation of the sensor in comparison to the non-ground points.

The proposed model employs weighted l2_norm loss on Euler angles and translation parameters. The loss function utilizing 2 weights for each rotation and translation components is indicated with comp, while the indiv represents the usage of individual weights for each transformation parameter.

Figure 10 shows comparative results of this experiment. We observe that in the first two iterations, component based weighting provides better results. However, in the third iteration, individual weighting achieves comparable results to the component based function. It is worth mentioning that the scale of reduction in error between iteration 2 and 3 is small, and the majority of the error reduction is achieved in the first 2 iterations.

Normalized Euler angles are used as the primary labels along with normalized translation parameters in our experiments. Normalization is utilized in order to remove the scaling effects for various parameters in calculation of the L2-norm. To validate our decision, we compare our choice of label representation to the transformation matrix representation with 12 parameters ($3\times 3$ rotation and $3$ translation). We employ the component-based weighting on rotation and translation components of this representation. The trained model is shown as 4k2k_tmat_comp in Figure 9. Results show that normalized Euler angles are a much better representation than the $3\times 3$ rotation matrix, which is an over-parameterized representation of the rotation.