Non-parametric Memory for Spatio-Temporal Segmentation
of Construction Zones for Self-Driving

Recent semantic segmentation approaches successfully exploit deep convolutional neural nets  [4, 5].

End-to-end trainable structured prediction techniques model dependencies between the predictions at multiple pixels  [6, 7, 8]. We draw inspiration from this to refine our segmentation results by reasoning about spatio-temporally related 3D points.

Initial attempts for 3D point cloud segmentation  [9, 10] used clustering methods to identify individual objects. Convolutional networks using a layered 2D representation for the point cloud in bird’s-eye view have also been proposed [11], where the sparsity of the data can be further exploited  [12]. Additionally, voxelization of point clouds is popular. Reductions to computational cost by casting voxel segmentation as 2D convolutions [13] or improved memory efficiency  [14] have been proposed. Some recent works directly learn convolutional networks over raw point clouds for segmentation. The pioneering work [15] aggregates global and point-wise deep features to predict class-wise labels. It was later extended to a hierarchical aggregation framework [16]. An alternative approach is to convert point coordinates to an intermediate space in which convolution operators are possible such as permutohedral lattice [17], tangent space [18] or over graphs [19].  [20] aggregates information across different spatial axis of a single observation using a recurrent structure. Recently, parametric continuous convolutions [21] were shown to significantly improve over the state-of-the-art by reasoning directly over neighborhood graphs constructed from points with their natural coordinates in continuous domain. However, these techniques use single observations, while our proposed method uses continuous convolutions and occlusion reasoning to fuse information in the memory with current observations to directly merge accumulated past beliefs with new observations at each time step.

Due to the sparse nature of LiDAR point clouds, temporal reasoning by accumulating multiple observations over multiple timesteps have shown to be more robust [22, 23]. Exploiting all three cues (spatial, semantic, and temporal) within a Bayesian framework, Held et al.  [24] demonstrated higher performance gains. Tokmakov et al.  [25] utilized a recurrent convolutional unit to realize a visual memory in image space, where the end result is used for segmenting objects in a video stream.

The majority of the earlier work in sensor fusion focused on early fusion where depth data is used as an additional channel in the 2D image space [26, 27, 28]. While early fusion appears promising in indoor scenes with high resolution depth sensors, LiDAR point clouds are very sparse, reducing the efficacy of early fusion frameworks. More recent end-to-end approaches exploit fusion at multiple levels downstream in the network architecture where the point cloud is rasterized into a birds-eye view  [29, 30, 31].  [32] fused a bird’s-eye view projection of a camera image with the top down view of the point cloud for lane detection. Qi et al.  [33] use a late fusion strategy with a two-stage pipeline where 2D detections in the image space is used as a hard prior in the form of a frustum to further perform binary segmentation of the foreground object in the point cloud. Our method leverages both camera and LiDAR data. However, contrary to the aforementioned approaches, we exploit an external memory which results in robust modelling of static objects that allows us to explicitly remember existing objects rather than re-discovering them with the next observation.

Building dynamically changing semantic maps is a active topic in robotics. It is crucial for the robot to complete many tasks in a dynamic evolving environment, such as navigation [1], localization [3] and manipulation [34]. There is a large body of work in semantic 3D reconstruction using RGB-D data for indoor scenes [35, 36, 37]. Cameras and LiDAR have been also exploited to build semantics maps from urban environment [1, 2, 3]. Most of these methods use a pre-defined update rule to fuse online perceived information with past estimates. However, the optimal trade-off between these two sources in a dynamic and noisy scene is difficult to encode. On the other hand, we use a learned model to aggregate information that takes into account factors that influence noise and occlusion. This allows us to handle cases with mis-alignment, noisy perceived semantics, distant small objects and dynamic changes in a smarter manner.

The majority of approaches utilizing memory rely on recurrent neural networks based on a long short-term memory (LSTM)  [25, 38], where the memory is represented as a 1D-vector. More recently,  [39, 40] have shown techniques using ConvLSTMs to explicitly remember spatial information. Such convolutional approaches to LSTM encoding exploits the spatial relationships in the image space. However, they do not explicitly encode the 3D structure. Explicit external memory with differentiable write and read operations [41] have been exploited as an external data structure in settings where a direct mapping between the memory and physical quantities is missing. In contrast, our memory representation is an explicitly external 4D spatio-temporal buffer that both has a direct mapping to the real world in a local coordinate frame, as well as being compatible with the continuous convolutions [21] for sparse computation in that space. Our memory is essentially a graph of ”points”, which is a different memory representation than existing approaches which rely more on canvases.

We select the construction elements and traffic sign classes as the focus of our technique due to their importance for autonomous driving and relative difficulty for traditional approaches arising from their small physical size. Human labelling efforts of point-wise LiDAR point clouds is expensive, as complex and sparse 3D points are difficult to visualize and select. Instead, we employ the same 2D semantic segmentation model and aggregation for automatic ground truth generation. For this, we train the image semantic segmentation model on a dataset of 10k training, 1.3k validation, and 1.3k testing images with pixel-wise semantic segmentation annotation collected in various North American cities and highways. The model is trained on 4 NVIDIA Titan Xp GPUs with ADAM and a learning rate of $1e-5$ until convergence at 240k iterations. This method achieves 64.1% and 63.3% pixel-wise intersection over union on the traffic sign and construction elements classes in the validation set, respectively.

Our joint LiDAR and camera image dataset consists of $3141$ training, $430$ validation, and $606$ testing sequences of LiDAR sweeps and associated RGB front camera images. Each sequence is defined by a key frame, and contains frames sampled at 0.5 meters of vehicle displacement (or 0.1 seconds if the vehicle moves farther between frames than 0.5 meters). To enable training of the memory system, the sequence includes frames prior to the key frame sampled over a displacement of 30 meters. The locations of the 3D points are expressed in a globally consistent coordinate system using an online pose estimation system.

We extend the data sequence with observations for 100 meters with the aforementioned sampling strategy. For each frame, the LiDAR points are labeled with the 2D semantic segmentation result. To reduce noise, we select only points within 25 meters of the vehicle. By nature, construction cones and signs are largely isolated objects that are easily separated from its environment in 3D. Thus, we use the density-based spatial clustering [44] as a denoising step over the initial point-wise semantic labels. Following this, we produce a training sample / ground truth pair by using only the sequence of frames over the 30 meters preceding the key frame, while removing the distance filter. We use the denoised ground truth results to label this set with k-nearest neighbor search for points that are measured at beyond 25m. We observe in the second row of Fig 5 that this method is able to provide us with highly accurate ground truth segmentations.

We estimate the sample mean and variances for the $C+3=6$ point-wise features and normalize them relative to each other. This is used to standardize the features, which are then used as input to the continuous convolution model. The model parameters are randomly initialized. At the output, we minimize the average of the standard cross-entropy loss at each point.

We train our model with a batch size of 1 and the ADAM [45] optimizer with a learning rate of $1e-3$ on a single NVIDIA Titan Xp GPU until convergence as determined by the validation set at 28k iterations. We then fix the batch normalization statistics, and further finetune the model at a learning rate of $1e-4$ for an additional 64k iterations until convergence.

We use standard point-wise classification intersection-over-union (IoU) for evaluating our method and the comparison. Moreover, we examine the performance differences between the baselines and our method at different distances from the vehicle. As this model is to be used in the online driving setting, we evaluate over only the LiDAR points that fall within the forward camera view at the final timestep. This has the effect of increasing the difficulty, as the elements in this region have less accumulated measurements than elements next to and behind the vehicle.

We analyze the performance of our model with two baselines. The first baseline consists of using the same 2D semantic segmentation model to label the 3D points projected into the image, which we call the Image Baseline. Additionally, we implement and train one of the recent fast point cloud segmentation techniques [43] according to the settings described in the paper. We refer to this technique as Voxel NN. This model uses 2D convolutions on a simple occupancy grid representation to produce semantic segmentation predictions for 3D point clouds. Specifically, they discretize a given point cloud into a $W\times H\times Z$ voxel grid, and compute predictions for each voxel using a 2D U-Net model by treating the $Z$-dimension as the feature channel. Per-point predictions are then recovered via nearest neighbour interpolation.

We also present results for a model where our memory approach is removed from the continuous convolution framework. This model uses a single sweep of the LiDAR point cloud and the corresponding image segmentation. We train this model in a similar fashion as our proposed model, with the exception that a single frame is used as an example instead of an accumulated sequence.

Table I show the validation and test dataset IoU metrics for traffic signs, construction elements and the mean for all models including the baselines, and the single sweep continuous convolution baseline. We observe that our continuous convolution approach outperforms results from other 3D processing methods [43]. Moreover, the use of the non-parametric memory presented in our paper consistently improves the results across the validation and test sets both for the traffic signs and the construction elements.

We further dissect the results by slicing them into object bins according to distance from the vehicle, as shown in Figure 4 for the test set. It is evident that the segmentation performance at larger distance is lower, as the image segmentation is noisier while the LiDAR sweeps return fewer points. These results also show that our method consistently performs better compared to the previous work, the image baseline, as well as a version of our method without the memory. The significant performance gap between our method and the baselines highlights the effectiveness of our model in appropriately aggregating noisy spatio-temporal information especially at range.

Fig 5 shows the qualitative comparisons on the validation set across the baseline methods, single LiDAR sweep ablation study, and our proposed method. Note that the automatic ground truth generation procedure described above is able to generate accurate ground truth point-wise semantic segmentation for the classes of interest. Additionally, we observe that the single LiDAR sweep continuous convolution method without the non-parametric memory tends to miss small elements that are far away, for example the distant traffic sign on the left in first column.

In comparison with the baseline methods, our model achieves much better performance. The Voxel NN  [43] model’s output is far noisier, and often overestimates the extent of the foreground objects. This is likely due to the limited resolution after discretization of the voxel-based model, which is necessary for the model to avoid exceeding memory and processing constraints. The noisy object boundaries can have a significant impact on the autonomous vehicle’s motion planning systems. In contrast, the continuous convolutions operation avoids the decrease in resolution by directly operating on the 3D points, and produces very accurate foreground extents.

Finally, the baseline method where the image semantic segmentation output is directly projected onto the LiDAR sweeps often results in trails of foreground labels on background points at occlusion boundaries. The construction signs in the first column and the closest cone in the third column show this issue most prominently. Again, the effect on autonomous vehicle operation can be significant, as drivable regions would be labelled as obstacles. In contrast, our method is able to remove these artefacts.

In our experiments, we used a relatively large and unoptimized image semantic segmentation technique with a runtime of approximately 400ms on a single GPU. The continuous convolution over the non-parametric memory module generally runs in under 20ms, with the exact runtime dependent on the number of foreground points in a scene. Therefore, with an optimized semantic segmentation method that runs at  130ms, our method will be running at 150ms end-to-end.

One area of potential future improvement is as follows. If there are false negatives in the instance segmentation model, the corresponding LiDAR points will not be passed to the continuous convolution model and hence be absent in the foreground estimation. Additionally, there may be elements of interest outside of the LiDAR return range, which would not be registered. Finally, the proposed method’s reliance on the image segmenter suggests that its performance may be limited in the dark.