Enhancing Vectorized Map Perception with Historical Rasterized Maps

At early stages, map perception mainly focuses on lane detection [7, 40, 16], road topology reasoning [2, 3, 39, 24, 21] or map segmentation [34, 52, 10], which are mainly constructing rasterized maps and need post-processing to produce desired vectorized map elements for downstream applications. For example, predicted rasterized maps are clustered to acquire final vectorized maps in HDMapNet [20].

VectorMapNet [27] and MapTR [25] are pioneer works to predict vectorized map elements directly. VectorMapNet designs a map element detector and a polyline generator to produce final vectorized maps. MapTR proposes a unified permutation-equivalent modelling and utilizes the DETR [4] paradigm to predict vectorized map elements directly. Following these breakthroughs, vectorized map perception becomes popular in autonomous driving research, leading to the development of many methods for improving performance. The evolved version MapTRv2 [26] adds decoupled self-attention in the decoder and auxiliary losses, improving the accuracy largely. ScalableMap [47] exploits the structural property of map elements and designs a progressive decoder for long-range perception. MapVR [50] introduces differentiable rasterization and rending-based loss for superior sensitivity. Furthermore, instead of simple point set representation, BeMapNet [32] predicts Bézier control points and PivotNet [6] predicts pivot points instead of a fixed number of points for accurate results.

The above methods predict map elements using a single frame, which limits further improvements. Recent advancements have expanded beyond single-frame perception, incorporating complementary information. For instance, extra standard-definition (SD) maps are explored to help HD map perception [18] and lane-topology understanding [29]. Satellite maps are used to augment onboard camera data for map perception in [9]. These methods require extra data for online perception and thus increase cost in practical applications.

Temporal information is a more accessible complement for online perception. It has been widely used in BEV feature learning [22] and object detection [41]. For vectorized map perception, StreamMapNet [48] leverages temporal information through query propagation and BEV feature fusion. SQD-MapNet [42] introduces stream query denoising to facilitate temporal consistency. In these methods, short-term previous frames in temporal are utilized for perception.

One step further, if all temporal information is collected and utilized, a map can be constructed. In [46], a map of past LiDAR scans is utilized for object detection in autonomous driving. NMP [45] constructs a map of past BEV features for map segmentation. But BEV features take substantial memory footprint, which limits its practical usage. Take the Boston map in nuScenes dataset [1] as an example, BEV features require over 11 GB memory in NMP [45]. By contrast, we propose to maintain a low-cost historical rasterized map for vectorized map perception, which takes only 120 MB memory for the same Boston map.

Our proposed HRMapNet is designed as a complement to SOTA online vectorized map perception methods. As illustrated in Fig. 2, HRMapNet maintains a global historical rasterized map (described in Sec. 3.2) to aid online perception. Given surrounding images as input, 2D features are extracted from a shared backbone and transformed to BEV space. We introduce a map encoder and feature aggregation module (described in Sec. 3.3) to obtain enhanced BEV features from both onboard cameras and retrieved local maps. Additionally, we also design a novel query initialization module (described in Sec. 3.4), working before the original map decoder. This module aims to endow base queries with prior information from local maps, enabling more efficient search for desirable map elements. Finally, vectorized map elements are predicted directly from the prediction head and can be rasterized to merge into the global map.

We firstly introduce the rasterized map used to save historical information. As illustrated in the bottom left corner of Fig. 2, vectorized maps are rasterized to update a global map. Here we adopt the rasterization method used in [19]. Briefly, the label of each pixel in the rasterized map is determined based on its distance to vectorized elements’ boundary. From each online prediction at $i$-th timestamp, we can obtain a semantic mask, referred as local rasterized map $M^{l}_{i}\in\{0,1\}^{H\times{W}\times{N}}$, where $H$ and $W$ denote the spatial shape of the perception range in BEV space; $N$ is the number of map element categories and $N=3$ as in previous methods [25, 48], representing lane divider, pedestrian crossing and road boundary, respectively. Therefore, $M^{l}_{i}(p)\in\{0,1\}^{1\times{N}}$ indicates whether and what map elements exist at position $p=(x,y)$ for each category; the value 1 indicates existence and the value 0 indicates non-existence.

Such rasterized map we utilize is analogous to occupancy grid map [30], a well-established concept in robot navigation and mapping [11, 12]. Occupancy grid mapping [37] is extensively studied for updating a global map from local observations. We use a similar method to update the global rasterized map $M^{g}\in\mathbb{R}^{H^{g}\times{W^{g}}\times{N}}$, where $H^{g}$ and $W^{g}$ denote the spatial shape of the global map. For map updating, the local coordinate $p$ of each pixel of $M^{l}_{i}$ is firstly transformed to the global coordinate $p^{g}$ based on the ego-pose $T_{i}=\left[R_{i},t_{i}\right]$:

where $R_{i}$ and $t_{i}$ are relative rotation and translation, and $\operatorname{round}(\cdot)$ denotes the rounding function which converts a continuous coordinate to a discrete coordinate in the rasterized map. Alternatively, given global coordinate $p^{g}$ and pose $[R_{i},t_{i}]$, its corresponding local coordinate $p$ is

Then the global map $M^{g}$ can be updated based on the local map $M^{l}_{i}$ for each category. Take one category for example:

where $p$ is determined by Eq. 2, $M^{g}$ records the status for each map element category, $S^{+}$ and $S^{-}$ are defined values to update status based on local prediction results. This simple method integrates local results into a global map efficiently, facilitating continuous refinement and updating. For online perception, a local rasterized map is retrieved from the global map based on the ego-pose $T_{i}$, and a threshold $S_{th}$ is used to determine whether map elements exist for each category:

where $p^{g}$ is determined by Eq. (1).

In our implementation, the global map $M^{g}$ is scaled to 8-bit unsigned int values to reduce memory consumption. As a result, a rasterized map consumes only a small memory footprint, about 1 MB per kilometer.

Various methods [31, 15, 22, 28, 23] have been proposed to transform features from perspective view to BEV space, serving as a fundamental module in map perception. For example, MapTRv2 [26] utilizes BEVPoolv2 [15] to acquire BEV features $F_{I}\in\mathbb{R}^{H\times{W}\times{C}}$, where $C$ is the number of feature channels. We keep this module unchanged and introduce an aggregation module to enhance BEV features with prior information from local maps.

In HRMapNet, the retrieved local map $M^{l}$ serves as priors indicating where deserves more attention for map element perception. Therefore, inspired by FB-BEV [23], we add additional BEV queries at locations where map elements exist in the local map (i.e., $M^{l}(p)\neq\mathbf{0}$). These additional BEV queries are projected onto images to extract relevant features through spatial cross-attention [22]. Then, additional BEV features are acquired where map elements exist in the local map. For the locations where no map elements exist (i.e., $M^{l}(p)=\mathbf{0}$), corresponding additional BEV features are set as zeros. These additional BEV features are formulated as $F_{M}\in\mathbb{R}^{H\times{W}\times{C}}$.

In the feature aggregation module, $F_{I}$, $F_{M}$ and $M^{l}$ are fused together:

Here, $F_{M}$ is added to $F_{I}$ to compensate for deficiencies. Additionally, $M^{l}$ can be regarded as special BEV features with clear semantic information. Thus we concatenate it with BEV features and use convolution to acquire the enhanced BEV features $F_{BEV}\in\mathbb{R}^{H\times{W}\times{C}}$ for further processing.

In addition to fusing the retrieved rasterized map into BEV features, a novel query initialization module is designed to facilitate efficient search for desirable map elements. Within a DETR [4] paradigm, a set of learnable queries will interact with extracted features to search for desirable elements. Without prior information, queries would search from random and prediction results are refined gradually through several decoder layers.

In HRMapNet, the retrieved rasterized map provides prior information about where map elements may exist and thus can facilitate map element queries search for desirable elements efficiently. As illustrated on the right of Fig. 2, the proposed query initialization works before the original map decoder. Base queries will firstly interact with prior features embedded from the local map.

In detail, for a valid location where map elements exist in the retrieved local map $M^{l}$, its position $p$ is related to a learnable position embedding $PE(p)\in\mathbb{R}^{1\times{C}}$; and the semantic vector $M^{l}(p)$ is projected to a label embedding $LE(p)\in\mathbb{R}^{1\times{C}}$ using a linear projection. Then, we acquire a map prior embedding for each valid position:

Map prior embedding $ME(p)\in\mathbb{R}^{1\times{C}}$ encodes where map elements may exist based on the retrieved local map. To fuse priors, base queries interact with a set of map prior embeddings through cross-attention [38]. Then, initialized queries search desirable elements in BEV features through original decoder layers. Moreover, to improve efficiency and save memory consumption, the retrieved local rasterized map $M^{l}$ is downsampled before extracting map prior embeddings.

Training. The prediction head and training loss remain identical to SOTA vectorized map perception methods. Taking MapTRv2 [26] as an example, the prediction head predicts a class score and sequential 2D point positions for each element. Classification loss, point-to-point loss and edge direction loss are used for training; one-to-many loss [17], dense prediction loss and depth loss are used as auxiliary supervision. In each training epoch, the global map is set from empty and updated gradually from prediction results.

Each map element is modelled as 20 sequential points. The perception range is set as [-30m, 30m] from rear to front and [-15m, 15m] from left to right, and the resolution of BEV features is 0.3m$\times$0.3m. The shape of a local rasterized map is set the same as BEV features; the resolution of both local and global rasterized maps is also 0.3m$\times$0.3m. We set $S^{+}$ as 30 and $S^{-}$ as 1 to update the global map.

Our model is trained on 8 NVIDIA A100 GPUs with a batch size of $8\times 4$, the learning rate is set to $6\times 10^{-4}$. ResNet50 [14] is used as the backbone to extract image features.

In comparison to methods introducing complementary information, HRMapNet also stands out for its comprehensive utilization of all historical information. Besides onboard images, P-MapNet adds extra SDMap from OpenStreetMap [13], but the improvement is lower than ours. SQD-MapNet is a concurrent work which leverages stream query denoising strategy to benefit from the results of previous frames. Since all predicted results are preserved in a global rasterized map, HRMapNet leverages not only temporal information but all past results for online perception, and thus achieves superior performance.

Moreover, HRMapNet maintains high efficiency in terms of inference speed when integrated with StreamMapNet and MapTRv2, running at 21.1 FPS and 17.0 FPS, respectively. This ensures HRMapNet not only enhances performance but also remains practical for real-time applications in autonomous driving.

In this subsection, we provide some ablation studies of our method, which are conducted on nuScenes with 24 epochs and use MapTRv2 [26] as the baseline.

To further demonstrate that the proposed query initialization helps search for map elements more efficiently, an ablation study of decreasing decoder layers is provided in the supplementary material.

The results clearly demonstrate the robustness of HRMapNet to localization errors, particularly in terms of translation. One contributing factor is the relatively small map resolution (0.3m) utilized in our method. Despite varying levels of localization errors, HRMapNet consistently achieves comparable results, experiencing only a 1 mAP drop in most cases. Even in the worst case with the largest noise, a historical map still brings benefits (63.8 mAP) compared to the baseline, MapTRv2 (61.5 mAP).

Considering localization with 0.1 m error for translation and 0.01 rad error for rotation is a common requirement in autonomous driving [33], these extra results indicate the effectiveness of HRMapNet in practical usage.

Here, we provide extra results with pre-built initial maps in Tab. 7. The model used here is the same as in Tab. 1, integrated with MapTRv2 and trained 24 epochs. Note that the model is not re-trained or finetuned, and we only test it with different initial maps. Here are two kinds of maps can be provided.

For the “validation map”, the same model is tested with validation data twice. The first time is running with an empty initial map. The map is updated gradually as validation data comes in and the final global map is saved. This global map is loaded for the second validation. There is actually no extra data input, the model constructs a global map by itself and use it again for validation. With the help of this more complete map, the performance of the same model is further enhanced by +5.4 mAP.

Besides, the global map constructed during training is saved and loaded again for validation. As stated in Tab. 3, there are overlaps in location between training and validation data. Thus this training map can also benefit online perception for validation. Because this training map is more accurate, the performance is improved largely by +16.5 mAP.

We provide these extra results to show the potential of HRMapNet for practical usage in autonomous driving, including crowdsourcing online map perception. Provided with an easily maintained global map, which may even be constructed by other vehicles, the performance of online vectorized map perception can be improved largely.

In Fig. 3, we show some qualitative comparisons in three challenging scenarios: severe occlusion, rainy day and poor lighting at night. The online map perception relying only on onboard sensors can be easily affected by these inevitable factors. Leveraging a historical rasterized map, HRMapNet helps to improve online map perception ability to handle such challenges. More visualized results and analysis are included in the supplementary material.