OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird’s-eye-view Vehicle Semantic Segmentation

In recent years, bird’s eye view (BEV) perception algorithms have made a significant contribution in the fields of intelligent transportation and autonomous driving. BEV semantic segmentation is critical as it provides essential local maps in today’s intelligent traffic systems, where perception is prioritized over High-definition (HD) mapping.

Camerea-based BEV Perception Methods. VPN [21] is one of the pioneering approaches in cross-view semantic segmentation. It introduced the View Parsing Network in simulated 3D environments, utilizing multiple MLP layers to regress the transformation matrix from PV to BEV. Based on VPN, PON [22] introduced a semantic Bayesian occupancy grid to establish map relationships across multiple frames. It also utilized a feature pyramid to extract multi-resolution features. CVT [7] is a Transformer-based method that utilizes cross-attention to perform the conversion from PV space to BEV space. BEVFormer [6] and BEVSegFormer [23] leverage deformable attention in a DETR-style [15] approach to enhance view transformation and feature representation. BEVFormer makes full use of spatiotemporal information by introducing a spatial cross-attention and temporal self-attention to aggregate spatial and temporal features. BEVSegFormer, by contrast, focuses on flexibly handling BEV segmentation tasks under arbitrary (single or multiple) cameras without requiring intrinsic and extrinsic camera parameters.

Multimodal Perception Methods. Cameras and radars are two of the most common sensors in the field of autonomous driving. Radar point cloud features typically contain information that complements camera data. Fishing Net [24] employs the same MLP as VPN for view conversion and extends BEV segmentation through posterior multimodal fusion using radar and LiDAR features. CRN [25] utilizes a well-designed multimodal fusion module and proposes a two-stage fusion method, which employs deformable attention to integrate camera and radar data effectively. In contrast to CRN, simpleBEV [10] proposes a lifting method that does not rely on depth estimation. It highlights that batch size and input resolution significantly impact BEV segmentation performance. The study also demonstrates that using a simple method to concatenate camera and rasterized radar data can effectively enhance segmentation performance.

State Space Model (SSM) have shown promising performance in long sequence modeling for its linear-time inference, parallelizable training, and robust performance. Unlike CNNs and Transformers, State Space Model (SSM) is inspired by traditional control theory, which maps input sequences to state representations. Structured State Space for Sequences (S4) [20] effectively handles discretized data by utilizing the zero-order hold method. The use of Highest Polynomial Powered Operator (HiPPO) [26] initialization significantly enhances the long-range modeling capabilities of the S4 model. However, S4 cannot adaptively adjust based on the input, meaning it cannot selectively focus on important parts of the input. Mamba incorporates selective information processing, enabling dynamic handling of input data. This allows the model to selectively remember or ignore different parts of the input, thereby improving the utilization of historical information.

Recent works, VMamba [27] and Vim [28], introduce the Mamba model to computer vision tasks. VMamba proposes a Cross-Scan Module to conduct selective scan in four different directions, which allows the model to maintain a global receptive field while achieving linear computational complexity. Vim introduces a Bidirectional SSM that performs global modeling in both forward and backward directions. [29, 30, 31], demonstrate the effectiveness of the Mamba model in medical and remote sensing image segmentation. Inspired by these works, we introduce Mamba into BEV semantic segmentation task to enhance the model’s global perception capabilities to the long-distance surrounding enviroments of the BEV space.

In autonomous driving task, BEV perception involve segmenting by projecting the vehicle’s surrounding environment into a bird’s-eye view. As illustrated in Fig.2, the inputs typically consist of images $I_{S}=\left\{I_{c,1},I_{c,2}...,I_{c,k}\right\},I_{S}\in\mathbb{R}^{C\times H\times W}$ from multiple cameras and radar point cloud $P\in\mathbb{R}^{N\times Z\times X}$ , where $k$ and $N$ denote the number of cameras and the channels of the radar data, respectively. $X$, $Y$, $Z$ are used to represent the left-right, up-down, and front-back axes, respectively. After the PV sapce feature $\left\{F_{p,k}\right\}_{k=1}^{K}\in\mathbb{R}^{d\times h\times w}$ is extracted by the encoder, it is lifted to 3D space by a parameter free bilinear sampling method[10]. Then $F_{3D}\in\mathbb{R}^{d\times Z\times Y\times X}$ is aggregated to get the BEV feature $F_{B}\in\mathbb{R}^{d\ast Y,Z,X}$, where $d$ is the dimension of the PV sapce feature. The 3D perception range represented by $F_{B}$ is $100m\times 100m$ in the $Z$ and $X$ axes, with a corresponding resolution of $200\times 200$. The up/down span of $Y$ is set to $10m$, with a resolution of 8. The fusion of multimodal features $F_{B}$ and $F_{P}$ is achieved in BEV space to obtain the fused features $F_{f}\in\mathbb{R}^{(d*Y+N)\times Z\times X}$. In this paper, we propose the Environment-aware BEV Compressor (EBC) and Center-Informed Object Enhancement (CIOE) modules to enhance the model’s environment perception ability and enrich the detail target object information. Finally, the Decoder performs upsampling and includes a multi-task head designed for segmentation and predicting centerness and offsets, which outputs the final segmentation mask $M\in\left\{0,1\right\}^{1\times H\times W}$.

The overall architecture of our proposed OE-BevSeg illustrated in Fig.3. To capture rich contextual information, we utilized a large backbone VOVNetV2 followed by [32] as the encoder for PV perspective images, and the first three layers of a small backbone ResNet18 [33] for upsampling. The input image $\left\{I_{in,k}\right\}_{k}^{S}\in\mathbb{R}^{B*S\times C_{1}\times H\times W}$ first passes through the stem, generating features with $1/4$ width and height of the $\left\{I_{in,k}\right\}_{k}^{S}$. $B,S,C_{1}$ represent the batchsize, camera number, input channel. The stem is composed of three layers of $3\times 3$ convolutions with strides of $2,1,$ and $2$. The input channels of each stage $\left\{C_{i}\right\}_{i=0}^{3}=\left\{256,512,768\right\}$. OSA block number of each stage $\left\{B_{i}\right\}_{i=0}^{3}=\left\{1,3,9\right\}$. The One-Shot Aggregation (OSA) block employs the principle of dense interaction, aggregating the outputs of all previous layers in one step at the end to capture rich multi-scale texture information. The eSE Module is a crucial component of the OSA block, similar to the SENet [34] structure, which can further enhance segmentation performance. The eSE Module can be represented as follows:

Where $\mathcal{A}$ represents average pooling, $FC$ denotes fully connected layer, and $\delta$ refers to the hsigmod operation, which specifically consists of ReLU6. $\mathcal{W}_{c}$ represents learnable channel weights, and $\otimes$ indicates element-wise multiplication. $F_{in}$ and $F_{out}$ represent the input features and output features of the OSA block, respectively.

In the Decoder part, we use simple ResNet layers $\left\{L_{i}\right\}_{i=1}^{n}$, along with bilinear upsampling $\left\{Upsample_{i}\right\}_{i=1}^{n},n=3$ and cross-layer skip connections to map the features to the output mask. By utilizing this layer-by-layer summative skip connection method, more multi-scale detail information is preserved during the upsampling process. The upsampling process can be summarized in the following equation:

where $\left\{U_{F,i}\right\}_{i=0}^{2}$ denotes the feature after upsampling in $L_{i}$, and $\left\{S_{F,i}\right\}_{i=0}^{2}$ denotes the skip connection feature map in $L_{i}$. Upsampling starts at the $L_{3}$ and proceeds sequentially forward, ultimately generating an upsampled feature map $U_{F,0}$.

Bird’s-Eye View (BEV) features encapsulate extensive environmental semantics, which significantly influences a model’s ability to differentiate between foreground and background elements. Previous research lacks further processing of BEV features after completing the perspective transformation. Inspired by the powerful long sequences modeling and computationally efficient State Space Model (SSM), we employed the recently prominent Mamba to model the global context of the environment in the BEV space.

With full consideration of the characteristics of the BEV perspective, we find that as the distance from the ego-vehicle increases, the surrounding environment sampled by the BEV features will change significantly. Specifically, at the closest distance to the ego-vehicle, such as $\operatorname{0-20m}$, the main component of the environment is the road surface. At a further distance of $\operatorname{20-35m}$, the main component of the environment is usually buildings. At $\operatorname{35-50m}$, the main component in the environment from multiple camera views should be the sky-related elements. Experiments in distance-phase segmentation [17, 7] have also demonstrated this point, the segmentation results vary significantly at different distances from the ego-vehicle. Therefore, if only global features are learned while ignoring the characteristics of the driving environment, the model cannot effectively understand and perceive the BEV space. Based on the aforementioned prior knowledge, we have improved Mamba by proposing the Bi-Surround Scan mechanism to serialize the BEV features, enhancing the model’s environment-aware capabilities.

State Space Model (SSM). State Space Model originates from control theory and represents a method of mathematically describing a problem using the minimal number of variables that fully describe the linear time-invariant system, i.e., $x(t)\in\mathbb{R}\longmapsto y(t)\in\mathbb{R}$. In continuous time $t$, the current hidden state $h_{t}\in\mathbb{R}^{N}$ is calculated using the previous hidden state $h_{t-1}\in\mathbb{R}^{N}$ and the current input $x_{t}\in\mathbb{R}$. $y_{t}\in\mathbb{R}$ is the prediction of $h_{t}\in\mathbb{R}^{N}$. Specifically, the model can be represented by the following ordinary differential equations:

Where $A\in\mathbb{C}_{N\times N}$ is the evolution parameter, $B\in\mathbb{C}^{N}$, $C\in\mathbb{C}^{N}$ and $D\in\mathbb{C}^{1}$ are the weighting parameters.

Discretization of SSM. The above SSMs cannot handle discrete data. To be applicable in deep learning, the SSMs of the continuous system need to be discretized. Structured State Space for Sequences (S4) achieves the discretization of SSMs using the zero-order hold technique, as shown in Eqn.(6)-(8):

$\overline{\textbf{A}}$, $\overline{\textbf{B}}$ and $\overline{\textbf{C}}$ are the discrete parameters. The discretized Eqn.(4)-(5) can be transformed as:

However, the parameters of $\overline{\textbf{A}}$, $\overline{\textbf{B}}$, and $\overline{\textbf{C}}$ in S4 cannot adaptively change based on different inputs, which limits the ability to effectively select important information. Mamba introduces a selection mechanism to enhance adaptability to different inputs and more effectively utilize previous information.

Bi-Surround Scan mechanism

Existing studies explore new scan methods suitable for specific tasks, such as cross-scan [27] and backward scan [28] to improve Mamba’s image understanding capability. Inspired by the BEV space’s surround sampling strategy and the typical urban environment’s main components based on distance intervals, we designed a bidirectional surround scan mechanism (Bi-Surround Scan) to improve Mamba’s adaptability in processing BEV features.

In this paper, the resolution of the BEV space is $200\times 200$. The BEV features are divided into $2\times 2$ patches in the $X$ and $Z$ axes, resulting in $100\times 100$ patch embeddings. As shown in Fig.4, according to the distance intervals from the ego-vehicle, we divided the BEV feature into three parts:$\left\{A,B,C\right\}$, corresponding to the distances of $\left\{\operatorname{0-20m},\operatorname{20-35m},\operatorname{35-50m}\right\}$. These three parts are hierarchically correspond to the road surface, buildings, and sky elements. Simply learning the global contextual information through serialization ignores this prior information. Therefore, based on the forward scan, we propose the forward surround scan and backward surround scan. These methods prioritize the modeling of environment-related components, allowing Mamba to learn hierarchical environmental features. The detailed process is illustrated in Algo.1. Given the BEV feature $F_{B}$ and the Surround-Scan Mamba block with three distinct scan branches, $S_{f}$, $S_{fs}$, and $S_{bs}$. $SiLU$ is employed as the activation function, and $z$ serves as the gating mechanism for the outputs ${y}^{\prime}_{f}$, ${y}^{\prime}_{fs}$, ${y}^{\prime}_{bs}$. Finally, $F_{EB}$ is obtained through skip connections.

Inspired by object-centric modeling [18], we introduced a center head that uses vehicle instance centerness as supervisory information, and designed a corresponding center loss based on L2 distance. In addition to the loss function, we explicitly utilized centerness information in both PV and BEV space to enhance and improve the output of the segmentation head. As shown in Fig.5, we take the output of ConvBlock after Decoder and before center head in BEV space as center feature $C_{F}$. Taking Center Feature $C_{F}$ as input, we use spatial attention [34] to obtain the center masks $C_{mask}$ based on the

Where $F_{max}$ and $F_{min}$ represent max pooling and average pooling, respectively. ⓒ denotes concatenation along the channel dimension, and $\sigma$ is the Sigmoid operation.

In the PV space, we introduce Multi-view deformable cross attention based on the deformable attention in the BEVFormer [6]. Whereas in our framework, we replace the predefined queries with confidence heatmap output by the center head $Q_{c}\in\mathbb{R}^{(B,Z\ast X,C)}$, Where $B$ represents batch size, $Z$ and $X$ are left-right and front-back axes of BEV space, respectively, and $C$ denotes the number of channels in the confidence heatmap. The multi-view features $M_{F}=\left\{F_{i}\right\}_{i=1}^{n}$ are used as the key $K$ and value $V$, where $F_{i}$ is the PV feature from the $ith$ view. $n$ is the number of all camera views. $\mathcal{P}$ for positional embeddings. Multi-view image features can be represented as $\delta_{1},\delta_{2},\dots\delta_{n}$, which we combine into a single vector $\delta=\left[\delta_{1},\delta_{2},\dots\delta_{n}\right]$. For center queries, $Q_{c}\supset[\phi_{1},\phi_{2},\dots,\phi_{n}]$. The specific formula is expressed as follows:

Through multi-view deformable cross attention $MCA$, $Q_{c}$ inquires about the vehicle centerness information from the multi-view image features $M_{F}$, which enables the model to focus more on the spatial regions where vehicles are located in the image, thereby better distinguishing between foreground and background. The specific formula can be expressed as follows:

Our OE-BevSeg includes a multi-task prediction head. In addition to the segmentation head, we introduce auxiliary task heads to predict vehicle instance centerness and offset. The centerness indicates the probability of finding the vehicle center [35], which is represented by the 2D Gaussian distribution and ranges from zero to one. The offset head output is a vector field, which points to the center of the instance.

The segmentation head uses the cross-entropy loss function, the center head is supervised using the $L2$ loss, and the offset head uses the $L1$ loss. To balance multi-task learning, we employ learnable weights based on uncertainty [36] to balance the three loss functions. The specific formulae are as follows:

Where $N$ represents the total number of pixels, and $P$ is the probability of predicting the $ith$ pixel as a vehicle. $w_{CE}$, $w_{L2}$ and $w_{L1}$ are the learnable uncertainty weights corresponding to the respective loss functions.

In this study, we conducted extensive experiments on the challenging nuScenes [37] urban scenes dataset to demonstrate the effectiveness of our OE-BEVSeg in large-scale autonomous driving scenarios. All data were collected using 6 cameras, 1 LiDAR, and 5 radar sensors, and then sampled at a well-synchronized frequency of 2Hz for the keyframes. The data were collected from 1,000 complex driving scenarios in Boston and Singapore, with 850 scenarios used for training and 150 for testing. There are 28,130 samples in the training set, while the validation set comprises 6,019 samples. We conducted research on BEV segmentation specifically for vehicle objects and performed experiments combining radar/LiDAR multimodal data. The ”vehicle” superclass consists of eight categories, i.e., bicycle, bus, car, construction vehicle, emergency vehicle, motorcycle, trailer, and truck. We use the intersection-over-union (IOU) as evaluation metric.

For the experimental implementation, We used the PyTorch framework and trained on four 40 GB NVIDIA A100 GPUs for a total of 20,000 iterations. The height and width of the input surround-view images are 448 and 800, respectively. The learning rate was set to 5e-4, using the Adam-W optimizer and the one-cycle learning rate schedule [44]. The resolution of the BEV space is $200\times 200$, with a channel dimension of 128, corresponding to a perception range of $100m\times 100m$. The encoder of our OE-BevSeg network adopts the first three stages of VOVNetV2, and the decoder uses the first three layers of ResNet18 as the backbone. Following the report of [10], a larger batch size yields superior results. With limited GPU memory, we accumulate the forward and backward over 5 iterations before performing a parameter update during training. This approach leads to more stable gradient updates and better model performance. Without increasing memory usage, we achieved training with a batch size of 40.

We compared our approach with other competitive algorithms for BEV segmentation, including recent most advanced methods such as PETRv2[42], SimpleBEV [10], and PointBEV [43]. To ensure fair comparison, we also listed the viewpoint lift methods, whether temporal information was used, and the resolution used during training for each algorithm. As shown in Table I, our OE-BEVSeg demonstrates superior performance. Compared to SimpleBEV, which uses the same perspective transformation method and resolution, our method outperforms by $5.2\%$ in IOU under camera-only conditions. Additionally, compared to the latest PointBEV, which utilizes temporal information, we achieve a $3.9\%$ improvement in IOU. Compared to TIIM, BEVFormer and PETRv2, which use the original image resolution of $900\times 1600$, our method significantly outperforms these high-resolution training and testing algorithms even when the input is at $1/4$ resolution. Specifically, our method surpasses PETRv2 by $6.3\%$ in IOU. This substantial improvement in BEV segmentation performance demonstrates the effectiveness of our approach. Additionally, Table I illustrates the superiority of our multi-modal branches. By fusing radar data, our method further improves IOU by $5.4\%$ compared to the camera-only approach. When integrating LiDAR data, the IOU reaches $65.3\%$, exceeding the camera-only approach by a large margin about $12.7\%$.

In this section, we conduct extensive ablation studies on the core ingredients proposed in this paper: the EBC Block and CIOE Block, the combination of loss functions, different image resolutions, the combination of different modal data, and different vehicle perception distances.

As shown in Fig.7, we visualized the BEV vehicle segmentation results and selected some challenging scenarios, such as rainy, occlusion, and night. Our OE-BevSeg achieved segmentation results closest to the ground truth, even in these difficult scenarios. The blue mask represents vehicles, the red annotations indicate negative results, and the green annotations indicate good results. It can be seen that OE-BevSeg provides better segmentation details. Compared to SimpleBEV, we reduce the number of true negative samples in night and occlusion scenarios, as well as the number of false negative samples in rainy scenarios. The fusion of multi-modal information, such as radar and LiDAR, enriches the segmentation details, better delineating vehicle contours. Multi-modal information provides valuable cues for measuring scene structure, enhancing the BEV segmentation performance.

To more intuitively demonstrate the benefits of centerness in enhancing object detail information, as shown in Fig.8, we visualized the BEV feature and center feature using heatmaps based on the V2Res-Net baseline. We can observe a clear distinction between the visualization of the BEV and center feature. The BEV feature focuses more on global perception but lacks sufficient attention to the target object. In contrast, the center feature focuses more on the vehicle area, showing higher activation values. Therefore, utilizing centerness to supervise and guide the segmentation head enables the model to better distinguish between the foreground and background.

Fig.9 shows the heatmap visualizations of the BEV features from the decoder of different models. It can be seen that in challenging scenarios, our OE-BevSeg can effectively focus on the vehicle object regions and better understand the environment, demonstrating improved differentiation between relevant areas with target objects and irrelevant areas without targets. This indicates that our model not only achieves better perception and understanding of the BEV long-distance environment but also enriches the information of target objects, thereby achieving superior vehicle segmentation performance.