GitNet: Geometric Prior-based Transformation for Birds-Eye-View Segmentation

Learnable Geometric Relation. The transformation from perspective view (PV) to BEV can be given by a projection matrix $P$. We first introduce the coordinate systems: a certain point in the camera coordinate system is represented by $\mathbb{x}_{c}=[x_{c},y_{c},z_{c}]^{T}\in\mathbb{R}^{3}$. The ground space is by setting the y-coordinate of the camera coordinate system to $h$ and a certain point lying on the ground plane turns out to be $\mathbb{x}_{c}=[x_{c},h,z_{c}]^{T}$, where $h$ denotes the height of the mounted camera from the ground. The BEV coordinates simply remove the $y$-dim and can be denoted as $\mathbb{x}^{B}=[x_{c},z_{c}]^{T}\in\mathbb{R}^{2}$. In the following, we do not particularly distinguish the BEV space from the ground space in the camera coordinate system. The homogeneous image coordinates $\mathbb{x}_{i}=[x_{i},y_{i},1]^{T}$ have a one-to-one correspondence with the ground coordinates, which can be expressed by:

where $K$ is the camera intrinsic matrix: $K=[[f_{x},0,c_{x}],[0,f_{y},c_{y}],[0,0,1]]^{T}$, and the inverse transformation from image to ground coordinates is formulated as:

Visibility-aware Perspective Feature Learning. The BEV semantic segmentation is an implicit mapping-segmentation coupling task. Here we decouple the BEV segmentation into the geometric prior-based mapping and perspective segmentation. The latter is supervised by an explicit segmentation loss with projecting the BEV GT labels onto the image plane following the transformation $P({\boldsymbol{x_{c}}}\xrightarrow{}{\boldsymbol{x_{i}}})$ in the Equation (1) to generate the projected labels ${PV}^{proj}_{gt}$. ${PV}^{proj}_{gt}$ reflects the whole perspective-view ground including visible or invisible regions. However, the perspective features extracted from images only reflect the visible foreground regions. Therefore, the projected labels ${PV}^{proj}_{gt}$ can be used to obtain the visibility-aware image features by fusing the information of projected labels and image features. In specific, the pyramid features $\boldsymbol{F_{\{1,2,3,4\}}}\in{\mathbb{R}^{H_{i}\times W_{i}\times 64}}$ are separately fed into the weight-shared segmentation head to generate the corresponding probability maps $\boldsymbol{P_{\{1,2,3,4\}}}\in{\mathbb{R}^{H_{i}\times W_{i}\times C}}$ under the supervision of our depth-aware Dice (DA-Dice) segmentation loss with projected labels. We concatenate the feature maps and the corresponding probability maps, and learn the visibility-aware features ${\boldsymbol{A_{i}}}$ with pixel-wise fusion (MLP) by:

Geometry-based Warping. From the Equation (2), we can derive that $z_{c}/x_{c}=f_{x}/(x_{i}-c_{x})$, which indicates that pixels of perspective view lying on the same column (i.e., with the same x-coordinate $x_{i}$) map onto the same polar ray in BEV space with a slope of $f_{x}/(x_{i}-c_{x})$. Following the transformation $P(y_{i}\xrightarrow{}z_{c})$ in Equation (2), the $\boldsymbol{j}$-th column of augmented image features $\boldsymbol{A_{m}^{j}}$ are warped into the BEV space with the learned camera height $h$ by inverse projection. That is:

where $\{\boldsymbol{S_{m}^{1},S_{m}^{2},...,S_{m}^{W_{m}}}\}$ are computed in parallel in our implementation, and are concatenated to output the tensor ${\boldsymbol{S_{m}}}\in\mathbb{R}^{Z_{m}\times W_{m}\times 64}$. The warped BEV features take advantage of both the appearance from multi-scale perspective-view features and the visibility with BEV projected-to-PV labels guidance. The geometry-guided transformation module provides initial queries for the follow-up transformer stage for further tuning the features for the BEV segmentation task.

Differences with the Original Transformer. Our method draws on the core idea of Transformer, i.e., employing the multi-head attention mechanism. But we have two new designs for our BEV segmentation task. Firstly, We use column-wise attention so that we can perform on high-resolution feature maps, which is indispensable in our pixel-wise recognition. Secondly, we introduce the pre-aligned features encoding the appearance and visibility, along with BEV positional encoding, as queries in the cross attention. The ablation study in the Sec. 4.3 validates the superiority of our designs. In the following part, we detail two attention mechanisms in our transformer, and omit other components such as normalization for the sake of simplicity. The complete structure can be seen in our supplementary material.

Column Context Augment(CCA) in Encoder. As illustrated in Fig.3, the inputs for the transformer encoders are perspective features ${\{\boldsymbol{F}_{1},\boldsymbol{F}_{2},\boldsymbol{F}_{3},\boldsymbol{F}_{4}\}}$ extracted from the FPN, where $\boldsymbol{F_{m}}$ has a spatial resolution of ${H_{m}}\times{W_{m}}$. In the CCA, each pixel adaptively integrates the information from other pixels of the same column, by using multi-head self-attention. We further introduce spatial positional encodings $\boldsymbol{P_{m}}$ to the input $\boldsymbol{F_{m}}$ to distinguish the positions of the input features. We use a sine function to generate spatial positional encoding. Let $\boldsymbol{F_{m}^{j}}$, $\boldsymbol{P_{m}^{j}}$ $\in\mathbb{R}^{H_{m}\times 64}$ denote the $\boldsymbol{j}$-th column of $\boldsymbol{F_{m}}$ and $\boldsymbol{P_{m}}$, respectively. The mechanism of CCA can be summarized as

Ray-based Cross-Attention (RCA) in Decoder. RCA in the transformer decoder aims to refine the output of pre-alignment block, ${{\{\boldsymbol{S}_{1},\boldsymbol{S}_{2},\boldsymbol{S}_{3},\boldsymbol{S}_{4}\}}}$, based on the augmented image features ${\{\boldsymbol{\widetilde{F}_{1}},\boldsymbol{\widetilde{F}_{2}},\boldsymbol{\widetilde{F}_{3}},\boldsymbol{\widetilde{F}_{4}}\}}$. As depicted in Fig.3, the RCA receives the pre-aligned BEV feature as Query, the augmented features built from the encoder as Key and Value. Similar to CCA, spatial positional encoding $\boldsymbol{P_{m}^{\prime}}$ is also adopted in RCA. The difference is that $\boldsymbol{P_{m}^{\prime}}$ represents the position in the BEV, while $\boldsymbol{P_{m}}$ is on the image plane. The mechanism of RCA can be summarized as

Since the columns of ${\boldsymbol{\widetilde{S}_{m}}}$ are still in the image coordinate, we warp them to rays in the camera coordinate following the transformation $P(x_{i}\xrightarrow{}x_{c})$ in the Equation (2) to obtain $\{{\boldsymbol{M}_{1},\boldsymbol{M}_{2},\boldsymbol{M}_{3},\boldsymbol{M}_{4}}\}$, which are responsible for different depth ranges. We concatenate all features along the depth axis to obtain the feature maps of the whole scene:

The final BEV feature maps $\boldsymbol{M}$ are fed to the downstream convolutional segmentation network. Thanks to the CCA and RCA-based transformer, the network take appearance and visibility knowledge into account, and further tunes the invisible regions of the pre-aligned BEV features, based on the context information from the perspective features.

Occlusion-free Depth-aware Dice Loss: As discussed in Sec. 3.2, we project the BEV labels onto the image plane to generate the projected labels ${PV}^{proj}_{gt}$, which supervises the segmentation on the perspective view. To tackle the first problem caused by the domination of nearer objects in perspective images, we propose the novel Depth-aware Dice loss by re-weighting the loss in a depth-aware manner. In specific, the Jacobian determinant gives the ratio of the area ratio between image ground plane ($\Delta A_{i}$) and the BEV ground plane ($\Delta A_{c}$) as:

We find that the area ratio is proportional to $(1/z_{c})^{3}$, thus we re-weight the pixels with the weight $z_{c}^{3}$ to solve the imbalance. The depth-aware dice loss is:

Self-Weighted Dice Loss: To further alleviate the second problem, i.e., the dominating influence from easy samples in training, we propose to associate training samples with dynamically adjusted weights to emphasize hard examples. We first propose a weighting function $I_{i}^{k}$ to adjust the hard-mining strength by a parameter $\alpha$ in Equation (11) and then utilize $I_{i}^{k}$ to reweight the Dice loss and obtain the self-weighted dice loss in Equation (12).

Dataset We conduct extensive experiments on two large-scale datasets: The nuScenes [1] and Argoverse [3] road-scene datasets. Since the two datasets are predominantly collected for 3D object detection task rather than BEV semantic segmentation task, we follow the data generation method in [14] to convert the ground truth 3D bounding box annotations and the vectorized road maps into GT semantic maps in BEV. In addition, for fair comparisons, we also follow the same training and validation splits with other methods. The nuScenes includes 4 road layout categories and 7 object categories, and the Argoverse includes 7 object categories as well as drivable road. For both datasets, the ground truth of birds-eye-view expands from 1m to 50m in front of the camera i.e., along the $z$-direction and 25m to either side (i.e., along the $x$-direction). Due to the greater diversity of nuScenes, we choose this dataset for all ablation studies.

Implementation details For fair comparisons, we adopt a pretrained ResNet-50 with a feature pyramid on top as the backbone. We adopt a simplified HRNet [22] as the BEV segmentation head. In our implementation, we use a simplified HRNet32 by halving the number of blocks in each stage. We use two encoder layers and four decoder layers in the Ray-based transformer. The hyperparameter $\alpha$ in Equation 11 for the SW-Dice loss is set as 0.5. We adopt a similar depth-interval assignment strategy with [14], but we only use the former four scales of the FPN. The concatenated BEV feature maps from different depth intervals are of $98\times 100$ pixels, with each pixel covering 0.5m. We obtain the final output map with a resolution of $196\times 200$ pixels by upsampling, which is consistent with other methods. The model is trained using four Tesla V100 cards, each with 32G memory. We optimize the network with Adam policy for gradients accumulated over every 8 iterations and train for 40 epochs. The initial learning rate is set to 0.0002, with a weight decay of 0.99 and batch size 12.

Evaluation metric Our evaluation metric is the Intersection over Union (IoU) score, which we compute by binarizing the output probability maps with the threshold of 0.5. Invisible regions are ignored during evaluation following [14].

Effects of different components. To analyze the effects of the key designs, we try different combinations and summarize the ablation results in Table 3.

• $\bullet$

  Baseline. Group (a) is the baseline that is similar to [13]. We transform the image features onto the ground plane via a homography matrix. The difference between it and our GPA is that it adopts a fixed camera height and is not supervised by the projected semantic maps. The transformed features are further processed by a segmentation network that is the same as our best model for fair comparisons. From Row 1 in Table 3, we can see that the baseline achieves reasonable results in road layout areas, but fails to distinguish the objects that standing above the road.

• $\bullet$

  Network. In Group (b), the GPA provides a reliable prior for feature transformation and relieves the effects of occlusion by supervision of projection, improving the mIoU by +9.7% in total. The RT (c) transforms the image features to the BEV space by multi-scale column-based attention, which improves the mIoU by +11.2% in total. If we combine the GPA and RT as disscussed in Sec. 3.1, their joint effect (d) further enhances the performance by +13.6% in total. It shows the geometric prior provides complementary information for the RT.

• $\bullet$

  Loss. Groups (e)(f)(g) show the improvements in our proposed loss function. The SW-Dice loss (e) automatically puts higher weights on these pixels that are hard to classify in birds-eye-view, improving the mIoU by +0.7% in total. The DA-Dice loss (f) balances the pixels of different depth ranges in the perspective view by reweighting the Dice loss under the guidance of the cubic depth when learning the geometric prior-based pre-alignment module, which improves the mIoU by +1.2% in total. The joint of both losses (g) brings a further mIoU gain of +1.9% in total.

Effects of components of GPA: Three key designs are presented in Geometry-guided Pre-Alignment to better convert the perspective image features to BEV features, including the learnable camera height, the projection supervision, and pixel-wise fusion between probability maps and image features. Comparing Group I and II, where we enforce the network to learn the offset of the camera height, we observe a 0.7% mIoU gain. Group III leverages the projected labels from BEV space to image space to supervise the feature learning procedure, which further improves the performance by 1.1% mIoU. The further pixel-wise fusion between perspective features and segmentation probability maps in Group IV further lifts the performance, resulting in a total of 2.6% gain.

Effects of $\alpha$ in SW-Dice loss: The SW-Dice loss introduces the hyperparameter $\alpha$ to control the strength of the modulating term with respect to the predicted probability. As is shown in Table 5(a), $\alpha=0$ means our loss is equivalent to the plain Dice loss. As $\alpha$ increases, the predicted probability gets dominant in the weighting function. Under all settings of $\alpha$, the proposed SW-Dice loss stably outperforms the baseline ($\alpha=0$). With the best setting, the SW-Dice loss yields a 0.7% improvement over the plain Dice loss.

Effects of decoder layers in RT: Table 5(b) shows the performance with various number of decoder layers within the ray-based transformer. Our model can yield 4.1% improvement even using one layer. The gain reflects that the pre-aligned BEV features can provide a good initialization for the decoder. With the decoder layers increasing, higher performance can be achieved. We observe that it becomes saturated when adopting more than three layers.