Progressive Query Refinement Framework for Bird’s-Eye-View Semantic Segmentation from Surrounding Images

Suppose that an autonomous vehicle (AV) is equipped with $N_{c}$ surrounding cameras. The $i$-th camera image acquired at time $t$ is denoted as $I_{i,t}\in\mathbb{R}^{H_{I}\times W_{I}\times 3}$, where $H_{I}$ and $W_{I}$ denote the height and width of the image, respectively. Our target is to obtain a BEV feature map $\mathbf{B}_{t}\in\mathbb{R}^{X\times Y\times C}$, which is utilized as input to subsequent NNs for the semantic BEV map prediction, from the images $\{I_{i,t}\}_{i=1}^{N_{c}}$. Here, $X$ and $Y$ respectively denote the length and width of the BEV grid space spanned by the vehicle pose at $t$, and $C$ denotes the BEV feature dimension. In the rest of this paper, we omit $t$ for the sake of simplicity. Our proposed VT module takes a set of MR BEV query maps $\{\mathbf{Q}_{s}\}_{s=1}^{N_{s}}$ and MR image feature maps $\{\mathbf{F}_{i}^{l}\}_{i=1,l=1}^{N_{c},N_{l}}$ as input and produces $\mathbf{B}$ by updating and merging the query maps progressively. Here, $N_{s}$ is the number of the MR query maps and $\mathbf{Q}_{s}\in\mathbb{R}^{\frac{X}{2^{s-1}}\times\frac{Y}{2^{s-1}}\times C}$. The image feature map $\mathbf{F}_{i}^{l}$ is the $l$-th layer output of an image backbone (e.g., ResNet [7]) when $I_{i}$ is used as input.

Figure 2 briefly illustrates the overall architecture of our proposed BEV segmentation network. First, MR feature maps $\{\mathbf{F}_{i}^{l}\}$ are extracted from $\{I_{i}\}$ using a backbone network. The proposed visual feature interaction (VFI) module is then applied to the MR feature maps to promote interactions between the features both across levels and across images. Next, the proposed VT module, which consists of VT encoders and upsamplers, updates and merges $\{\mathbf{Q}_{s}\}$ progressively to produce $\mathbf{B}=\mathbf{Q}_{1}+\mathbb{UpScale}(\mathbf{Q}_{N_{s}})\in\mathbb{R}^{X\times Y\times C}$. Finally, a class header network, which is dedicated to a specific object or road element class, is applied to $\mathbf{B}$ and the final BEV map $\mathbf{BEV}\in\mathbb{R}^{X\times Y}$ is predicted using a subsequent map decoder network. As we mentioned, supervising both $\mathbf{B}$ and $\mathbf{Q}_{N_{s}}$ during training can be regarded as residual learning [7], resulting in a significant improvement in segmentation performance.

Visual feature interaction (VFI) module is devised to promote interactions between the MR image features. It comprises two sub-modules: 1) Intra-camera interaction module (Intra-CIM), 2) Inter-camera interaction module (Inter-CIM). Intra-CIM is introduced to promote interactions between features located at the same pixel position across different feature levels and is implemented by the FPN [15]. Inter-CIM is introduced to promote interactions between features across different camera images. Given a set of image features at the same feature level, $\{\mathbf{F}_{i}\}$, the interactions are promoted through our proposed multi-head deformable attention as follows:

$$f_{DA}(\mathbf{F}_{i}(\mathbf{p}),\{\mathbf{F}_{j}\}_{j=1}^{N_{c}})=\sum_{h\in\mathcal{H}}\sum_{\mathbf{p}\in\mathcal{P}_{h}}\sum_{i=1}^{N_{c}}\alpha_{\mathbf{p},i}^{h}\mathbf{F}_{i}(\mathbf{p}+\Delta\mathbf{p}_{i}^{h}),$$

(1)

where $\mathcal{H}$ is a set of head indices, $\mathcal{P}_{h}$ is a set of 2D reference points for a head of index $h$, and $\alpha_{\mathbf{p},i}^{h}$ denotes an attention weight and satisfies $\sum_{\mathbf{p},i}\alpha_{\mathbf{p},i}^{h}=1$. Our deformable attention distinguishes itself from the original [32] through the design of the reference points pattern and the calculation of offsets $\{\Delta\mathbf{p}_{i}^{h}\}$ and weights $\{\alpha_{\mathbf{p},i}^{h}\}$. Let us elaborate on our reference points pattern using Fig. 3b. For the deformable attention operation, we assign each head four reference points, covering specific areas in an image. We illustrate in Fig. 3b the reference points for each head with different colors. As a consequence, with the eight heads having distinct reference points, which cover different image areas, our model can easily locate the image regions to be attended to. On the other hand, in the conventional design, the reference points for each head are the same as the pixel position of the query feature, $\mathbf{p}$, as seen in Fig. 3a.

In the conventional deformable attention [32], the offsets and weights are predicted directly from $\mathbf{q}_{\textbf{emb}}=\mathbf{F}_{i}(\mathbf{p})+\mathbf{P}(\mathbf{p})$ through MLPs, where $\mathbf{P}(\mathbf{p})$ denotes a sinusoidal positional embedding for the pixel position $\mathbf{p}$. We, in this paper, predict the offsets and weights as proposed in [32] with the following two modifications: 1) We introduce trainable embedding vectors $\{\mathbf{c}_{i}\}_{i=1}^{N_{c}}$ to encourage the VFI module to distinguish image features across different images and to enhance the prediction of the offsets and weights. As a result, the offsets and weights are predicted from $\mathbf{q}_{\textbf{emb}}=\mathbf{F}_{i}(\mathbf{p})+\mathbf{P}(\mathbf{p})+\mathbf{c}_{i}$ through MLPs. 2) We limit the maximum allowable magnitude of the offsets to ensure that a point $\mathbf{p}+\Delta\mathbf{p}_{i}^{h}$ can cover a specific area of an image as follows:

$$\Delta\mathbf{p}_{i}^{h}=\delta\cdot\text{Tanh}(\text{MLP}(\mathbf{q}_{\textbf{emb}})),$$

(2)

where $\text{Tanh}()$ denotes the hyperbolic tangent function. $\delta$ is a positive constant and is set to 0.25 in this paper.

View Transformation (VT) Encoder repeatedly updates $\mathbf{Q}_{s}$ using $\{\mathbf{F}_{i}^{l}\}$. We deploy the Transformer encoder proposed in [13], which benefits from the deformable attention [32] in terms of complexity and training speed. Specifically, let $\mathbf{V}$ be the projection of $\mathbf{Q}_{s}$ through a MLP. Then, the self-attention module in the VT encoder updates $\mathbf{Q}_{s}$ via the following deformable attention:

$$f_{DA}(\mathbf{Q}_{s}(\mathbf{p}),\mathbf{V})=\sum_{h\in\mathcal{H}}\sum_{z=1}^{Z}\alpha_{z}^{h}\mathbf{V}(\mathbf{p}+\Delta\mathbf{p}_{z}^{h}),$$

(3)

where $\mathbf{Q}_{s}(\mathbf{p})$ denotes the element of $\mathbf{Q}_{s}$ at the spatial location $\mathbf{p}$ in the BEV grid space. $Z$ is the number of the value sampling operation. By adding the result of (3) to $\mathbf{Q}_{s}(\mathbf{p})$ and applying a feed forward network (FFN) to the addition, the interaction between queries in $\mathbf{Q}_{s}$ is promoted.

The cross-attention module in the VT encoder updates $\mathbf{Q}_{s}$ using the visual features $\{\mathbf{F}_{i}^{l}\}$ through the deformable attention defined as follows:

$$f_{DA}^{h}(\mathbf{Q}_{s}(\mathbf{p}),\{\mathbf{F}_{i}^{l}\},\{\mathcal{T}_{i}\})\\ =\frac{1}{|\mathcal{V}_{hit}|}\sum_{i\in\mathcal{V}_{hit}}\sum_{z=1}^{Z}\sum_{l=1}^{N_{l}}\alpha_{z,l}^{i}\mathbf{V}_{i}^{l}(\mathcal{T}_{i}(\mathbf{p},i,z)+\Delta\mathbf{p}_{z,l}^{i}),$$

(4)

where $f_{DA}^{h}$ denotes the deformable attention dedicated to the $h$-th header and $\mathbf{V}_{i}^{l}$ is the projection of $\mathbf{F}_{i}^{l}$ through a MLP. $\mathcal{T}_{i}$ is the transformation function that projects a 3D point in the BEV space into a 2D point in the $i$-th image space. Consequently, $\mathcal{T}_{i}(\mathbf{p},i,z)$ is a 2D point in the $i$-th image space projected from the $z$-th 3D reference point defined on the position $\mathbf{p}$ in the BEV space. $\mathcal{V}_{hit}$ is the set of image indices where at least one of the 3D reference points is projected to. Finally, we note that the attention weight $\alpha$ and offset $\Delta\mathbf{p}$ are predicted directly from $\mathbf{Q}_{s}(\mathbf{p})$ through MLPs after a positional sinusoidal embedding is added to $\mathbf{Q}_{s}(\mathbf{p})$.

Query Map Merge Module fuses a previously updated LR query map $\mathbf{Q}_{s-1}$ with a higher resolution query map $\mathbf{Q}_{s}$ for a subsequent query update. We explored diverse options for the fusion and empirically found that a simple addition operation yields the best performance. Specifically, $\mathbf{Q}_{s-1}$ first undergoes bilinear interpolation to match the size of $\mathbf{Q}_{s}$ and then is added to $\mathbf{Q}_{s}$. Finally, $\mathbf{Q}_{s}$ goes through a subsequent VT encoder.

Auxiliary Task is introduced to encourage our model to capture the global characteristics of driving scene more effectively. Our auxiliary decoder depicted in Figure 2 progressively increases the resolution of the lowest resolution query map up to the target resolution to obtain the final output $\mathbf{BEV}_{aux}\in\mathbb{R}^{X\times Y}$. It comprises a series of NN modules, each of which consists of Up-scaler, Conv, BN [12], and Relu. During the training, $\mathbf{BEV}_{aux}$ is forced to match the ground-truth BEV semantic map, $\mathbf{BEV}_{GT}$. As shown in the section IV, our model achieves the best performance when only the lowest resolution query map undergoes the auxiliary task.

To train our model, we minimize the final loss $\mathcal{L}$ defined as follows:

$$\mathcal{L}=\sum_{c\in\mathcal{C}}\lambda_{c}(\mathcal{L}_{main}^{c}+\alpha\mathcal{L}_{aux}^{c}),$$

(5)

where $\mathcal{L}_{main}^{c}$ and $\mathcal{L}_{aux}^{c}$ denote the focal losses [16] for a specific class $c$ calculated from $\mathbf{BEV}$, $\mathbf{BEV}_{aux}$, and $\mathbf{BEV}_{GT}$. $\alpha$ and $\lambda_{c}$ are pre-defined constants. $\alpha$ balances the contributions of $\mathcal{L}_{main}$ and $\mathcal{L}_{aux}$ to the final loss. $\lambda_{c}$ balances the contributions of each class to the final loss.

A large-scale real-world dataset, nuScenes [2], is used to evaluate our model. nuScenes is a collection of 1000 scenes acquired over diverse weather, time-of-day, and traffic conditions. Following the previous works [13, 31, 6], 28,130 and 6,019 key frames are used for the training and test sets, respectively. To generate the ground-truth BEV semantic maps at a resolution of $(200\times 200)$, 3D bounding boxes of vehicles and pedestrians or HD map elements in $100m\times 100m$ area around the ego-vehicle are orthographically projected onto the ground plane, following the standard practice [23, 8]. For an evaluation metric, we use Intersection-over-Union (IoU) score between the ground-truth and its prediction.

We use as an image backbone ResNet-50 [7] pre-trained on ImageNet [26]. Three consecutive mid-layer feature maps ($N_{l}=3$), each of which corresponds to down-scaling factors of $\times 4$, $\times 8$, and $\times 16$ from the original size, are used as the visual feature maps. The feature maps further undergo a shallow CNN to match the channel dimension with that of the query maps. For the class headers, we use shallow CNNs consisting of Conv, BN [12], and Relu. For the map decoders, we deploy the mask decoder [14]. Our model is trained with a batch size of 4 for 28 epochs. We optimize our model using AdamW [11] optimizer with learning rate $2e^{-4}$ and weight decay $1e^{-7}$. We also set $N_{s}=3$ and $\alpha=1$ in Eqn. 5. Lastly, in accordance with common practice [31, 21], we trained a separate NN for each class to achieve the results presented in the subsequent sections. This separation is adopted due to the potential underperformance of jointly trained models, a phenomenon known as negative transfer in multi-task learning [13].

We objectively compare our model with the SOTA models in Table I. Since our model does not consider previously obtained image data for the current prediction, we report the prediction performance of the static version of the SOTA models in the table for fair comparisons. We can see in the table that our proposed model achieves the best performance across almost all class categories. Specifically, our model outperforms BEVFormer-s [13] despite the input image size for our model being four times smaller than that for BEVFormer-s [13]. This result demonstrates that our model effectively captures objects of various sizes in driving scenes. TBPFormer-s [6] also shows the performance comparable to ours. However, it has $1.6$ times more trainable parameters than ours ($117.9\textbf{M}$ v.s. $73.7\textbf{M}$), which mainly originates from its repeated deployments of SwinT [18] for the hierarchical query refinement.

In Figure 4, we visualize the prediction results of our model alongside the ground-truth. We can see in the figure that our model captures the shape of the dynamic road agents and static road elements well. Specifically, our model effectively distinguishes vehicles gathered owing to our query refinement framework and its dedicated training method. Figure 5 further demonstrates the effectiveness of the framework.

In Table II, we demonstrate the effectiveness of the VFI module and our progressive refinement framework in terms of the prediction performance. In the table, M1 and M2 denote the proposed model without the VFI module and with the Inter-CIM removed, respectively. For M3, we replace the proposed deformable attention for the Inter-CIM with the conventional one. The table shows that the proposed deformable attention enhances the interaction between the features across images, leading to a significant improvement in prediction performance compared to the conventional attention (M3 v.s. Ours). M4 and M5 respectively denote the proposed model with a single query map ($N_{s}=1$) and MR query maps ($N_{s}=3$) without applying the auxiliary task during the training. In contrast, for M6 ($N_{s}=3$), we exclude the addition of the lowest resolution map to the highest one and apply the auxiliary task for all query maps except for the final resolution. The table and Figure 5 demonstrate that the proposed progressive query refinement framework, along with its dedicated training (the auxiliary task), enhances our network’s ability to capture the global and local characteristic of driving scenes.