MatrixVT: Efficient Multi-Camera to BEV Transformation for 3D Perception

Perception of 3D objects and scenes takes a key role in autonomous driving and robotics, thus attracting increasing attention nowadays. Camera-based perception [27, 26, 6, 11, 12] is the most commonly used method for varies scenarios due to its low cost and high accessibility. Comparing with 2D perception (object detection [25], semantic segmentation [23], etc.), 3D perception requires additional prediction of the depth information which is an naturally ill-posed problem [22].

Existing works either predict the depth information explicitly or implicitly. FCOS3D [27] simply extend the structure of the classic 2D object detector [25], predicting pixel-wise depth explicitly using an extra sub-net that is supervised by LiDAR data. CaDDN [19] propose treating depth prediction as classification task rather than regression task, and project image feature into Bird’s-Eye-View (BEV) space to achieve unified modeling of detection and depth prediction. BEVDepth and following works [10, 9, 29] propose several techniques to enhance depth prediction, these works achieve outstanding performance due to precise depth. Meanwhile, methods like PON [21] and BirdGAN [24] use pure neural networks to transform image features into BEV space, learning object depth implicitly using segmentation or detection supervision. Currently, methods that explicitly learn depth show prior performance than implicit approaches thanks to the supervision from LiDAR data and depth modeling. In this paper, we use the DepthNet same as in BEVDepth for high-performance.

The concept of BEV is firstly proposed for processing LiDAR point cloud [33, 7, 31, 32], and found effective for fusing multi-view image features. The core component of vision-based BEV paradigm is the view-transformation. OFT [22] firstly propose mapping image feature from Perspective View into BEV using camera parameters. This method project a reference point from an BEV grid to image plane, and sample corresponding features back to the BEV grid. Following this work, BEVFormer [11] propose using Deformable Cross Attention to sample features around the reference point. These methods fail to distinguish BEV grids that are projected to same position on the image plane, thus show inferior performance than depth-based methods.

Depth-based methods, represented by LSS [18] and CaDDN [19], predict categorical depth for each pixel, the extracted image feature on a specific pixel is then projected into 3D space by doing per-pixel outer product with corresponding depth. The projected high-dimensional tensor is then “collapsed” to a BEV reference plane using convolution [19], Pillar Pooling [18, 6], or Voxel Pooling [10]. Lift-Splat based methods [18, 10, 6, 9] show outstanding performance for down-stream tasks, but introduces two extra problems. Firstly, the intermediate representation of image feature is large and in-efficient, making training and application of these methods difficult. Secondly, the Pillar Pooling [18] introduces random memory access, which is slow and device demanding (extremely slow on general-purpose devices). In this paper, we propose a new depth-based view transformation to overcome these problems while retaining the ability of producing high-quality BEV features.

Lift-Splat-like methods exploit the idea of pseudo-LiDAR [28] and LiDAR-based BEV methods [33, 7] for View Transformation. Image features are “lifted” to 3D space and being processed like pointcloud. We define $N_{c}$ as the number of cameras for a specific scene; $N_{d}$ as the number of depth bins; $H_{I}$ and $W_{I}$ to be the height and width of image features; $H_{B}$ and $W_{B}$ be the height and width of BEV features (i.e., shape of BEV grids); $C$ to be the number of feature channels. Consequently, let multi-view image features to be $F\in\mathbb{R}^{N_{c}\times H_{I}\times W_{I}\times C}$; categorical depth to be $D\in\mathbb{R}^{N_{c}\times H_{I}\times W_{I}\times D}$; BEV features to be $F_{BEV}\in\mathbb{R}^{H_{B}\times W_{B}\times C}$ (initialized to zeros). The Lift-Splat VT first “lift” the $F$ into 3D space by doing per-pixel outer product with $D$, obtaining the high-dimensional intermediate tensor $F_{inter}\in\mathbb{R}^{N_{c}\times H_{I}\times W_{I}\times N_{d}\times C}$.

The intermediate tensor can be treated as $N_{c}\times H_{I}\times W_{I}\times N_{d}$ feature vectors, each vector corresponding to a geometric coordinate. The “splat” operation is then adopted using operators like Pillar Pooling [18], during which each feature vector is summed to a BEV grid according to geometric coordinates (see Fig. 2).

We find that the “splat” operation is a fixed mapping between the intermediate tensor and BEV grids. Therefore, we use a Feature Transporting Matrix $\mathrm{M_{FT}}\in\mathbb{R}^{(H_{B}\times W_{B})\times(N_{c}\times H_{I}\times W_{I}\times N_{d})}$ to describe this mapping. The FTM can be strictly equal to Pillar Pooling in LSS [18] or encodes different sampling rules (i.e., Gaussian sampling). The FTM enables feature transportation with Matrix Multiplication (MatMul):

where $\mathrm{F_{inter}}\in\mathbb{R}^{(N_{c}\times H_{I}\times W_{I}\times N_{d})\times C}$ is the “lifted” feature $F_{inter}$ in matrix format, $\mathrm{F_{BEV}}\in\mathbb{R}^{(H_{B}\times W_{B})\times C}$ is the BEV feature $F_{BEV}$ in matrix format.

Replacing the “splat” operation with FTM eliminates the need for customized operators. However, the sparse mapping between the 3D space and BEV grids can lead to a massive and highly sparse FTM, harm the efficiency of matrix-based VT. To address the sparse mapping problem without customized operators, we propose two techniques to reduce the sparsity of FTM and speed up the transformation.

The high-dimensional intermediate tensor is the primary cause of the sparse mapping and the low efficiency of Lift-Splat-like VT. An intuitive way to reduce sparsity is reducing the size of $F_{inter}$. Therefore, we propose Prime Extraction — a compression technique for autonomous driving and other scenarios where information redundancy exists.

As shown in Fig. 4, the Prime Extraction module predicts Prime Depth Attention for each direction (a column of the tensor) guided by the image feature. It generates the Prime Depth as a weighted sum of categorical depth distributions. Meanwhile, the Prime Feature is obtained by the Prime Feature Extractor (PFE), which consists of position embedding, column-wise max-pooling, and convolutions for refinement. By applying Prime Extraction to matrix-based VT, we successfully reduce the FTM to $\mathrm{M^{\prime}_{FT}}\in\mathbb{R}^{(H_{B}\times W_{B})\times(N_{c}\times W_{I}\times N_{d})}$, which is $H_{I}$ times smaller than the raw matrix. The pipeline of generating the BEV feature using the Prime Feature and the Prime Depth is demonstrated in Fig. 5 (yellow box).

The prime extraction is motivated by an observation: The image feature’s height dimension has a lower response variance than the width dimension. This observation indicates that this dimension contains less information than the width dimension. We thus propose to compress image features on the height dimension. Previous works have also exploited reducing the height dimension of image features [21, 1], but we firstly propose compressing both image features and corresponding depth to boost VT.

In Sec. 4.4, we will show that the extracted Prime Feature and Prime Depth effectively retain valuable information from the raw feature and produce BEV features of the same high quality as the raw feature. Moreover, the Prime Extraction technique can be individually adopted to existing Lift-Splat-like VTs to enhance their efficiency at almost no performance cost.

The sparsity FTM can be further reduced by matrix decomposition. To this end, we propose the “Ring & Ray” Decomposition. Without loss of generality, we can set the $N_{c}$ to 1 and view $\mathrm{M^{\prime}_{FT}}$ as a tensor $M^{\prime}_{FT}$. In that case, the shape of $M^{\prime}_{FT}$ would be $H_{B}\times W_{B}\times W_{I}\times N_{d}$. We note that its dimension of size $W_{I}$ can be viewed as the direction in a polar coordinate, since each column of the image feature represents information of a specific direction. Likewise, the dimension of size $N_{d}$ can be viewed as the distance in the polar coordinate. In other words, the image feature required for a specific BEV grid can be located by direction and distance. We thus propose to orthogonally decompose the $M^{\prime}_{FT}$ into two separate matrices, each encoding the directional or distance information. Specifically, we use a Ring Matrix $\mathrm{M_{Ring}}\in\mathbb{R}^{N_{d}\times(H_{B}\times W_{B})}$ to encode distance information and a Ray Matrix $\mathrm{M_{Ray}}\in\mathbb{R}^{W_{I}\times(H_{B}\times W_{B})}$ to encode directional information (see Appendix 1.1 for pseudo code). The Ring & Ray Decomposition effectively reduces the size of static parameters. The number of predefined parameters (size of FTM) is reduced from $W_{I}\times N_{d}\times H_{B}\times W_{B}$ to $(W_{I}+N_{d})\times H_{B}\times W_{B}$, which is typically 30 to 50 times less¹¹1Under feature width 44 and 88, with 112 depth bins..

Then we show how to use these two matrices for VT. Given the Prime Feature and Prime Depth, we first do the per-pixel outer product of them as in Lift-Splat to obtain the “lifted feature” $F_{inter}\in\mathbb{R}^{W_{I}\times N_{d}\times C}$.

Then, as illustrated in Fig. 5, instead of using $\mathrm{M^{\prime}_{FT}}$ to directly transform it into a BEV feature (yellow box), we transpose $F_{inter}$, view it as a matrix $\mathrm{F_{inter}}\in\mathbb{R}^{N_{d}\times(W_{I}\times C)}$, and do MatMul between the Ring Matrix and $\mathrm{F_{inter}}$ to obtain an intermediate feature. The intermediate feature is then masked by doing a Hadamard Product with the Ray Matrix, summed on the dimension of $W_{I}$ to obtain the BEV feature. (Fig. 5, blue box, summation omitted).

where $F^{*}_{inter}\in\mathbb{R}^{W_{I}\times C\times H_{B}\times W_{B}}$ is $\mathrm{F^{*}_{inter}}$ in tensor form.

However, this decomposition do not reduce the FLOPs during VT and introduces the Intermediate Feature in Fig. 5 (blue) whose size if huge and depends on feature channel $C$. To reduce the calculation and memory footprint during VT, we combine Eq. 3 to Eq. 5 and rewrite them in a mathematically equivalent form (see Appendix 1.2 for proof):

With this reformulation, we reduce the calculation during VT from $2\times W_{I}\times C\times N_{d}\times H_{B}\times W_{B}\text{~{}FLOPs}$ to $2(C+N_{d}+1)\times W_{I}\times H_{B}\times W_{B}\text{~{}FLOPs}$; the memory footprint is also reduced from $W_{I}\times N_{d}\times H_{B}\times W_{B}$ to $(W_{I}+N_{d})\times H_{B}\times W_{B}$. Under common setting ($C=80,N_{d}=112,W_{I}=44$), the Ring & Ray Decomposition reduces calculation by 46 times and saves 96% memory footprint.

With above techniques, we reduce the calculation and memory footprint using FTM by hundreds times, making VT with FTM not only feasible but also efficient. Given the multi-view images for a specific scene, we conclude the overall pipeline of MatrixVT as follows:

1. 1.

   We first use an image backbone to extract image features from each image.

2. 2.

   Then, a depth predictor is adopted to predict categorical depth distribution for each feature pixel to obtain the depth prediction.

3. 3.

   After that, we send each image feature and corresponding depth to the Prime Extraction module, obtaining the Prime Feature and the Prime Depth, which is the compressed feature and depth.

4. 4.

   Finally, with the Prime Feature, Prime Depth, and the pre-defined Ring & Ray Matrices, we use Eq. 6 (see also Fig. 1, lower) to obtain the BEV feature.

We conduct our experiments based on BEVDepth [10], the current state-of-the-art detector on the nuScenes benchmark. In order to conduct a fair comparison of performance and efficiency, we re-implement BEVDepth according to their paper. Unless otherwise specified, we use ResNet-50 [3] and VoVNet2-99 [8] pre-trained on DD3D [17] as the image backbone and SECOND FPN [31] as the image neck and BEV neck. The input image adopts pre-processing and data augmentations same as in [10]. We use BEV feature size $128\times 128$ for low input resolution and $256\times 256$ for high resolution on detection. Segmentation experiments use $200\times 200$ BEV resolution as in LSS [18]. We use the DepthNet [10] to predict categorical depth from 2m to 58m in nuScenes, with uniform 112 division. During training, CBGS [34] and model EMA are adopted. Models are trained to converge since MatrixVT converges a little slower than other methods, but no more than 30 epochs.

We compare the efficiency of View Transformation in two dimensions: Speed and Memory Consumption. Note that we measure the latency and memory footprint (using fp32) of view transformers since these metrics are affected by backbone and head design. We take the CPU as a representative general-purpose device where customized operators are unavailable. For a fair comparison, we measure and compare the other two Lift-Splat-like view transformers. The LS-BEVDet is the accelerated transformation used in BEVDet (with default parameters); the LS-BEVDepth uses the CUDA [15] operator proposed in BEVDepth [10] that is not available on CPU and other platforms. To demonstrate the characteristics of different methods, we define six transformation settings, namely S1$\sim$S6, varies in image feature size, BEV feature size, and feature channels that are closely related to model performance.

As can be seen in Fig. 6, the proposed MatrixVT boost the transformation significantly on the CPU, being 4 to 8 times faster than the LS-BEVDet[6]. On the CUDA platform [15], where customized operators enable faster transformation, MatrixVT still shows a much faster speed than the LS-BEVDepth under most settings. Besides, we calculate the number of intermediate variables during VT as an indicator of extra memory footprint. For MatrixVT, these variables include the Ring Matrix, Ray Matrix, and the intermediate matrix; for Lift-Splat, intermediate variables include the intermediate tensor and the predefined BEV grids. As illustrated in Fig. 6 (right), MatrixVT consumes 2 to 15 times less memory than LS-BEVDepth and 40 to 80 times less memory than LS-BEVDet.

In this section, we validate the effectiveness of Prime Extraction by performance comparison and visualization.

In the PFE, we propose using max pooling followed by several 1D convolutions to reduce and refine the image feature. Before reduction, the coordinate of each pixel is embedded into the feature as position embedding. We conduct experiments to validate the contribution of each design.

A possible alternative of the max pooling is the CollapseConv as in [19] and [21], which merges the height dimension into the channel dimension and reduces the merged channel by linear projection. However, the design of CollapseConv brings some disadvantages. For example, the merged dimension is of size $C\times H$, which is high and requires extra memory transportation. To address these problems, we propose using max pooling to reduce the image feature in Prime Extraction. Tab. 4 shows that reduction using max pooling achieves even better performance than CollapseConv while eliminating these shortcomings.

We also conduct experiments to show the effectiveness of the refine sub-net after reducing in Tab. 4. The results indicate that the refine sub-net plays a vital role in adapting the reduced feature to BEV space, without which a performance drop of 0.9% mAP will occur. Finally, the experimental results in Tab. 4 have shown that the position embedding brings an improvement of 0.4% mAP, which is also preferable.

Here, we give a brief description of symbols used in the below sections in Tab. 5.

Besides, letters of the normal script (e.g. $F$) denote tensors, letters of Roman script (e.g. $\mathrm{M}$) denote matrices, $\mathrm{A}\cdot\mathrm{B}$ denotes matrix multiplication, $\mathrm{A}\odot\mathrm{B}$ denotes Hadamard Product.

The generation of the Ring Matrix and the Ray Matrix relies on the intrinsic and extrinsic parameters of the camera setting. These parameters determine the geometrical relationship between the “lifted” features and the BEV grids. Note that these matrices need to be generated only once for real-world applications - as the camera positions are usually fixed.

To simplify the algorithm, we take the geometry (2D coordinate, $(x,y)$) of the “lifted” Prime Feature and the BEV grids (2D grids, $(x_{0},y_{0},x_{1},y_{1})$), instead of intrinsic and extrinsic parameters as input. One of the algorithms that generate these matrices is described in Alg. 1.

In MatrixVT, we reformulate our pipeline into a new form to eliminate huge intermediate tensors. Now we prove that these two forms are mathematically equivalent. For clarity, we mark the shape of variables in their upper right corner.

We omit the $C$ in the below equations for simplicity, the Prime Feature $\mathrm{F}_{BEV}^{S\times C}$ is therefore viewed as a vector $\boldsymbol{f}_{BEV}^{S\times 1}$. Since the “lift” operation is a per-pixel outer product of the feature and categorical depth, we rewrite the whole pipeline as follows:

where $\mathrm{F}_{I}^{W\times W}$ denotes a diagonal matrix that satisfies $diag(\mathrm{F}_{I}^{W\times W})=\boldsymbol{f}_{I}^{W\times 1}$.

Taking Eq. 8 into Eq. 7, the prime feature $\boldsymbol{f}_{BEV}$ can be derived as:

where $\mathrm{M}_{FT}^{S\times W}=\left(\mathrm{M}_{Ray}^{S\times W}\odot\left(\mathrm{M}_{Ring}^{S\times N_{d}}\cdot\mathrm{D}^{N_{d}\times W}\right)\right)$.

Since the size of $\mathrm{M_{FT}}$ is invariant to the feature channels, the memory footprint during transformation is saved.