SOGDet: Semantic-Occupancy Guided Multi-view 3D Object Detection

has emerged as a popular task in the past two years [5; 16; 20; 28; 39; 37]. It involves assigning an occupancy probability to each voxel in the 3D space. The task offers useful 3D representations for multi-shot scene reconstruction, as it ensures the consistency of multi-shot geometry and helps obscured parts to be recovered [32].

The existing methods are relatively sparse in the literature. MonoScene [5] is the pioneering work that uses monocular images to infer dense 3D voxelized semantic scenes. However, simply fusing multi-camera results with cross-camera post-processing often leads to sub-optimal results. VoxFormer [20] devises a two-stage framework to output the full 3D volumetric semantics from 2D images. The first stage uses a sparse collection of depth-estimated visible and occupied voxels, followed by a densification stage that generates dense 3D voxels from the sparse ones. TPVFormer [16] performs end-to-end training by using sparse LiDAR points as supervision, resulting in more accurate occupancy predictions.

It is worth noting that the methods discussed above are specifically designed to tackle the 3D OC task. However, our focus in this paper is to improve 3D OD by incorporating 3D semantic-occupancy as a supportive branch. As a result, we empirically resort to a simple network structure of OC to validate the key idea and do not adopt the standard OC labeling protocols following [16].

The overall architecture of our proposed method is illustrated in Figure 2. It is composed of three main components: an image backbone, a view transformer, and a task stage that predicts both OC and OD simultaneously. Specifically, the multi-view input images are first encoded by the image backbone, and then aggregated and transformed into the Bird-Eye-View (BEV) feature by the view transformer. With inherent camera parameters, the view transformer conducts depth-aware multi-view fusion and 4D temporal fusion simultaneously. Thereafter, the task stage generates both OC and OD features, which are interacted using a modality fusion module. We finally predict the OD and OC outputs using their respective features.

To ensure the clearance and consistency throughout our presentation, we first define the following notations following the order of data flow within our pipeline.

${I}$ represents an image group with same height and width from $N$ cameras with the same timestamp. $\boldsymbol{F_{img}}\in\mathbb{R}^{N\times C\times H\times W}$ represents feature map produced by the image backbone. ${H}$, ${W}$ and ${C}$ means the height, width and channels of the feature map, respectively. $\boldsymbol{F_{d}}\in\mathbb{R}^{N\times D\times H\times W}$ represents depth estimation of the multi-view image group ${I}$. $\boldsymbol{F_{bev}}\in\mathbb{R}^{C_{bev}\times X\times Y}$ represents BEV features extracted by the view transformer. The dimensions of the BEV plane are represented by $X\times Y$, and the number of channels in the BEV feature is $C_{bev}$, which is set to 128 following [14]. $\boldsymbol{F_{od}}$ and $\boldsymbol{F_{oc}}$ represent task-specific intermediate features of OD and OC branches in task stage.

For the camera parameters, we combine the offset vector and rotation matrix to represent the translation $\boldsymbol{TR}\in\mathbb{R}^{4\times 4}$ from source coordinate system to target coordinate system. For example, $\boldsymbol{{TR}_{cam}^{lid}}$ means a translation from camera coordinate system to lidar coordinate system. And $\boldsymbol{TR_{in}}$ represents the intrinsic parameters of all cameras.

For the output, the OD branch has two outputs: Bounding Box $\boldsymbol{B}\in\mathbb{R}^{M\times(3+3+2+2+1)}$ and Heatmap $\boldsymbol{H}$, where $M$ is the total number of bounding boxes and the second dimension of $\boldsymbol{B}$ represents location, scale, orientation, velocity and attribute. $\boldsymbol{Occ}\in\mathbb{R}^{O\times X\times Y\times Z}$ represents the OC branch output, which means that for the different grids from voxel grid $\boldsymbol{V}\in\mathbb{R}^{X\times Y\times Z}$, there are $O$ semantic labels in total. And we generate the occupancy voxel grid from point cloud $\boldsymbol{P}\in\mathbb{R}^{K\times 3}$ of $K$ points.

The image backbone encodes the multi-view input images ${I}$ into the feature map $\boldsymbol{F_{img}}$. Following previous work [15; 14], we sequentially concatenate ResNet [13] and FPN [22] as our image backbone to extract the image feature. Moreover, we empirically found that using ShapeConv [6] instead of traditional convolutional layers in the image backbone leads to improved accuracy in the OD task without increasing model complexity during inference. In view of this, all ResNet-50 and -100 models in our method and baseine are replaced with ShapeConv for a fair comparison. Detailed ablation studies on the performance obtained by ShapeConv can be found in the Supp.

The view transformer converts the image feature $\boldsymbol{F_{img}}$ to the BEV feature $\boldsymbol{F_{bev}}$. We implement this module with the combination of BEVDepth [19] and BEVDet4D [14] for better performance, namely BEVDet4D-depth, which jointly conducts depth-aware multi-view fusion and 4D temporal fusion based on BEVDepth and BEVDet4D, respectively.

The task stage consists of two branches that take the BEV feature $\boldsymbol{F_{bev}}$ as the input to obtain the Bounding Box $\boldsymbol{B}$ and Heatmap $\boldsymbol{H}$ outputs for OD branch and the Occpancy output $\boldsymbol{Occ}$ for OC branch, respectively.

On the one hand, the OD branch is our primary task branch, which performs a 10-class object detection on car, truck, etc. On the other hand, the OC branch is to facilitate object detection by generating a 3D geometrical voxel around the ego vehicle.

To refine the BEV feature $\boldsymbol{F_{bev}}$ in both branches, we first apply a 3-layers ResNet [13] to extract intermediate features $\boldsymbol{F_{od}}$ and $\boldsymbol{F_{oc}}$ in three different resolution, which are 1/2, 1/4, 1/8 of the height, width respectively. A pyramid network [22] is then employed to upsample the features to the same size as the original one. For the OD branch, we use CenterPoint [41] to produce the final OD bounding box prediction heatmap $\boldsymbol{H}$ and bounding box $\boldsymbol{B}$ from $\boldsymbol{F_{od}}$. For the OC branch, a simple 3D-Conv Head [11] is used to generate occupancy voxel grid $\boldsymbol{Occ}$ from $\boldsymbol{F_{oc}}$.

We leverage two types of supervision signals from the occupancy prediction for the OC branch. One is binary occupancy label BO, whose supervision is binary with 0 and 1 representing empty and occupied voxels, respectively. The other is semantic label SE, containing 16 semantic labels such as barrier, bicycle, etc. Figure 3 illustrates the two types of label.

To generate the binary occupancy labels, we consider only the geometry features of each voxel and illustrate this procedure in Algorithm 1. This approach is cost-friendly and require no extra manual annotations.

For semantic label, we observe that directly using the sparse semantic occupancy points as ground-truth labels leads to unstable training. Therefore, we follow TPVFormer [16] to optimize the supervision voxel generation, where the voxels without semantic labels are masked and ignored. We detail this labeling process in the Supp.

We evaluated the performance of our SOGDet model against other state-of-the-art multi-view 3D object detectors on the nuScenes validation and test sets.

Table 1 reports the results for the validation set using Swin-Tiny, -Small, ResNet-50 and -101 backbones (detailed results can be found at the Supp.). As shown in the table, our method achieves highly favorable model performance, with NDS scores of 50.2% and 55.4% for SOGDet-BO and 50.6% and 56.6% for SOGDet-SE on ResNet-50 and -101, respectively. These results surpass current state-of-the-art multi-view 3D object detectors with a large margin, including BEVDepth [19] (3.1% improvement in NDS at both ResNet-50 and -100) and AEDet [12] (0.5% improvement in NDS at both ResNet-50 and -100).

In Table 2, we present the results obtained by SOGDet with the ResNet-101 backbone on the nuScenes test set, where we report the performance of state-of-the-art methods that use the same backbone network for a fair comparison. We follow the same training strategy of existing approaches [19; 12] that utilize both the training and validation sets to retrain the networks and without any test-time augmentation. SOGDet shows improved performance in multi-view 3D OD task with 58.1% NDS and 47.4% mAP, further verifying the effectiveness of our proposed approach.

Figure 4 illustrates qualitative results of our approach on the nuScenes [3] dataset using ResNet-50 as the backbone for both OD and the OC branch, more results can be found in the Supp. Pertaining to the object detection task, we focus only on occupied voxels, and therefore, locations marked as “empty” are not shown. The hybrid features reveal strong correlations between the physical structures and the location of the detected objects, such as vehicles, bicycles, and pedestrians. For example, vehicles are typically detected in drive surface, while bicycles and pedestrians are often detected on sidewalk. These findings are consistent with the observations and motivations of our paper and demonstrate that the integration of the two branches can lead to a better perception and understanding of the real world.