FB-OCC: 3D Occupancy Prediction based on Forward-Backward
View Transformation

Here, we give an introduction to FB-BEV. It is known that view transformation is a central module of camera-based 3D perception. In the literature, this module is based on two dominant view transformation strategies: forward projection (represented by List-Splat-Shoot [5]) and backward projection (represented by BEVFormer [2]). FB-BEV adopts a unified design that leverages both methods, promoting the benefits from each method with improved perception results while overcoming their limitations. In the case of FB-OCC, we use forward projection to generate the initial 3D voxel representation. We then condense the 3D voxel representations into a flattened BEV feature map. The BEV feature map is treated as queries within the BEV space and attends the image encoder features to acquire dense geometry information. The fusion features of the 3D voxel representation and the optimized BEV representations are then fed into the subsequent task head.

In the forward projection module, we adhere to the principles of Lift-Splat-Shoot (LSS) [5] to account for the uncertainty in the depth estimation of each pixel. This allows us to project the image features into the 3D space based on their corresponding depth values. In contrast to LSS, which models BEV features, we directly model 3D voxel representations to capture more detailed information in the 3D space. Additionally, we adopt BEVDepth [4] to utilize point clouds in generating accurate depth ground truth, which helps supervise the depth prediction of our model for improved accuracy. LSS tends to produce relatively sparse 3D representations. To tackle this issue, we incorporate a backward projection method to optimize these sparse 3D representations. Considering the computational burden, we employ BEV representation instead of 3D voxel representation at this stage. The backward projection method draws inspiration from BEVFormer [2]. However, unlike BEVFormer, which employs randomly initialized parameters as BEV queries, we compress the obtained 3D voxel representation into a BEV representation, thereby incorporating stronger semantic priors. Furthermore, our backward projection method leverages the depth distribution during the projection phase, enabling more precise modeling of projection relationships.

Following the acquisition of the 3D voxel representations and optimized BEV representation, we combine them through the process of expanding the BEV features, resulting in the final 3D voxel representations. The voxel encoder and the occupancy prediction head, as depicted in Figure 1 and Figure 2, are outlined below.

To train the model, we use a distance-aware Focal loss function $L_{fl}$ inspired by M2BEV [6], Dice loss $L_{dl}$, affinity loss $L_{scal}^{geo}$ and $L_{scal}^{sem}$ from MonoScene [7], lovasz-softmax loss $L_{ls}$ from OpenOccupancy [8]. In addition, we also need a depth supervision loss $L_{d}$ and a 2D semantic loss $L_{s}$, which will be introduced in the next section.

Scaling the model size has traditionally been a convenient approach to improving model accuracy. However, in the field of 3D vision-only perception, researchers have discovered that employing a more powerful 2D backbone often leads to overfitting [3]. For instance, on the nuScenes 3D object detection task, the largest backbone, such as VIT-L [9] with approximately 300M parameters, and commonly used backbones like ConvNext-B [10] and VoVNet-99 [11] with around 100M parameters, tend to encounter this issue. To address this challenge, we explore the utilization of the 1B-parameter backbone, InternImage-H [12], for multi-camera 3D perception tasks. However, directly applying this backbone would result in severe overfitting due to the limited number of samples available for training, specifically the 40K samples in the nuScenes dataset [13]. To overcome this limitation, we leverage the opportunity provided by this competition, which allows participants to utilize additional public data. By augmenting our data resources, we can train our large-scale model more effectively. Building upon the open-source InternImage-H checkpoint, we conduct model training on the Object365 dataset [14], which is a vast 2D object detection dataset comprising 2 million images. This pre-training on large-scale 2D detection tasks enhances the model’s semantic perception capabilities. However, there still exists a certain domain gap when applying the pre-trained model to downstream 3D perception tasks. Therefore, we further perform targeted pre-training on the model specifically for 3D perception tasks. An effective approach for pre-training is to enhance the model’s geometric awareness through depth estimation tasks. Consequently, we conduct extensive pre-training on the nuScenes dataset, primarily focusing on depth estimation. It is worth noting that depth pre-training lacks semantic-level supervision. To mitigate the risk of the model becoming excessively biased towards depth information, potentially leading to the loss of semantic priors (especially given the large-scale nature of the model, which is prone to overfitting), we simultaneously aim to predict the 2D semantic segmentation labels alongside the depth prediction task, as shown in Figure 3.

However, nuScenes does not provide semantic segmentation labels for 2D images. To address this issue, we employ the popular Segment Anything Model (SAM) [15] for automatic labeling. For thing categories with bounding box annotations provided by nuScenes, we utilize box prompts to generate high-quality semantic masks for each object. Unfortunately, for stuff categories such as road surfaces or buildings, bounding box annotations are not available. Nonetheless, nuScenes offers corresponding point cloud semantic segmentation labels for these categories.

To generate semantic masks for the stuff categories, we project LiDAR points belonging to these categories onto the image. For each category, we randomly select three points as point prompts to generate the corresponding semantic masks, which yields satisfactory mask quality. With the 2D image semantic mask labels and the ground truth depth maps, we train the model using a joint depth estimation task and semantic segmentation task. This pre-training task closely aligns with the final occupancy prediction task, enabling the direct generation of 3D occupancy results using depth values and semantic labels. The pre-trained model serves as an improved starting point for the subsequent training of the occupancy prediction task.

Training large-scale models requires huge computing resources. In our exploration, we first verify the effects of different models at a smaller scale. In this setting, the input scale is $256\times 704$, the resolution is $100\times 100\times 8$, and the image backbone is ResNet-50 [19]. We list the milestones of our exploration in Table. 1. Version A is our vanilla baseline. In Version B, we use depth supervision following BEVDepth. For Version C, we ignore the invisible voxels from cameras during the training phase. Version D model fixes several major bugs in Version C, especially regarding the abnormal IoU on the other category. For Version E, we use temporal information from the previous 16 frames. We leverage the joint depth and semantic pre-training in Version F. For Version G, we optimize the loss function by adding the Dice loss and using 3D transformation to align voxel features from different timesteps. The results of Version H is the test-time augmentation results of Version G.

After exploring the basic design of FB-OCC, we scale up the model size by using larger backbones and image input size, as shown in Table 2. Compared to Version H, version I uses VoVNet-99 backbone and other modifications, including using $960\times 1760$ image input and a voxel resolution of 0.4m. Compared to Version I, Version J leverages a ViT-L backbone and ViT-Adapter [20]. For our most powerful Version K, we scale up the model to over 1 billion parameters with the InternImage-H backbone.

In our final submission, we adopt our ensemble strategy with seven models for the best accuracy. The main difference between different models is the use of different backbones. By combining the results of all the models through ensemble techniques, we were able to achieve our best result on the test set with a mIoU score of 54.19%.