LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera

LiDAR/Camera-based methods. LiDAR-based methods [8, 9, 7, 6, 10, 11, 12] use LiDARs for precise 3D semantic occupancy prediction. SSCNet [13] and UDNet [8] utilize 3D U-Nets but face computation overhead from empty voxels. LMSCNet [9] and SGCNet [14] improve efficiency with 2D CNNs and spatial group convolutions, respectively. Advanced methods focus on multi-view fusion [15], local implicit functions [16], knowledge distillation [10], and BEV representation [11, 7]. Recently, Pasco [12] further extends the SSC task with instance-level information to produce a richer 3D scene understanding. Camera-based methods [17, 18, 19, 20, 21, 22, 23] have become popular due to their rich visual cues and cost-effectiveness. Many methods have explored effective 3D scene representation learning for outdoor surrounding SSC. For instance, TPVFormer [17] introduces a tri-perspective view for detailed 3D structure representation. OccFormer [19] utilizes transformers for multi-scale voxel features. PanoOcc [21] achieves a unified occupancy representation for comprehensive 3D scene understanding.

Multi-modal methods [5, 4, 24] combines multi-source sensor data (e.g., images, LiDARs, and radars) to perform robust outdoor SSC. OpenOccupancy [4] provides a surrounding benchmark and establishes camera-based, LiDAR-based and LiDAR-camera baselines. Recently, radar perception [25, 26, 27, 28, 29] has garnered wide attention in multi-modal 3D detection task due to its cost-effectiveness, robustness against adverse weather conditions, and ability to detect distant objects. However, there are only a few works [24] to incorporate radar for outdoor SSC tasks. For instance, the recent OccFusion [24] devises a sensor fusion framework to integrate features from LiDARs, surround view images, and radars for robust and accurate SSC. We focus more on the distillation design of LC modes distilled to RC or R than on the direct fusion of multiple modes.

The concept of Knowledge Distillation (KD) was initially proposed for model compression and performance improvement in image classification tasks [30]. It has since been extended to other fields, such as 3D object detection [25, 31, 32], 3D segmentation [33, 34], and semantic scene completion [10, 35].

In SSC tasks, SCPNet [10] introduces Dense-to-Sparse Knowledge Distillation (DSKD) to transfer dense, relation-based semantic knowledge from a multi-LiDAR teacher to a single-LiDAR student, significantly boosting the representation learning of the student model. Similarly, MonoOcc [35] leverages knowledge distillation to transfer temporal information and visual cues to a monocular semantic occupancy framework. Different from the above methods that apply knowledge distillation in the same modality, our LiCROcc proposes fusion-based cross-modal knowledge distillation in a shared BEV space for outdoor SSC, passing informative cues from the LiDAR-camera features to the radar-camera features for performance improvement.

As mentioned above, we construct the radar-based baseline and design the radar-camera fusion network (the bottom part of Fig. 2) to boost the baseline’s performance. To leverage the guidance of detailed geometric structure and point representation in LiDAR-camera fusion, we further utilize the fusion-based KD (Section III-C) to transfer the knowledge from the LiDAR-camera fusion network (the top part of Fig. 2) to the radar-based baseline and radar-camera fusion network. We employ the same architecture, i.e., the multi-modal fusion network (Section III-B), to establish the above two fusion networks.

The multi-modal fusion network mainly consists of the image branch for extracting image features, the point branch for encoding LiDAR/radar points, and the multi-modal BEV fusion network for effectively and efficiently integrating point and image features hierarchically.

Image branch. Following FlashOcc [20], we propose to project surrounding image features to BEV space for subsequent processing, alleviating the memory overhead while maintaining the high accuracy of occupancy prediction. As shown in Fig. 2, the image branch mainly consists of three components: the camera encoder for image features, the PV-to-BEV projection layer for BEV representation of the 3D scene, and the BEV encoder for hierarchical BEV features $(\bm{F}_{c,0},\bm{F}_{c,1},\bm{F}_{c,2}$, where $\bm{F}_{c,i}\in\mathbb{R}^{C_{i},H_{i},W_{i}})$ that contain the rich semantic context. The extracted multi-scale BEV features are fed into the BEV fusion model to interact with the point features, which will be elaborated on below.

Point branch. Without losing generality, we adopt the recent BEV-based SSC-RS [7] as our point branch. This branch uses two independent branches for semantic and geometric encoding. The BEV fusion network with an ARF module [7] aggregates features from these branches, resulting in the final semantic scene completion. Due to its disentangled design, SSC-RS is lightweight and has strong representation ability, making it very suitable for use as the point branch. The point branch takes the LiDAR/radar point cloud $P$ and outputs the multi-scale BEV features $(\bm{F}_{p,0},\bm{F}_{p,1},\bm{F}_{p,2}$, where $\bm{F}_{p,i}\in\mathbb{R}^{C_{i},H_{i},W_{i}}))$. For LiDAR point cloud, the $\bm{P}\in\mathbb{R}^{N\times 3}$ is in the range of $[R_{x},R_{y},R_{z}]$. Moreover for the radar point cloud, $\bm{P}\in\mathbb{R}^{N\times 6}$ is the concatenation of the $xyz$ coordinates, radar cross section $\sigma$, and the $xy$ velocities compensated by the ego-motion.

Multi-modal BEV Fusion network. Due to the computational burden associated with 3D convolutions for dense feature fusion, we introduce a multi-modal bev fusion network, drawing inspiration from BEV perception tasks. This network efficiently combines semantically rich visual BEV representations $(\bm{F}_{c,0},\bm{F}_{c,1},\bm{F}_{c,2})$, geometrically informative LiDAR features or weather-resistant radar features. To streamline the fusion process, we unify LiDAR or radar point cloud features with $(\bm{F}_{p,0},\bm{F}_{p,1},\bm{F}_{p,2})$. Similarly to [7], our BEV fusion network employsa 2D convolutional U-Net architecture. Each residual block reduces the input feature resolution by a factor of 2 to maintain consistency with semantic/complementary features. Before each subsequent block, we integrate the previous stage’s $\bm{F}_{b,i-1}$ with the current stage’s $\bm{F}_{p,i}$ using ARF [7] to obtain $\bm{F}_{b_{i}}$, and then the scaled $\bm{F}_{c,i}$ is fused to $\bm{F}_{b,i}$ by addition. The decoder upsamples the encoder’s compressed features three times by a factor of two by skipping connections. The final decoder convolution generates the SSC prediction $\bm{Y}\in\mathbb{R}^{((C_{n}+1)\cdot Z)\times H\times W}$, where $C_{n}$ denotes the number of semantic classes. To represent the voxel-wise semantic occupancy probabilities, $\bm{Y}$ is reshaped into $(({C_{n}+1})\times{Z}\times H\times W)$.

To train the proposed fusion model, cross-entropy loss $\mathcal{L}_{ce}$ is used to optimize the network. In addition, following [22], we also utilize affinity loss $\mathcal{L}_{\rm{scal}}^{\rm{geo}}$ and $\mathcal{L}_{\rm{scal}}^{\rm{sem}}$ to optimize the metrics in the scene and the class (i.e., geometric IoU, and semantic mIoU). Therefore, the BEV fusion loss function can be derived as:

As shown in Fig. 2, our teacher and student model share the same network structure. Thanks to the fact that both the fusion and distillation processes are under BEV, the image branching of the student model is optional. In this section, we use $(\bm{F}_{s,0},\bm{F}_{s,1},\bm{F}_{s,2},\bm{F}_{s,3})$ to represent the four BEV features of the student model, and $(\bm{F}_{t,0},\bm{F}_{t,1},\bm{F}_{t,2},\bm{F}_{t,3})$ to represent the corresponding features of the teacher model (illustrated in light blue in Fig. 2 and Fig. 3).

During the training phase, we adopt a multi-task training strategy to guide the various components effectively. For 3D semantic scene completion, we utilize the $\mathcal{L}_{bev}$ loss. To enhance the distillation process, we combine three distillation components: $\mathcal{L}_{red}$, $\mathcal{L}_{brd}$, and $\mathcal{L}_{prdd}$. Additionally, we retain the semantic ($L_{s}$) and occupancy ($L_{c}$) losses from SSC-RS [7] to supervise feature point cloud extraction. The overall loss function is represented as follows:

where $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$ and $\lambda_{5}$ are hyper-parameters.

Datasets. We evaluate our approach on the nuScenes-Occupancy [4] benchmark, which comprises 850 scenes (700 for training and 150 for validation) with 34,000 keyframes at the 3D volume with the size of $40\times 512\times 512$. The dataset covers an area of -51.2 to 51.2 meters in the $xy$ plane and -5 to 3 meters in the $z$-axis. It provides the voxel annotations with 17 categories and a $0.2m\times 0.2m\times 0.2m$ resolution.

Metrics. Following [13], we employ Intersection-over-Union (IoU) for scene completion (excluding semantics), and the mean Intersection-over-Union (mIoU) for semantic scene completion (no “noise” class), as the validation metrics.

For LiDAR inputs, we concatenate 10 LiDAR sweeps as a keyframe similar to [4, 6]. We leverage a pre-trained ResNet50 [39] on ImageNet [40] as the image backbone to process the camera images with the input resolution of $256\times 704$. During the training, we project the point cloud onto the camera view to provide depth supervision of the LSS [41]. For radar input, we adopt the preprocessing process in CRN [1], using radar scans stitched from 5 radar sensors of the car. For data augmentation, we randomly apply horizontal and vertical flips and cropping to the images. The point clouds are augmented by the random flipping on the $x$-axis and $y$-axis. We employ the AdamW optimizer with a weight decay of 0.01 and an initial learning rate of 2e-4. We use the cosine learning rate scheduler with linear warming up in the first 500 iterations. All experiments are conducted on 8 NVIDIA A100 GPUs with a total batch size 32 for 24 epochs.

Tab. I shows the comparison results of our method against the state-of-the-art methods on the nuScenes-Occupancy [4] benchmark. Compared with all previous methods, our LiCROcc achieves the best performance under the same configuration. For example, our LiDAR-camera fusion model LC-Fusion shows significant improvements, with a 23% increase in mIoU and an 18% increase in IoU compared to the baseline (M-CONet [4]). Meanwhile, LC-Fusion achieves the improvements of 3.3% and 2.3% in terms of mIoU and IoU scores over PointOCC [6], which emphasizes the effectiveness of our proposed multi-modal BEV fusion.

To comprehensively evaluate the effectiveness of our proposed method, we have modified several existing LiDAR-based and multi-modal methods (CONet [4], SSC-RS [7], PointOcc [6], and CoOCC [5]), to accommodate radar input, serving as our comparisons. As shown in the second part of Tab. I, our R-LiCROcc outperforms the second-best one (PointOcc) and the baseline (R-SSC-RS) by 13.3% on the mIoU scores, demonstrating the effectiveness of our proposed fusion-based knowledge distillation. We find the IoU score of our R-LiCROcc is slightly lower than PointOcc. We explain that this can be attributed to the fact that PointOcc projects features on three planes and uses a larger model, which may be more beneficial for occupancy prediction. For radar-camera fusion, we take CONet and CoOCC as our baselines. The results are presented in the third part of Tab. I and show that our radar-camera fusion version RC-Fusion has achieved comparable performance to these baselines. The proposed fusion-based knowledge distillation further boosts the performance by 1.5 and 0.8 in terms of mIoU and IoU. We also provide the visualization in Fig. 4 illustrates that our RC-LiCROcc and R-LiCROcc enable more complete scene completion and precise object segmentation.

We conduct a series of experiments to validate the proposed module and the potential of radar as a sensor for semantic scene completion tasks. All experiments are conducted under the same training configuration and evaluated according to nuScene-Occupancy [4] validation dataset.