A Novel Multi-Teacher Knowledge Distillation for Real-Time Object Detection using 4D Radar

To provide the context of our work, we briefly outline the Radar processing procedure illustrated in Fig.2, which transforms electromagnetic wave responses into Radar point clouds. The process begins by converting the received signal from analog to digital, followed by two Fast Fourier Transforms (FFT) to generate a 4-dimensional tensor (4DRT) containing range, azimuth, elevation, and Doppler measurements, as introduced in Sec.I. Finally, preprocessing (Alg.1) averages the Doppler dimension and retains a high-percentile subset of measurements, reducing noise while balancing sparsity, as shown in Fig.3.

As we exploit the dense 4D Radar tensor that have not undergone the preprocessing, our method belongs to the category of methods using raw Radar data. Since Radar data undergoes a multi-stage processing before yielding sparse point clouds, methods in this category take the ADC signal, or the Range-Doppler Map (RDM), or the 4DRT, as their input. The early work FFT-RadNet [16] directly detect objects in the RDM. Since this tensor lacks the explicit azimuth, which is crucial for 3D object detection, FFTRadNet implicitly estimate the azimuth in its latent space by forcing an axis of its feature map to represent the azimuth value. ADCNet [17] and T-FFTRadNet [18] go one step further by using the ADC signal. Based on a learnable form of the discrete Fourier transform, they devise a learnable signal processing module that transforms the input ADC signal into a latent representation of the range-azimuth-Doppler map. DPFT [19] takes the 4DRT as its input and project this tensor to the range-azimuth plane and the azimuth-elevation plane for feature extraction. The two resulting feature maps are fused with the camera’s through the deformable attention mechanism [20].

While the methods above use the raw Radar data for both training and inference, our method only requires the raw data for training. Specifically, we vary the percentile $r$ of the preprocessing to obtain high-density point clouds for training teacher models. During inference, the student model uses directly the preprocessed sparse point cloud. This ensures a low memory consumption and a high inference speed of the student model.

The idea of the knowledge distillation framework [21] is to use the feature map of a model, referred to as the teacher, to guide the feature computation of another model, referred to as the student. This is implemented by complemeting the task-specific loss function of the student with addtional terms measuring the similarity between its feature maps and the teacher’s. Prior works on knowledge distillation for Radar-based object detection distill LiDAR-based or camera-based models to leverage their relatively dense representation. RadarDistill [22] align the representation of a Radar and a LiDAR in BEV. Then, the Radar model is forced to emulate the representation of the LiDAR model in key regions. CRKD [23] take the similar key-region-based approach to transfer knowledge from a camera-based model. A development made by CRKD is the enforcement of the similarity at the global level through the application of the L1 loss on the affinity matrix of the representation of camera and of Radar.

Unlike cross-modal distillation methods, our approach requires no extrinsic calibration between Radar and other modalities, enhancing practicality given the complexity of Radar calibration [24, 25]. Moreover, differences in sensing capabilities, such as Radar’s ability to measure radial velocity versus RGB cameras’ light intensity perception, create unobservable features for the student, complicating distillation.

Our knowledge distillation scheme named 4DMT, shown in Fig.4, consists of $N$ teacher models and a student model. These models take Radar point clouds as their input. It worths noticing that each model (teachers and the student) use point clouds having a different density level. Teacher models and the student model use an identical (but not shared) backbone to compute the feature maps of their input Radar point cloud in BEV. Their backbone has a multi-stage architecture that progressively increases the semantic level of the feature map at each stage while reducing its spatial resolution. This is done through a chain of sparse 3D convolutions [26] similar to RTNH. An anchor-based detection head [27] detects objects using the feature map of the last stage.

To make input point clouds for each teacher, we choose a different percentile of power that the preprocessing module extracts from the 4DRT. Teacher models are trained using the object detection loss of RTNH. During the training of the student model, teacher models are not subjected to the weight update. An Aggregation Module, detailed in Sec.III-B, combines the output of the backbone of every teacher into a single feature map that plays the role of the target feature map of the student.

Similar to teacher models, we use a different percentile to make input point clouds for the student model. As the student model is intended to be deployed on resource-constrained hardware and has to satisfy a strict real-time requirement, we choose a high percentile, resulting in very sparse point clouds. Beside the backbone and the detection head, the student has a Densify Module, detailed in Sec.III-C, that upsamples the output of its backbone to alleviate the impact of the sparsity of its input. The loss function of the student is the weighted sum of the distillation loss, presented in Sec.III-D, and the same detection loss that used to train teacher models. During inference, the teachers and the aggregation module are removed.

The use of multiple teachers, each of which operates on a different level of point cloud densities, results in a diverse set of features for distillation. On the other hand, the sparsity of the student’s point clouds make certain features to be distilled uncomputable as certain regions of the environment are unobservable to the students. Therefore, it is necessary to select appropriate teacher features for distillation based on the student’s input. We achieve this with the Aggregation Module.

For a teacher $i$-th, its feature map at every stage is upsampled to obtain the same spatial dimension, then concanated along the channel dimension to make the tensor $\mathbf{F}^{T_{i}}$. Aggregation Module adaptively selects the useful features for distillation at each location in BEV from the set of all $\mathbf{F}^{T_{i}}$ through a series of 1-by-1 convolutions and channel concatenation operation, resulting a unified tensor $\mathbf{F}^{T}$.

The backbone of our models (teachers and the student) use a mix of two implementations of the sparse convolution: [28] and [29]. The former allows occupancy leaking from one layer to another by performing the convolution on any location whose neighborhood (defined by the convolution filter) has at least one active (non-zero value) sites. The result of the occupancy leaking is a reduction in the sparsity level of the feature map in deeper stages of the backbone.

Thanks to the occupancy leaking and a relatively high density of their input point clouds, the last feature map $\mathbf{F}^{T_{i}}$ of the backbone of each teacher has a relatively high density. On the other hand, input point clouds of the student have so few points that the last feature map of its backbone is still sparse, despite the occupancy leaking. As these feature maps are used for knowledge distillation, their density mismatch makes the optimization of the distillation loss more challenging. Inspired by prior works [30, 31], we propose a densify module that upsample the output of the student’s backbone.

Our densify module first reduces the resolution of its input using a deformable convolution [32] and a ConvNeXt layer [33], where the deformable convolution adapts to irregular inputs, and the ConvNeXt layer efficiently refines the extracted features. It then restores the resolution with a transposed convolution, producing a denser feature map better aligned with the teacher’s representation for improved knowledge distillation and object detection. The output of this module, referred to as $\mathbf{F}^{S}$, is used to compute the distillation loss and to detect objects.

The loss function $L_{total}$ used to train the student model is a weighted sum of the object detection loss $\L_{detect}$ and the knowledge distillation loss $L_{distill}$

Here, $\alpha$ and $\beta$ are hyperparameters that adjust the balance between the object detection loss and the knowledge distillation loss. They are set to 1 in our experiments.

$L_{distill}$ is the Mean Squared Error (MSE) between the aggregation of teachers’ feature map $\mathbf{F}^{T}$ and the student’s densified feature map $\mathbf{F}^{S}$. Since features in regions where objects exist are more important for detecting objects than background, we mask both $\mathbf{F}^{T}$ and $\mathbf{F}^{S}$ using ground truth when calculating the distillation loss. Specifically, we transform each ground truth object into a 2D Gaussian and splat it to the BEV to obtain a heatmap. This heatmap is multiplied element-wise with $\mathbf{F}^{T}$ and $\mathbf{F}^{S}$ to realize the masking operation.

We use the loss function defined by RTNH [7] as the detection loss $L_{detect}$.

To showcase the impact of knowledge distillation on the student model, we compare the detection performance and the resource efficiency of our 4DMT, which is an RTNH model enhanced by our knowledge distillation framework, and two original RTNH. Our 4DMT and one RTNH use point clouds produced at the 99.9th percentile while while the other RTNH baseline uses a lower percentile of 95. The lower percentile results in point clouds that are, on average, 50 times denser. The result presented in Tab.I shows that our 4DMT achieves the highest 3D AP in the Sedan category, especially 25% improvement compared to the RTNH baseline with the same density level.

Furthermore, our 4DMT has the second highest inference speed which is 30 FPS on NVIDIA RTX 3090. These results show that our distillation framework strikes a good balance between precision and efficiency. The underperformance of our 4DMT compared to RTNH using denser point clouds (95-th percentile) in the ”Bus or Truck” category is explained by the misclassification of large vehicles as multiple smaller vehicles due to the sparsity of point clouds at 99.9-th percentile. This is illustrated in Fig.5,