Enhanced K-Radar: Optimal Density Reduction to Improve Detection Performance and Accessibility of 4D Radar Tensor-based Object Detection ^††thanks: ^†co-first authors, ^∗corresponding author

Perception is an essential module for autonomous driving systems because the information acquired by the perception module will be used as inputs for the subsequent planning and control modules. Therefore, a robust perception module that can operate under challenging driving conditions (e.g., adverse weather conditions) is urgently needed.

In recent years, numerous studies have introduced deep learning-based perception modules with remarkable accuracy for various autonomous driving tasks such as lane detection [1, 2, 3, 4] and object detection[5, 6, 7, 8, 9]. These studies often rely on RGB images as inputs to the neural networks, mainly due to the abundance of camera-based datasets available to the public. In addition, RGB images have a relatively straightforward data structure with low dimensionality and high correlation between neighboring pixels, which enable the neural networks to learn high-dimensional representations efficiently. However, RGB cameras are vulnerable to low illumination conditions, can be easily obstructed by raindrops and snowflakes, and lack of depth information that is crucial for proper 3D understanding of the surroundings. In contrast, the LiDAR sensors use infrared signals to measure the surroundings with up to cm-level resolution and without being affected by the illumination conditions. However, infrared signals with a wavelenght of about $\lambda$ = 850nm $\sim$ 1,550nm cannot pass through raindrops or snowflakes, which results in unreliable measurements under adverse weather conditions [10].

Radar sensors, on the other hand, utilizes signals with a longer wavelength ($\lambda\approx$ 4mm) compared to LiDAR sensors. This enables Radar signals to pass through raindrops and snowflakes, allowing for accurate measurements even under adverse weather conditions. The robustness of Radar sensors (particularly Frequency Modulated Continuous Wave (FMCW) Radars) in adverse weather conditions has been studied in several works [11, 12, 13]. Additionally, FMCW Radars can be easily implemented into hardwares, resulting in the widespread use of FMCW Radar in the automotive industry.

As illustrated in Fig. 1, an FMCW Radar output is a Radar Tensor (RT), a dense tensor that is populated by non-zero power measurements in all axes. The RT is obtained by applying the Fast Fourier Transform (FFT) algorithm on the hardware-processed FMCW signals. Due to the density, an RT provides rich information regarding the environment, but at a cost of a large amount of memory for storage and computations.

With the availability of dense RTs, many studies [11, 14, 15, 13] have proposed RT-based object detection networks that achieve similar detection performance to camera and Lidar-based object detection networks. In particular, the K-Radar dataset [11] provides a collection of 4D Radar Tensor (4DRT) that consists of power measurements along the Doppler, range, azimuth, and elevation dimensions. This is in contrast to the conventional 3D Radar tensor (3DRT) [14, 15, 13] that do not provide elevation information. The importance of the additional elevation information has been shown in [11], where the 4DRT-based Radar Tensor Network with Height (RTNH) significantly outperforms the Radar Tensor Network without height (RTN) in the 3D object detection task. In addition, the 4DRT-based RTNH achieves similar 3D object detection results to LiDAR point cloud-based (LPC-based) neural network, PointPillars [5], in road environments under clear weather conditions, and significantly outperforms the LPC-based network in adverse weather conditions such as sleet and heavy snow. These results indicate the importance of 4D Radar sensors for a robust perception in adverse weather conditions.

While the advantages of 4DRT-based networks are clear, it remains challenging to conduct experiments on the 4DRT data. This is mainly because of the size of the 4DRT data is prohibitively large (i.e., $\sim$12TB). In the prior work [11], the size of the 4DRT data is reduced by performing a density reduction online during training, where the ouput density level is chosen arbitrarily. The effects of density reduction on the detection performance and memory consumption, however, remains largely unknown.

In this paper, we aim to address this issue by conducting extensive hyperparamter tuning on the density reduction level. As expected, experimental results show that increasing the density level from 0.01% to 50% of the original 4DRT density level proportionally improves the detection performance, at a cost of memory consumption. Interestingly, however, when the density level is increased beyond 5%, only the memory consumption increases, while the detection performance oscillates below the peak point. This optimized density reduction level could act as a guide for the automotive radar industry for designing the pre-processing steps at hardwarde-level implementations.

In addition to the optimized density reduction hyperparameter, we introduce the 4D Sparse Radar Tensor (4DSRT), a new representation for 4D Radar data. Unlike 4DRT, we perform the polar-to-Cartesian transformation and the density reduction offline, which significantly reduce the raw data size of 4DSRT. When the 4DSRT is used along with the optimized development kit for training the neural networks, the training speed is improved by a factor of 17.1 times compared with the original 4DRT-based neural networks.

In a summary, our contributions are as follows:

• •

  We conduct an extensive hyperparameter tuning for the density reduction level of the 4DRT. We observe that increasing the density level up to 5% proportionally improves the detection performance, but provides there is no clear benefits beyond that. This insight can serve as a hardware guide for the 4D Radar industry.

• •

  We propose 4D sparse Radar Tensor (4DSRT), a new representation for 4D Radar data that allows for a reduction in memory size. The new representation could improve accessibility of 4D Radar data, especially for resource-limited environments.

• •

  We provide an optimized devkit for 4DSRT which, when used along with the 4DSRT, could improve the training speed by a factor of 17.1 compared to the 4DRT-based neural networks.

The rest of this paper is structured as follows: Section 2 provides an overview of the K-Radar dataset and the 4DRT-based baseline neural network, RTNH. Section 3 explains the 4DSRT data presented in this paper. Section 4 presents the experimental results of the RTNH using 4DSRT with various density. Finally, Section 5 summarizes the paper.

In this section, we provide an overview of the related works, especially on the K-Radar dataset and the Radar Tensor Network with Height (RTNH) [11].

As shown in Fig. 3, a 4D Sparse Radar Tensor (4DSRT) is the sparse representation of a 4D Radar Tensor (4DRT) that can be used as an input to 4D Radar-based neural networks such as the RTNH. To construct a 4DSRT, we transform the 4DRT from a polar coordinate into a Cartesian coordinate, and then perform a pooling operation, where the top-$N$% elements with the highest power measurements are retained. These post-pooling values are then used as the input to the nural networks. Note that unlike [11] that performs the coordinate transformation and pooling operation at every iteration of the training process, we only need to perform the transformation and pooling once for each unique 4DRT, and reuse the corresponding 4DSRT for every subsequent iterations.

Compared with a 4DRT, a 4DSRT requires significantly lower memory and number of computations because the number of elements in a 4DSRT is only $N$% of a 4DRT. As a result, the advantages of utilizing 4DSRT are twofold. First, the 4DSRT representation improves the accessibility of the K-Radar dataset. Since the 4D Radar data size is significantly reduced, we can easily host the complete dataset in a commercial cloud-based storage service. This would bring the advantage of higher uptime and download bandwidth compared with hosting the dataset in a local server, as is the case of the original K-Radar. Therefore, a higher number of uninterrupted parallel access to the dataset can be supported.

Second, utilizing 4DSRT significantly improves the training speed. This is because when we train using 4DRTs, the majority of the training time is used for reading the 4DRT elements from the disk, and for pre-processing the 4DRT with Polar-to-Cartesian transform with interpolation that requires a large number of computations. Because the number of elements in the 4DSRT is significantly smaller, and the pre-processing is only performed once, we observe that the training speed of 4DSRT-based networks can be improved by a factor of 17.1 compared with 4DRT-based networks.

One of the most important hyperparameter for 4DSRT is the density reduction level $N$. In prior work [11], for the online density reduction of 4DRT, $N$ is arbitrarily chosen as 10%. However, we observe that $N$ has profound effects on both detection performance and memory consumption, and therefore should be chosen carefuly. We discuss the most optimal value for $N$ in Subsection IV. C.

In this section, we first describe the experiment setup and metrics that are used in the experiments. Then, we discuss the experiment results of 4DSRT-based RTNH with various density of 4DSRT. We also provide a comparison of training speed between 4DSRT-based RTNH and 4DRT-based RTNH.

In this paper, we have proposed the 4D Sparse Radar Tensor (4DSRT), a sparse representation of 4D Radar data with significantly lower data size compared with the dense 4D Radar Tensor (4DRT). Unlike prior work, that arbitrarily choose the density reduction level, we have conducted an extensive hyperparameter tuning to find the most optimal density reduction level for the 4DSRT. We have found that a 5% density reduction would result in the best performance in terms of $AP_{3D}$, and further increases in the density level do not result in better detection performance, indicating that a denser 4DSRT does not guarantee a better detection performance. In addition, we have provided a highly-optimized development kit that, when used along with the 4DSRT, can improve the training speed by a factor of 17.1.

This work was supported by Institute of Information $\And$ communications Technology Planning $\And$ Evaluation (IITP) grant funded by Korea government (MSIT) (No.2020-0-00440, Development of Artificial Intelligence Technology that Continuously Improves Itself as the Situation Changes in the Real World).