ColoRadar: The Direct 3D Millimeter Wave Radar Dataset

Robots are being deployed in ever more challenging roles. In many environments GPS is unavailable so autonomous robots must be able to localize and map using only their onboard sensors. These tasks are normally accomplished using some combination of visual, lidar, and inertial sensors Hess et al. (2016); Leutenegger et al. (2013, 2015). However, the quality of the estimates from visual and lidar methods is severely handicapped in so-called visually degraded environments (VDEs). These include darkness, fog, smoke, and similar conditions that adversely affect sensing in the visible or near-visible spectrum. Such conditions are common in a variety of scenarios that are of keen interest for robotics research including disaster response and subterranean exploration.

Millimeter wave radars, of the type used in advanced driver assistance systems (ADAS), are an attractive option for perception in VDEs. These sensors do not require ambient light to operate and are unaffected by airborne particulates like smoke and fog. Additionally, automotive-grade system-on-chip (SoC) radars are lightweight and have low power requirements, making them attractive for vehicles with limited payload and power capacity, e.g. micro aerial vehicles (MAVs). Because of these attributes the number of robotic perception papers using automotive millimeter wave radar sensors has increased drastically in the last few years Schuster et al. (2016b, a); Rapp et al. (2017); Kramer et al. (2020); Lu et al. (2020); Xiaoxuan Lu et al. (2020).

However, several factors make the development of new radar-based methods extremely difficult. First, there are very few radar sensors available that are suitable for robotics work. Large 2D scanning sensors Cen and Newman (2018, 2019); Lakshay et al. (2020); Stefan Saftescu et al. (2020), and small 3D automovotive sensors dominate the market Schuster et al. (2016b, a); Rapp et al. (2017); Kramer et al. (2020); Lu et al. (2020); Xiaoxuan Lu et al. (2020). Second, almost all of the available radar sensors implement their own signal processing and target detection systems onboard Iovescu and Rao (2020). These are well adapted for ADAS but ill-suited to robotic perception tasks Lu et al. (2020). It would be ideal if researchers could implement their own target detection and/or signal processing algorithms, but most sensors do not allow the user access to their full data stream and those that do are not designed for streaming in real time. Hacking these sensors to stream data in real time takes both domain-specific expertise and a large time investment which is prohibitive for most researchers. The final nail in the coffin for most nascent radar-based robotics projects is there are no publicly available robotics datasets that include this data.

To remove these obstacles we present the first multi-sensor dataset designed for 6DoF radar-based state estimation and mapping. The dataset includes dense 4D data from millimeter wave radar sensors, 3D lidar pointclouds, IMU measurements, and groundtruth pose information. The dataset is available for download at arpg.colorado.edu/coloradar. We include radar data in 3 different forms, providing entry points of varying sophistication and difficulty for users. First, we provide unprocessed ADC samples from each radar measurement. Second, we provide dense 3D heatmaps for each radar measurement generated from signal processing on the raw ADC samples. Lastly, we provide point targets derived from the sensor’s onboard signal processing and target detection process. Although as previously stated, these point targets are not ideal for robotic state estimation and perception applications, we include point targets because several recent methods have made use of them Kramer et al. (2020); Lu et al. (2020); Xiaoxuan Lu et al. (2020). So including point targets provides a familiar entry point for many researchers.

While there exist at least three publicly available robotics datasets containing radar data Barnes et al. (2020); Caesar et al. (2020); Leung et al. (2017), each dataset is limited in some manner. The Oxford Radar RobotCar dataset Barnes et al. (2020) and NuScenes dataset Caesar et al. (2020) both focus on autonomous road vehicles, limiting sensor movement largely to a 2D horizontal plane. In contrast, the Radar RobotCar dataset uses a different style of radar sensor, the NavTech CTS350-X, which provides dense 2D scans in the horizontal plane only. The NavTech sensor’s 2D output makes the Radar RobotCar dataset unsuitable for developing methods for 3D environments. Also, the NavTech sensor weighs 6kg and draws 24W, making it impractical for small, agile autonomous vehicles like MAVs. The Chilean underground mine dataset provides a different environment, it also uses a 2D scanning radar and is mostly confined to 2D movement. Leung et al. (2017). Lastly, a major limitation common to all of these datasets is that they do not include the raw ADC sample data from their radar sensors. This means users of these datasets must be satisfied with the signal processing and target detection algorithms implemented onboard their sensors. These algorithms are often very ill-suited to robotics tasks as previously stated.

The Euroc MAV dataset Burri et al. (2016) and the UZH FPV dataset Delmerico et al. (2019) are examples of datasets that are useful for the development of algorithms for small, agile robots operating in 3D environments. However, these are visual-inertial datasets that do not include radar. Currently there are no similar datasets that include radar data in any form.

Like the Euroc MAV and UZH FPV datasets Burri et al. (2016); Delmerico et al. (2019), our dataset includes both slow and rapid 6DoF motion. This means our dataset can be used in the development of state estimation and mapping methods for small, agile robots that operate in diverse 3D environments. Also, our dataset includes both raw ADC data and dense images from multiple types of 4D radar sensors. This allows users to develop new methods for signal processing and target detection which are specifically designed for robotics tasks. Lastly, we include data gathered in highly diverse indoor, outdoor, and subterranean environments. This allows users to develop methods for a wide variety of environments and scenarios. These are clear advantages over the currently-available radar datasets.

To understand the different types of radar data provided in our dataset, it is necessary to give a brief description of the typical signal processing pipeline for multi-input-multi-output (MIMO) frequency modulated continuous wave (FMCW) radar sensors.

Data was collected in several unique environments with the goal of providing a diverse range of sensor data. The Intelligent Robotics Laboratory (IRL) and engineering center (EC) hallways sequences were taken in an indoor built environment and the EC courtyard sequences capture data in an outdoor built environment. In the Boulder Creek Path sequences the sensor rig moves rapidly through an outdoor environment with dynamic obstacles including pedestrians, cyclists, and road vehicles. There are two sets of sequences taken in Edgar Experimental Mine, a former silver and gold mine managed by Colorado School of Mines: Edgar Army sequences were taken in large, open areas of the mine and the Edgar Classroom sequences were taken in the mine’s tighter, more constrained passageways. The ASPEN Lab sequences were taken in a motion capture space with a variety of obstacles.

Example images of the various environments are shown in Figure 6. The amount of data collected at each location, along with their associated location file names are shown in Table 2.

All sensor readings for a given sequence are grouped in a single compressed file named <date>_<location>_<sequence>.zip, where <date>, <location>, and <sequence> are placeholders for the recording date, location, and sequence number respectively. The directory structure is shown in Figure 7.

Timestamps for each sensor reading are stored in a timestamps.txt file for each sensor. Each line in timestamps.txt gives the time of a sensor measurement in seconds.

As previously stated we provide three levels of processing for radar data: unprocessed ADC samples, 3D range-azimuth-elevation intensity-Doppler velocity heatmaps, and target detections from the single-chip radar sensor. The cascaded sensor does not do any signal processing or target detection onboard, so point targets are not provided for the cascade sensor. By providing these different data types will enable users to develop application specific methods for signal processing, target detection, scan alignment, etc.

We provide development tools for manipulating our raw data on the ColoRadar website. These include example Python tools for reading and plotting the provided data files. These are described in detail in the README.md file provided with the development tools. Here we will briefly describe the tools and their functions. dataset_loaders.py provides functions to load sensor measurements and calibrations from dataset files. These functions parse the raw data and organize it into logical data structures. For instance, the function get_heatmap() reads bytes from a radar heatmap file, converts those bytes to intensity and range rate values, and returns them in a numpy array of dimension $N_{e}\times N_{a}\times N_{r}\times 2$. The function get_cascade_params() reads config files for the cascaded radar sensor and returns a dictionary where each key-value pair is a parameter for the radar sensor. plot_pointclouds.py demonstrates how the functions in dataset_loaders.py can be used to create an animation of the sensor readings and groundtruth poses in a sensor-centric reference frame. MATLAB equivalents of these tools are also provided.

Four types of calibration parameters are provided for the radar sensor data. The first is a mapping of the location of the TX and RX antennas on each radar board. These are stored in text files found in calib/cascade/antenna_cfg.txt and calib/single_chip/antenna_cfg.txt.

The second type of calibration is the waveform parameters. These include the start frequency, frequency slope, ADC sample rate, and other parameters necessary for signal processing. These parameters are found in calib/cascade/waveform_cfg.txt and calib/single_chip/waveform_cfg.txt.

The third type is the antenna coupling calibration. Antenna coupling occurs when a current in one antenna element unintentionally induces a voltage in adjacent elements. Antenna coupling calibration compensates for this as

Antenna coupling calibration files are provided in calib/cascade/coupling_calib.txt and calib/single_chip/coupling_calib.txt and Python code is provided to show how to apply the calibration.

The fourth type is phase and frequency calibration. These parameters are needed for the cascaded sensor only. This is because the cascaded sensor uses 4 separate chips to control different parts of the antenna array and there may be small mismatches in the frequency, phase, and amplitude across these 4 chips. Phase and frequency calibration corrects for this.

In frequency calibration, we start with a calibration vector $\mathbf{f}$ of dimension $N_{\text{rx}}N_{\text{tx}}$. From this we calculate a frequency calibration matrix $\mathbf{F}$ of dimension $N_{\text{rx}}N_{\text{tx}}\times N_{\text{range}}$ as

Phase and frequency calibration vectors can be found in calib/cascade/phase_frequency_calib.txt and Python code is provided to show how to apply the calibration.

Extrinsic transforms between sensors are given relative to the sensor rig’s base frame. The sensor rig’s base frame is coincident with the IMU in translation and oriented to align with the ENU frame ($x$ right, $y$ forward, $z$ up). Transformations between the base frame and each sensor are given in calib/transforms/base_to_<sensor>.txt. Each of these files has two lines. The first gives the translation $(x,y,z)$ and the second gives the rotation as a quaternion in the order $(x,y,z,w)$. The extrinsic transformations for all of the sensors are hand measured. The transformation between the lidar and radar sensors is further refined via the following process: we took a dataset in which the sensor rig was held static and two obstacles are clearly visible in the lidar pointcloud and the heatmaps for the single chip and and cascaded radar sensors. The extrinsic transformations for the lidar and single chip sensors are then manually adjusted until the locations of the obstacles in the lidar pointcloud and single chip heatmap coincide with those in the cascaded heatmap. The extrinsic transforms do not change appreciably between runs, so we include only one set of extrinsic transforms.