RadarCam-Depth: Radar-Camera Fusion for Depth Estimation
with Learned Metric Scale

Monocular depth estimation is a challenging task due to the inherent scale ambiguity. Many researchers have tried to address this issue by integrating it with optical flow [14], uncertainty estimation [15], semantic segmentation [16], instance segmentation [17] and visual odometry [18]. Although some previous studies [19, 20, 4, 3] have achieved promising results in affine-invariant scaleless depth estimation across diverse datasets, recovering the metric scale remains a significant challenge. Some existing methods rely on inertial data to provide scale. To enhance the generalization, VI-SLAM [21] warps the input image to match the orientation prevailing in the training dataset. CodeVIO [22] proposes a tightly coupled VIO system with optimizable learned dense depth. It jointly estimates VIO poses and optimizes the predicted and encoded dense depth of specific keyframes efficiently. Xie et al. [23] utilize a flow-to-depth layer to refine camera poses and generate depth proposals. They solve a multi-frame triangulation problem to enhance the estimation accuracy. Recently, Wofk et al. [5] introduced a framework for metric dense depth estimation from the VIO sparse depth and monocular depth prediction, which inspires our work. They first globally align the scaleless mono-depth with the metric VIO sparse depth and then learn to refine the dense scale of the globally aligned mono-depth.

The fusion of Radar and camera data for metric depth estimation is an active research topic. Lin et al. [8] introduce a two-stage CNN-based pipeline that combines Radar and camera inputs to denoise Radar signals and estimate dense depth. Long et al. [9] propose a Radar-2-Pixel (R2P) network that utilizes radial Doppler velocity and induced optical flow from images to associate Radar points with corresponding pixel regions, enabling the synthesis of full-velocity information. They also achieve image-guided depth completion using Radar and video data [10]. Another approach, DORN [11] proposed by Lo et al., extends Radar points in the elevation dimension and applies deep ordinal regression network-based [24] feature fusion. Unlike other methods, R4dyn [13] creatively incorporates Radar as a weakly supervised signal into a self-supervised framework and employs Radar as an additional input to enhance the robustness. However, their method primarily focuses on vehicle targets and does not fully correlate all Radar points with a larger image area, resulting in lower depth accuracy. Recently, Singh et al. [12] present a method that relies solely on a single image frame and Radar point cloud. Their first-stage network infers the confidence scores of Radar-Pixel correspondence, generating a semi-dense depth map. They further employ a gated fusion network to control the fusion of multi-modal Radar-Camera data and predict the dense depth. However, all the above methods directly encode and concatenate the ambiguous Radar depth and images, confusing the learning pipeline and resulting in suboptimal depth estimation.

We employ off-the-shelf networks to predict robust and accurate scaleless depth from a single-view image. The high quality of the mono-depth prediction furnishes a solid foundation for scale-oriented learning. In this research, we harnessed SOTA mono-depth networks, like MiDaS v3.1 [19, 3] and DPT-Hybrid [20] with pre-trained weights on mixed diverse datasets. Both networks are built upon transformer architecture [28] and trained with scale and offset-invariant losses, ensuring strong generalization. They infer the relative depth relationship between pixels, producing dense depth (see Fig.3). Notably, our framework is versatile and compatible with arbitrary mono-depth prediction networks that predict depth $\mathbf{\hat{d}}_{m}$, inverse depth $\mathbf{\hat{z}}_{m}$ or others.

We align the scaleless mono-depth prediction $\mathbf{\hat{d}}_{m}$ with the Radar depth originating from projecting raw Radar points $\mathbf{P}$, by a global scaling factor $\hat{s}_{g}$ and optional offset $\hat{t}_{g}$. The global aligned metric depth is calculated by $\mathbf{\hat{d}}_{ga}=\hat{s}_{g}\cdot\mathbf{\hat{d}}_{m}+\hat{t}_{g}$. Then, it is fed into the subsequent scale map learner (SML). There are many options for performing this global alignment between the projected Radar depth and mono-depth prediction, including: (i) Var: A varying $\hat{s}_{g}$ for individual frame of mono-depth, calculated via root-finding algorithms [29, 30]. (ii) Const: A constant $\hat{s}_{g}$ for all frames of mono-depth prediction, deemed as the mean of scale estimates on the entire training samples. (iii) LS: $\hat{s}_{g}$ and $\hat{t}_{g}$ for individual frames, computed with linear least-squares optimization [19]. (iv) RANSAC: $\hat{s}_{g}$ and $\hat{t}_{g}$ for individual frames, computed with linear least-squares while incorporating RANSAC outlier rejection of the Radar depth. We randomly sample 5 Radar points with valid depth values, estimate $\hat{s}_{g}$ and $\hat{t}_{g}$ with the sampled Radar depth, and adopt the first pair of $\hat{s}_{g}$ and $\hat{t}_{g}$ that yields an inlier ratio over $90\%$. The inlier is the one where the discrepancy between the Radar point depth and aligned mono-depth is under $6$m or the inverse depth discrepancy is under $0.015$.

Due to inherent sparsity and noises in Radar data, additional enhancement of raw Radar depth is crucial before conducting the scale map learner. To densify the sparse Radar depth obtained from projection, we exploit a transformer-based Radar-Camera data association network (shorthand RC-Net), which predicts the confidence of Radar-Pixel associations. Pixels without a direct correspondence of Radar point during projection might be associated with the depth of neighboring Radar point, thereby densifying the sparse Radar depth to a quasi-dense depth map, denoted as $\mathbf{\hat{d}}_{q}$.

For the nuScenes dataset, following [12], we accumulate the individual $\mathbf{d}_{gt}$ of 160 frames nearby to get $\mathbf{d}_{acc}$, which is then interpolated to yield $\mathbf{d}_{int}$. Dynamic objects are masked out during the above process. For the ZJU-4DRadarCam dataset, we directly interpolate $\mathbf{d}_{gt}$ to obtain $\mathbf{d}_{int}$ with linear interpolation [34] in the log space of depth.

For training the RC-Net on nuScenes, with an input image size of $900\times 1600$, the size of the cropped patch during confidence map formation is set to $900\times 288$. For the training on ZJU-4DRadarCam, the input image size is $300\times 1280$, while the patch size is $300\times 100$. We employ the Adam optimizer with $\beta_{1}$=0.9 and $\beta_{2}$=0.999, and a learning rate of $2e^{-4}$ for 50 epochs. Data augmentations, including horizontal flipping, saturation, brightness, and contrast adjustments, are applied with a 0.5 probability. We train our RC-Net for 50 epochs with an NVIDIA RTX 3090 GPU, taking approximately 14 hours with a batch size of 6.

We adopt MiDaS-Small architecture for our SML, where the encoder backbone is initialized with pre-trained ImageNet weights [35], and other layers are randomly initialized. The input data is resized and cropped to $288\times 384$. We use an Adam optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The initial learning rate is set to $2e^{-4}$ and reduced to $5e^{-5}$ after 20 epochs. Training SML for 40 epochs takes about 24 hours with a batch size of 24.

Some widely adopted metrics from the literature are used for evaluating the depth estimations, including mean absolute error (MAE), root mean squared error (RMSE), absolute relative error (AbsRel), squared relative error (SqRel), the errors of inverse depth (iRMSE, iMAE), and $\delta_{1}$ [36]. To better illustrate our experimental details, the demo video is available at https://youtu.be/JDn0Sua5d9o.

We evaluate the metric dense depth against $\mathbf{d}_{gt}$ within the range of 50, 70, and 80 meters (see Tab.I). Our proposed RadarCam-Depth outperforms all the compared Radar-Camera methods and surpasses the second best method [12] by a large margin at all ranges. Specifically, we observe 25.6%, 23.4%, and 22.5% reductions in MAE and 20.9%, 20.2%, and 19.4% drops in RMSE for 50m, 70m, and 80m, respectively. We attribute the outstanding performance of RadarCam-Depth to the reasonable monocular prediction $\mathbf{\hat{d}}_{m}$ and our scale learning strategy. Notably, RadarCam-Depth solely relies on a single-frame image and a Radar point cloud, obviating the need to aggregate multi-frame data. The “Radar” and “Image” columns in Tab.I specify the quantities of point clouds and images used as inputs for various methods. For nuScenes dataset, we adopt the sky-sensitive pre-trained model, MiDaS v3.1, as our monocular depth prediction network, which can accurately differentiate the sky from others (see Fig.3). However, since the depth estimations in sky regions are not associated with corresponding LiDAR depth, they are not counted during metric evaluations. Some snapshots of depth estimations from different methods are shown in Fig.7.

We follow a similar way to Sec.IV-C for the evaluations on the ZJU-4DRadarCam dataset. For the mono-depth prediction in our framework, we tried both MiDaS v3.1 [3], and DPT-Hybrid [20] models. The evaluations of the metric dense depth estimations from various methods are presented in Tab.II, where our approach at different configurations of mono-depth network and global alignment options are marked in bold. After a comprehensive evaluation, we observe that our methods with the DPT model perform better for depth metrics, while the methods with MiDaS demonstrate higher accuracy for inverse depth metrics. Overall, our proposed methodology exhibits significant improvements compared to existing Radar-Camera methods [12] and [11] (Fig.5(b)). Compared to the second best [12], the best configuration of our method shows 40.2%, 40.1%, and 40.2% reductions in MAE within ranges of 50m, 70m, and 80m, respectively.

We report our proposed method’s runtime at the DPT-based mono-depth prediction configuration. The average processing times per frame are shown in Tab.III. Note that Mono-Pred and GA can run simultaneously with RC-Net. Regarding different scale global alignment methods, GA (Var) and GA (LS) exhibit relatively fast speeds, while GA (RANSAC) is significantly slow and not advocated.