MIPI 2022 Challenge on RGB+ToF Depth Completion: Dataset and Report

Depth completion [11, 8, 9, 12, 13, 16, 20, 25, 3, 6, 22, 24, 15] aims to recover dense depth from sparse depth measurements. Earlier methods concentrate on retrieving dense depth maps only from the sparse ones. However, these approaches are limited and not able to recover depth details and semantic information without the availability of multi-modal data. In this challenge, we focus on the RGB+ToF sensor fusion, where a pre-aligned RGB image is also available as guidance for depth completion.

In our evaluation, the depth resolution and RGB resolution are fixed at $256\times 192$, and the input depth map sparsity ranges from $1.0\%$ to $1.8\%$. As a reference, the KITTI depth completion dataset [32] has sparsity around $4\%\sim 5\%$. The target of this challenge is to predict a dense depth map given the sparsity depth map and a pre-aligned RGB image at the allowed running time constraint (please refer to Section 2.5 for details).

The training data contains 7 image sequences of aligned RGB and ground-truth dense depth from 7 indoor scenes (20,000 pairs of RGB and depth in total). For each scene, the RGB and the ground-truth depth are rendered along a smooth trajectory in our created 3D virtual environment. RGB and dense depth images in the training set have a resolution of 640$\times$480 pixels. We also provide a function to simulate the sparse depth maps that are close to the real sensor measurements^††https://github.com/zhuqingpeng/MIPI2022-RGB-ToF-depth-completion. A visualization of an example frame of RGB, ground-truth depth, and simulated sparse depth is shown in Fig. 1.

The testing data contains, a) Synthetic: a synthetic image sequence (500 pairs of RGB and depth in total) rendered from an indoor virtual environment that differs from the training data; b) iPhone dynamic: 24 image sequences of dynamic scenes collected from an iPhone 12Pro (600 pairs of RGB and depth in total); c) iPhone static: 24 image sequences of static scenes collected from an iPhone 12Pro (600 pairs of RGB and depth in total); d) Modified phone static: 24 image sequences of static scenes (600 pairs of RGB and depth in total) collected from a modified phone. Please note that depth noises, missing depth values in low reflectance regions, and mismatch of field of views between RGB and ToF cameras could be observed from this real data. RGB and dense depth images in the entire testing set have the resolution of 256$\times$192 pixels. RGB and spot depth data from the testing set are provided and the GT depth are not available to participants. The depth data in both training and testing sets are in meters.

The challenge consisted of the following phases:

1. 1.

   Development: The registered participants get access to the data and baseline code, and are able to train the models and evaluate their running time locally.

2. 2.

   Validation: The participants can upload their models to the remote server to check the fidelity scores on the validation dataset, and to compare their results on the validation leaderboard.

3. 3.

   Testing: The participants submit their final results, code, models, and factsheets.

The proposed algorithms are required to be able to process the RGB and sparse depth sequence in real-time. Participants are required to include the average run time of one pair of RGB and depth data using their algorithms and the information in the device in the submitted readme file. Due to the difference of devices for evaluation, we set different requirements of running time for different types of devices according to the AI benchmark data from the website^††https://ai-benchmark.com/ranking_deeplearning_detailed.html. Although the running time is still far from real-time and low power consumption on edge computing devices, we believe this could set up a good starting point for researchers to further push the limit in both academia and industry.

Team ZoomNeXt proposes a lightweight and efficient multimodal depth completion (EMDC) model, shown in Fig. 3, which carries their solution to better address the following three key problems.

1. 1.

   How to fuse multi-modality data more effectively? For this problem, the team adopted a Global and Local Depth Prediction (GLDP) framework [11, 33]. For the confidence maps used by the fusion module, they adjusted the pathways of the features to have relative certainty of global and local depth predictions. Furthermore, the losses calculated on the global and local predictions are adaptively weighted to avoid model mismatch.

2. 2.

   How to reduce the negative effects of missing values regions in sparse modality? They first replaced the traditional upsampling in the U-Net in global network with pixel-shuffle [27], and also removed the batch normalization in local network, as they are fragile to features with anisotropic distribution and degrade the results.

3. 3.

   How to better recover scene structures for both objective metrics and subjective quality? They proposed key innovations in terms of SPN [5, 4, 24, 17, 12] structure and loss function design. First, they proposed the funnel convolutional spatial propagation network (FCSPN) for depth refinement. FCSPN can fuse the point-wise results from large to small dilated convolutions in each stage, and the maximum dilation at each stage is designed to be gradually smaller, thus forming a funnel-like structure stage by stage. Second, they also proposed a corrected gradient loss to handle the extreme depth (0 or inf) in the ground-truth.

GAMEON team proposed a multi-scale architecture to complete the sparse depth map with high performance and fast speed. Knowledge distillation method is used to distill information from large models. Besides, they proposed a stability constraint to increase the model stability and robustness.

They adopt NLSPN [24] as their baseline, and improve it in the following several aspects. As shown in the left figure of Fig. 4, they adopted a multi-scale scheme to produce the final result. At each scale, the proposed EDCNet is used to generate the dense depth map at the current resolution. The detailed architecture of EDCNet is shown in the right figure of Fig. 4, where NLSPN is adopted as the base network to produce the dense depth map at 1/2 resolution of the input, then two convolution layers are used to refine and generate the full resolution result. To enhance the performance, they further adopt knowledge distillation to distill information from Midas (DPT-Large) [26]. They distill information from the penultimate layer of Midas to the penultimate layer of the depth branch of NLSPN. A convolution layer of kernel size $1$ is used to solve the channel dimension gap. Furthermore, they also propose a stability constraint to increase the model stability and robustness. In particular, given a training sample, they adopt the thin plate splines (TPS) [7] transformation (very small perturbations) on the input and ground-truth to generate the transformed sample. After getting the outputs for these two samples, they apply the same transformation to the output of the original sample, which is constrained to be the same as the output of the transformed sample.

Team Singer proposed a depth completion approach that separately estimates high- and low-frequency components to address the problem of how to sufficiently utilize of multimodal data. Based on their previous work [18], they proposed a novel Laplacian pyramid-based [29, 2, 14] depth completion network, which estimates low-frequency components from sparse depth maps by downsampling and contains a Laplacian pyramid decoder that estimates multi-scale residuals to reconstruct complex details of the scene. The overall architecture of the network is shown in Fig. 5.

They use two independent mobilenetv2 to extract features from RGB and Sparse ToF depth. The features from two decoders are fused in a resnet-based encoder. To recover high-frequency scene structures while saving computational cost, the proposed Laplacian pyramid representation progressively adds information on different frequency bands so that the scene structure at different scales can be hierarchically recovered during reconstruction. Instead of the simple upsampling and summation of the two parts in two frequency bands, they proposed a global-local refinement network (GLRN) to fully use different levels of features from the encoder to estimate residuals at various scales and refine them. To save the computational cost of using spatial propagation networks, they introduced a dynamic mask for the fixed kernel of CSPN termed as Affinity Decay Spatial Propagation Network (AD-SPN), which is used to refine the estimated depth maps at various scales through spatial propagation.

Team NPU-CVR submitted two results with a difference in network structural parameter setting. To efficiently fill the sparse and noisy depth maps captured by the ToF camera into dense depth maps, they propose an RGB-guided depth completion method. The overall framework of their method shown in Fig. 6 is based on residual learning. Given a sparse depth map, they first fill it into a coarse dense depth map by pre-processing. Then, they obtain the residual by the network based on the proposed guided feature fusion block, and the residual is added to the coarse depth map to obtain the fine depth map. In addition, they also propose a depth map optimization block that enables deformable convolution to further improve the performance of the method.

Team JingAM improves on the FusionNet proposed in the paper [33]. This paper takes RGB map and sparse depth map as input. The network structure is divided into global network and local network. The global information is obtained through the global network. In addition, the confidence map is used to combine the two inputs according to the uncertainty in the later fusion method. On this basis, they added the skip-level structure and more bottlenecks, and canceled the guidance map and replaced it with the global depth prediction. The loss function weights of the prediction map, global map, and local map are 1, 0.2, and 0.1 respectively. In terms of data enhancement, the brightness, saturation, and contrast were randomly adjusted from 0.1 to 0.8. The sampling step size of the depth map is randomly adjusted between 5 and 12, and change the input size from fixed size to random size. The series of strategies they adopted significantly improved RMSE and MAE compared to the paper [33].

Team Anonymous proposed their method based on a fully convolutional neural network (FCN) which consists of 2 downsampling convolutional layers, 12 residual blocks, and 2 upsampling convolutional layers to produce the final result as shown in Fig. 7. During the training, 25 sequential images and their sparse depth maps from one scene of the provided training dataset will be selected as inputs, then the FCN will predict 25 dense depth maps. The predicted dense depth maps and their ground truth will be used to calculate element-wise training loss consisting of L1 loss, L2 loss, and RMAE loss. Furthermore, RTSD will be calculated with the predicted dense depth maps and regarded as one item of the training loss. During the evaluation, the input RGB image and the sparse depth map will also be concatenated along the channel dimension and sent to the FCN to predict the final dense depth map.

As shown in Fig. 8, Team Mainhouse113 uses a multi-scale joint prediction network (MSPNet) that inputs RGB image and sparse depth image, and simultaneously predicts the dense depth image and uncertainty map. They introduce the uncertainty-driven loss to guide network training and leverage a course-to-fine strategy to train the model. Second, they use an uncertainty attention residual learning network (UARNet), which inputs RGB image, sparse depth image and dense depth image from MSPNet, and outputs residual dense depth image. The uncertainty map from MSPNet serves as an attention map when training the UARNet. The final depth prediction is the element-sum of the two dense depth images from the two networks above. To meet the speed requirement, they use depthwise separable convolution instead of ordinary convolution in the first step, which may lose a few effects.

Team UCLA Vision Lab proposed a method that aims to learn a prior on the shapes populating the scene from only sparse points. This is realized as a light-weight encoder-decoder network (ScaffNet) [34] and because it is only conditioned on the sparse points, it generalizes across domains. A second network, also an encoder-decoder network (FusionNet) [31], takes the putative depth map and the RGB colored image as input using separate encoder branches and outputs the residual to refine the prior. Because the two networks are extremely light-weight (13ms per forward pass), they are able to instantiate an ensemble of them to yield robust predictions. Finally, they perform an image-guided bilateral filtering step on the output depth map to smooth out any spurious predictions. The overall model architecture during inference time is shown in Fig. 9.