FSNet: Redesign Self-Supervised MonoDepth for Full-Scale Depth Prediction for Autonomous Driving

Monodepth2 [6] sets up the baseline framework for self-supervised monocular depth prediction. The core idea of the training of Monodepth2 is to teach the network to predict dense depths that reconstruct the target image from source views. ManyDepth [7] utilizes the geometry in the matching between temporal images to improve the accuracy of single image depth prediction. However, the cost volume construction slows the inference speed and makes it difficult to accelerate on robotics platforms.

To obtain the correct scale directly from monocular depth prediction, researchers use either LiDAR [14, 10, 15] or stereo cameras [8, 9, 16, 17, 18] to provide supervision. Among stereo methods, Depth Hints [8], and Wavelet Depth [9] use traditional stereo matching algorithms to provide direct supervision to the depth prediction network. Monorec [16] uses stereo images to provide scale for the network and to identify moving object pixels in the sequence. DNet [19] uses the ground plane estimated from the normal of the image to calibrate the global scale of the predicted depth, but it fails in the nuScenes dataset which contains many image frames without a clear ground plane.

Our proposed FSNet uses poses from robots to regularize the network to predict correct scales without requiring a multiview camera or LiDARs. We point out that relative poses between frames are commonly available for robotic platforms because of sensors like IMUs and wheel encoders.

Knowledge distillation (KD) is a field of pioneering training methods that transfers knowledge from teacher models to target student models [20]. KD has been applied in various tasks, including image classification [21], object detection [22, 23], and incremental learning [24]. The philosophy of knowledge distillation is training a small target model with an additional loss function to mimic the output of a teacher model. The teacher network could be an ensemble of the target models [25], a larger network [26], or a model with additional information inputs or different sensors [27].

Self-distillation (SD) is a special case where the teacher model is completely identical to the student model, and no additional input is applied to train the teacher model. Some existing works have demonstrated its performance on image classification [28].

In this paper, we first investigate the training process of an unsupervised monocular depth predictor, then we apply an offline-SD to our proposed model to regularize the training process and improve the final performance.

Visual odometry (VO) systems [29, 30, 31] are widely used to provide robot centric pose information for autonomous navigation [32] and simultaneously estimate the ego-motion of the camera and the position of 3D landmarks. With IMU or Global navigation satellite system (GNSS) information, the absolute scale of the estimations can also be recovered [33, 34].

ORB-SLAM3 [31] is a typical indirect VO method in which the current input frame is tracked online with a 3D landmark map built incrementally. In this tracking process, the local features extracted are first associated with landmarks on the map. Then, the camera pose and correspondences are estimated within a Perspective-n-Point scheme. The depth of the local feature point can be retrieved from the associated 3D landmark and the camera pose, yielding an image-aligned sparse depth map for each frame. In this work, we use these sparse depth maps as an input for the full-scale dense depth prediction post-optimization procedure.

Monocular depth completion means predicting a dense depth with a monocular image and sparse ground truth LiDAR points. Learning-based methods usually follow the scheme of partial differential equations (PQEs) that extrapolate depth between sparse points based on the constraints learned from RGB pixels [35]. Other new methods without deep learning or extra ground truth labels exist. IPBasic [36] produces dense depth with morphological operations. Bryan et al. [37] introduce superpixel segmentation to segment images into different units, and each unit is considered as a plane or a convex hull.

In our proposed post-processing step, the merging between dense network depth prediction and visual odometry can be formulated into the same optimization problem as monocular depth completion but with extra uncertainty and hints. The proposed method is formulated without extra LiDAR point labels.

Multichannel Output: In order to preserve organic reconstruction results, we select reformulate the output as multi-channel outputs, to allow larger depth values during initialization and also unsaturated gradients at large depth values.

Assuming the predicted depth $d$ should be within $[d_{min},d_{max}]$ and the output is decoded from $N$ channels of the output network, then $q=\frac{d_{max}}{d_{min}}^{1/N}$ is defined as the proportion between consecutive depth bins, and the $i$th bin will represent $d_{i}=d_{min}\times q^{i-1}$. Considering the softmax activated output of the network being $Z\in\mathbb{R}^{N}$, we interpret the weighted mean of each depth bin as the predicted depth $d=\sum_{i}z_{i}\cdot d_{i}$.

At initialization, the network will predict an almost-uniform distribution for each depth bin, so the initial decoded depth will be the algorithmic mean of the depth bins $d^{\prime}=\frac{1}{N}\sum_{i}d_{i}$. To guide the selection of hyperparameters, we analyze the initial depth values $d^{\prime}$ in our framework.

Theorem : The algorithmic mean $d^{\prime}$ of a proportional array, bounded by $[d_{min},d_{max}]$ will converge to the following limit as the number of bins $N$ grows:

Proof: Denote the boundary of the predicted depth as $[d_{min},d_{max}]$, the $i$ th bins will represent $d_{i}=d_{min}\times q^{i-1}=d_{min}(\frac{d_{max}}{d_{min}})^{i/N}$. As presented in the paper, the initial depth predicted by the network will be the mean of all depth bins $d^{\prime}=\frac{1}{N}\sum_{i}{d_{i}}$.

The mean of the proportional array will be

L’Hôpital’s rule is applied at $\overset{\mathrm{H}}{=}$ to obtain the final result.

We set $d_{max}=100m$ for the autonomous driving scene. Based on (4), we choose $d_{min}=0.1m$ for our network to obtain $d^{\prime}>10m$ to ensure stable training initialization.

Camera-Aware Depth Adaption: To train one single depth prediction network on all six cameras of the nuScenes dataset, we point out that we need to take the difference of camera intrinsic matrix into account. Similar object appearance produces similar features in the network, while depth predictions should vary based on the focal length of the camera. As a result, we adapt the value of the depth bins according to the input camera $f_{x}$ as $d_{i}=d_{i}^{0}\cdot\frac{f_{x}}{f_{base}}$, where $f_{base}$ is a hyper-parameter chosen according to the front camera in nuScenes.

With the introduction of ego-vehicle poses at training times, we have a stronger prior for dynamic object removal. We know that the reprojection of a static 3D point on neighboring frames must stay on an epipolar line regardless of the distance between that point to the camera. The epipolar line $L$ can be computed as a vector with a length of three $L=\mathbf{F}\cdot[p_{x},p_{y},1]^{T}$, where $(p_{x},p_{y})$ are the the coordinate of the pixel in the base frame and $\mathbf{F}$ is the fundamental matrix between the two frame. The fundamental matrix $F$ is further calculated from $\mathbf{F}=K^{T}\cdot[t]_{x}\cdot R\cdot K$, with camera intrinsics parameters $K$, and relative pose $(R,t)$ between two image frames.

Therefore, if the reprojected point was far from its epipolar line, we can estimate that this point is probably related to a dynamic object and should probably be omitted during training loss computation.

To construct the optical flow mask for dynamic object removal, we first compute the optical flow $(d_{x},d_{y})$ between the two image frames using a pretrained unsupervised optical flow estimator ArFlow [39]. The distance $dis_{l}$ between the reprojected point from the epipolar line is computed as with

where $L_{0},L_{1}$ represents the first and second element of vector $L$. Pixels with a deviation larger than 10 pixels are considered dynamic pixels. In this way, we produce a loss mask for photometric reconstruction loss computation.

As discussed in Section III, the monocular depth prediction network starts training with a generally uniform depth map. The loss function, with a kernel size of $3\times 3$, only produces guaranteed optimal gradient directions to the network when the reconstruction error is within several pixels. As a result, at the start of the training, the network is first trained on pixels with ground truth depths close to the initialization state, and the gradients at other pixels will be noisy. The noise in the gradient will affect the stability of the training process. In order to stabilize the training process by increasing the signal-noise ratio (SNR) of the training gradient, we introduce a self-distillation framework.

Using the baseline self-supervised framework, we first train an FSNet $\Theta_{teacher}$. The first FSNet will suffer from the noisy learning process mentioned above and produce sub-optimal results. Then, we train another FSNet $\Theta_{student}$ from scratch using the self-supervised framework, with pseudo label $d_{pseudo}=\Theta_{teacher}(I_{0})$ from the first FSNet to guide the training. With the pseudo label from the teacher net, the student network will receive an additional meaningful training gradient from the beginning of the training phase. Thus, the problem of noisy gradients introduced above can be mitigated. The training process is summarized in the upper half of Fig.  2.

However, since the pseudo label predicted from teacher FSNet is not always accurate, we encourage the student network to predict the uncertainty $\sigma$ for each pixel adaptively, alongside the depth $d_{0}$. We observe that close-up objects have smaller errors in depth prediction than far-away objects, and we encode this empirical result in our uncertainty model. Thus, we assume that the logarithm of the depth prediction $\log(d)$ follows a Laplacian distribution with mean $\log(d_{pseudo})$ predicted by the teacher. The likelihood $p$ of the predicted depth can be formulated as:

Practically, the convolutional network directly predicts the value of $\log(\sigma)$ instead of $\sigma$ to improve the numerical stability. We adopt $l_{distill}=-p(\log(d)|\log(d_{pseudo}),\sigma)$ as the training loss.

To deploy the algorithm on a robot, we expect the system to be robust to online permutation to extrinsic parameters that could affect the accuracy of $D_{0}$ predicted by the network. We introduce a post-processing algorithm to provide an option to improve the run-time performance of the depth prediction module by merging the sparse depth map $D_{vo}$ produced from VO.

We formulate the post-processing problem as an optimization problem. The optimization problem over all the pixels in the image is described as:

The consistency term $L_{consist}$ is defined by the change in relative logarithm between each pixel compared to $D_{0}$. The visual odometry term $L_{vo}$ works only at a subset of points with sparse depth points, and $\lambda_{1}^{i}$ is zero for pixels without VO points.

The problem above is a convex optimization problem that can be solved in polynomial time. However, the number of pixels is too large for us to solve the optimization problem at run time.

Therefore, we downscale the problem by segmenting images with super-pixels and simplifying the computation inside each super-pixel. The full test-time pipeline is summarized in the second half of Fig.  2.

We first present the dataset and background settings of our experiments.

We utilize the following datasets in our experiments to evaluate the performance of our approach:

• •

  KITTI Raw dataset [11]: This dataset was designed for autonomous driving and many existing works have produced official results on it. We mainly evaluate FSNet on the Eigen Split [14]. It contains 39810 monocular frames for training and 697 images from multiple sequences for testing. Images are sub-sample to $192\times 640$ during training and inference for the network.

• •

  NuScenes dataset [13]: This dataset contains 850 sequences collected with six cameras around the ego-vehicle. We separate the dataset under the official setting with 700 training sequences and 150 validation sequences. We only select scenes without rain and night scenes during both training and validation. We uniformly sub-sample the validation set for validation. Images are sub-sampled to $448\times 762$. Unlike [42], we treat images at each frame as six independent samples during both training and testing. The model will have to adapt to different camera intrinsic and extrinsic parameters. Furthermore, because the lidar and the cameras are not synchronized, there will be noise in the poses between frames. FSNet needs to overcome these problems to achieve stable training, and also predict scale-aware depths.

Depth Metrics: Previous works, including [6, 7, 16, 8] mainly focus on metrics where the depth prediction is first aligned with the ground truth point clouds using a global median scale before computing errors. In this paper, we first present data on the scaled metrics for comparison with existing methods. Then, we focus on metrics without median scaling in ablation studies.

Data Augmentation: Besides photometric augmentation adopted in MonoDepth2 [6], we implement horizontal flip augmentation for image-pose sequences, where we also horizontally flip the relative poses between image frames.

FSNet Setting: We adopt ResNet-18 [43] as the backbone encoder for KITTI datasets following prior mainstream works for a fair comparison, and we adopt ResNet-34 for the nuScenes dataset because of the increasing difficulty. The PoseNet is dropped and poses from the dataset are directly used in image reconstruction, and no additional modules are created except for a frozen teacher net during distillation. In summary, we basically share the inference structure of MonoDepth2, and we do not train a standalone PoseNet. In the KITTI dataset, FSNet spends 0.03s on network inferencing and 0.04s on post-optimization, measured on RTX 2080Ti. We point out that multichannel output does not noticeably increase network inference time with additional width only in the final convolution layer.

The performance on the KITTI dataset is presented in Table I. We point out that, even though FSNet spends extra network capability to memorize the scale of the objects in the scenes, it can produce even better depth maps than baseline models by utilizing multi-channel output, flow mask, and self-distillation.

We present results on both single-frame settings and multi-frame settings. We could appreciate the improvement from post-optimization in the result table. We further point out that our method decouples the prediction of a single frame and post-optimization into standalone modules, which makes it flexible for different application settings.

The performance on the nuScenes dataset is presented in Table  I. We also present the detailed performance result of RMSE and RMSE log on all six cameras on the nuScenes dataset. Some other methods [42, 38] train with all six camera images in a frame at once, and propagate information between images during training and inference. We treat the data as six independent monocular image streams to train and infer with a single FSNet model. The single FSNet model aligns its predicted depth with different cameras using their intrinsic camera parameter.

Because the cameras and LiDAR data are not synchronized, the relative poses between frames in the nuScenes dataset are not as accurate as those in the KITTI dataset. However, experiment data show that FSNet trained with noisy poses can still produce depth predictions with the correct scale as the 3D scenes and it is robust to different camera setups.

Finally, we present some qualitative results on the validation split of the nuScenes dataset in Figure 4. FSNet predicts depth in each image independently, and we concatenate the predictions together to form the results here. The first three rows demonstrate the network’s ability to identify different objects of interest in urban road scenes. The final row presents a failure case where a bus with a huge reflective glass dominates the image. More 3D visualization is presented at the project page https://sites.google.com/view/fsnet/home, which can further show that the depths predicted from the six cameras are consistent, and we can obtain a detailed 3D perception result of the surrounding environment.

In Section IV-A, we introduce a multi-channel output to allow organic image reconstruction at initialization to boost-trap the training. Here, we experiment with two other output settings: (1) original monodepth2 output with a bias in the output layer; (2) exponential activation. As presented in Table III, multi-channel output performs better at most error metrics. The original output representation of monodepth2 saturates in a common depth range like $d>10m$, which makes it difficult to train the network. We highlight that the un-scale baseline MonoDepth tends to maintain the output in range with sufficient gradients by adjusting the scale of the pose prediction. The exponential activation, though effective in monocular 3D object detection [44], results in un-controllable activation growth in background pixels like the sky, where the correct depths are essentially infinity, corrupting the training gradients. The experiments and analysis show that the proposed multi-channel output enables stable training and produces accurate predictions.

We further present the results with a flow mask and self-distillation in Table III. Each proposed method incrementally improves the depth estimation results. Squared Relative (Sq Rel) and Root Mean Square Error (RMSE) improve the most, which means that the proposed methods mostly prevent the network from making significant mistakes like with dynamic objects. We note that both methods do not introduce extra cost in the inference time, but it regulates the training process to improve performance.

There are many factors contributing to the performance of the post-optimization step. This section investigates two factors: (1) the usage of depth in the 3D SLIC algorithm and (2) the importance of each loss weight in the optimization step.

The results are presented in Table IV. The proposed 3D SLIC algorithm improves the segmentation quality by utilizing the depth predicted by the network, thus improving the post-optimization results. When $\lambda_{0}=0$, the depth prediction of each super segment will be independent, and the optimization from visual odometry cannot fully propagate throughout the map. However, when $\lambda_{2}=0$, we completely ignore the scale predicted by the network, and errors in visual odometry, especially at dynamic objects, will corrupt the prediction result.