Towards Better Data Exploitation in Self-Supervised Monocular Depth Estimation

Fully supervised methods for depth estimation in the field of robotics or autonomous driving require depth ground truth when training. Among this category of methods, Eigen et al. [1] first propose to use convolutional neural network (CNN) for monocular depth estimation. They employ a multi-scale neural network to estimate the depth map. Liu et al. [2] combine a depth CNN with continuous conditional random fields (CRFs) for depth estimation from a single image. Li et al. [3] propose a multi-scale approach, which uses a deep neural network to regress depth at the super-pixel scale and applies CRFs for post-processing. Laina et al. [4] introduce an innovative up-sampling module and utilize the reverse Huber loss to improve training. And Fu et al. [5] employ a multi-scale network and regard depth estimation as an ordinal regression task, resulting in improved accuracy and faster convergence.

The high expense of collecting large annotated datasets for depth estimation has led to the emergence of self-supervised methods that do not rely on the ground truth. Garg et al. [9] first consider the training of monocular depth estimation as a view synthesis problem with a photometric-consistency loss between stereo pairs. Building upon this idea, Godard et al. [10] introduce a left-right disparity consistency loss, which can improve the accuracy of depth estimation. For training with monocular video frames, Zhou et al. [11] propose a pioneering method which jointly learns monocular depth and ego-motion from monocular videos in a self-supervised way. They also use an explainability prediction network to exclude pixels that violate view synthesis assumptions, to improve the robustness when meeting with occlusion and moving objects. For dynamic scenes, other works introduce multi-task learning such as optical flow estimation [13] and semantic segmentation [14], or introduce additional constraints, such as uncertainty estimation [15]. Then Godard et al. [16] propose Monodepth2, which employs a minimum reprojection loss to mitigate occlusion issues and an auto-masking loss to filter out moving objects with the similar velocity as the camera. This work achieves competitive results without introducing extra tasks, which presents promising prospects for robotic perception. Therefore many subsequent methods including ours follow and base on it.

For data augmentation, many existing methods just simply utilize horizontal flipping and color jittering. Some approaches [28, 24, 27] additionally use resizing and cropping. And Peng et al. [29] proposes a data augmentation approach called data grafting, which forces the model to explore more cues to infer depth besides the vertical image position. Among them, [24], [28] and [29] only utilize augmented images for training. Especially, PlaneDepth [24] achieves striking performance when using stereo pairs, but poor with monocular videos. The most similar to our work is RA-Depth [27], which uses resizing and cropping to generate images with arbitrary scales for the same scene. It modifies the camera intrinsics to keep the world coordinates, whereas we follow the assumption in [24] that all input images, including augmented views and original one, are captured by the same camera system.

Self-distillation is commonly used in self-supervised learning which means using a pretrained teacher network to generate pseudo labels for training a better student network. In depth estimation, for example, Pilzer et al. [30] obtain multiple predictions from teacher network to generate pseudo labels for the student network training. Self-distillation can also be used to reduce the uncertainty of depth prediction [31], and solve the artifact problem caused by occlusion especially under stereo training [24], which actually feeds a flipped version of the original image into a depth network to synthesize pseudo labels. In our work, we also use two data augmentation methods and generate pseudo labels from augmented images.

Self-supervised monocular depth estimation paradigm aims at training a depth network $\theta_{\rm{depth}}$ to predict a per-pixel depth map $D$ from the input RGB image $I$, without providing any depth ground truth. Specifically, considering a target view $I_{t}$ in a consecutive image sequence from a monocular video, the neighbour frames are used as the source views $\{I_{s}\}_{s=1}^{N}$ to carry out view synthesis. For each source view $I_{s}$, an auxiliary pose network $\theta_{\rm{pose}}$ is utilized to estimate the relative camera pose $T_{t\rightarrow s}=[R_{t\rightarrow s}|t_{t\rightarrow s}]$, where $R_{t\rightarrow s}$ and $t_{t\rightarrow s}$ represent the rotation and translation component respectively. Note that the pose network is only used for training and could be discarded in the inference, and it is not needed when training with only stereo pairs since the stereo baseline is already known. Then the target depth map $D_{t}$ and the relative camera pose $T_{t\rightarrow s}$ from the target view to the source view can be derived from:

And the scene at the viewpoint of $I_{t}$ can be synthesized from the source view $I_{s}$ as:

where $I_{s\rightarrow t}$ is the synthesized image, $\langle\rangle$ is the sampling operator, $proj()$ returns the 2D coordinates of the depths in $D_{t}$ when reprojected into the viewpoint of $I_{s}$, and $K$ is the camera intrinsics matrix assumed to be known. Finally the per-pixel photometric loss between $I_{s\rightarrow t}$ and $I_{t}$ is used to optimize the full network. We follow [16] to use the combinaton of structural similarity (SSIM) [43] and L1 distance as the photometric error function and take the minimum across all the source views at each pixel, which is formulated as:

where $\alpha=0.85$. And also, the edge-aware smoothness loss is used to cope with depth discontinuities:

where $d_{t}^{*}=d_{t}/\overline{d_{t}}$ is the mean-normalized inverse depth from [32] to discourage shrinking of the estimated depth.

In this paper, We adopt two data augmentation strategies, Resizing-Cropping and Splitting-Permuting, to generate another two augmented views $I_{t}^{rc}$ and $I_{t}^{sp}$ from the original target image $I_{t}$ for self-distillation. It is worth noting that both the original image and augmented images are involved in the training process. The augmentation detail for a single image view is provided in Algorithm 1 and the overview of our training pipeline is depicted in Fig. 1.

To enhance the fine details in depth maps, in this paper we propose to add a full-scale branch to the encoder and use grid-style decoder, forming the detail-enhanced DepthNet, as illustated in Fig. 2.

Our models are trained and evaluated on the KITTI [39] and Cityscapes [40] datasets, which are two widely used outdoor datasets for depth estimation. KITTI consists of 200 street scene videos captured with RGB cameras. For fair comparison, we use the data split in [6] and follow pre-processing operation in [11] to remove static frames, resulting in 39,810 monocular triplets for training and 697 for evaluation. For Cityscapes, we pre-process and train on 69,731 images from the monocular sequences following [11, 37]. And We evaluate on the 1,525 test images using the provided groundtruth by [37].

Furthermore, we use Make3D [41] and NYUv2 [42] datasets for evaluation to validate the generalization ability of the models trained on KITTI. Specifically, Make3D is a small outdoor dataset containing 134 test images with aligned depth information. And NYUv2 is an indoor scene for depth estimation with 654 labeled test images.

For the PoseNet to estimate camera ego-motion when using monocular video sequences, we follow [16] to construct it from ResNet18 [45] and modify it to accept a pair of color images (or six channels) as input and to predict a 6-DoF relative pose. For the encoder of our DepthNet, we experiment with two kinds of backbone, ResNet18 and HRNet18 [46], to validate the effectiveness of our methods. Each of the two backbones is initialized with a pretrained model on ImageNet [47]. Different from [16], we only predict a single depth map at the full scale to save memory and accelerate training. Our experiment framework is implemented in PyTorch [48]. And we train our models on KITTI with two types, monocular (M), and monocular plus stereo (MS). For each type, the models are trained for 20 epochs using AdamW [49] optimizer, with a learning rate of $10^{-4}$ for the first 15 epochs and $10^{-5}$ for the remainder. The training resolution for KITTI is set to $640\times 192$, the batch size is set to 10, and the maximum depth is 100m. As for Cityscapes, only the monocular (M) setting is trainable. And all the training settings are the same as KITTI, except for a different resolution of $416\times 128$ following [37]. All the experiments are conducted on a single NVIDIA RTX 3090 GPU. For evaluation, we adopt error-based metrics for which lower is better (Abs Rel, Sq Rel, RMSE, RMSE log, $\rm log_{10}$) and accuracy-based metrics for which higher is better ($\delta<1.25$ , $\delta<1.25^{2}$, $\delta<1.25^{3}$). And we employ median scaling before evaluation to tackle the scale ambiguity when using monocular videos.

For evaluation on KITTI [39], we cap the depth to 80m following [16]. We compare the results of our models on KITTI benchmark with several recent methods, which are listed in Table I. On the raw ground truth under the monocular (M) type, our model with ResNet18 backbone already performs better than several approaches [16, 17, 18, 19, 21, 23, 24]. Furthermore, when using a more powerful backbone, HRNet18, our model can surpass them by a large margin. Besides, under the same backbone of HRNet18, our method also shows superior performance than DIFFNet [26] and RA-Depth [27]. Overall, whether the training is under monocular (M) or monocular plus stereo (MS) type, our method can always achieve state-of-the-art performance.

We also provide some qualitative comparison results on different scenes of the KITTI dataset, which are shown in Fig. 3. Our model is qualitatively compared with Monodepth2 [16], R-MSFM6 [19], Lite-mono [23] and RA-Depth [27]. It can be seen in the box region of each scene that our model can obtain higher quality depth maps with finer depth edges.

To validate the generalization ability of our models trained on KITTI, we directly evaluate them on Make3D [41] and NYUv2 [42] datasets without any fine-tuning. The maximum detph is set to 70m and 10m for Make3D and NYUv2 respectively as common practice. The results, as summarized in Table II and Table III, demonstrate that our models still outperform other methods when transfering to out-of-distribution datasets, including outdoor scenes different from KITTI and indoor scenes. It is worth noting that monocular plus stereo (MS) training has poorer generalization performance than monocular (M). We also show the qualitative results on Make3D in Fig. 4 and It can be seen that our model can perceive objects more accurately.

In Table IV we train and evaluate our models on the Cityscapes [40] dataset. We also check the generalization ability to Cityscapes of our models trained on KITTI, without any finetuning. When evaluating on Cityscapes, we cap the depth to 80m as with KITTI. Note that all models in Table IV are trained under monocular (M) setting. And the evaluating resolution is the same as the training ($640\times 192$ for KITTI and $416\times 128$ for Cityscapes). It can be seen that out models still show significant improvement in the two cases.

In this section, we verify the effectiveness of each component in our method. All ablation experiments are trained with ResNet18 backbone and monocular setting, and are evaluated on the KITTI raw ground truth. We add each component independently to the baseline model, Monodepth2 [16], and also experiment with each component removed individually on the basis of our full method. The ablation results are listed in Table V, which demonstrate that all the components can bring significant gains individually. One thing to note is that the self-distillation loss plays the most important part in our method. However, it cannot do without data augmentation.

Besides, we explore more about the two data augmentation strategies and detail-enhanced DepthNet in our method.