Semantics-Driven Unsupervised Learning for Monocular Depth and Ego-Motion Estimation

In previous methods, image reconstruction loss is used, which is a fundamental supervision signal widely used in unsupervised tasks. For two adjacent frames, $I_{t}$ and $I_{t^{\prime}}$, if the depth map of $I_{t}$ and the relative pose between the two views are given, then $I_{t}$ view can be reconstructed from $I_{t^{\prime}}$. Taking $I_{t}$ as input, depth prediction network generates depth map for $I_{t}$, denoted as $\hat{D}_{t}$. The relative camera pose between two views can be estimated from the pose estimation network, denoted as $\hat{T}_{t\to t^{\prime}}$. Denote $p_{t}$ as the homogeneous coordinates of a pixel in $I_{t}$, and $p_{t^{\prime}}$ as the corresponding pixel in $I_{t^{\prime}}$. Using epipolar geometry, the projected coordinates can be expressed as:

where $K$ is the camera intrinsic matrix, $\hat{T}_{t\to t^{\prime}}$ is the camera coordinate transformation matrix from the $I_{t}$ frame to the $I_{t^{\prime}}$ frame, $\hat{D}_{t}(p_{t})$is the depth value of the $p_{t}$ pixel in the $I_{t}$ frame, and the coordinates are homogeneous.

According to the projection relationship, a new synthetic frame $\hat{I}_{t^{\prime}\to t}$ can be obtained from $I_{t^{\prime}}$ frame by using the differentiable bilinear interpolation mechanism proposed in [15].

Structural similarity (SSIM) [47] can be used to evaluate the quality of image prediction. A widely used image reconstruction error function is as follows:

where the superscript $uv$ represents the image pixel at coordinates $(u,v)$ and $\alpha$ usually is set to 0.85.

The image reconstruction loss can be formulated as:

[13] proposes a minimum reprojection loss to handle occlusions. [13] applies a per-pixel mask $\mu$:

where $t^{\prime}\in\{t-1,t+1\}$ and $\left[\cdot\right]$is the Iverson bracket. The minimum reprojection loss is:

In order to solve the gradient-locality issue in motion estimation and eliminate the discontinuity of the depth learned in low texture regions, depth smoothness loss is used to adjust the depth estimation. We adopt the depth gradient smoothness loss in [12] which uses image gradient to weight depth gradient:

where $\nabla$ is the vector differential operator, $T$ denotes the transpose of image gradient weighting and $|\cdot|$ denotes elementwise absolute value.

Similar to photometric consistency, semantic consistency should be satisfied between the adjacent frames. The semantic segmentation of $I_{t}$ is denoted as $S_{t}$. According to equation (1), the semantic segmentation $\hat{S}_{t^{\prime}\to t}$ of the $I_{t}$ frame can be synthesized from $S_{t^{\prime}}$. Different from differentiable bilinear interpolation mechanism in image reconstruction, nearest neighbor interpolation is used in semantic segmentation synthesis, because the value of semantic segmentation represents the class of each pixel. The semantic segmentation reconstruction loss is as follows:

where $\left[\cdot\right]$is the Iverson bracket.

The projection process in equation (1) implies an assumption: the scene is static and there is no occlusion between the two views, but the actual scene obviously does not meet this assumption. The projection rule will make mistakes at the pixels of dynamic or occluded objects. Pixels that do not obey the semantic consistency may be dynamic objects or occlusion, whose pixels should be removed when calculating the reconstruction loss. A mask $M_{t^{\prime}}$, in which the value of dynamic or occluded pixels is 1 and the value of the remaining pixels is 0, is used to indicate these pixels:

where $\left[\cdot\right]$is the Iverson bracket.

The improved image reconstruction loss is:

where $mre(I_{t}^{u,v},\hat{I}_{t^{\prime}\to t}^{u,v})=re(I_{t}^{u,v},\hat{I}_{t^{\prime}\to t}^{u,v})+bM_{t^{\prime}}^{u,v}$ and $b$ is a large constant greater than all possible $re(I_{t}^{u,v},I_{t^{\prime}}^{u,v})$ values.

For the depth of some objects, we can give some prior knowledge constraints. For roads and sidewalks in autonomous driving datasets, between the two adjacent upper and lower pixels in a image, the upper pixel has a longer distance and a larger depth. Therefore, we can introduce the following loss:

where $R^{u,v}$ is 1 if pixel $(u,v)$ is roads or sidewalks, otherwise 0.

The photometric loss is mainly concerned with the 2D pixel coordinate system. The spatial structure information of 3D points can be used as an effective supervisory signal to improve the performance of depth prediction. We propose 3D point loss to make full use of 3D information. [26] aligns 3D point clouds with ICP(Iterative Closest Point algorithm. ICP algorithm needs many iterations in the process of point cloud registration, which results in time-consuming network training. We use two frame depth maps and transformation matrix to calculate the 3D coordinates of corresponding points.

The depth map prediction of $I_{t}$ can be obtained from depth prediction network, and 3D coordinates of each pixel in $I_{t}$ in the camera coordinate system can be further obtained. For a pixel $p_{t}$ whose coordinates are $(u,v)$ in $I_{t}$, denote $P_{t}$ as the coordinates of 3D point corresponding to $p_{t}$ in $I_{t}$ frame camera coordinate system. $P_{t}$ can be expressed as:

Pixel coordinates of the corresponding point $p_{t^{\prime}}$ in $I_{t^{\prime}}$ can be obtained from equation (1).

Denote $\hat{D}_{t^{\prime}}$ as the depth prediction of $I_{t^{\prime}}$ by the network. The coordinates of $p_{t^{\prime}}$ are not integers, so the depth value of $p_{t^{\prime}}$ can’t be obtained directly. Similar to the image reconstruction, we use bilinear interpolation to estimate the depth value of $p_{t^{\prime}}$. Denote $P_{t^{\prime}}$ as the coordinates of 3D point corresponding to $p_{t^{\prime}}$ in $I_{t^{\prime}}$ frame camera coordinate system. Same as equation (11), $P_{t^{\prime}}$ can be expressed as:

Note that $P_{t}$ and $P_{t^{\prime}}$ are in different camera coordinates. They need to be transformed into the same coordinate system.

Transform $P_{t^{\prime}}$ to $I_{t}$ frame camera coordinate system:

where $\hat{P}_{t}$ is the coordinate after transformation, $\hat{T}_{t^{\prime}\to t}$ is the camera coordinate transformation matrix from $I_{t^{\prime}}$ frame to $I_{t}$ frame.

As shown in Figure 3, $P_{t}$ and $\hat{P}_{t^{\prime}\to t}$ are corresponding points and should be as close as possible. The 3D position error is: $pe(P_{t},\hat{P}_{t^{\prime}\to t})=\|P_{t}-\hat{P}_{t^{\prime}\to t}\|_{1}$. Occluded or dynamic pixels should be ignored and the 3D point loss can be expressed as:

where $mpe(P_{t},\hat{P}_{t^{\prime}\to t})=pe(P_{t},\hat{P}_{t^{\prime}\to t})+hM_{t^{\prime}}^{u,v}$ and $h$ is a large constant greater than all possible $pe(P_{t},\hat{P}_{t^{\prime}\to t})$ values.

Compared with the 2D loss using only one frame depth map, the 3D loss using two depth maps based on 3D point consistency, can make better use of 3D spatial structure information.

The framework is divided into two parts: depth prediction network and pose estimation network. Input of the networks includes RGB images and semantic segmentation. We adopt the pre-trained semantic segmentation network in [55], which performs fine-tuning on the 200 training images of KITTI dataset [11] and can output 19 classes, including road, sidewalk, building, wall, fence, etc. Input each image into the network in [55] to get the result of semantic segmentation. The introduction of semantic segmentation network is similar to network pre-training. We use noisy semantic segmentation results, which avoids labeling cost.

Depth prediction network is composed of encoder and decoder networks with skip connections similar to DispNet architecture [27]. Two parallel encoder networks are used to input a single image and semantic segmentation respectively to extract their feature maps. The two feature maps are concatenated and input to the decoder network. Each encoder network has 14 convolution layers and kernel size is 3 for all layers, except the first 4 layers for which the sizes are 5, 5, 7, 7 respectively. The semantic segmentation, which includes 19 classes, inputs the encoder network in the form of 19 channels. The decoder network uses skip-connections to fuse low-level features from different stages of the encoder networks consisting of 7 convolution layers and 7 deconvolution layers.

Pose estimation network takes as input a sequence of adjacent frames concatenated along the color channels, which is similar to the Pose network in [54]. Different from the relative poses between the target view and each of the source views in [54], the relative poses between every two adjacent frames is predicted.

The total loss function is:

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$ and $\lambda_{5}$ are weights for the different losses. Through experiments, we find that setting the weights to $\lambda_{1}=1$, $\lambda_{2}=0.1$, $\lambda_{3}=0.1$, $\lambda_{4}=0.1$ and $\lambda_{5}=0.001$ makes training more stable.

We train and evaluate the proposed method on commonly used KITTI benchmark dataset [11], which includes a full set of input sources including raw images, 3D point cloud data from LIDAR and camera trajectories. We use monocular image sequences for training and test, 3D point cloud and camera trajectories are only used to evaluate training models. The original image size is $375\times 1242$, and images are downsampled to $256\times 832$ during training. In order to compare fairly with other methods, we use two different splits of the KITTI dataset to evaluate depth prediction and pose estimation respectively.

We evaluate the single-view depth estimation performance on the test split composed of 697 images from 28 scenes as in [8]. About 40,000 pictures of the remaining 33 scenes were used for training and validation.

The KITTI odometry benchmark [11] consists of 22 stereo sequences, 11 sequences (00-10) with ground truth trajectories and 11 sequences (11-21) without ground truth. We follow [54] to split the KITTI odometry dataset. We train the model on KITTI odometry sequence 00-08 and evaluate the pose error on sequence 09 and 10.

We evaluate the performance of our monocular depth prediction. The Velodyne laser scanning point is projected into the image plane to obtain the ground truth. Since we use only monocular image for training, absolute scale information can’t be recovered. We multiply the predicted depth map by a scale factor, which is the ratio of the median of ground truth to the median of the predicted depth map, the same as [54]. Our depth estimation results are compared quantitatively with previous works (some of which use certain type of supervision information). All methods are evaluated on the same training images and test images. The split of dataset is described in section 4.1. The error measurements are consistent with those used in [8].

Table 1 shows the comparison between our method and other methods. Abs Rel, Sq Rel, RMSE and RMSE log are error metrics, and small values mean better performance. $\delta<1.25$, $\delta<1.25^{2}$ and $\delta<1.25^{3}$ are accuracy metrics, and large values mean better performance. In order to make a fair comparison with other methods, we use the maximum depth thresholds of 80 meters to evaluate. Our method achieves the best performance on most metrics.

Figure 4 shows some visualization examples compared with other methods. As can be seen from Figure 4, the depth maps predicted by our method are clearer at the boundary of objects and better recover the depth of cars and trees.

We use KITTI odometry dataset to evaluate the proposed approach, and compare the results with Zhou et al. [54], Mahjourian et al. [26], Yin et al. [51], Luo et al. [25], Godard et al. [13], Ranjan et al. [32] and ORB-SLAM [29] proposed by Mur-Artal et al.. We also use the dataset mean of car motion (using ground truth odometry) for 5-frame snippets as another baseline for comparison. Among them, [54, 26, 51, 25, 32, 13] are unsupervised deep learning methods, while ORB-SLAM is a traditional geometry-based method. The methods based on deep learning use the same training data. The models are trained on the KITTI odometry dataset 00-08 sequences and the relative pose estimation is evaluated on the sequences 09 and 10. In the experiment, we fixed the length of the input image sequences to 3 frames, which is the same as [26]. We compared two versions of ORB-SLAM. “ORB-SLAM (full)” accepts all frames of the whole sequence as input, which involves global optimization steps, such as loop closure detection and bundle adjustment. Note that there is no loop in sequence 10, so loop closure detection is not used. “ORB-SLAM (short)” only accepts five consecutive frames as input. Because of the scale uncertainty of monocular VO, we optimize the scale to make the trajectory consistent with the ground truth.

In order to make a fair comparison with other methods, like [54, 26, 51, 25, 32], we measure the Absolute Trajectory Error (ATE) [39] over 3 or 5 frame snippets as the metric for pose evaluation. As shown in Table 2, our method is superior to other methods. The output of pose estimation network is the relative poses between 3 or 5 frames snippets. Compared with the geometry-based method, the camera trajectory predicted by this kind of method has a larger cumulative error for a long time image sequence.

We investigate the contribution of several components proposed in our unsupervised architecture. As shown in Table 3, in order to demonstrate the importance of each component of the losses, we conducted ablation studies on depth prediction. We trained and evaluated three models with different losses. The experimental results show the importance of each component.

Figure 5 shows the depth maps generated by the models under different loss function training. Using all losses can get the best performance. The boundary of objects is clearer, and it is better to predict the depth of small or thin objects such as poles and traffic signs.

As shown in Table 4, we compare the effects of different loss functions on pose estimation results. We can see that semantic loss and 3D point loss improve the accuracy of pose estimation, although not so significant compared to depth estimation.