3D Object Aided Self-Supervised Monocular Depth Estimation

To ease illustrating the geometric relationships of camera frames and object frames, the world coordinate system is built at the target image frame, as illustrated in Fig. 3. The camera pose at time $s$ is denoted as $T_{s}$. Suppose an object $O$ is observed both in the source and target frame with poses $L_{s}$ and $L_{t}$ respectively. Further suppose it moves from time $s$ to $t$ with transformation matrix $V$. Then we have

$$L_{t}=T_{t{\rightarrow}s}^{-1}\;L_{s}\;V.$$

(2)

For a physical point $P=[x,y,z,1]^{T}$ in the object frame, the position of $P$ relative to the object does not change assuming the object is rigid. Then the 3D coordinate of point $P$ at different times can be obtained by

$$P_{t}=L_{t}\;P$$

(3)

$$P_{s}=L_{t}\;V^{-1}\;P$$

(4)

The coordinates are different when the object is moving, that is, $V$ is not the identity matrix. Then the homogeneous coordinate of the corresponding pixel in view $I_{s}$ can be acquired through projection:

$$\bar{p}_{s}=K\;T_{t{\rightarrow}s}\;P_{s},$$

(5)

Notice that there is an implicit conversion from homogeneous coordinates to non-homogeneous coordinates before multiplying to K. After substituting $V$ from (2) and $P$ from (3) into (4) and further substituting $P_{s}$ from (4) into (5), we have

$$\bar{p}_{s}=K\;L_{s}\;L_{t}^{-1}\;D(p_{t})\;K^{-1}\;\bar{p_{t}}$$

(6)

where $D(p_{t})\;K^{-1}\;\bar{p_{t}}=P_{t}$ is the back projected 3D coordinates for pixel $p_{t}$. Comparing (6) with (1), we can find that the projected pixel of dynamic object points in source frames dependent only on object poses from 3D object detector. It does not dependent on camera ego-motion $T$ nor object motion $V$. Indeed, when the object is not moving, then $T_{t{\rightarrow}s}$ and $L_{s}L_{t}^{-1}$ are equal according to (2). Therefore, the per-pixel correspondence between $\bar{p}_{s}$ and $\bar{p_{t}}$ which corresponding to dynamic objects is established through object pose change $L_{s}L_{t}^{-1}$.

After the rigid background pixels and the dynamic object pixels are projected into the source frames, the synthesized target view $\tilde{I}_{s{\rightarrow}t}$ can be acquired through sampling from each source view $I_{s}$:

$$\tilde{I}_{s{\rightarrow}t}=M^{(0)}\;I_{s}\langle p_{s}^{(0)}\rangle+\sum_{i=1}M^{(i)}\;I_{s}\langle p_{s}^{(i)}\rangle,$$

(7)

where the superscript $(0)$ denotes the pixel from the rigid background and $(i)$ denotes the pixel from the object $i$, and $M^{(0)}$ is the binary mask of the rigid background, $M^{(i)}$ is the binary mask of the $i$-th object. All the masks are acquired in the source image, therefore not overlapping with each other. $\langle\rangle$ is a differentiable bilinear sampling operator. $p_{s}$ can be computed with (1) if the pixel is static, and (6) if it is dynamic. Now the photometric error between the composite view and the target view is formulated as

$$\mathcal{L}_{ap}=\min_{s}\mathit{pe}(I_{t},\tilde{I}_{s{\rightarrow}t})$$

(8)

Herein, $\mathit{pe}$ is the appearance similarity function and the minimum reprojection loss of all source views for each point is applied. The function $pe$ is defined as:

$$\mathit{pe}(I_{t},\tilde{I}_{s{\rightarrow}t})=\frac{\alpha}{2}(1-\mathit{SSIM}(I_{t},\tilde{I}_{s{\rightarrow}t}))+(1-\alpha)\|I_{t}-\tilde{I}_{s{\rightarrow}t}\|_{1},$$

(9)

which considers both $\mathit{SSIM}$ [13] and $L1$ differences. $\alpha$ is a balancing constant. In projection geometry, the same color image corresponds to numerous different real scenes in the physical world. To make the depth estimation problem solvable, an edge-aware smoothness regularization term [7] is adopted here:

$$\mathcal{L}_{sm}=|\partial_{x}d_{t}^{*}|exp(-\partial_{x}I_{t})+|\partial_{y}d_{t}^{*}|exp(-\partial_{y}I_{t})$$

(10)

where $d_{t}^{*}$ is mean value normalized depth of $D_{t}$. We follow Godard et al. [14] to predict the depth map at multiple image scales. Finally, the combined loss function is formulated as:

$$\mathcal{L}_{photo}=\lambda\mathcal{L}_{sm}+\sum_{l}\mathcal{L}_{ap}^{l},$$

(11)

where $\lambda$ is a weighting constant. The superscript $l$ denotes the appearance loss is calculated at image scale $l$. Then, each subnetwork can be trained by minimizing the final loss $\mathcal{L}_{photo}$ by gradient descent algorithms.

Now we discuss how to acquire the object pose in the source and target views. We use a 3D region proposal network similar to [11] and follow the convention of autonomous driving literature to use 3D cuboids to represent objects in monocular images. Each cuboid $C$ is comprised of 2D bounding box $B_{2d}=[x,y,w,h]_{2d}$, 3D bounding box $B_{3d}=[w,h,l]_{3d}$, center location $t_{c}=[x,y,z]_{c}$ and rotation angle $\theta$ with respect to the camera frame, as illustrated in Fig. 1. The object pose $L$ relative to the camera frame is

$$L=\begin{bmatrix}\cos\theta&0&\sin\theta&x_{c}\\ 0&1&0&y_{c}\\ \sin\theta&0&\cos\theta&z_{c}\\ 0&0&0&1\end{bmatrix}.$$

(12)

Before the detected object pose $L$ can be used to model the dynamic point, the objects detected in every single source and target view must be associated. The problem is also known as data association. Following Wei et al. [15], we consider both the object cuboid shape, location, and 2D box for the association. The score $s$ for each potential association is calculated as

$$s(C^{i},C^{j})=IoU_{2D}(C^{i},C^{j})+D(C^{i},C^{j})+\alpha\;S(C^{i},C^{j}),$$

(13)

where $\alpha=0.5$ for balancing the similarity for object distance and shapes. $IoU_{2D}$ is the Intersection over Union (IoU) of the two 2D boxes, defined as:

$$D(C^{i},C^{j})=exp(-|t_{c}^{i}-t_{c}^{j}|),$$

(14)

which considers the object centroid distance. While $S$ is defined as

$$S(C^{i},C^{j})=\sum_{s\in w,h,l}exp(-|B^{i}_{3d}[s]-B^{j}_{3d}[s]|),$$

(15)

which considers the shape differences of each dimension. As shown in Fig. 4, each object that appeared in the target view is associated with the object wih the highest score in the source view. Then, the projection model proposed in (6) can be applied to all the associated objects.

To further improve the precision of the projection model, we take advantage of panoptic segmentation to distinguish whether every pixel in the 2D box is on or off the object. As shown in Fig. 1, some pixels in the 2D bounding box are actually from the background. To remedy this problem, we use the off-the-shelf tool Panoptic-DeepLab [16] to perform panoptic segmentation for each view. All the training images are preprocessed and are fed together with the color images into our network.

The detected object poses and sizes have true scale because the object detection network is trained with the real size and distance of objects in similar scenes. As a result, the estimated depths of dynamic points that minimize the reprojection appearance loss are consistent with the true scale. For the static points, we design a scale loss to constrain the translation of the camera ego-motion instead of using the additional camera velocity information as proposed in [5]. Specifically, we check the reprojection error of objects with camera ego-motion $T_{s,t}$ against the relative object pose change $L_{s}L_{t}^{\textnormal{-}1}$ . If the two errors are close enough, the object is then determined as static since the reprojection error can also be explained with the camera ego-motion. Then we use the translation of static objects to constrain the translation of the camera. This can be justified by Equation (2), as $L_{s}L_{t}^{\textnormal{-}1}$ equals to $T_{t{\rightarrow}s}$ for static objects with $V$ set to identity matrix. Therefore, we propose the scale loss as follows:

$$\mathcal{L}_{scale}=|tran(T_{t{\rightarrow}s})-\frac{1}{N}\sum_{i}tran(L_{s}^{(i)}L_{t}^{(i)\textnormal{-}1})|$$

(16)

where $N$ is the number of detected static objects and $tran$ is the function to extract the translation part from a transformation matrix. The translation part of object pose changes is averaged over all static objects. Finally, the total loss function is

$$\mathcal{L}_{final}=\mathcal{L}_{photo}+\beta\mathcal{L}_{scale}$$

(17)

where $\mathcal{L}_{photo}$ is from Equation (11) and $\beta$ is a weighting constant set to $0.05$. The scale loss part is effective to recover the true scale of the predictions without decreasing the network performance.

Our network is comprised of three subnetworks, DepthNet, PoseNet, and ObjectNet, as illustrated in Fig. 2. The three subnetworks are connected by the appearance loss between the reconstructed view and the target view. Specifically, the reconstructed view is segmented into two parts: static background and dynamic objects. The dynamic part is obtained by warping the source with $D_{t}$ together with the detected object pose $L$ by ObjectNet. While the static part is obtained by warping the source view $D_{t}$ together with $T$ predicted by PoseNet. The DepthNet and PoseNet are similar with [1]. The DepthNet is an auto-encoder like structure for single view depth estimation. It accepts a single color image and outputs inverse depth map $Disp_{t}$ with the same resolution. The inverse depth is converted to depth map $D_{t}$ by $1/(a*Disp_{t}+b)$, with specifically chosen parameters $a$ and $b$ to constrain the depth to be in the valid range $[0.1,80]$ for the KITTI dataset [17]. The ResNet18 [18] with pre-trained weights on ImageNet [19] is used as the encoder part for feature extraction. The decoder part has 4 layers of building blocks each consisting of convolution and upsampling. The PoseNet accepts two temporally connected image sequence and outputs the 6 DoF relative camera pose $T_{t{\rightarrow}s}$. The 3D object detection network is similar to [11] which adopts DenseNet121 [20] as backbone feature extractor and followed by a Region Proposal Network (RPN) layers that use 3D anchor boxes to perform monocular 3D object detection.

Our proposed network is implemented with PyTorch [21] framework. The panoptic segmentation of color images are precomputed with [16]. The ObjectNet is also pretrained on KITTI object dataset [17]. Regarding data augmentation for DepthNet and PoseNet, we use common random horizontal flips and color augmentation with random brightness, contrast, saturation, and hue dithering, ranging from $\pm 0.2$, $\pm 0.2$, $\pm 0.2$ and [14] proposed $\pm 0.1$. ObjectNet has random flipping disabled, but detected object poses are flipped when necessary. The network is trained for 15 epochs with a learning rate of 0.0001 and drops to 0.00001 for the last 5 epochs. Training takes about 48 hours on an Nvidia RTX 3090 24G graphic card. ObjectNet is frozen for the first 15 epochs to make training more stable. In the last 5 epochs, ObjectNet is fine-tuned together with DepthNet and PoseNet, excluding feature extraction layers to reduce computational cost.

There are total 61 scenes in KITTI raw dataset [17] consisting of city, residential and road. Following Eigen et al. [22] and Zhou et al. [1], a total number of 39810 training images and 4424 validation images from both the left and right camera with static frames removed are used for training. The test set contains 697 images from different scenes from the training set. We compare our work with other networks using similar depth encoder layers. A qualitative comparison of depth estimation with baseline Monodepth2 [14] is given in Fig. 5. We can find that our network can better predict the depth of dynamic objects as marked in the figure. In some highly dynamic scenes, our network can predict reliable depth for moving objects while the baseline network fails in some frames. We also quantitatively evaluate the depth estimation results with standard metrics as defined in [22]. A detailed comparison with other state-of-the-art work is shown in Table I. Our method achieves improved performance without scaling the depth with median value, which is commonly adopted by other works. Following the convention on KITTI depth evaluation, only certain area $A=\{(x,y)|x\in(0.03594771,0.96405229),y\in(0.40810811,0.99189189)\}$ of meaningful points in the depth map is used when calculating all the metrics, where the pixel coordinates are normalized to $[0,1]$ according to the depth map width and height.

To verify the proposed network predicting the right scale in the depth map, we record the scale at each training step. The scale is defined as $scale=med(D_{t}^{*})/med(D_{t})$, where $D_{t}^{*}$ and $D_{t}$ are the ground truth depth map and the predicted depth map, respectively and $med$ is the median value operator. The $med$ is taken over the entire batch of inputs with batch size as $6$. To ease the numerical convergence of the network, we scale the detected objects with $0.02$, which constrains the right scale to be $50$. As shown in Fig. 6, the predicted scale of our network is stabilized at 50 while the baseline network converges to the wrong scale.

We also performed an ablation study of how scale loss affects the depth estimation as shown in Table. II. We first train the network without scale loss and manually set the object scale to a fixed number. Not surprisingly, the results are even a little worse than baseline because the scales of the camera motion and the detected objects are inconsistent. The scale of camera motion (translation) is from PoseNet and the scale of object pose is from a separate ObjectNet. Without scale loss, the depths of rigid background and dynamic objects which are indirectly determined by the scales of camera motion and objects respectively are with different scales, hence leading to a worse result. With scale loss which enforces the predicted scales are consistent, our model is performed as expected.

The 3D object detection network is fine-tuned in a self-supervised manner with depth estimation. After $5$ epochs of fine-tuning together with DepthNet and PoseNet, we observed improvements in object orientation angles, as shown in Fig. 7. Some objects which are partially occluded are detected more consistently on orientations. We also reevaluate the fine-tuned ObjectNet on the original KITTI 3D object dataset. The evaluation is conducted on the val1 [11] data splits. We compared the Average Precision (AP) [27] on 3 core tasks including Bird Eye View (BEV), 3D Object detection and 2D detection under 3 different settings: Easy, Moderate and Hard [27][17]. The AP is calculated with IoU $\geq$ 0.7. All the metrics are improved after fine-tuning as shown in Table III. This shows the effectiveness of our proposed self-supervised training of monocular 3D object detection.

A detailed ablation study of our model is given in Table. II. We adopt Monodepth2 [14] (monocular variation) as the baseline and gradually apply object detection, scale loss, and joint training technique to the baseline. We observed improvements when object detection is combined together with scale loss as discussed in Sec. IV-C. Furthermore, the joint training of depth prediction and object detection improves the results even more as proposed.