SIDE: Center-based Stereo 3D Detector with Structure-aware
Instance Depth Estimation

Given the input stereo RGB images, our goal is to predict 3D bounding boxes and category labels for each object of interest. The attributes of predicted bounding box include the position of the object center (x, y, z), the three-dimensional size (w, h, l), and the steering angle $\theta$. Our method only needs the labels of 3D bounding boxes as supervision for each image.

The complete framework is shown in Fig.2. For stereo detection, we associate the position of objects in the left and right images through the heatmap of objects’ center. The details will be introduced in Section 3.2. For instance depth estimation, we only pay attention to the depth information of the center point of each object, so we construct local cost volume based on the RoIs of objects. Since the target depth information has strong sparsity and locality in the local cost volume, we make use of the structure and internal similarity of local cost volume to aggregate features. As a result, match reweight and structure-aware attention are introduced to make the information more concentrated and thus enhance the accuracy of depth estimation. The details will be introduced in Section 3.3. Finally, given the preliminary accurate estimated depth, we devise an efficient geometric post-processing scheme with the 3D-2D projection formula to further correct objects’ positions. The details are in the Section 3.4.

Usually, object’s position is corrected in stereo pictures to ensure the vertical position of the same object is the same, so $y1$ and $y2$ in the bounding box of the stereo image are the same and we only need to predict a shared $y$ when predicting the position. Therefore, the detection of the object’s stereo 2D box is completed by the heatmap and two other branches, whose predictions are $(o_{ul},o_{ur},o_{v})$, $(w_{l},w_{r},h)$. As shown in Fig.3, $o_{ul}$ and $o_{ur}$ represent the respective offset from the horizontal $u$ coordinate of the object’s center in the heatmap to stereo images, and $o_{v}$ represents the offset of vertical $v$ coordinate. $w_{r}$ and $w_{l}$ represent the respective widths of object’s stereo 2D box, $h$ represents the same height of the 2D box.

For the generation of the image center heatmap, we use the same method in CenterNet [43] and set $Y\in[0,1]^{\frac{W}{R}\times\frac{H}{R}\times C}$ to label the image, where $W$ represents the width of image, $H$ represents the height of image, $R$ represents the multiple of downsampling, and $C$ represents the type of classification. In order to alleviate the imbalance problem of positive and negative samples, we use focal loss [23] to train this network. Since the network down-sampled the image by $R$, when the feature map is remapped to the original image, it will cause errors. Therefore, an additional offset needs to be predicted for each center and we predict the offset from the object’s center in the heatmap to its center in the left and right image. In addition, we regress the length and width of the 2D stereo detection boxes after downsampling which are calculated in advance. All of these values are trained with L1 loss.

For the prediction of object’s dimensions, because 3D dimensions of an object are three scalars, We directly regress to their absolute values in meters using a separate head. For the prediction of the steering angle $\theta$ of the object, we predict the allocentric angle $\alpha$ of each object instead of directly predicting the egocentric steering angle $\theta$. The two angles can be transformed by $\theta=\alpha+\arctan(\frac{x}{z})$. To avoid the discontinuity, the training targets are $[\sin\alpha,\cos\alpha]$ instead of the raw angle value. both values are trained with L1 Loss.

Depth estimation is a very challenging problem in 3D detection. Next, we will elaborate the details of our devised structure-aware instance depth estimation module for tackling this problem in stereo-based 3D detection. Most previous work[38, 41, 32] construct cost volume of entire picture to calculate the disparity relationship corresponding to every pixel, then transform it to get the entire picture’s depth information, which requires the use of depth map or disparity map for supervision during the training process. Compared with the previous, our depth estimation only pays attention to the information of the center depth of each detected object, and only needs to estimate the center depth of the object. Therefore, the cost volume needed to be constructed is smaller, and only the annotated object depth information in the 3D ground truth is required for supervision.

The detail of structure-aware instance depth estimation module is shown in Fig.4. We use RoI Align[11] to select the area of target object on the feature map of the stereo images, build 4D cost volume from the selected feature, and then feed it into a 2-stage 3D convolutional network. After reweighting the cost volume by calculating the similarity of the feature area, it will be fed into the first-stage 3D convolutional network, where we introduce a structure-aware attention mechanism to make the information more concentrated. In the second stage, we use Max Pooling to perform 2 times downsampling after passing 3D convolutional network twice respectively, then we use Max Pooling to perform 4 times downsampling and SoftMax to normalize the prediction. Relying on the network’s normalization, the down-sampled features are finally merged into depth probability of the 3D box center. Therefore, the final result of the object’s center depth can be calculated by $\hat{z}=\sum_{i=1}^{N}z_{i}\times P(i)$, where $N$ denotes the number of depth levels and $P(i)$ is the normalized probability. We train our model with supervised learning using ground truth depth of 3D box center, where supervised regression loss is defined using the error between the ground truth depth $z$ and the model’s predicted depth $\hat{z}$

In addition, since we estimate the depth for each specific object individually, so we do not need to construct cost volume in such a large depth range. According to the width of the 2D box in stereo image, we can roughly calculate a more precise depth range for depth estimation by the camera intrinsic parameters.

This equation uses cosine function to calculate the similarity of the corresponding regions of the feature map in the cost volume of each level. Then we use this similarity to reweight corresponding depth level in the cost volume. Our reweight method mainly comes from the idea that, in the construction of cost volume at different depth levels, the contents contained in the corresponding selected feature maps are different. When the similarity between the two feature maps is high, it means that there are more corresponding regions in the two feature maps at this depth level, which also means that this level is more likely to represent the center depth of the object.

As shown in the structure-aware attention part of Fig.4, after feeding the local cost volume into a 3D convolutional network, we averaged it on the Y-axis to get a depth feature space in 2D bird’s-eye-view, then we use a 2D convolution and Sigmoid function $\sigma$ to determine which parts in the feature space of bird’s-eye-view are useful. Finally, the convolution result is multiplied by the original cost volume to achieve the effect of reducing space noise. The attention process can be expressed by Eq.3

After obtaining object’s 2D box, 3D dimension, steering angle, and the depth of object’s center, object’s 3D position can be roughly calculated. Subsequently, we combine the objects’ predicted perspective keypoints and visual keypoints to further correct the 3D position. In the post-processing, we use box estimation and dense alignment to get more accurate results.

Note that with regard to the perspective keypoints, only one of the bottom corners can be projected within the vertical border of the 2D box visually and the keypoints observed in different perspectives about the same car are different. Therefore, when predicting key points, we not only predict the distance of the keypoint relative to the border, but also predict the type of keypoint. In addition, when the object is truncated on the image, We use the viewpoint angle $\alpha$ to compensate the unobservable states.

We conduct experiments both qualitatively and quantitatively. For comparison, we summarize the results mainly into two groups, monocular-based and stereo-based methods, then we set the downsampling factor R = 2 and R = 4 respectively to train the model and evaluate the performance of our 3D detection method by Average Precision for bird’s eye view $AP_{BEV}$ and 3D box $AP_{3D}$ as shown in Tab.1.

Compared with stereo-based methods without depth map supervision, we obtain the highest $AP_{3D}$ and $AP_{BEV}$. Specifically, we outperform 3DOP over 30% for both $AP_{BEV}$ and $AP_{3D}$ across all kinds of cases. The performance of our model is also much better than Stereo R-CNN in the $AP_{BEV}$ and $AP_{3D}$ because Stereo R-CNN calculates the center depth of the object through a geometric method instead of directly predicting the center depth of the object, which causes inaccurate depth prediction. For IDA-3D, since this method will estimate the depth of object center, its performance is also better than Stereo R-CNN. But our method introduces the match reweight and attention mechanism to make the information more aggregated as well as uses geometric and pixel-level constraints to refine detection results in the post-processing, so the performance of our method are also comparable with IDA-3D, and it is better than IDA-3D in the easy and hard case with IoU=0.5 and all kinds of cases with IoU=0.7.

We also list some stereo-based methods that require depth map as supervision. Although some of them perform better than our method, they all run much slower than our method. Compared with previous monocular-based methods, although the running time of these methods is lower than ours, our method outperforms previous monocular-based methods by a significant margin in all kinds of cases. These comparisons show that our method achieves a good balance between performance and efficiency. Furthermore, we report evaluation results on the KITTI testing set in Tab.2, compared with the testing set results of Stereo-RCNN, our method also shows superiority especially on the hard class.

We conduct some ablation experiments to show the contribution of our proposed network modules.These experiments are performed on the car category in the KITTI dataset and we set the downsampling factor R = 4.