YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection

Stereo 3D object detection is usually considered as a tractable but computationally hard problem. Recent advances in stereo 3D object detection algorithms are based on the idea of pseudo-LiDAR [15]. DispRCNN [5], ZoomNet [6], and OC Stereo [7] applied instance segmentation on binocular images to construct a local point cloud for each detected instance to improve the accuracy of disparity estimation on foreground objects. Pseudo-LiDAR++ [3] recognized that uniform 3D convolution might not be suitable to process the disparity cost volume, and transformation to the depth cost volume may be needed.

We point out that all the aforementioned algorithms require more than 0.3 seconds runtime per frame. Moreover, Pseudo-LiDAR++ [3], ZoomNet [6], OC Stereo [7] and DSGN [16] required point cloud data during training or need point cloud data to help the training process. DispRCNN [5] and the baseline Pseudo-LiDAR [15] required off-the-shelf disparity modules, which are usually trained with depth images or point cloud data.

YOLOStereo3D is a light-weight model that performs most of the convolution operation in the perspective view, and the training and inference are significantly lighter and faster than all methods mentioned above. Moreover, the training process of YOLOStereo3D does not depend on point-cloud data.

Monocular 3D object detection is an ill-posed problem, but it provides many insights into how depth information can be estimated from a single image. Tom et al. [17] demonstrated that a typical monocular depth estimation network mainly estimates depth from the vertical position of an object. The authors [17] provided the theoretical background for pseudo-LiDAR in monocular detection [18][19]. YOLOStereo3D is built upon the inference structure of M3D-RPN [20] and GAC [21] and further enhances the final features with stereo matching results.

Each anchor is described by 12 regressed parameters including $[x_{2d},y_{2d},w_{2d},h_{2d}]$ for the 2D bounding boxes; $[c_{x},c_{y},z]$ for the 3D centers of objects on the left image; $[w_{3d},h_{3d},l_{3d}]$ corresponding to the width, height and length of the 3D bounding boxes respectively; and $[sin(2\alpha),cos(2\alpha)]$ to estimate the observation angle/orientation of objects.

We observe that $[sin(\alpha),cos(\alpha)]$ and $[sin(\alpha+\pi),cos(\alpha+\pi)]$ correspond to the same rectangular bounding box results in 3D object detection. As a result, we instead predict $[sin(2\alpha),cos(2\alpha)]$ in the regression branch. We also add a classification channel to predict if $|\alpha|>\frac{\pi}{2}$ to eliminate ambiguity, which intuitively means whether or not the object is facing the camera.

We incorporate 3D statistic priors into 2D anchors to improve the regression results. To collect prior statistics of the anchors, we iterate through the training set, and for each anchor box of different shapes, collect all the objects assigned to this anchor based on the IoU metric. Then, we compute the mean and the variance of $z$, $sin(2\alpha)$, $cos(2\alpha)$ for each box.

Furthermore, we explicitly exploit scene-specific knowledge for autonomous vehicles and utilize the statistical information from the anchor boxes. During training, we project dense anchor boxes into 3D with the mean depth value $\hat{z}$ and filter out anchor boxes that are far away from the ground plane based on their, as displayed in Figure 2.

For multi-class training, since the statistics for different types of obstacles, e.g., cars and pedestrians, are significantly different, we compute 3D priors for each category, separately. During training, we filter out anchor boxes dynamically based on the categories assigned. During inference, we also filter out anchor boxes dynamically based on the anchors’ local categorical predictions.

Data augmentation is useful to improve the generalization ability in deep learning applications. However, the nature of stereo 3D detection limits the number of possible augmentation choices. We follow [20] to apply photometric distortion concurrently on binocular images. We also follow [22] to apply random flipping online during training. Random flipping includes flipping both RGB images, flipping the position and orientation of objects, and then switching left/right images.

Current state-of-the-art stereo matching algorithms usually construct 3D cost volume with concatenation, where the module iteratively shifts the right feature map horizontally over the left feature map, and at each step, concatenate the two features at each overlapping pixel. For binocular feature maps with the shape $[B,C,H,W]$, the shape of the output tensor $f_{i}$ is $[B,2\cdot C,max\_disp,H,W]$. In this paper, we follow [10] and [13] to apply a normalized dot-product to construct a thin cost volume. Such a module compute correlation between two overlapping pixels of the feature maps instead. The shape of the output tensor $f_{i}^{\prime}$ becomes $[B,max\_disp,H,W]$.

The stereo matching process can be much faster. Consider two input feature maps of $[1,64,72,320]$, which is a common shape of a KITTI image down scaled by 4. The forward pass of concatenation-based cost volume construction takes about 200 ms while the correlation-based cost volume takes about 7 ms on an Nvidia-1080Ti.

However, the number of output channels is smaller, which could cause the network to be numerically skewed towards monocular features during the fusion stage and downsampling the stereo matching results could induce further information loss. We ease these two problems with densely connected ghost modules [23] and a hierarchical fusion structure.

As mentioned in Section III-B1, we need to expand the width of the features to guide the network to skewed towards features produced by stereo matching.

Han et al. propose the ghost module, which is an efficient module to produce redundant features [23]. It applies depthwise convolution to produce extra features, which requires significantly fewer parameters and FLOPs. We go one step further and densely concatenate the original input features with the output of the original ghost module, thereby tripling the number of channels. As indicated in Figure 1, the mauve blocks in (b) are the results from ghost module and others denote densly connected residuals.

Such a module preserves more information before downsampling and also rebalances the number of channels between stereo features and monocular semantic features during the fusion phase.

To minimize the information loss during the stereo matching phase while keeping the computational time tractable, we engineer a hierarchical fusion scheme. At the downsampling level of $\frac{1}{4}$ and $\frac{1}{8}$, we construct a light-weight cost volume of a max-disparity of 96 and 192, respectively. As shown in Figure 1, they are fed into a densely connected ghost module, downsampled, and concatenated with features at a smaller scale. At a downsampling level of $\frac{1}{16}$, we first downsample the number of channels with $1\times 1$ convolution. We then construct a small concatenation-based cost volume (also flattened to be a 2D feature map) to preserve more semantic information from the right images.

This arrangement can also be justified with high-level reasoning. Features with higher resolution are usually local features with higher frequency portions, which are suitable for dense and accurate disparity estimation. In contrast, features with low resolution contain semantic information at a larger scale.

As pointed out by Chen et al. [16], disparity supervision is important to improve detection performance. We also observe a similar phenomenon in our framework. Without disparity supervision, the network may not be guided to produce local features useful in stereo matching to fully utilize the geometric potential of binocular images, and the network could be trapped in a local minimum similar to that of a monocular detection network.

We upsample the output of the final stereo features to $[W/4,H/4]$, and supervise the prediction with a sparse ”ground truth” disparity derived from the traditional block matching algorithm in OpenCV [24] during training. During evaluation and testing, this disparity estimation branch is disabled to improve efficiency.

Though the disparity from the block matching algorithm is coarse and sparse, we empirically show that it significantly improves the network’s performance.

We apply focal loss [26][27] on classification, and smoothed-L1 loss [28] on bounding box regression. We follow the scheme of [14] to apply stereo focal loss on the auxiliary disparity estimation. First, we compute the expected distribution of disparity with a hard-coded variance $\sigma=0.5$:

$$P(d)=\operatorname{softmax}\left(-\frac{\left|d-d^{gt}\right|}{\sigma}\right)=\frac{\exp\left(-c_{d}^{gt}\right)}{\sum_{d^{\prime}=0}^{D-1}\exp\left(-c_{d^{\prime}}^{gt}\right)}.$$

Where $d$ represents the disparity and $c_{d}$ indicates the predicted confidence at disparity $d$. Then, following [14], stereo focal loss is defined as:

$$\mathcal{L}_{SF}=\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}\left(\sum_{d=0}^{D-1}\left(1-P_{p}(d)\right)^{-\alpha}\cdot\left(-P_{p}(d)\cdot\log\hat{P}_{p}(d)\right)\right)$$

where $D$ is the max-disparity, $\alpha$ is the focus weight, $\mathcal{P}$ presents the set of pixels involved, and $P_{p}(d)$, $\hat{P}_{p}(d)$ represents the expected and predicted distribution map of disaparity $d$.

The final loss function is simply the sum of the three losses.

We first test the effectiveness of including statistical information in each anchor. In the first experiment, instead of predicting a depth value normalized by the depth prior, the network outputs a transformed depth output $\hat{z}$, where $z=1/\sigma(\hat{z})-1$, following [34]. In the second experiment, we do not filter out anchors during training, and the training loss is evaluated with all anchors. We conduct these two experiments in both the stereo setting and the monocular setting. The results are presented in Table IV respectively.

From the two table, we can observe that anchor priors significantly boost the performance of the network, and filtering out irrelevant anchors during training is also helpful. The performance gain can be observed in both monocular 3D detection and stereo 3D detection. We suggest that we can ease the difficulty of depth inferencing by properly defining and preprocessing anchors specifically for 3D scene understanding in autonomous driving.

The improvement we apply on anchors can also be applied and verified in monocular 3D detection. We further provide ablation experiments to validate the effectiveness of these processing methods under monocular 3D detection setting.

Densely-connected ghost modules are useful in expanding the number of channels in stereo processing. We conduct two experiments to verify its effectiveness. In the first experiment, we use a BasicBlock in resnet [30] to replace the ghost module without expanding the number of channels, resulting in fewer channels during the fusion between RGB features and stereo features. In the second experiment, we directly upsample the number of channels with $1\times 1$ convolution before feeding the tensor into a BasicBlock.

We can observe from Table IV that the densely-connected ghost module is useful in improving the network’s capability. From the first experiment, we demonstrate that expanding the number of channels is crucial for the network’s performance. In the second experiment, we further show that the densely-connected ghost module is better at preserving information than the naive $1\times 1$ convolution.

We respectively disable the stereo matching output on scale 8/16 to produce two networks to justify the usage of multi-scale fusion. We can observe from Table IV that the results of the two ablated models are inferior to that of the baseline model. We also point out that the forward pass of stereo matching modules on scale 8/16 is several times faster than that on scale 4.

As a result, we argue that hierarchically fusing stereo features from scale 8 and 16 is worth the effort. The baseline structure of hierarchical fusion in YOLOStereo3D achieves a fair balance between speed and performance.

We also have an ablation study on the importance of disparity supervision. Similar to the conclusion in DSGN [16], disparity supervision significantly boosts the performance of the network.

In the experiment, we show that such supervision is essential, but the results are not sensitive to the accuracy of the ”target” disparity map. The insight is that the network may only need slight regularizations in stereo matching submodules. The auxiliary loss is required to drive the network from falling back to a naive local optimal of monocular 3D object detection.