DCDet: Dynamic Cross-based 3D Object Detector

VoxelNet Zhou and Tuzel (2018) encodes voxel features using PointNet Qi et al. (2017a), and then extracts features from 3D feature maps through 3D convolutions. SECOND Yan et al. (2018) efficiently encodes sparse voxel features by proposed 3D sparse convolution. PointPillars Lang et al. (2019) divides a point cloud into pillar voxels, avoiding the use of 3D convolution and achieving high inference speed. 3DSSD Yang et al. (2020) significantly improves inference speed by discarding upsampling layers and refinement networks commonly used in point-based methods. PointRCNN Shi et al. (2019) produces proposals from raw points using PointNet++ Qi et al. (2017b), and then refines bounding boxes in the second stage. PV-RCNN Shi et al. (2020a) uses features of internal points to refine proposals. Voxel R-CNN Deng et al. (2021) replaces the features of raw points in the second-stage refinement with 3D voxel features in the 3D backbone.

Label assignment, which is fundamental to 2D and 3D object detection, significantly influences the optimization of a network. Its development is more mature in 2D object detection, with RetinaNet Lin et al. (2017) assigning anchors on the output grid map, FCOS Tian et al. (2019) designating grid points within the range of ground truth boxes as positive samples, and CenterNet Zhou et al. (2019b) identifying center points of ground truth boxes as positive samples. ATSS Zhang et al. (2020) and AutoAssign Zhu et al. (2020) propose adaptive strategies for dynamic threshold selection and dynamic positive/negative confidence adjustment, respectively. YOLOX Ge et al. (2021) introduces the SimOTA scheme for dynamic positive sample selection. Conversely, 3D object detection label assignment is less developed, grappling with unique challenges such as maintaining a balance of positive samples across various object sizes. Current methods in 3D object detection typically use either anchor-based Yan et al. (2018); Lang et al. (2019); Deng et al. (2021) or center-based Yin et al. (2021); Ge et al. (2020); Hu et al. (2022) label assignment schemes. However, these schemes have drawbacks: the anchor-based label assignment often results in unbalanced assignments, and the center-based label assignment may overlook high-quality samples. To simultaneously overcome the above two drawbacks, we propose the dynamic cross label assignment (DCLA). Details about the DCLA are described in the methodology section.

IoU-based losses Rezatofighi et al. (2019); Zheng et al. (2020); Zhang et al. (2022a) without rotation have been well studied in 2D object detection. These methods not only ensure consistency between the training objective and the evaluation metric but also normalize object attributes, leading to enhanced performance compared to the $L_{norm}$ loss. Due to their success in 2D object detection, some 3D object detection methods Zhou et al. (2019a); Sheng et al. (2022); Shi et al. (2022) incorporate IoU-based losses. 3DIoU Zhou et al. (2019a) extends IoU calculation from 2D to 3D by considering rotation. However, the optimization direction of 3DIoU-based loss can be opposite to the correct direction. To address this, RDIoU Sheng et al. (2022) decouples rotation from 3DIoU. It considers rotation as an attribute similar to object location, but it doesn’t consider object direction. A direction loss is needed for classifying object directions. ODIoU Shi et al. (2022) combines $L_{1}$ metric and axis-aligned IoU to regress objects. Our proposed RWIoU incorporates both rotation and direction into the IoU metric, eliminating the need for $L_{norm}$ and direction losses. Details of RWIoU will be explained in the next section.

The label assignment schemes used in existing 3D object detection methods are generally based on prior information such as spatial ranges or object scales to manually select positive samples. For example, the anchor-based label assignment uses object scales to set the sizes of anchors and then uses anchors with IoU greater than a certain threshold as positive samples. The anchor-based label assignment generally produces unbalanced positive samples for different-scale objects, causing the model to prioritize large-scale objects. The center-based label assignment usually takes the center points of grounding truths as positive samples. This can result in a large number of good-quality samples being discarded, leaving inefficient utilization of training data.

The above label assignment schemes have a common property, they all use static prior information as the selection criteria. And the prior information is determined by human experience. Dynamic label assignment schemes Zhang et al. (2020); Zhu et al. (2020); Ge et al. (2021) have shown their advantages in 2D object detection. However, directly transferring these schemes to 3D object detection is not trivial. There are some challenges: 1) There is no space to dynamically select positive samples for small objects (e.g. pedestrians). Because small objects generally cover one or two grid points on the output map; 2) The coverage of objects with different scales varies greatly. This easily results in an imbalance of positive samples between different scale objects.

To dynamically select sufficient high-quality positive samples while maintaining the balance between different scale objects, we propose a dynamic cross label assignment (DCLA) scheme. Specifically, it limits the positive sampling range in a cross-shape region for each object. Typically, an object’s center region on a feature map contains enough features to identify it Tian et al. (2019), and objects in point clouds have regular shapes. Therefore, we only use the center point and its surrounding points for positive sampling in the DCLA scheme. We refer to this sampling range as the cross region. It can be adjusted by a parameter $r$ to adapt to outputs with different grid cell sizes as illustrated in Figure 1. $r$ is the Manhattan distance away from the center point. When $r=1$, the cross region covers the center and its top, down, left, and right neighbors. And when $r=0$, the DCLA degenerates to the center-based label assignment.

The implementation steps of the DCLA are described in detail next. Given a ground truth $\mathbf{b}^{t}$ and positions $P$ in its cross region, calculate the selection cost as follows:

Specifically, given a point cloud input and the ground truth boxes $\{\mathbf{b}_{1}^{t},\mathbf{b}_{2}^{t},\cdots,\mathbf{b}_{n}^{t}\}$, we assume that $f(\mathbf{b}^{t}_{i},\mathbf{b}^{o}_{ij})$ represents the regression loss function, where $\mathbf{b}^{t}_{i}$ and $\mathbf{b}^{o}_{ij}$ denote the $i$-th ground truth box and its $j$-th predicted box, respectively. Therefore, the regression loss $\ell$ for the point cloud is calculated as follows:

We adopt the heatmap target for the classification task. The weights of positive samples are set to 1, and the weights of negative samples in cross regions are set to the values of IoU between predicted boxes and ground-truth boxes. As for the rest negative samples, the weights are all set to 0.

In general, different object categories exhibit significant scale variations, and various attributes such as location, size, and rotation also possess scale differences. Many existing methods employ the $L_{norm}$ loss as the regression loss. However, this loss function renders the model sensitive to differences in both object and attribute scales. Consequently, large objects and attributes dominate the total loss. The IoU metric can normalize object attributes, making it immune to scale differences. Moreover, the optimization objective of the IoU-based loss aligns with the evaluation metrics of detection models. Therefore, substituting the $L_{norm}$ loss with the IoU-based loss often yields accuracy improvement Zheng et al. (2020); Rezatofighi et al. (2019); Sheng et al. (2022).

Utilizing IoU-based loss in 3D object detection poses several challenges. Firstly, calculating traditional IoU requires the computation of polyhedron volumes, which is a complex and computationally expensive task. Secondly, the traditional IoU-based loss, due to its tight coupling with rotation, can lead to misdirection in optimization, resulting in training instability Sheng et al. (2022). Lastly, integrating the traditional IoU metric with object directions is not trivial. Therefore, the inclusion of $L_{1}$ loss or direction loss becomes necessary to aid models in classifying object directions.

To tackle the aforementioned challenges, we propose a rotation-weighted IoU (RWIoU). It thoroughly decouples the rotation from the IoU calculation, making the computation similar to the axis-aligned IoU computation. RWIoU can be implemented with just a few lines of code. By integrating sine and cosine values of rotations of objects into a rotation weighting item, our RWIoU can penalize rotation and direction errors simultaneously.

The RWIoU calculation process is shown in Figure 3. It first considers two rotation boxes $\mathbf{B_{1}}$ and $\mathbf{B_{2}}$ as axis-aligned boxes, and then calculates the intersecting volume of the two axis-aligned boxes as follows:

Single-stage detectors typically encounter misalignment between classification confidence and localization accuracy. To solve the misalignment problem, we follow Zheng et al. to introduce an extra IoU prediction branch. The classification loss $L_{cls}$ and IoU prediction loss $L_{iou}$ are the same as those of CIA-SSD Zheng et al. (2021).

The regression loss $L_{reg}$ is based on the RWIoU. It is calculated as follows:

The baseline models presented in Table 1 primarily utilize either center-based or anchor-based label assignment. Moreover, they commonly employ $L_{norm}$ regression loss. As depicted in Table 1, the center-based label assignment demonstrates a significant advantage over the anchor-based label assignment on the Waymo Open dataset. Nevertheless, our DCDet, featuring a lightweight single-stage network, surpasses the state-of-the-art center-based method DSVT, which employs a heavy backbone network. Notably, even our DCDet model trained on only 20% of the training samples outperforms both the center-based and anchor-based methods trained on the entire dataset. These results demonstrate the superior performance of our DCDet framework which employs DCLA and RWIoU-based regression loss.

We also evaluated our DCDet on the Waymo Open test set by submitting the results to the official server. The performance comparisons are presented in Table 2, revealing that our DCDet surpasses previous state-of-the-art methods significantly. Particularly, in the case of small-scale categories such as pedestrians and cyclists, our method demonstrates a substantial advantage due to the balanced and sufficient positive samples provided by DCLA.

To assess the generality of our DCLA and RWIoU, we conduct experiments by incorporating them into several widely used backbone networks, namely SECOND, PillarNet, and DSVT. All models are reproduced using the OpenPCDet Team (2020) codebase. We train these models using both 20% and 100% of the training data from the Waymo Open dataset and present the results in Table 3. As evident from the table, the integration of our DCLA and RWIoU yields significant improvements across all model groups. This underscores the generality and effectiveness of our proposed DCLA and RWIoU techniques. Notably, the DCLA and RWIoU-based regression loss belong to the learning strategies of models, resulting in cost-free improvements. Even when trained on only 20% of the training data, the models integrated with our DCLA and RWIoU techniques either surpass or catch up to the performance of models trained on the entire training data without these enhancements. This demonstrates that our learning strategies enhance the utilization of training data, which is particularly valuable considering the high cost associated with labeling 3D bounding boxes.

To further study the influence of each component of DCDet, we perform a comprehensive ablation analysis on the Waymo Open and KITTI datasets. For the Waymo Open dataset, following prior works Shi et al. (2020a); Wang et al. (2023), models are trained on 20% training samples and evaluated on the whole validation samples. For the KITTI dataset, models are trained on the train set and evaluated on the val set.