2nd Place Solution for
Waymo Open Dataset Challenge - 2D Object Detection

With the renaissance of deep learning based object detector, two mainstream frameworks, i.e., one-stage detector and two-stage detector, have dramatically improved both accuracy and efficiency. To fully leverage different detection frameworks, we employ the state-of-the-art two-stage detector Cascade R-CNN [7] and one-stage detector CenterNet [2] with anchor-free fashion. Cascade R-CNN employs a cascade structure for classification and box regression of proposed candidates, which is good at precisely localizing object instances. In contrast to Cascade R-CNN, CenterNet is anchor-free and treats objects as points with properties, which may be better suited for detecting small objects and objects in crowded scenes. We argue that these two different mechanisms have fair diversity on their detections so that the results can be complementary. We respectively produce detections using these two frameworks and then fuse them as the final detection results.

We design a greedy version of the auto ensemble scheme [5] that automatically merges multiple groups of detections according to their detection accuracies, as shown in Figure 3. We note that a group of detections represents the detection results generated from a unique detector framework or a specific inference scheme (e.g., testing with specific image scales) of the same detector framework. As in [5], we consider each group of detections as a node of a binary tree. Let $\mathcal{S}_{l}=\{\mathcal{D}^{k}_{l}\}_{k=1}^{N_{l}}$ be all the detection groups at the $l$-th level of the binary tree, where $\mathcal{D}^{k}_{l}$ is the $k$-th group of detections and $N_{l}$ is the number of detection groups at the $l$-th level. We denote $L$ as the number of levels of the binary tree. Note that $\mathcal{S}_{L}=\{\mathcal{D}^{k}_{L}\}_{k=1}^{N_{L}}$ indicates all the leaf nodes whose detection results are generated from different models and $N_{L}$ stands for the total number of detector frameworks and inference schemes used for the ensemble. For each node $\mathcal{D}^{k}_{l}$, it will be evaluated on the validation set and the corresponding accuracy is $A^{k}_{l}$ that is calculated based on the mAP metric. We iteratively merge every pair of children nodes into one parent node in each level of the binary tree until the root node is reached, where the root node serves as the final detection results. Different from [5], our method determines the hierarchical relations of the binary tree dynamically and greedily and therefore reduces much search space. To be more specific, at the $l$-th level, we treat two nodes $\mathcal{D}^{i}_{l}$ and $\mathcal{D}^{j}_{l}$ as siblings if $\mathcal{D}^{i}_{l}$ and $\mathcal{D}^{j}_{l}$ are available in the candidate node set $\mathcal{C}_{l}$ and the $D^{k}_{l-1}=merge(\mathcal{D}^{i}_{l},\mathcal{D}^{j}_{l})$ yields the best accuracy so far, in which $D^{k}_{l-1}$ is the parent node of $\mathcal{D}^{i}_{l}$ and $\mathcal{D}^{j}_{l}$ and $merge(\cdot,\cdot)$ is the merge operation. After merging $\mathcal{D}^{i}_{l}$ and $\mathcal{D}^{j}_{l}$, we delete them from $\mathcal{C}_{l}$ and add $D^{k}_{l-1}$ into $\mathcal{C}_{l-1}$.

For the merge operation $merge(\cdot,\cdot)$, we search several candidate operations and employ the operation that yields the best accuracy. Given two nodes $\mathcal{D}^{i}_{l}$ and $\mathcal{D}^{j}_{l}$, we define $merge(\cdot,\cdot)$ as:

where $\mathcal{O}$ is the operation set used in our method. $nms$ and $adj{\text{-}}nms$ denotes the traditional NMS and the Adj-NMS [5] respectively. $nmw{\text{-}}naive$ is a simplified version of non-maximum weighted (NMW) [3] that only use confidence scores as weights to merge multiple boxes into one box. In case the detection performance may be degraded after merging, we also introduce $o1(\mathcal{D}^{i}_{l},\mathcal{D}^{j}_{l})=\mathcal{D}^{i}_{l}$ and $o2(\mathcal{D}^{i}_{l},\mathcal{D}^{j}_{l})=\mathcal{D}^{j}_{l}$. The overall algorithm of the greedy auto ensemble is presented in Algorithm 1.

Data distribution is highly imbalanced in the Waymo Open Dataset [11]. For example, there are $1.7M$, and $6M$ instances for the vehicle and pedestrian classes but only $50K$ instances for the cyclist class in the training set. As a result, the cyclist class could be overwhelmed by pedestrian or vehicle samples in training, leading to poor performance on the cyclist class. To solve this problem, we train multiple expert models for the cyclist, pedestrian, and vehicle classes, respectively. Since the Waymo Open Dataset also provides context information for each image frame such as time of the day (e.g. daytime and nighttime). We also train additional daytime and nighttime expert models using only daytime and nighttime training images, respectively.

In Cascade R-CNN, the anchors are predefined manually. By default, the aspect ratios are set to 0.5, 1, and 2. We add two more anchor aspect ratios of $0.25$ and $0.75$ for the vehicle expert model since we observe some vehicles with very elongated shapes. CenterNet is free of the anchor selection problem.

After visually inspecting the annotations, we noticed some hard examples and inaccurate or missing annotations as shown in Figure 4, which may cause problems for the training. Therefore, we employ label smoothing to handle this problem during training.

During inference, we resize the long dimension of each image to $2400$ pixels and keep its original aspect ratio. We use multi-scale testing with $3$ scale factors of $0.8,1.0,1.2$ as well as the horizontal flipping for all models, except for the vehicle expert which only adopts the horizontal flipping. For each model, we first use the class-aware soft-NMS to filter out overlapped boxes. To merge detections generated by different models, we employ the greedy auto ensemble for the pedestrian and cyclist classes, respectively, and use the Adj-NMS for the vehicle class.

To study the effect of each module used in our solution, we perform ablation experiments on the mini-val set as shown in Table 1. We first evaluate the Cascade R-CNN baseline with label smoothing that achieves $59.71$ AP/L2. We further improve performance from $59.71$ to $61.04$ on AP/L2 by utilizing the commonly used inference schemes of class-aware soft-NMS and multi-scale testing. To assess the greedy auto ensemble, we merge the results of baseline with those of expert models, which leads to a notable improvement of $2.24$ AP/L2. Finally, we fuse the detections of the Cascade R-CNN and CenterNet, which further improves $1.44$ AP/L2 compare to the results of CenterNet. It demonstrates the effectiveness of combining the one-stage and two-stage detectors.

The 2D detection track is quite competitive among all the five tracks in the Waymo Open Dataset Challenge. To compare our final submitted result with other competitors, we show the leaderboard of the Waymo Open Dataset Challenge - 2D Detection Track in the Table 2. It is seen that our overall detection system achieves superior detection results and ranks the 2nd place among all the competitors.