PLoc: A New Evaluation Criterion Based on Physical Location for Autonomous Driving Datasets

The object detection task in autonomous driving is mainly to classify and localize objects around the vehicle in real time. With the development of deep learning in recent years, object detection algorithms have evolved from two-stage detection to one-stage detection.

Two-stage Object Detection  [16] first proposed R-CNN as the research basis for two-stage target detection algorithms in 2014. R-CNN divides the target detection task into two steps of candidate region extraction and detection, thus increasing the target detection accuracy significantly. However, the fixed input size limitation and a large number of repetitive convolutional operations lead to long training time and slow inference speed of R-CNN. In order to improve the detection speed  [17] proposed Fast R-CNN in 2015, which is a fast version of R-CNN. Unlike R-CNN, Fast R-CNN introduces region of interest pooling operation to extract candidate features, which significantly improves the detection accuracy, but the process of generating candidate regions greatly inhibits the detection speed of the algorithm. However, the candidate region generation process greatly inhibits the detection speed of the algorithm. To address this problem,  [18] introduces region proposal network in Faster R-CNN, so that region proposal, feature extraction and target detection can be done in a unified network, which greatly improves the detection efficiency.

One-stage Object Detection The two-stage type algorithms of RCNN series still have the drawbacks of slow inference speed and complex model, which leads to insufficient detection efficiency and cannot be applied to automatic driving scenarios that require fast detection. In order to solve this problem,  [23] proposed the single-stage target detection algorithm named YOLO in 2015, which is different from the two-stage detection method, and eliminates the region proposal step, and directly performs feature extraction and target detection on the input image, thus predicting the bounding box and the class probability of the target at one time. Then researchers proposed several more efficient versions of YOLO algorithm [24, 25, 26] in terms of inference speed and resource consumption to achieve higher accuracy and faster target detection.

Autonomous driving vehicles rely on a variety of sensors to perceive surrounding objects and make driving decisions. In order for a self-autonomous driving car to make accurate decisions, the model need to be trained on a large amount of effective data, so that the corresponding datasets are created.

KITTI [27] Object Detection Evaluation 2012 dataset contains real image data captured from urban, rural, and highway scenes, with up to 15 vehicles and 30 pedestrians in each image. It consists of 7481 training images and 7518 test images. All images are in color and saved as png.

Apollo [28] presented by Baidu in 2019 consists of 73 city street view videos from all over China and under different weather conditions. It contains over 140K images and their corresponding 2D/3D semantic annotations. Appllo dataset has 26 categories including cars, bicycles, pedestrians, buildings, streetlights and so on.

BDD100K [29] launched in 2020 by University of California, Berkeley, contains 100k videos and 10 tasks. The videos are annotated with a variety of annotations including image tagging, lane detection, drivable area segmentation, road object detection, semantic segmentation, instance segmentation, multi-object detection tracking, multi-object segmentation tracking, domain adaptation, and imitation learning. BDD100K has geographic, environmental, and weather diversity and ten categories of ground truth labels.

SODA10M [30] launched in 2021 by Huawei Lab, which includes 10 million unlabeled images of road scenes collected from 32 cities, and 20k images with complete annotation, annotated with six major types of human and vehicle scene categories. The unlabeled images were selected from 32 different cities, covering most regions of China. Additionally, the images encompass a variety of road scenes, weather conditions and times of the day.

A highly effective deep learning object detection model hinges not only on algorithmic design but also on the quality of training data, directly influencing the model’s performance. Consequently, it is crucial to focus on enhancing the training data’s utility to improve model accuracy. In autonomous driving scenarios, the accuracy of object detection models is particularly critical as it directly impacts the ability of autonomous vehicles to make safe driving decisions [15]. Inaccurate identification of pedestrians by these models may result in vehicles failing to avoid them promptly, potentially leading to traffic accidents.

However, prevailing object detection datasets typically treat all pedestrians equally, disregarding the varying importance based on their physical positions. In the real world, the position of pedestrians directly influences the decision-making process of autonomous vehicles. For instance, pedestrians on the roadway pose a more immediate threat to autonomous vehicles compared to those on sidewalks. Neglecting this distinction, models fail to recognize the importance of pedestrians in different locations, diminishing the efficiency of driving decisions.

Because of the limitations of existing measurements standard, we propose a new evaluation criteria(PLoc) that combines physical location of pedestrians into the consideration. It is beneficial for adapting models more effectively to real-world application scenarios by changing datasets. In PLoc, pedestrians are categorized into two different classes based on their physical location in the autonomous driving scenario: 1. Pedestrians on the roadway: Since these pedestrians are in the direct path of the autonomous driving car, there is a high possibility of accidents occurring and get pedestrians in danger. 2. Pedestrians on the sidewalk: Although overall driving environment awareness is still important, these pedestrians pose less threat to autonomous driving cars. Therefore, these types of pedestrians are given a lower weight under the guidance of PLoc. This can be achieved by setting different weight for pedestrian, such as a weight of 1.0 for pedestrians on the roadway and 0.5 for pedestrians on the sidewalk.

Generally, the loss function of an object detection model comprises three main components: Box Loss, Classification Loss, and Distribution Focal Loss. These components are utilized to assess the accuracy of the bounding box, the effectiveness of target identification, and the ability to handle category imbalance and indistinguishable samples. The mathematical expression of this loss function is as follows:

where:

• •

  $N$ is the total number of cells.

• •

  $\lambda_{\text{box}}$, $\lambda_{\text{cls}}$ and $\lambda_{\text{dfl}}$ are the weights of Box Loss, Classification Loss and Distribution Focal Loss respectively.

• •

  $\mathcal{L}_{\text{box}}(\theta)$, $\mathcal{L}_{\text{cls}}(\theta)$ and $\mathcal{L}_{\text{dfl}}(\theta)$ represent the functions of Box Loss, Classification Loss and Distribution Focal Loss respectively.

Building on the concept we just proposed, we categorize the pedestrians in the dataset into two groups: those on the road and those on the sidewalks. We use Eq. 1 to calculate the loss for each type of pedestrian separately. The total loss function of the new model can then be represented as a weighted sum of these two losses. Specifically, the loss attributed to pedestrians on the road is expressed as follows:

The loss caused by pedestrians on the sidewalks is partially expressed as:

Then, the total loss function is in the form of:

where $\lambda_{\text{r}}$ and $\lambda_{\text{s}}$ are the weights of the loss caused by pedestrians on the road and sidewalks respectively.

By adjusting the weights of the two losses, the model can focus more on the loss caused by certain pedestrians. In the previous example, if we give a higher weight (like 1) to pedestrians on the road and a lower weight (like 0.5) to those on the sidewalk, then the model will focus more on issues caused by pedestrians on the road. By making this distinction, autonomous driving models can be motivated to pay more attention on pedestrians in the roadway, while still maintaining the ability to recognize all pedestrians.

To implement PLoc and validate its applicability in real-world scenarios, we introduce the ApolloScape-R dataset, derived from the ApolloScape [28] dataset. Specifically, we selected images from ApolloScape containing pedestrians and re-annotated them based on PLoc, considering the physical location of pedestrians. Subsequently, we conducted experiments on ApolloScape-R using ten object detection models to assess their effectiveness in detecting pedestrians at different locations under PLoc. Examples of ApolloScape-R are depicted in Fig. 2.

To validate the new evaluation criteria PLoc, we proposed a new dataset that includes the physical location information of pedestrians and conducted experiments using 10 popular object detection models. Experimental result shows in Table I, II. These models include: ATSS [35], Cascade R-CNN [32], DDQ [39], Deformable-DETR [36], Faster R-CNN [18], FCOS [33], RepPoints [34], RetinaNet [31], Sparse R-CNN [37], and YOLOX [38]. We categorized pedestrian targets into two types, including pedestrians on the road and pedestrians not on the road, in order to get the performance of different models in recognizing these two types of pedestrians. In the experiments, we used the standard mean Average Precision (mAP) and calculated at different Intersection over Union (IoU) thresholds, such as mAP, $\text{mAP}_{\text{50}}$ and $\text{mAP}_{\text{l}}$ (for large-sized objects) as evaluation metrics. By comparing three different mAP values, all tested models in the ”Pedestrians on the Road” category have better performance in the ”Pedestrians not on the Road”.

Among them, the DDQ [39] model has the best pedestrian detection performance, be cause its three mAP values for person not on road are higher than other models, which are 0.120, 0.236 and 0.262 respectively. However, when this model detects person on the road, its detection effect drops significantly, only 0.066, 0.125 and 0.164. This means that the model has obvious performance advantages in detecting pedestrians not on the road compared to detecting pedestrians on the road. Similar significant performance differences also appear between the Cascade-RCNN [32] model and Sparse-RCNN [37] model. Although compared with this huge difference, the performance difference of ASTT [35], Faster-RCNN [18], FCOS [33], RepPoints [34] and YOLOX [38] models in detecting two different pedestrians has been reduced, but they are still better at detecting pedestrians on the road. The Deformable-DETR [36] and RetinaNet [31] models have shown relatively good performance consistency in the detection of different types of pedestrians. They have similar effects in detecting pedestrians on the road and pedestrians not on road. However, their mAP values are at a low level compared to other models, which means that although their performance does not fluctuate significantly due to changes in pedestrian locations, their detection effects are not outstanding.

Experiments across different models confirm a significant difference in the accuracy of object detection models for pedestrians at different physical locations. This observation extends beyond pedestrian detection, constituting a new evaluation criterion for autonomous driving object detection based on physical location. This location-based evaluation measurement can be further extended to other object detection tasks in the realm of autonomous driving, enhancing the accuracy of these models in real-world applications.

For instance, trees in different locations may impact a vehicle’s movement differently, as illustrated in Fig.  3. In most scenarios, the tree on the roadside may only be part of the environment and does not directly affect the driving path. While, in some special cases, a tree may grow in the middle of the road, which requires the detection model to be able to accurately recognize and avoid it. Therefore, evaluating the importance of trees depending on their physical location can help autonomous driving vehicles make better decisions.

Autonomous driving vehicles are not only influenced by objects located on the road, but signal-type objects also play a crucial role in driving decisions. Distinguishing between different signals based on their location is essential to ensure that autonomous driving cars follow traffic rules correctly. For instance, when a vehicular traffic signal is in a position similar to a pedestrian crossing signal, the vehicular traffic signal should be given more attention to ensure accurate signal recognition.

Additionally, considering the design of entrances and exits on a road, traffic signs in different physical locations can pose a threat to the accurate decision-making of an autonomous driving car. The autonomous driving object detection model should accurately recognize signs at different locations to avoid mistakes that could lead to dangerous situations.

In summary, the physical location-based evaluation criterion holds practical application value, effectively improving the accuracy of the autonomous driving object detection model and enhancing overall safety. Moreover, this criterion can be generalized to various fields such as urban planning, intelligent monitoring, and robot navigation, providing new perspectives and methods for solving practical problems in complex environments.