S2O: An Integrated Driving Decision-making Performance Evaluation Method
Bridging Subjective Feeling to Objective Evaluation

As introduced in the literature review, a risk energy field is selected to assess safety because of its continuity, generalization ability, and comprehensiveness. Based on the study of Wang et al. [28], we revise the Driving Safety Model (DSF) to fit the demands of safety evaluation.

The basic idea of DSF is to treat each agent on the road as a risk source radiating risk energy. The risk a vehicle is exposed to is the sum of the risk energy generated by all agents at that location, as shown in Fig. 3. In this study, we improve the definition of the safety energy model by considering the concept of virtual mass and equivalent distance.

The safety risk a vehicle at position ($x$,$y$) has from an agent at position ($x_{a}$,$y_{a}$ ) is defined as (1):

The virtual mass $M_{eq}$ is the revised mass of a vehicle, since a vehicle with higher velocity will cause more severe collision results. The expression of $M_{e}q$ is shown in (2):

The equivalent distance $r_{eq}$ is a revision of the vanilla distance calculation considering the motion sensitivity of the ego vehicle in different directions. The definition is shown in (3):

$R_{ego}$, the overall risk of the ego vehicle at position $(x,y)$ is the superposition of the risks generated by all agents in the scene at position $(x,y)$, as expressed in (4) where $A$ is the agent set. And the final safety term score $S_{safety}$ is the average risk energy intensity during driving, as shown in (5) where $t_{f}$ and $t_{s}$ are the finishing and starting time.

Note that when agents are too far away from the ego vehicle, the risk energy they generate can be considered negligible compared to those that are closer. To improve computational efficiency, in this study, we consider an area of interest (ROI) of 100 meters to the front and 50 meters to the rear. In (5), only agents within the ROI are taken into calculation.

Time efficiency is usually measured by the ratio of the current velocity and expected speed which in most cases is the road speed limit. When speed is slow, the quicker the better. However, penalties should be given for excessive speeding. Taking China’s traffic law as an example, speeding less than 20% can be tolerated, while more than that will be seen as a violation. In addition, exceeding the speed limit by over 50% is considered as severe speeding. In our time efficiency evaluation design, we follow these regulations, as shown in (6):

The final evaluation score of time efficiency is the average value of $s_{eff}$ during the driving event, as illustrated in (7).

As introduced in related works, basic comfort evaluation components involve jerk, trajectory curvature, etc. Meanwhile, velocity also influences the discomfort level for passengers while riding, since centripetal force experienced by people is proportional to the product of the vehicle’s angular velocity and linear speed.

Apart from these vehicle state-related quantities, specific behaviors such as repetitive U-turns and emergency stops also significantly decrease the comfort level. Hence, in this study we add a discrete term to punish the unpleasant motions.

The overall comfort model is described in (8), where $\omega$ is the angular velocity, $v$ is the linear velocity, $j$ is the jerk, $L_{upm}$ is the loss caused by unpleasant motions which is a behavior counter, and $k$ is a constant weighting coefficient. Similar to other factors, the final comfort score of a driving case $S_{comf}$ is the temporal weighted average value of $s_{comf}$, as shown in (9).

In our study, we use the total energy consumed for driving as the indicator to evaluate the energy efficiency so that differences on powertrain systems, which are irrelevant to the decision-making algorithms, will not influence the assessment results. According to vehicle dynamic theories, the total consumed energy can be decomposed into four parts: acceleration, wind resistance, road gradient resistance, and rolling friction [18].

The acceleration term represents the energy used for velocity increasing. The calculation is given by (10), where $\delta$ is the conversion coefficient of the rotating mass, $m$ is the vehicle mass, $u_{a}$ (km/h) is the vehicle speed, and $u$ (m/s) is also the vehicle speed.

As for the wind resistance, the calculation is shown in (11), where $C_{D}$ is the air resistance coefficient, and $A$ is the windward area of the car.

Road gradient resistance can be computed by (12), where $g$ is gravitational acceleration and $i$ is the road gradient.

Road rolling friction can be derived by (13), where f is the road surface coefficient.

The total consumed energy is the sum of (10) to (13), as shown in (14). And the final energy term score of the whole driving $S_{eng}$ event is computed by (15).

One critical problem is that the magnitudes of the four single-term scores are different, ranging from single digits to hundreds. However, what matters is the relative variance instead of absolute values. Therefore, in this study we first normalized all terms into an identical range, which is $[60,100]$.

Common normalization methods include max-min linear normalization, $arctan$ curve, robust normalization, z-score, etc. After comparison, the result of which will be shown in the experiments section, we selected max-min linear normalization as the normalization style. Max-min linear normalization is a common method of data normalization that transforms data into a unified value range. The standardized value of term $i$ $S_{i}^{std}$ can be calculated by (16), where $S_{i}$ is the original value of the term, $S_{i}^{min}$ is the minimum value of the data, and $S_{i}^{max}$ is the maximum value. The value ranges from 60 to 100.

Decision-making evaluation is not a simple linear problem. Based on the analysis of the human rater evaluation score distribution, it can be observed that for different levels of driving events, people pay varying degrees of attention to each factor.

Fig. 4 demonstrates the distribution between the overall evaluation ground given by human raters in the D2E dataset and four normalized single-term scores. Each point represents a driving event, with the horizontal axis representing the corresponding term score and the vertical axis representing the rater’s score. The original human score range is $[0,100]$. According to human rating criteria in the D2E dataset, 60 is considered as the borderline of acceptable driving, and events with extremely low scores are often associated with collisions and other incidents, leading to anomalies in the values of the four individual metrics. To eliminate the disturbances caused by abnormal events and observe the patterns in regular driving evaluations, we exclude all events involving collisions from the distribution. Hence, the lower bound of human rating scores for the remaining events is close to 60. In the later integrated evaluation method design, we will use a revision multiplier to restore the impact of collisions.

According to the holistic analysis of the distribution, we can observe that no single element exhibits a significant linear or simple monotonic relationship with human scoring, which further proves that driving decision-making evaluation is a complicated process influenced by a multitude of factors and cannot be simplistically represented by a single term. The distribution results demonstrate the necessity of an integrated evaluation method considering multiple factors.

On the other hand, from the perspective of each factor, we can discover that humans have varying factor sensitivity to driving events with various score levels. Generally speaking, comfort is strongly correlated with the high score range. Only when the comfort level is high can the case achieve a good overall score. Comparatively, the energy efficiency factor has good discrimination ability on all driving events with scores above 60. The safety term, with a wider distribution, is more likely a prerequisite of good driving, thus having little impact on the high score range. However, for the mid and low range safety generates many differences. Similarly, the time efficiency factor shows great impact on the mid and low range, while making little difference in the high range. Considering the fact that humans’ evaluation mechanisms of various driving event levels are different, in this study we separate driving evaluation into three segments and fit a set of weights for each one to reflect diverse human sensitivity towards driving events of different levels.

Fig. 5 shows the frequency of human evaluation scores. According to the distribution, we divide the evaluation into three intervals: low: [0, 75], mid: (75, 85], and high: (85, 100], accounting for 24.7%, 38.3%, and 37.0% of the full set of evaluation scores respectively. This segmentation also aligns with human intuitive impressions of high, medium, and low scores.

Since in the later linear fitting module, we will use different weights for various segments, it is necessary to determine which interval an event falls into first, which is a classification problem. To balance the interpretability and accuracy, we use Support Vector Machine (SVM) as the classifier model. The primary objective of SVM is to find the hyperplane that best separates the data points into different classes while maximizing the margin. The decision boundary or hyperplane is determined by a linear combination of the input features, represented by  (17) where $\bm{w}$ is the weights and $b_{c}$ is the bias term. $\bm{x}$ is the input vector, which is the combination of the four single-term evaluation scores in this study.

The goal of SVM is to find the optimal hyperplane that maximizes the margin, which can be formulated as the following optimization problem (18) where $y_{i}$ is the segment label of driving events, $N$ is the total number of data points.

Apart from the SVM, a pre-screening for collision is conducted to filter the crashed cases. Considering that safety should always be the priority of driving, once a collision happens the classifier will label the driving event as the lowest level.

After training the SVM classifier, the segmental linear fitting can be realized to precisely assess driving decisions.

With the four normalized single factor scores and the segment classification, a segmental linear fitting model is applied to give the integrated decision-making evaluation score, as shown in (19), where $w_{safe}$, $w_{eff}$, $w_{comf}$, and $w_{eng}$ are the corresponding weights of the terms, and $b$ is the constant offset. The offset is used to compensate for the static offset in human ratings. For instance, in many events, raters assign extremely high scores exceeding 95 points. However, it’s hard for objective algorithms to achieve scores close to the maximum. Therefore, the offset helps to decrease the error.

As a regression problem, the weights of three segments are derived by fitting on the D2E dataset, the process and results of which will be introduced in the experiment section.

Apart from the four mentioned terms above, the human evaluation of driving experience also includes a veto mechanism for crashes.

Although the safety term can partly reflect the risk when a crash happens, since it is an average value on the whole driving case, more severe punishment should be given when sudden extreme danger happens. Otherwise, even when an accident occurs, the score can still be tens of points, which does not meet the human’s priority for absolute safety. Referring to the Nuplan simulation [5] and Carla leaderboard [37], crash revision can be expressed by a punishment multiplier as shown in (20). The final revised integrated evaluation score $S_{int}^{rev}$ is given by (21).

The first category of baselines are the four single-factor models designed in this research, which are safety, time efficiency, comfort, and energy consumption. The comparison between the single-factor methods and comprehensive ones will demonstrate whether considering multiple factors is essential for decision-making evaluation.

Based on the Least Action Principle, this method merges risk, efficiency, comfort, and smoothness into one defined action quantity $S_{int}$ of ego vehicle $i$ [20], as shown in from (23) to  (24):

After defining the action quantity $S_{int}$, this method labels the average $S_{int}$ value of excellent drivers as the pole score, denoted as $S_{int}^{*}$, and normalizes the distribution of other driving case’s $S_{int}$ to a range between 0 and 1 linearly. The final position in the [0,1] range is the final evaluation result. In this study we also transfer it into a percentage scale.

This baseline divides driving evaluation into four major categories: safety, comfort, intelligence, and efficiency. Then, each category is further broken down into a linear combination of several physical quantities. Subsequently, the weights between physical quantities within each category are determined using the Analytic Hierarchy Process (AHP), followed by calculating the weights between categories using the criteria importance through intercriteria correlation (CRITIC) algorithm [21]. Through these two procedures, the needed relative weights are derived and confirmed. In this experiment, we use the weights recorded in the paper.

Similar to the second baseline, this method also selects the best single categories from human-driven cases as the pole scores and utilizes the Grey Relational Analysis (GRA) to output a similarity of the tested trajectory terms to the pole scores. Then combined with the previously derived weights, a final evaluation score is given.

Currently many AD planning challenges use a penalty-based evaluation system, and we select the Onsite Competition’s published judging rules as a baseline in this experiment. This system considers three categories of driving: safety, efficiency, and comfort, with a point ratio of 50 to 30 to 20. Each aspect involves many penalty conditions, and if one condition is met in one timestep, a penalty will be given. At the end of the entire driving event, the final score is obtained by subtracting the penalty points from all-time steps from the total score. The detailed penalty conditions are illustrated in Table  6.

To verify the feasibility of our proposed method, we embed the evaluation algorithm into SUMO, a simulation platform widely used in transportation and vehicle-related research, to conduct online driving decision-making evaluation, as shown in Fig. 9 which shows the visualization interface and the original SUMO GUI.

In this example, we run a default decision-making algorithm within SUMO. The physical states of agents and the roadmap are continuously updated and transferred into the evaluation module, and assessment results are computed in real time. The visualization interface, which is divided into four sections, can output real-time scores for the four individual factor scores, the total score, the Driving safety field heat map that is used for safety evaluation, and the diagram of the traffic scenario. The implementation of S2O on the SUMO platform demonstrates the feasibility of our algorithm.

To directly validate the precision and correctness of the decision-making evaluation function, based on the above real-time platform we test the driving performance of three different autonomous driving decision methods and compare with the alignment of human subjective evaluations.

The three decision-making methods used in this experiment are Intelligent Driver Model (IDM) [44] and two optimization-based models with different weights on driving factors [45, 46]. In the validation, we run the three tested AD planning algorithms on eight interactive scenarios which could generate differentiated behaviors, as shown in Fig. 10. In these test scenarios, the ego vehicle needs to driving forward facing the interactions from surrounding vehicles (SV) ranging from 2 to 4. To maintain high passing efficiency and safety level, the ego vehicle may conduct complicated behaviors such as overtaking preceding slow SV and yielding to following quick SV.

Similar to the collection of D2E dataset, in the validation phase we recruit 10 third-party volunteers with diverse distributions to give a subjective evaluation score on each of the driving events and analyze whether the results derived from S2O aligns with human feelings. The results are shown in Tabel. 8, where $S_{h}$ denotes the average human subjective evaluation score, $S_{S2O}$ denotes the score caluated from the proposed S2O model, $MAE$ denotes the deviation from $S_{S2O}$ to $S_{h}$, $R_{h}$ denotes the rank of the algorithm according to human evaluation, and $R_{S2O}$ denotes the rank according to S2O.

According to the validation results, the proposed S2O model can evaluate autonomous driving decision-making algorithms precisely, with a mean absolute error of 3.77 under a percentage scale. Note that the MAE in the validation phase is less than the results on D2E dataset, because the tested AD algorithms all ensure a basic acceptable performance and scores of most cases range from 75 to 90. As discussed in Section 4.4, S2O has better evaluation precision on regular cases than extreme ones. From the perspective of ranking, the average ranking error is 0.50 compared with human subjective results. Although S2O evaluate most of the cases accurately, the evaluation of Scene 8 suffers from relatively large MAE. The reason is that Scene 8 is a complicated situation where all three fast vehicle from behind try to overtake the ego vehicle, which is not common in daily driving experience. Therefore, various raters have difference comments on the driving performance. This phenomenon demonstrates that performance evaluation of extremely complex scenarios is still a challenge for both human and objective models. Nevertheless, overall S2O can effectively evaluate AD algorithm performances in a real-time and comprehensive view.