Learning to Evaluate Perception Models Using Planner-Centric Metrics

We wish to measure how the future state of a multi-agent system operating under some dynamics changes due to a noisy agent, which is our self-driving car. For the purpose of measuring perception performance, we consider that the noise in our agent comes only from noisy perception. Let $x_{t}^{i}$ denote the position of agent $i\in\{1\ldots N\}$ at time $t$ and $o_{t}^{i}$ denote the observation (coming from perception) for agent $i$ at time $t$. We will denote the perfect perceptual observation as $o_{t}^{i*}$. The joint probability of the “ideal” system state over a time horizon of $T$ time steps starting from $t=1$ is,

$\displaystyle P=p(x^{1}_{1}\ldots x^{N}_{T}|o^{*1}_{1}\ldots o^{*N}_{T})$

(1)

Without loss of generality, we will consider the first agent to be our noisy agent. The metric we want to compute is the change in this distribution given noisy observations from our agent, which can be measured using the KL Divergence [9] as follows,

	$\displaystyle D_{KL}(P\;||\;Q)$		(2)
	$\displaystyle P=p(x^{1}_{1}\ldots x^{N}_{T}|o^{*1}_{1}\ldots o^{*N}_{T})$
	$\displaystyle Q=p(x^{1}_{1}\ldots x^{N}_{T}|o^{1}_{1}\ldots o^{1}_{T},o^{*2}_{1}\ldots o^{*N}_{T})$

To measure the perception performance at $t=1$, we first assume that all agents make future predictions given observations only at $t=1$. We discuss this assumption in detail at the end of this section. The joint probability can then be written as,

$\displaystyle P=p(x^{1}_{1}\ldots x^{N}_{T}|o^{*1}_{1}\ldots o^{*N}_{1})$

(3)

Since the agents do not get any new observations in this time horizon of $T$ steps, they can only act independently of each other (since their future states are not observable to each other). The joint probability then becomes a product of the marginal distributions over the future of every agent,

$\displaystyle P=\prod_{i=1}^{N}p(x^{i}_{1}\ldots x^{i}_{T}|o^{*i}_{1})$

(4)

Finally, we assume that the system moves independently at each time step, given its observations. This amounts to factorizing the joint probability as,

$\displaystyle P=\prod_{t=1}^{T}\prod_{i=1}^{N}p(x^{i}_{t}|o^{*i}_{1})$

(5)

Under these assumptions, the joint distribution Q under noisy observations from our agent factorizes as,

$\displaystyle Q=\prod_{t=1}^{T}p(x^{1}_{t}|o^{1}_{1})\prod_{i=2}^{N}p(x^{i}_{t}|o^{*i}_{1})$

(6)

Substituting these in the KL divergence, we get,

	$\displaystyle D_{KL}(P\;||\;Q)$
	$\displaystyle=\mathbb{E}_{P}\bigg{[}\log\frac{\prod_{t=1}^{T}\prod_{i=1}^{N}p(x^{i}_{t}|o^{*i}_{1})}{\prod_{t=1}^{T}p(x^{1}_{t}|o^{1}_{1})\prod_{i=2}^{N}p(x^{i}_{t}|o^{*i}_{1})}\bigg{]}$		(7)
	$\displaystyle=\mathbb{E}_{P}\bigg{[}\log\frac{\prod_{t=1}^{T}p(x^{1}_{t}|o^{*1}_{1})}{\prod_{t=1}^{T}p(x^{1}_{t}|o^{1}_{1})}\bigg{]}$		(8)
	$\displaystyle=D_{KL}(P^{1}\;||\;Q^{1})$		(9)

where, $P^{1}$, $Q^{1}$ represent the marginal distribution over the future states of our agent, given perfect and noisy perception, respectively. In practice, these assumptions make computing the metric tractable, since we can train a parametric model of possible future states of an agent $p_{\theta}(x_{t}|\mathbf{o})$. The specific instantiation of state $x_{t}$, observations $\mathbf{o}$, model $p_{\theta}$ and its training is presented in Section 3.2.

Discussion on assumptions: To obtain a tractable estimate of the metric, and to measure the performance of perception at a particular time $t$, we assumed that predictions over a time horizon $T$ from $t$ are made given only the initial observation at $t$, and that every agent acts independently of each other and at every time step in this time. The first assumption is the most important and entails that all agents in the scene are not “reactive” in the time horizon specified by $T$. This enables us to measure how well perception till (or at) a current time step can help in driving with “anticipation”. Within a short time horizon $T$, this is indeed intuitive, since perfect perception should result in a best-case scenario for anticipatory driving. This is reflected in the PKL metric, which is zero when $o_{1}^{*1}=o_{1}^{1}$. Moreover, imperfect perception in irrelevant parts of a scene, such as in a nearby parking lot will not affect the metric since it does not affect how the whole system would have progressed in time. The second assumption follows from the first, since given no new sensory information, the agents can only act independently of each other. The last assumption is not necessary to our derivation, but is used in our particular implementation – where we model the marginal likelihood of an agent’s location at every time step within the time horizon $T$ independently, as explained in Section 3.2.

Let $s_{1},...,s_{t}\in S$ be a sequence of raw sensor observations, $o^{*}_{1},...,o_{t}^{*}\in O$ be the corresponding sequence of ground truth object detections, and $x_{1},...,x_{t}$ be the corresponding sequence of poses of the ego vehicle. Let $A:S\rightarrow O$ be an object detector that predicts $o_{t}$ conditioned on $s_{t}$. We define the PKL at time $t$ as

	$\displaystyle\mathrm{PKL}$	$\displaystyle(A)$		(10)
		$\displaystyle=\sum_{0<\Delta\leq T}D_{KL}(p_{\theta}(x_{t+\Delta}|o^{*}_{\leq t_{0}})\;||\;p_{\theta}(x_{t+\Delta}|A(s_{\leq t_{0}})))$

where $p_{\theta}(x_{t}|o_{\leq t})$ models the distribution of ground truth trajectories in the dataset $D$,

$\displaystyle\theta=\operatorname*{arg\,min}_{\theta^{\prime}}\sum_{x_{t}\in D}-\log p_{\theta^{\prime}}(x_{t}|o^{*}_{\leq t}).$

(11)

Intuitively, the PKL is a way to measure how similar a set of detections in a scene are from the ground truth detections. It does so by measuring how differently the ego car would plan if it only saw the predicted objects versus seeing the actual objects in the scene.

We model the marginal likelihoods of future positions with a similar approach to other end-to-end planning architectures [31, 3]. We discretize the grid -17.0 meters behind the ego to 60.0 meters in front of the ego and $\pm 38.5$ on either side into voxels of size 0.3 meters by 0.3 meters. We form the input $x\in\mathbb{R}^{8\times X\times Y}$ by binarizing the 3 map layers “ped_crossing”, “walkway”, and “carpark_area” and concatenating with binarized birds-eye-view projections of the detections for $t\in\{t_{0}-2.0,t_{0}-1.5,t_{0}-1.0,t_{0}-0.5,t_{0}\}$ where all coordinates are transformed to the frame of the ego car from time $t_{0}$. To form the target, we discretize the ground truth trajectory of the ego for timesteps $\{t_{0}+0.25i\mid 0<i<16,i\in\mathbb{N}\}$ and train with cross entropy loss as is standard for segmentation. We train using all non-zero trajectories of all annotated cars in nuScenes training set (1,216,412 trajectories) with batch size 16 for 100k steps using Adam [15, 8] with learning rate 2e-3 and weight decay 1e-5. We validate only on ego trajectories from the validation set (4,135 trajectories). To find the PKL over the full dataset, we average over all 2 second chunks. Note this is one possible instantiation of a neural planner, and other parametrizations and designs are possible. Our key contribution is in exploiting (neural) planner in evaluating perception models.

We present a short analysis on the planner’s performance w.r.t. different training hyperparameters in Tab. 1. Due to class imbalance, we find that weighting positive examples and clipping the loss function [33] provides accuracy boosts. Although we report accuracies exclusively on ego vehicle drives from the validation set, we find that training on the trajectories of all annotated vehicles in the dataset results in the largest boost to performance. We measure Top $k$ accuracy by calculating the Top $k$ locations in each heat map $p(x_{t+\delta}|o)$ and averaging over $\delta$ and $t$. The more accurately the planner is able to approximate the distribution of feasible future trajectories, the better the ranking induced by the planner will be.

We qualitatively demonstrate our planner in Figure 5. We sample frames from the validation set and visualize the planner’s predictions for all vehicles that have existed for longer than 2 seconds in the current frame. More examples can be found on the project page.

The nuScenes object detection benchmark uses a heavily engineered evaluation metric, called the nuScenes Dataset Score (NDS) to rank object detectors [5]. NDS is defined as:

	NDS		(12)
	$\displaystyle=\frac{1}{2}\left[\text{mAP}+\frac{1}{|TP|}\sum_{mTP\in TP}(1-\min(1,mTP))\right]$

where $TP$ is a collection of “true positive” error functions that are only measured on detections that are matched with a ground truth detection. NDS is designed to penalize false positives, false negatives, orientation error and translation error for all ground truth boxes within a distance $d_{k}$ to the ego car for each class of object $k$. This behavior is chosen because it aligns well with human intuition on what is important to perceive in order to drive safely. We show that our metric is also sensitive to these errors. More importantly, in our metric, these properties emerge because the planner implicitly learns that these variables are strong signals for predicting the distribution of future trajectories.

Results are shown in Fig. 4. We show that our metric possesses these properties by evaluating NDS and PKL on detectors with synthetic noise. To test translation error, we add gaussian noise to the center coordinate of every ground truth box. To test orientation error, we add gaussian noise to the 2D heading of every ground truth box. To test size noise, we add gaussian noise to the width, length, and height of every box. To test response to false negatives, we drop every detection with some fixed probability $p$. To test response to false positives, we add $N$ boxes of random size, orientation, and location into the scene at each timestep.

We see that for all noise models, NDS and PKL decrease with more error. Interestingly, PKL penalizes orientation more strongly than NDS. In the recently released Waymo Open Dataset [2], a new metric named “Mean average precision weighted by heading” or mAPH was proposed. mAPH is designed to weigh heading more heavily than the size, center of the bounding box because future prediction is generally more sensitive to the heading. We find it compelling that our metric implictly learns this weighting.

While NDS and mAP are guaranteed to agree with intuition about the importance of detecting objects accurately, they do not condition on a specific scene to determine how important each detection is in context. For instance, an object detector that always predicts a false positive directly in front of the ego vehicle receives roughly the same score under mAP as a detector that predicts a false positive 30 meters behind it. If the downstream task for the detector is unknown, it is difficult to justify weighing certain detections more than others.

In Fig 3, we show that our metric learns to take these factors into account. In the first row of Fig. 3, for each scene, we remove 5 vehicles with distance in the $p$ percentile of distances among all objects in the scene. As a result, in each trial, we get roughly the same number of false negatives, but the distribution of distances of removed cars to the ego car decreases with increasing $p$. For the second row, we rank the cars by speed in the global frame instead. In this case, the distribution of speeds of removed vehicles increases with increasing $p$. Unlike the noise models visualized in Fig. 4, these noise models are deterministic so we do not display error bars. Our metric penalizes missed detections closer to the vehicle as well as missed detections that are moving at high speed. However, the NDS score stays roughly the same in this experiment. The behaviour in PKL strongly correlates with intuition, where these detections would be considered critical to safe driving.

To gain insight into what the different metrics penalize, we rank the scenes in the dataset according to the performance of the state-of-the-art 3D object detector, MEGVII [35]. While ranking under PKL comes naturally given that the PKL is the expectation of KL over all scenes, NDS is not written as an expectation and therefore needs to be adapted. We adapt the NDS by calculating average precision (AP) only for classes that have ground truth boxes in each local chunk of a scene. Fig. 6 shows that this temporally local version of NDS is a well-behaved approximation of the global NDS.

Fig. 1 shows the time chunk on which the published MEGVII detections perform worst under the PKL metric. In the scene, a false positive appears right in front of the ego vehicle, giving the appearance that the truck in front of the ego is moving backwards. As a result, the planner expects the ego vehicle to stop instead of continuing forward, resulting in a huge penalty under the PKL metric.

Figure 9 shows the time chunk on which MEGVII performs best under PKL. In the time series, the detection of the car to the left of the ego is stable. There are several false positive humans detected in the scene, but these detections are irrelevant to the task of waiting at the light, which is why the scene still performs well. We recognize that for some downstream tasks, such as autonomous taxis, accurately detecting the humans on the sidewalk is a crucial subtask. Our goal is not to advocate for the sole adoption of PKL to evaluate object detectors but to propose PKL as an alternative to task-agnostic metrics that do not account for the context in which perceptual mistakes are made.

We submit a survey to the Amazon Mechanical Turk service asking humans to decide if one set of noisy detections is more dangerous than another set of noisy detections in a certain scene. Instructions provided to the workers are shown in Fig 11. We name the car “Herbie” to encourage workers to empathize with the car. We choose the scenes and noise such that NDS and PKL disagree on which scene has noise that is more dangerous. Maintaining the same scene for a given pair forces workers to differentiate between the two options based purely on the behavior of the detections as opposed to differences in the complexity of the scenes. Noise is added to the system to differentiate between metrics based on how they couple across error functions; we generate noisy detections by sampling translation noise with $\sigma=0.1$m, orientation noise with $\sigma=4^{\circ}$, size noise with $\sigma=0.1$m, missed detection with probability $p=0.05$, and exactly 1 false positive per frame.

While PKL is defined as the expectation of PKL for a single frame, there is no obvious way to obtain monte carlo estimates of mAP for single samples. In NDS, this problem is exacerbated by the fact that the mAP is normalized over classes which would mean that scenes with very few instances of a class would be unfairly penalized. We approximate mAP by evaluating mAP over a segment in time only for classes that have at least one ground truth box within that time segment. We visualize the histogram of these local NDS measurements in Figure 6 to verify that we can provide a competitive ranking under the local NDS metric.

We leave an optional comment box on the survey. Workers largely appear to pay attention to the correct mistakes made by the detectors. For instance, common comments include “failure to detect vehicle behind”, “The car to the left wasn’t detected but it’s off to the side”, and “something isn’t in Herbie’s path but it thinks something is”. However, it is not clear that all workers fully understand the task that they are being asked to enact. Other comments include “Herbie runs into an object”, “looks like it thought it had a collision”, “there looks to be a possible head on collision here”, suggesting that the concepts of false positive and false negative are not easily communicated through the survey to a naive crowd without technical expertise.