Pedestrian Trajectory Forecasting Using Deep Ensembles Under Sensing Uncertainty

In this section, we present an approach for estimating the state and covariance of a Bayes filter using a neural network (NN). Bayes filters are used to account for the uncertainty during sensing and localization with popular filters such as the KF, Extended Kalman filter, and Particle filter. These filters estimate the states of a system as well as their associated beliefs, which are updated by fusing information from raw sensor measurements. The belief, which characterizes the state uncertainty, can have either uniform variance, known as homoskedastic, or heteroscedastic noise. Capturing this upstream uncertainty during sensing data is critical for subsequent robust predictions. Therefore, the current objective is to design a NN model that can learn the heteroscedastic measurement covariance of a Bayes filter. We approach the problem of state estimation as a 2D object tracking problem, using the KF.

The KF module consists of two steps; prediction and update step (Figure 1). The prediction step takes the previous state $\mathbf{X_{k-1}}$ and covariance $\mathbf{P_{k-1}}$ and computes the prior distribution based on the constant velocity motion model (1).

Here, $\mathbf{F}$ is the state transition matrix, and $u_{k}$ is the control input. For the tracking problem, the control input has no significance. Further, $\mathbf{Q}$ represents the process noise covariance.

We model the innovation, $z_{k}$ based on the difference between actual and predicted measurement of the state

which has covariance,

It is the sum of measurement noise covariance and predicted state covariance, $\mathbf{\bar{P}_{k}}$. R represents the covariance matrix associated with measurement noise.

The state, $\mathbf{{X}_{k}}$ and associated covariance, ${\mathbf{P_{k}}}$ are updated at each step using the Kalman gain, $\mathbf{K_{k}}$ which resembles a weighting factor between predicted (prior) state and actual measurement (likelihood) of the state.

The posterior states, $\mathbf{{X}_{k}},\mathbf{P_{k}}$ (2) are updated recursively at each step by fusing the actual noisy measurement from sensors and predicted measurement using the motion model.

Usually, trajectories can be randomly sampled from the posterior distribution of states obtained using the KF. However, random sampling generate non-smooth and infeasible trajectories especially with high sensor noise. To address this in-feasibility, we propose conditional trajectory sampling (CTS) where we propagate an initial sampled point using a dynamics model [24] and then resample a new point from the adjacent posterior distribution. This process is repeated recursively till a trajectory is generated (Figure 1). In this way, the generated trajectories adhere to system dynamics and are not random.

The CTS technique is used to sample from the posterior distribution of the KF state covariance at each time step. We randomly sample the initial state as $x_{0}$ and then propagate the initial state based on some motion model as a Markovian process $x_{t}\sim P(x_{t}|x_{t-1},a)$ [25]. The prior distribution, $P(x_{t-1})$ is propagated based on some action, a. This transition predicts the the likelihood of next state based on the constant-velocity motion model. The Process is repeated recursively based on the Markovian dynamics to generate a trajectory. Further, we repeat the trajectory generation process according to a particular bootstrap, i $\in$ {1,…,M}, where M represents the number of trajectories. The ensemble of generated trajectories roughly represent the state uncertainty. For the current research, trajectory sampling can be considered as a data augmentation step for the training of the neural network to capture this perceptual uncertainty. Another benefit of CTS is that each sampled trajectory within the distribution can be fed into the independent NN of an ensemble model to capture total predictive uncertainty. Details of our implementation are discussed in sec.II-C and Figure 2.

We usually treat trajectory prediction as a regression problem. For regression problems with deterministic predictions, the NNs output a single value say $\mu(x)$. It is estimated by minimizing the mean squared error on the training set, MSE = $\sum_{n=1}^{N}(y_{n}-\mu({x_{n}}))^{2}$. However, the outputs, $y_{n}$ are point estimates and do not contain any information on associated uncertainty. To capture the uncertainty, we assume the outputs are sampled from a Gaussian distribution such that the final layer outputs two values, predicted mean, $\mu({x})$ and variance, $\sigma^{2}(x)$ of the distribution. The variance, $\sigma^{2}(x)$ of a NN model can be obtained using the Gaussian negative log-likelihood loss (NLL) on training samples with input $x_{n}$ and output $y_{n}$ as:

$\sigma(x)$ represents the model’s noise observation parameter - showing the amount of noise present in the model’s outputs. However, the standard NLL strongly depends on predictive variance, $\sigma(x)$ and scales down the gradient for ill-predicted points [11]. Hence, an alternative loss function called as the $\beta$-exponentiated negative log-likelihood loss ($\beta$-NLL) [23] has been used to minimize loss.

$\beta$ controls the dependency of gradients on predictive variance while stop() is the stop gradient operation that prevents the gradients from flowing. $\beta$ = 0 represents the standard NLL loss. Meanwhile, $\beta$ =1 completely removes the dependency of gradients on variance, $\sigma(x)$ and treats the loss function as standard mean-squared error (MSE). The $\beta$-NLL loss function allows us the flexibility to switch between NLL and MSE loss function.

Total predictive variance (7) can be disentangled into aleatoric uncertainty, associated with the inherent noise of the data, and epistemic uncertainty accounting for uncertainty in model predictions[11].

Equation (9) shows that across multiple output samples, the mean of variances represents aleatoric uncertainty, while the variance of mean represents the epistemic uncertainty. The predictive variance, $\sigma_{i}^{2}(x)$ is obtained using the Gaussian NLL loss (4). However, predictive uncertainty only accounts for the data and model uncertainty during future trajectory prediction and does not have the information of upstream perceptual uncertainty obtained using KF. In order to capture the perceptual uncertainty, the NN is trained on augmented trajectory samples that takes both state and associated covariance as inputs. The perceptual uncertainty is then estimated by minimising the MSE loss between actual covariance obtained using KF with the predicted covariance from NN. Details of the method and results have been shown in section IV-C.

Initially, we trained our model on the ETH dataset only which contains approximately 420 pedestrian trajectories under varied crowd settings. However, a small number of trajectories is insufficient for training. Therefore, we performed data augmentation using Taken’s Embedding theorem [31]. We used a sliding window of T = 1 step to generate multiple small trajectories out of a single large trajectory. For instance, a pedestrian’s trajectory of 29 steps will result in 10 small $\{x,y\}$ trajectory pairs of 20 steps each if past trajectory information of 8 steps is used for predicting 12 steps into future. In total, we constructed 1597 multivariate time series sequences which we split into 1260 training and 337 testing sequences for the ETH hotel dataset. Further, each trajectory was augmented using KF to generate posterior state and covariance distribution. Then, M trajectories were sampled from the distribution for each original trajectory. Details of data augmentation using KF and TS have been discussed in sec.II.

The encoder-decoder neural network was trained end-to-end using PyTorch. Adam optimizer with a learning rate of $1e-3$ was used to compute the MSE and NLL loss. MSE loss was minimized to estimate the covariance of KF while NLL loss was minimized to capture the predictive uncertainty. Each model was trained for 150 epochs with a batch size of 64. For the ensemble model, M=3 networks were considered while for the MC dropout, a single model with dropout probability, p = 0.5 was used based on our previous research [4]. The model was compiled and fit using train data and test data was used for predictions.

The trained model is then used to predict the distribution for pedestrian future states. Overall, the predictions are averaged to generate the mean predicted path along with the associated variance that quantifies uncertainty. We adopt the widely used performance metrics [27] namely average displacement error (ADE) and final displacement error (FDE) for prediction comparison between the ground truth and mean predicted path. Further, we define valid prediction intervals for regression problems based on performance metrics like prediction interval coverage probability (PICP) and mean prediction interval width (MPIW) [26].

(a) Prediction Interval Coverage Probability (PICP): Coverage probability for a single state shows whether the ground truth state, $\mathbf{y_{k}}$ lies within the predicted covariance ellipse $\Gamma(X_{k})$ for the state $\mathbf{X_{k}}$,

$\mathbbm{1}$ denotes an indicator function representing Boolean values.

(b) Mean Prediction Interval Width (MPIW): It refers to the average width of the confidence interval. For the current results, we consider MPIW as the average of the major and minor axes of the predicted covariance ellipse.

$u(x)$ and $l(x)$ refer to the lower and upper bounds for the prediction interval.

(c) Average Displacement Error (ADE): Mean Euclidean distance between predicted and ground truth.

(d) Final Displacement Error (FDE): Euclidean distance between the predicted and true final state across all trajectories.

where $\mathbf{\hat{y}}_{t}$ is the predicted location at timestamp t and $\mathbf{y}_{t}$ is the ground truth position.

For quantifying uncertainty, deep ensembles average predictions over an ensemble of independently trained networks. In the current scenario, each network is trained using the Gaussian negative log-likelihood (NLL) (4) loss function such that the network outputs probabilistic predictions with both mean ($\mu$) and variance ($\sigma^{2}$). As, a single network can generate probabilistic outputs when trained with NLL loss function, why consider averaging the predictions over an ensemble of M networks? To answer this question, we observe how the NLL and test MSE loss scale with the number of independent networks (M) within an ensemble. The losses were evaluated on the ETH [29] dataset for pedestrians.

Each neural network within the ensemble was randomly initialized at the beginning of the training. Additionally, for each network, a training trajectory was randomly sampled as input from the distribution of trajectories. Training was performed and the final NLL loss was averaged over the number of networks, M within the ensemble. Test MSE was evaluated on a set of test trajectories different from the training data after the model was fully trained (Table II). The results indicate an ensemble of five networks had the lowest NLL as well as test MSE. Indeed, this shows an ensemble network because of lower NLL captures better predictive uncertainty. Further, low test MSE shows predictions of an ensemble network are closer to ground truth as compared to a single network. Overall, any ensemble of networks with $M>2$ produced better NLL and MSE as compared to a single network.

In this section, we compare the predictive uncertainty of a single network with an ensemble of three networks (M=3) on a pedestrian trajectory from the ETH dataset. Figure 3 shows the predictive uncertainty. The model takes 8 input states ( $\bullet$, green dot) to predict 12 states into future. $\blacktriangle$ represents the actual ground truth trajectory of the pedestrian. Further, the plot shows the mean predicted path ( $\blacklozenge$, blue diamond) alongwith the $1\sigma$ covariance ellipse to quantify uncertainty during prediction.

Figure 3b shows the predictive uncertainty for a single network, which fails to generate accurate prediction interval with respect to the ground truth. A significant portion of the ground truth trajectory lies outside of the $1\sigma$ predictive covariance. On the other hand, the ensemble network (Figure 3a) produced better predictive uncertainty and mean path by averaging the mean and variance of predictions over an ensemble of networks. The plot shows that the ground truth completely lies within the confidence interval at each time step. Further, the ADE/FDE for the ensemble network (0.618/1.137) was significantly lower compared to the ADE/FDE for a single network (0.704/1.394).

Figure 3a shows the total predictive uncertainty, which is due to the combination of aleatoric and epistemic uncertainty. The aleatoric uncertainty represents the inherent noise in the data, while the epistemic uncertainty arises due to the variation in model predictions. Since, the test data is sampled from the same distribution as the train data, the model uncertainty highlighted in yellow, is negligible compared to the aleatoric uncertainty. In contrast, a single network has no model uncertainty, and the total predictive uncertainty and aleatoric uncertainty are the same. Thus, the ensemble network is better suited to handle epistemic uncertainty, which is critical for robust real-world applications.

Further, we compare the performance metrics, coverage probability PICP (10) and prediction interval width MPIW (11) for different ensemble networks with the ETH dataset (Figure 4). Results show that the ensemble network with M = $\{3,5\}$ networks provide better predictive uncertainty estimates than a single network, with an average coverage probability of $\approx 63\%$ compared to $45\%$. In addition, current results also reveal that the MPIW for an ensemble model with multiple networks is either comparable or less than that of a single network, indicating that even with a smaller confidence interval, the ensemble model can achieve a higher coverage probability for the predictions. We denote the average width of major and minor axes as $\mathrm{MPIW_{x}}$ and $\mathrm{MPIW_{y}}$ respectively.

Previous studies have focused on capturing predictive uncertainty while neglecting upstream state uncertainty during perception. Incorporating perception or state uncertainty into the prediction pipeline remains a challenge, as it is unclear how the total predictive uncertainty will be affected. To address this challenge, we propose incorporating and propagating state uncertainty by including the associated state covariance, $\bar{P_{k}}$, obtained at each step from the KF. We append the heteroskedastic noise associated with each state to the state, $\mathbf{X_{k}}$, and train them together. In section IV-B, the state $\mathbf{X_{k}}$ = $[x,y]$ contained only respective states sampled from the posterior distribution of state covariance using KF. Here, we have neglected the velocity, [u,v] in the states for training as no significant improvement was observed with their inclusion. In the current scenario, we append the states and the associated covariance together as $\mathbf{[X_{k},\Sigma_{k}]^{T}}$ and train them jointly. The outputs are $\mathbf{[\hat{y}_{k},\hat{\Sigma}_{k}^{s},\hat{\Sigma}_{k}^{p}]^{T}}$. $\mathbf{\hat{\Sigma}_{k}^{s}}$ corresponds to the estimated state covariance which is trained by minimizing the MSE loss with the actual covariance, $\mathbf{\Sigma_{k}}$ obtained using KF. This enables the NN capture the upstream perceptual uncertainty. Meanwhile, $\mathbf{\hat{\Sigma}_{k}^{p}}$ corresponds to the estimated predictive covariance by the NN and was obtained by minimising the NLL loss for the state $\mathbf{X_{k}}$. Overall, our method enables us to estimate the state uncertainty and incorporate it into the prediction pipeline, which can help improve the total uncertainty estimation.

In this section, we compare the prediction efficacy of Deep Ensemble with Monte Carlo (MC) dropout which is an approximate Bayesian inference method. The MC dropout leverages on the idea of training the NN using dropout layers and then performing inference at test time by randomly dropping weights. This generates a distribution of varying outputs instead of a single deterministic prediction. Like ensembles, the mean and variance of the output distribution can be computed to obtain the mean predicted path and quantify uncertainty. Details of the method has been described in section II-C.

We show the the uncertainty plots for three test trajectories from the ZARA01 dataset, at $\mathbf{R}=5\%$ comparing both the methods (Figure 9). All trajectories start from origin. For deep ensemble, we consider M=3 networks while a dropout probability, p = 0.5 has been used on a single network for training using MC dropout method.

$\mathbf{Left:}$ The plot compares the $2\sigma$ perception uncertainty between the NN and KF for each method. Both the methods are slightly under confident in predictions and overestimate the uncertainty bounds when compared with the KF ground truth. This may arise due to the simultaneous training using the NLL and MSE loss function. The NN fails to minimise the MSE loss function accurately while estimating covariance. However, the deep ensemble model slightly outperforms the MC dropout in estimating the perception uncertainty at each state. The MC dropout model overestimates the covariance associated with initial states.

$\mathbf{Center:}$ The predictive uncertainty plot shows the $1\sigma$ distribution for future states with uncertainty bound for each method. The Predictive uncertainty can be disentangled into epistemic and aleatoric uncertainty. The uncertainty estimation is scalable and no significant difference was observed between the predictions of each models. However, the ADE and FDE for the mean predicted path of deep ensemble model (0.53/0.97) is closer to the ground truth when compared to the dropout model (0.58/1.00) as seen in Table III.

$\mathbf{Right:}$ The top and bottom right plots show the $1\sigma$ total uncertainty for deep ensemble and MC dropout respectively. As discussed, the total uncertainty is the Minkowski addition of the covariance ellipses representing the perception and prediction uncertainty. Since, the MC dropout overestimates the perception uncertainty, it affects the total uncertainty too. The MC dropout method makes under-confident predictions for total uncertainty. This phenomenon is more pronounced for the two trajectories along negative y-axis when compared to scalable predictions from deep ensembles.

Apart from uncertainty quantification, the current study also compared the performance metrics (sec. III-C) for both methods across the pedestrian datasets ETH[29] and UCY[30]. The ADE/FDE results indicate that the deep ensembles have a closer mean predicted path to ground truth compared to MC dropout across all the datasets. Meanwhile, the $1\sigma$ coverage probability results show deep ensembles have slightly higher coverage probability although not significant except for ETH dataset. We have combined the prediction interval width across major, $\mathrm{MPIW_{x}}$ and minor, $\mathrm{MPIW_{y}}$ axes of predicted covariance ellipse to obtain the mean prediction interval width,

Again, the deep ensembles have a lower MPIW compared to MC dropout which shows the deep ensembles are able to achieve slightly higher or equal coverage probability even with less prediction interval width. This shows that ensembles make robust predictions with scalable uncertainty and estimations closer to ground truth.

The current simulation results showed that the NN based end-to-end estimator yield good performance for prediction on trajectories which follow same distribution as the training data. However, one fundamental challenge for NN based prediction model has been out-of-distribution (OOD) robustness. Especially, if the test samples are based on real-world pedestrian trajectory with distributional shift, how effectively the trained NN model can predict the future state as well as the prediction and sensing uncertainty? To test this hypothesis, we studied multiple scenarios namely walking fast, walking slow, turning left, turning right, walking normal (Figure 11) which constitutes a set of edge case scenarios which are different from the training samples.