Attentional-GCNN: Adaptive Pedestrian Trajectory Prediction towards Generic Autonomous Vehicle Use Cases

Given observed pedestrian trajectories X from all time steps in period $t\leq T_{obs}$, where $\textbf{X}^{t}=[\textbf{X}_{1}^{t},\textbf{X}_{2}^{t}...,\textbf{X}_{N}^{t}]$ for $N$ pedestrians in a scene, our goal is to predict future trajectories $\hat{\textbf{Y}}^{t}=[\hat{\textbf{Y}}_{1}^{t},\hat{\textbf{Y}}_{2}^{t}...,\hat{\textbf{Y}}_{N}^{t}]$ over the prediction time period $T_{obs}<t\leq T_{pred}$.

The input position of the $i$th pedestrian at time $t$ is denoted as $\textbf{X}^{t}_{i}=(x^{t}_{i},y^{t}_{i})$, and in ground truth future trajectories Y as $\textbf{Y}^{t}_{i}=(x^{t}i,y^{t}_{i})$. During training, Y is compared to the predictive model’s output $\hat{\textbf{Y}}$. This output takes the form of a bi-variate Gaussian distribution $\hat{\textbf{Y}}_{i}^{t}\sim\mathcal{N}(\hat{\mu}^{t}_{i},\,\hat{\sigma}^{t}_{i},\hat{\rho}^{t}_{i})$ over the positional $x$ and $y$ dimensions in the probabilistic model version, or $\hat{\textbf{Y}}^{t}_{i}=(x^{t}i,y^{t}_{i})$ in the deterministic version.

Our proposed model consists of three parts: (1) a Near Pedestrian Attention Function (NPA); (2) the Spatio-Temporal Graph Convolution Neural Network (ST-GCNN); (3) and the Time-Extrapolator Convolution Neural Network (TXP-CNN), as shown in Fig. 2.

The model consists of 1 layer of ST-GCNN and 5 layers of TXP-CNN, where both temporal convolutions and graph convolutions use a kernel size of 3. The deterministic model and the probabilistic model both use PReLu activations [33] and are trained for 150 epochs with the batch size of 128. We use Adam optimiser [34] with learning rate at 0.0015 for training the deterministic model, however we use the SGD optimiser with learning rate at 0.01 in the training of the probabilistic model as this was seen to achieve better training results.

Data was captured from an Electric Vehicle (EV) platform similar to [35], as depicted in Fig 3. Sensing consists of three front NVIDIA 2Mega SF3322 automotive GMSL cameras and a 3D Velodyne Puck VLP-16 lidar. Lidar was captured at 10 Hz and RGB at 30Hz with a resolution of resolution of $1,928\times 1,208$. In testing, we downsample the raw camera data to 10 Hz to be consistent with the lidar sample rate. 2D cameras were intrinsically calibrated based on fish-eye camera model [36] before extrinsic calibration with lidar, performed automatically using 3D point and plane correspondences, proposed by Verma et al. [37].

The dataset was labelled using an automatic perception pipeline shown in Fig 3 based on the work proposed by [38]. This pipeline includes 2D object detection using YOLOv3 [39], a 3D lidar classification model based on [40] and [41], and a Deepsort visual tracker [42]. Each potential pedestrian image patch found by YOLOv3 is projected into the 3D pointcloud, associating it with a cluster of points that depicts the subject in the lidar pointcloud. This cluster is then voxelised and forwarded to a 3D CNN. The image patch is also passed through another CNN. Finally, the output of both convolutional pipelines are concatenated and passed to a fully connected layer that classifies the sample as either a true pedestrian or a false positive. We relied on this pipeline instead of a 2D-based one because the urban environment contains a range of objects that look like pedestrians but they are not pedestrians. For instance, we could find billboards, posters and digital displays depicting people. A 2D-based approach would eventually detect these objects as pedestrians and, thus, extracting a false positive. The original image patch is then encoded by secondary 2D CNN to extract features used by Deepsort, associating it in the 2D frame. For true pedestrians, the original image patch is then associated with prior tracks in the 2D frame using Deepsort [42], before the 3D coordinates of the pointcloud cluster are used to transform the track into the world frame. Linear interpolation is applied to allow trajectories to be continuous across short occlusions.

ETH [43] contains two scenes, ETH-Univ is collected from a University, and ETH-Hotel is collected in urban street. The dataset is converted to world coordinates with an observation frequency of 2.5 Hz.

UCY [24] includes UCY-Zara01, UCY-Zara01 and UCY-Hotel collected from an urban scene. The trajectories in the datasets are sampled every 0.4 seconds.

USyd Pedestrian Dataset is collected by a vehicle operating in a university campus as described in Sec. IV sampled at 10 Hz.

For both the ETH and UCY datasets we observe the last 8 timesteps (3.2 seconds) per trajectory and predict for the next 12 timesteps (4.8 seconds). For the Usyd Pedestrian dataset, as tracks tend to be of shorter length due to occlusion from the vehicles viewpoint, we only observe 4 timesteps (0.4s) per trajectory and predict for the next timesteps (1.2 seconds). We split all datasets into training, validation and testing sets, in ratios ${3}/{5}$, ${1}/{5}$ and ${1}/{5}$ respectively.

We additionally compare the parameter size and inference time of each of the predictive models, an important consideration for real-world applications of autonomous vehicles.

Probabilistic Predictions: Table II displays the comparison for all probabilistic methods on the ETH [43] and UCY [24] datasets. On average, our probabilistic model obtains better results than all other compared probabilistic models with $10\%$ improvement on ADE and $12\%$ improvement on FDE compared to the next best model Social-STGCN [14]. This result demonstrates that our proposed NPA module, which embeds relative distance based attention between pedestrians into the graph, improves performance greatly. Whilst our probabilistic model only achieves the best accuracy on 4 out of 10 individual metrics, it is clear from the average score that our approach is better able to generalise between datasets than other compared methods. Additionally, we see that both our proposed method and Social-STGCNN outperform the compared LSTM methods [8, 5, 10], demonstrating that GNN and CNN based methods can achieve significantly better results than LSTM based methods in pedestrian motion prediction tasks. It is clear from these results that the spatial-temporal graph neural network is able to successfully extract the features relevant to implicit interaction between pedestrians.

Table. III shows results from the Usyd pedestrian dataset. Our proposed probabilistic model and Social-STGCNN significantly outperform all other methods, including the probabilistic SocialGAN method, although Social-STGCNN achieves slightly better results in ADE and FDE. Due to the limited observation period of 4 timesteps, social interactions are not as prominent, and so our proposed NPA function was likely not able to better represent relationships. Additionally, significant noise exists compared to top-down datasets, as seen in the bottom right of Fig. 1. This is typical of real-world data collected from a vehicle, resulting from occlusions, sensor noise, and lidar firing cycles mismatching with 2D input. The noise is also a likely explanation for the minimal difference seen between ADE and FDE for our method and Social-STGCNN. The ground truth path contains significant noise however the predicted path directly leads to the final ground truth position, resulting in lower final but higher average error.

Deterministic Predictions: Table. LABEL:table_deterministic shows the results of testing the deterministic models on the ETH [43] and UCY [24] datasets, displaying both our deterministic model, and our probabilistic model that has been limited to outputting a single ‘most-likely’ prediction. We see that directly training our model in a deterministic fashion as opposed to using the ‘most-likely’ prediction of the probabilistic model results in increased prediction accuracy, an important consideration in time-critical tasks where only a single prediction without uncertainty is required. Whilst SR-LSTM outperforms our deterministic model in both ADE and FDE, we see in Table. IV that SR-LSTM inference takes significantly longer, limiting the applicability for use in time-critical tasks in autonomous driving systems. CVM also achieves good results, however as described in [4] some subsets of the data used, such as ETH-Hotel, appear to involve significantly fewer pedestrian interactions suggesting that linear models should perform well. This appears to be the case when we focus on examples of pedestrian interaction from the datasets, as described in Section VI-B and shown in Fig. 5.

Computation Speed Comparison: We bench-mark inference time for 8 observed and 12 prediction timesteps running on an Nvidia GTX 1080Ti GPU and Intel Xeon E5v4 CPU in Table IV. Our proposed method and Social-STGCNN achieve the fastest computation speed among all models with the least number of parameters, suggesting that GCNN-based methods not only outperform state of the art probabilistic RNN-based methods but also requires much less computational resources. Furthermore, our approach also outperforms the CPU implemented Kalman filter based CVM method. As per [14], we do not include graph computation time due to the use of an non-optimized CPU implementation, however this process takes approximately 22ms, resulting in speeds still comparable to the next fastest PCGAN method when included. Whilst our proposed method has the same size and speed as Social-STGCNN [14], our method achieves superior performance on ADE and FDE metrics as Table II shows.

Comparisons in interactive scenarios, shown in Fig. 5, confirms the robustness of our proposed method. We compare our proposed method with Social-STGCNN [14] to demonstrate the benefit of our proposed NPA function during pedestrian encounters, and also to CVM [44] to highlight instances when linear predictions fail.

Parallel Walking Generally, when a group of pedestrians are walking in parallel, they maintain similar walking speeds and direction. As it shows in Scenario 1 of Fig. 5 in which two pedestrians walk side by side, Social-STGCNN [14] falsely predicts the pedestrians will collide with each other while our approach predicts similar distributions for each pedestrian. In Scenario 3 and Scenario 4, Social-STGCNN [14] predicts the pedestrians will begin to diverge, whilst our method predicts future trajectory distributions with the same direction and speed. We can clearly see that CVM [44] fails in the situations when pedestrians change direction, or some noise exists, as in Scenario 3.

Collision Avoidance Scenarios 4 and 5 show situations when pedestrians are approaching another stationary (4) or moving (5) pedestrian. Our proposed model predicts that both the approaching pedestrians and the stationary pedestrian will change heading slightly to avoid a collision, while Social-STGCNN falsely predicts greater incorrect deviation by approaching pedestrians. Similarly, when a single individual interacts with a group, it is difficult to predict their response, as illustrated in scenarios 2, 4, and 5. In (4), the stationary pedestrian is likely to move aside to avoid an approaching group. Our proposed method correctly predicts this motion while Social-STGCNN predicts an incorrect direction. Likewise, in (5), both CVM and Social-STGCNN [14] predict a collision, while our predicted direction better follows the ground truth response.