Deep Learning and Statistical Models for Time-Critical Pedestrian Behaviour Prediction

Many practical applications can be represented as a sequence of various behaviours. These include detecting bull and bear financial markets [1], gesture recognition [2], animal behaviour recognition [3], and aircraft manoeuvres in military applications [4]. Detection time in such applications is often crucial. When data arrives sequentially, detection time becomes a problem of how many sequential samples the model requires to make an accurate prediction.

In this study, we consider the problem of pedestrian behaviour prediction or intent estimation. A review on the prediction of pedestrian behaviour in urban scenarios is presented in [5]. A large portion of the literature has been devoted to tracking and path prediction. Many studies use the SLDS as a framework [6], [7], [8], and [9]. Recently, the recurrent neural network (RNN) has been shown to be a promising approach [10, 11, 12, 13]. Owing to the significant advancement of the state-of-the-art in pedestrian detection [14, 15, 16, 17], we assume that the trajectories of the pedestrians are known in this study. Given the pedestrian trajectories, we predict a particular behavioural class.

There are studies have considered the problem of pedestrian behaviour prediction. Probabilistic models such as the latent dynamic conditional random field [18] and balanced Gaussian process dynamical models [19] have been applied. Various forms of the RNN have been also been considered. Hoy et. al. [20] propose a variational RNN which performs both tracking and behaviour prediction. Völz et. al. [21] compare neural networks, a support vector machine, and the LSTM. Though both statistical and machine learning models have been applied to the problem and some studies consider time-to-event analyses, to our knowledge, no specific analyses between these model types in terms of time-to-detection have been considered in the literature.

Our contribution is a comparison between a SLDS and a multi-layered bi-directional LSTM neural network in the context of time-to-detection. This is performed by classifying various pedestrian behaviours from the raw motion tracks under varying sequence lengths. Through the comparison, we gain novel insight into a key difference between the models: though the neural network is more accurate than the SLDS overall, it requires 10 times as many sequential samples to achieve this accuracy. The SLDS is able to provide its most accurate classification within the first few samples of the sequence. This result is important in situations where early detection is imperative.

The SLDS models a system that switches between various dynamical models. Each dynamical model is represented as a Linear Dynamic System (LDS) – which is widely associated with the Kalman filter. The SLDS has been extended in various ways, such as introducing variables representing behavioural context information. Such models have been applied to maritime piracy applications [22, 23] and abalone poaching applications [24]. Linderman et. al. [25] extend the SLDS by allowing the switching state to depend on the latent state and exogenous inputs through a logistic regression. The SLDS has also been extended to include multiple LDSs over several sequences under a single switching state [26, 27].

The joint distribution describing the SLDS is given by:

The switching state transition probability $p(s_{t}|s_{t-1})$ is a discrete distribution. It describes how the model switches between various states. The state transition distribution $p(h_{t}|h_{t-1},s_{t})$ and emission distribution $p(v_{t}|h_{t})$ are assumed to be Gaussian. These describe the dynamics of the system through the linear state space equations.

Inference in the SLDS involves inferring the latent variables $s_{t}$ and $h_{t}$ given the observations $v_{1:t}$. This is typically performed using filtering and smoothing methods. The filtering operation computes the filtered posterior $p(s_{t},h_{t}|v_{1:t})$. The smoothing operation computes the smoothed posterior $p(s_{t},h_{t}|v_{1:T})$. Exact inference in the SLDS is intractable [28, 29]. Approximate inference algorithms such as the Generalised Pseudo Bayesian (GPB) algorithm [30] and the Gaussian Sum Smoothing (GSS) algorithm [31] have been developed for the SLDS. In this study, the GPB algorithm is used.

Parameter learning in the SLDS can be performed using the Expectation Maximisation (EM) algorithm [30]. In the expectation step, a smoothing algorithm such as GPB can be used. In the maximisation step, the parameters are estimated using maximum likelihood.

The well-known Daimler Pedestrian Path Prediction Benchmark Dataset (GCPR’13) [6] is used in this study. The dataset comprises a collection of 68 pedestrian sequences with 4 different pedestrian behaviour types: crossing, stopping, starting to walk, and bending-in. Though the dataset seems relatively small, the LSTM has been shown to perform well in the path prediction application [11].

The dataset was acquired using stereo cameras. The stereo camera provides the means to produce three dimensional Cartesian coordinates for tracking purposes. The ground truth of the dataset provides bounding boxes, disparity, and $X$ and $Z$ coordinates of each target.

The dataset was developed to test recursive Bayesian filters for pedestrian path prediction for different behaviours [6]. In this study, this problem is inverted. The behaviour of a pedestrian is inferred from the tracked path.

The dataset is provided with a predefined training set and a test set. The training and test sets comprise 36 and 32 sequences respectively. The model parameters are estimated using the training dataset. The trained models are then applied to predict the behaviour class from the tracks provided in the test dataset. To measure the performance of the models, accuracy, precision and recall are used.

The models are tested on sequences of varying length. This is achieved by truncating the sequences in increments of 10 samples. That is, the models are tested on the first $10,20,30,\dots$ samples of each sequence in the test set. The classification results are stored for each sequence length. Limiting the number of timesteps provides an indication of how well the method is able to predict a behaviour class in a short period of time. Furthermore, it provides some form of consistency over the varying sequence lengths in the dataset.

The SLDS motion model is configured as a constant acceleration model. The tracked coordinates are provided as observations to the SLDS. The model parameters are learned using the EM algorithm. The switching state is defined to comprise the states BendingIn, Crossing, Starting, and Stopping. In the dataset, the pedestrians do not switch between behaviour classes over the sequences. The switching state transition distribution is set with a $0.97$ probability of remaining in the current switching state and a $0.01$ probability of transitioning to one of the other three switching states. The prior switching state probability distribution is set to the uniform distribution.

The RNN is configured with 32 hidden units in each LSTM cell. The ADAM algorithm [33] is used to minimise the cross entropy of the softmax outputs. The model is trained over 110 epochs with a learning rate of 0.0001 and a batch size of 1. The remaining ADAM parameters are set as recommended in [33]. The RNN is trained over the complete length of each sequence in the test set.

The accuracy over the set of truncated sequences is presented in Figure 3. The striking feature in the plot is that the RNN increases in accuracy with increasing sequence length, whereas the SLDS decreases in accuracy with increasing sequence length. The SLDS has the highest accuracy with a sequence of 10 samples. This implies that within the first 10 samples, the SLDS is able to classify the sequence. The RNN’s accuracy curve saturates around the 100 sample length mark. This indicates that the RNN requires a sequence of at least 100 samples to achieve the high accuracy.

These results are consistent with theoretical design of the models. The SLDS assumes a first order Markov model in both the dynamics and the switching state. A first order Markov model assumes that the current state is conditionally dependent only on the previous state. The result is that the SLDS is not designed to model long-term dependencies in the data. The SLDS thus performs better when provided with the shorter sequences. Furthermore, the SLDS performance decreases with sequence length as it is designed to switch between dynamics. It is more likely to switch behaviour class in a longer sequence. The LSTM cell in the RNN has been specifically designed to model both long and short-term dependencies in the data [32]. The result is that the RNN requires a longer sequence to achieve a higher accuracy. Another relevant difference between the models is that the SLDS is a structured model where the dynamics have been predefined. In the RNN, the dynamics are learned in a black-box approach, which often requires more data.

The precision and recall over the set of truncated sequences are presented in Figure 4 and Figure 5 respectively. Confusion matrices for the 10-sample-length and complete sequences are presented in Table 1. Precision is often viewed as a measure of the quality of the model. Recall describes the probability of correctly classifying the pedestrian behaviour.

As for the RNN accuracy, the precision and recall values are only high for sequences with 100 samples or more. The precision and recall for the SLDS are highest for sequences of 10 steps.

The RNN generally has a higher precision and recall than the SLDS. The RNN however struggles to correctly predict the starting behaviour class. Consider the confusion matrix for the complete sequence classification presented in Table 1. The majority of Starting samples are incorrectly associated with the BendingIn class. A possible reason for this is that the sequences for the Starting class are generally short in length. The poor results for the Starting class lowers the overall accuracy of the RNN.

The lowest recall is for the SLDS model is the BendingIn class, with a value of $63\%$. Considering the confusion matrix, the $30\%$ of the samples were misclassified as starting behaviour. The model performs well on the crossing and starting classes. For longer sequences, the precision and recall for the Stopping class decreases significantly. As also indicated in Figure 5, the recall for the Crossing and Starting classes remain fairly constant.

For the RNN with 10-sample sequences, $46\%$ of the BendingIn samples were incorrectly associated with the Crossing class as indicated in Table 1. When provided with the complete sequence, this reduces to $0\%$. Similarly, most of the Starting samples are incorrectly associated with the Crossing class with short sequences. When provided with the complete sequence, the incorrect classifications shift to the BendingIn class.

Figure 8 illustrates results for the Crossing behaviour class. The track in Figure 8b is approximately linear over the space. With such behaviour, both models generally perform well in this class.

An example of the Stopping behaviour class is presented in Figure 9. The SLDS correctly begins by classifying the stopping behaviour class and then transitions to the crossing class. This result corresponds to the complete sequence confusion matrix presented in Table 1. The RNN correctly classifies the stopping class for the entire sequence. This corresponds to the high recall for this class as illustrated in Figure 5.

In this study a SLDS and a three-layered bidirectional LSTM RNN are applied to predict pedestrian behaviour from motion tracks from the Daimler Pedestrian Path Prediction Benchmark Dataset (GCPR’13). The key result is that the RNN model’s accuracy increases with increasing sequence length, whereas the SLDS’s accuracy decreases with increasing sequence length. The best results for the SLDS are obtained when the first 10 samples of the sequence are provided to the model. This is possibly due to the SLDS being designed to model short-term behaviour as well as having a predefined model of the dynamics. The RNN is designed to model both short and long-term dynamics with a black-box approach. The result is that the RNN is more accurate, but over longer sequences (100 samples or more). This suggests that in situations where a decision is required to be made quickly, the SLDS may be the preferred model.

There is potential for improvement of the results for both models. One approach would be to include contextual information. This can be achieved in the SLDS using methods such as those described in [22, 23, 24]. Contextual information could include road signs, proximity to crossing areas, and traffic congestion levels. Additional information relating to the urban environment could also be influential. For example, a street may be residential or commercial.