Coupling Intent and Action for Pedestrian Crossing Behavior Prediction

Given 2-D RGB traffic video clips, visual features were first extracted from image patches around each target pedestrian using a pre-trained deep convolutional neural network (CNN) following Rasouli et al. (2019), as illustrated in the upper-left of Fig. 2. An image patch is a square region defined by an enlarged pedestrian bounding box (“bbox”) that includes a pedestrian as well as some context information such as ground, curb, crosswalk, etc. Meanwhile, the bounding box coordinates of the image patch were passed through a fully-connected (FC) network to obtain location-scale embedding features. Then, the intent-action encoder (ENC) cells took the concatenated feature vectors as input and recurrently encoded the features over time. To ensure the neural network learns features that represent both crossing intent and more complicated semantic actions, a multi-layer perceptron (MLP)-based intent classifier and an action classifier were designed to use the encoder hidden state to predict the present intent and action simultaneously.

Prior work such as the temporal recurrent neural network (TRN) Xu et al. (2019) have shown that using future action predictions can improve the performance of classifying an action in the present. Inspired by this, our action predictor module was designed to utilize features from future action predictions and pass this information to the next ENCcell. In the action predictor module, the hidden state at each iteration was classified by an MLP network to predict the action at each future time step. Then, the average of all decoder hidden vectors was computed and used as the “future” action feature. Such future action feature was concatenated with the visual and box features as described in the previous subsection to form the input to the next ENCcell.

The loss function of the proposed model consists of the binary cross entropy loss for the crossing/non-crossing intent detection, $H_{B}(\cdot)$, and the cross entropy loss for the multi-class action detection and prediction, $H(\cdot)$. The total loss can be written as

where $T$ is the total training sample length. Variables with and without $\hat{\cdot}$ indicate the predicted and ground truth values, respectively. The three loss terms refer to the intent detection loss for the current time step $t$, the action detection loss for the current time step $t$, and future action prediction loss for future time steps $t+1$ to $t+\delta$, respectively. The $\omega_{1}$, $\omega_{2}$ and $\omega_{3}$ are weighting parameters. In this work, the $\omega_{1}$ was designed to start with value $0$ and increase towards $1$ over the training process based on a sigmoid function of training iterations. This procedure ensures the model learns a reasonable action detection and prediction module, which can be used as a beneficial prior for the intent detector. For simplicity, $\omega_{2}$ and $\omega_{3}$ were both set to $1$ in training.

In the training stage, the network is updated in an online, supervised fashion, i.e., given observation $O_{1}$, the network detects the current intention $i_{1}$, current action $a_{1}$, and predicts future actions $[a_{2},a_{3},...,a_{6}]$ (if $\delta=5$, for example). Then, given the next observation $O_{2}$ and the encoded features from the previously predicted future actions $[a_{2},a_{3},...,a_{6}]$, we detect the intention $i_{2}$ and again predicts future actions $[a_{3},a_{4},...,a_{7}]$. This way, the predicted future actions of the pedestrian were leveraged and recurrently looped back as a prior to detect the present intent and action of that pedestrian, following our cognitive assumption that intent affects actions and actions in turn reflect the initial intent. This process goes on for all given training frames and the loss function was updated given the ground truth labels of intent and actions of training frames. In the testing stage, only image sequences were needed as input and the trained network generates the intent and action labels of all frames in the testing data.

$f_{ne}=[x^{j}_{1}-x^{i}_{1},y^{j}_{1}-y^{i}_{1},x^{j}_{2}-x^{i}_{2},y^{j}_{2}-y^{i}_{2}]$, which represents the bounding box differences between the observed traffic participants (e.g., vehicle, pedestrian, and cyclist) $j$ and the target pedestrian $i$.

$f_{tl}=[x^{j}_{c}-x^{i}_{c},y^{j}_{c}-y^{i}_{c},w^{j},h^{j},n_{l}^{j},r^{j}]$, where $x^{j}_{c}-x^{i}_{c}$ and $y^{j}_{c}-y^{i}_{c}$ are the coordinate differences between centers of traffic light $j$ and target pedestrian $i$, $w^{j},h^{j}$ are the traffic light bounding box width and height, $n_{l}^{j}$ is the light type (e.g., general or pedestrian light), and $r^{j}$ is the light status (e.g., green, yellow or red).

$f_{ts}=[x^{j}_{c}-x^{i}_{c},y^{j}_{c}-y^{i}_{c},w^{j},h^{j},n_{s}^{j}]$, where the first four elements are similar to the traffic light feature, $n_{s}^{j}$ is the traffic sign type (e.g., stop, crosswalk, etc.).

$f_{cw}=[x^{j}_{1}-x^{i}_{bc},x^{j}_{2}-x^{i}_{bc},y^{j}_{1}-y^{i}_{bc},x^{j}_{1},y^{j}_{1},x^{j}_{2},y^{j}_{2}]$, where $x^{j}_{1}-x^{i}_{bc},x^{j}_{2}-x^{i}_{bc},y^{j}_{1}-y^{i}_{bc}$ represent the coordinate differences between the bottom center of target pedestrian $i$ and the two bottom vertices of crosswalk $j$. We assume all pedestrians and crosswalks are on the same ground plane, thus, the bottom points better represent their depth in 3-D frame. $x^{j}_{1},y^{j}_{1},x^{j}_{2},y^{j}_{2}$ are the crosswalk bounding box coordinates to show the crosswalk location information.

$f_{st}=[x^{j}_{1}-x^{i}_{bc},x^{j}_{2}-x^{i}_{bc},y^{j}_{1}-y^{i}_{bc},x^{j}_{1},y^{j}_{1},x^{j}_{2},y^{j}_{2}]$, where the elements are similar to the crosswalk feature vector.

$f_{eg}=[v,a,v_{yaw},a_{yaw}]$, where $v$ and $a$ are the speed and acceleration, and $v_{yaw},a_{yaw}$ are the yaw rate and yaw acceleration, collected from on-board diagnostics (OBD) of the data collection vehicle (ego vehicle).

The traffic object-specific features were extracted and computed as described above and an attentive relation network was designed to extract the relational features of all objects for a target pedestrian, as illustrated in Fig. 3. Each feature vector was first embedded to the same size by a corresponding FC network (e.g., neighbor feature vector is embedded by $FC_{ne}$). For object types that may have multiple instances (e.g., there can be more than one cars in the scene), a soft attention network (SAN)  Chen et al. (2017) was adopted to fuse the features. The SAN predicts a weighing factor for each instance, which represents the importance of that instance to the desired task, and then use a weighted sum to merge all instances. The ego car feature is not processed by a SAN, since current work assumes there is only one ego car at any given time. Then, the extract features of each traffic object type were concatenated to form the final relation feature $f_{rel}=[f_{ne},f_{tl},f_{ts},f_{cw},f_{st},f_{eg}]$. Note that all elements of a feature vector were set to zeros if the corresponding object type was not observed in a scene. Finally, the relation features extracted through the ARN were concatenated with the visual, bounding box, and future action features as described in Section 3.1 to be passed together into the ENC network for intent and action encoding.

Compared to using visual features (e.g., semantic maps) to represent environmental information as used in prior work Rasouli et al. (2020a, b), our ARN directly uses pre-detected traffic object features, which is far more straightforward and informative. This is also inspired by human visual learning, as human drivers make decision based on observing objects rather than image pixels. The ARN is a very simple and fast network, since no deep CNNs were needed for visual feature extraction.

Pedestrian crossing intent detection is traditionally defined as a binary classification problem (cross/not cross) and we include previous benchmark evaluation metrics including accuracy, F1 score, precision, and the receiver operating characteristic (ROC) area under curve (AUC) Rasouli et al. (2020b). We introduce another metric in this work – the difference between the average scores $s$ of all positive samples $P$ and all negative samples $N$, $\Delta_{s}=\frac{1}{|P|}\sum_{i\in P}s_{i}-\frac{1}{|N|}\sum_{i\in N}s_{i}$. This $\Delta_{s}$ metric reflects the margin between the detected positive and negative labels. The greater the $\Delta_{s}$, the larger the margin, thus, the more easily (better) the model can separate the crossing (positive) and not crossing (negative) classes. Note that accuracy, F1 score and precision can be biased towards the majority class if the test datasets were imbalanced (which is the case with both datasets used). Besides, these three metrics were computed at a hard threshold at 0.5 (score$>$0.5 is classified as positive, otherwise negative). On the other hand, both AUC and $\Delta_{s}$ metrics are less sensitive to the data imbalance and can more accurately reflect the different performance comparisons between methods.

The proposed neural network model was implemented using PyTorch. We used gated recurrent units (GRU) with hidden size 128 for both ENCcell and DECcell, and a single-layer MLP as the classifier for intent detection, action detection and action prediction. The input image patch was prepared following Rasouli et al. (2019) to realize fair comparison, resulting in $224\times 224$ image size. An ImageNet-pre-trained VGG16 network acted as the backbone CNN to extract features from image patches Rasouli et al. (2019). During training, the pedestrian trajectories were segmented to training samples with constant segment length $T$ in order to make batch training feasible. We applied a weighted sampler during training to realize balanced mini-batch. All models were trained with sample length $T=30$, prediction horizon $\delta=5$, learning rate $1e-5$, and batch size $128$ on a NVIDIA Tesla V100 GPU. During inference, the trained model was run on one test sequence at a time and the per-frame detection results were collected.

We compared multiple variations of our method: The “I-only” model is a basic model without action detection and prediction modules, which is similar to PIE_int Rasouli et al. (2019) except that we used an average pooling feature extraction and a simpler GRU encoder father than the ConvLSTM network in PIE_int; The “I+A” model is a multi-task model with both intent and present action detection modules and the “I+A+F” adds future action prediction; The “I+A+F+R” or “full” is our full model with the attentive relation network. We also present an “A-only” and an “A+F” ablations to show that future action prediction improves present action detection.