A Real-Time Predictive Pedestrian Collision Warning Service for Cooperative Intelligent Transportation Systems Using 3D Pose Estimation

Cooperative-Intelligent Transportation Systems (C-ITS) collect and provide information in both directions between vehicles and road-side infra-structures [9]; in general, C-ITS utilizes on-board-unit (OBU) installed on vehicles and vision, radar and Lidar sensors on the road-side for information processing. C-ITS allows vehicles and transportation infra-structures to inter-connect, share information and coordinate pertinent actions [10]. Representative services of C-ITS includes slow-vehicle, pedestrian collision and abnormal condition on the road (accidents and construction) warning services.

It is true that the initial installation of OBU and setting of transportation infra-structures for C-ITS. However, C-ITS can secure time to preemptively respond to risks that occur far from the subject vehicle and generate accurate information regarding the hazardous area—minimizing traffic accidents. Therefore, various countries such as Europe, the United States, Republic of Korea and Japan are currently conducting C-ITS demonstration projects to compensates for blind spots that drivers cannot detect by in-vehicle sensors through infrastructure sensors installed on the roadside [11, 12, 13].

Among C-ITS services, the pedestrian collision warning service aims to improve pedestrian safety at intersections or road sections. The pedestrian collision warning service implemented in the demonstration projects in various regions by applying the C-ITS standard acquires image data from a camera installed on the road-side such as crosswalks, and streams the image data to the image processing unit. The image processing unit analyzes the received image data to detect possible collisions and transmit collision warning messages to nearby vehicles through the vehicle-to-everything (V2X) server and RSU. However, due to the large amount of streaming data and the communication latency, simplification of software architecture and processing speed beyond real-time operation have become a critical concern [14].

Prediction of pedestrian’s intention is under active research to realize intelligent transportation systems and autonomous driving. Especially, it plays a key role in realizing a pedestrian collision warning service. One of the works has utilized pedestrian’s demographic information, such as gender and age, and the movement of pedestrians to predict the intention [5]. Although such demographic information could help infer pedestrian’s intention, pedestrian’s demographic information is not always available and straightforward to recognize.

On one hand, hand-crafted features or statistical model design still perform better than deep neural networks in certain environments [15, 5] since a sufficient amount of data for training large deep neural networks is not available in the area of the pedestrian intention prediction task. On the other hand, deep neural networks trained on large public datasets could replace sub-modules of intention prediction systems [7, 1]. Although such methods can guarantee solid performance in predefined environments, they can hardly generalize to multiple sites and they in general require a fine-tuning process to get deployed in new sites.

Another stream of research incorporates deep-learning methods to maximize the performance [8, 16, 17]. At the current state, the resulting algorithms assume specific situations such as evacuations and thus do not generalize to common transportation scenarios. Moreover, an intention prediction algorithm in signalized environments takes the signal and elapsed time of the signal phase into account in addition to environmental context, vehicle features and pedestrian characteristics [18]. The work, however, does not guarantee universality over multiple sites. The work that is most relevant to our framework estimates 2D pose for predicting pedestrian intention [7]. However, the performance of intention prediction based on 2D pose degrades in the cases of occlusions and view variations.

Human pose estimation algorithms include two main categories: 2D and 3D pose estimations. Both 2D and 3D pose estimation algorithms take in RGB images and estimate the pose of humans within the images. The surge of deep-learning has resolved the limitations of classical methods in 2D human pose estimation [19, 20, 21]. The research on 2D pose estimation with deep-learning has become feasible with the collection of corresponding datasets [22, 23]. The most widely used COCO dataset contains over 200,000 images and 250,000 person instances with the labels of 17 keypoints.

The request for 3D coordinates of human joints has triggered the development of 3D pose estimation algorithms [24, 25, 26]. Exemplary applications of 3D pose estimation encompass AR/VR, human computer interaction, computer graphics and human action understanding. The research on 3D pose estimation has become active with the collection of large datasets [27, 28, 25], which is similar to the case of 2D pose estimation. Recent 3D pose estimation methods have incorporated pre-trained 3D human models [29, 30]. Incorporation of 3D human models improves the performance significantly and enables 3D pose estimation to overcome harsh conditions such as occlusions and view variations due to the injection of additional knowledge-base.

We designed the P2CWS framework to alert dangerous behaviors of pedestrians to vehicle drivers approaching intersections. Fig. 1 shows the overview of P2CWS. P2CWS require three hardware components: On-Board Unit (OBU) on vehicles, Road-Side Units (RSU), and CCTV installed at an intersection. For detecting pedestrians in real traffic scenes, we consider a camera (CCTV) as a sensor. P2CWS resides on RSU and consists of a real-time vision framework and a crossing intention prediction system. The overall data flow of P2CWS is as follows. The installed camera at an intersection shoots a fixed site including pedestrians before crossing the crosswalk, and collects a sequence of images at every 0.033 second (30 FPS). It transmits the collected image sequences to RSU. The OBU on each vehicle gather their own vehicles’ location data while uploading their locations to RSU. If RSU receives no information from OBU, the vision framework of P2CWS extracts the position and speed information of the vehicle. The real-time vision framework of P2CWS processes collected information and extracts features with which the crossing intention prediction system predicts the intention of pedestrians.

Implementing the P2CWS demands a high level of real-time data processing and transmission techniques. There are still a number of issues associated with the application of wireless cellular communication in practice, such as latency, reliability, data delivery ratio, and GPS accuracy of smartphones in vehicles. The main focus of our work is on studying a machine learning-based collision warning rather than on dealing with the communication problems—we do not consider such communication issues. Therefore, we develop the machine learning-based collision warning system under the assumption that P2CWS receives traffic information with acceptable latency and deliberated accuracy.

Fig. 2 displays the concept of collision warning provided by P2CWS. P2CWS extracts features of pedestrians, vehicles and other contexts every 0.033 seconds. Then, the crossing intention prediction system recognizes the pedestrian’s crossing intention to warn drivers approaching the intersection of possible collisions. Since the paths of vehicles are available, P2CWS could predict possible collisions. Moreover, P2CWS aims to predict the future crossing intention in 1.5 seconds.

Fig.3 illustrates the overall architecture of the proposed vision framework. The input sequence of images first passes through the object detection and object tracking modules which recognize object semantics and object identities. Then, the human pose estimation and orientation estimation modules extract pose and orientation features of pedestrians. Meanwhile, a set of first image frames goes through the semantic segmentation module for the analysis of environmental semantics. After extracting the environmental semantics, the semantic segmentation module becomes idle. Moreover, the distance measure module estimates distances between entities. After all the features necessary for the intention prediction get extracted, the process of intention prediction begins.

The proposed framework estimates both 3D and 2D poses of pedestrians (Fig. 4). For 3D pose estimation, the proposed framework utilizes one of the off-the-shelf 3D pose estimation algorithms [26] and we propose to derive 2D poses from the estimated 3D poses.

We define two categories of pedestrian orientation: head orientation and body orientation. By defining two types of orientation, we can specifically analyze the orientation of pedestrians. Moreover, we use line equations in the vector form to represent orientation.

Distances between objects offer a key context for the interpretation of interactions between the objects. Since 2D imaging modalities hinder the exact recovery of the 3D dimensions without prior knowledge [33], we linearly approximate distances from 2D images using a knowledge-base of object dimensions as follows:

where $h$ and $\bar{h}$ denote the measured height in pixels and the mean height of an object from the knowledge-base, respectively, $(u,v)$ represents the position of an object on the 2D image plane, and $1$ and $2$ refer to object identities, respectively. Table I displays the knowledge-base of the mean heights of the objects involved in our study. After measuring distances, we normalize them by $exp(-\hat{d}/n_{h})$ where $n_{h}$ is a normalization factor.

Table II and Fig. 5 summarize the features for intention prediction. We propose to extract three categories of features: pedestrian features, V2P interactions and environmental contexts. The pedestrian features derive the characteristic of pedestrians in three feature vectors (3D pose, group size and speed). The V2P interactions represent the effect of vehicles on pedestrians’ intention (distance, angle and speed). Last, the environmental contexts stand for the encoding of environment information (distance to a crosswalk, angle with a crosswalk and location of pedestrian). In total, we deal with nine types of features.

We categorize vehicles into two groups: approaching or non-approaching. The distances between the pedestrian of interest and the approaching vehicles decrease ($\Delta d<0$) over time and the distances increase ($\Delta d>0$) in the case of non-approaching vehicles. We only consider the closest approaching vehicle for the analysis of V2P interactions. This analysis setting simplifies the analysis process and the subsequent approaching vehicles get into consideration after the closest approaching vehicle becomes a non-approaching vehicle.

For intention prediction, the feature at time step $t$ becomes

We input a set of features from a specific length of time span (temporal context) to a classifier and retrieve the intention prediction result at different future time steps. We sample 15 features per second to account for the case when detectors fail to recognize entities. Furthermore, we could attach the current state information (crossing or not-crossing) at each time step to $F_{t}$.

Table III summarizes the comparative study results. The proposed vision framework outperforms or performs on par with baselines in the orientation recognition task. One thing to note is that our method does not involve any training or fine-tuning steps for orientation recognition with the TUD dataset, and we obtain the result solely with (4). This fact ensures the generality of the proposed pedestrian orientation recognition method and indicates that our method is free from the issue of overfitting. Other methods involve the process of fine-tuning for performance maximization, which could cause overfitting.

Next, our method seems to reveal weak performance in catching tiny details, although the proposed method entails much lower computational complexity (higher FPS) and displays superior performance on the overall view. The Accuracy $22.5^{\circ}$ of our method is not satisfactory compared to the performance in other metrics. We presume that the axis for measuring orientation angles might not match between the one provided by the TUD dataset and ours. Since our method measures orientations in a normalized 3D space and the TUD dataset has been annotated using 2D images, developing a calibration method for the measurement axis would result in performance enhancement in the Accuracy $22.5^{\circ}$ metric.

Moreover, our method can recognize the body orientation when pedestrians do not stand up straight while other baseline methods function with the assumption of the straight pose. The TUD dataset contains only the straight pose cases, and baseline methods trained or fine-tuned on the dataset would fail with other pedestrian postures. In a similar vein, we could easily extend our method to perform other tasks, unlike baseline methods, due to its extraction of generic 3D pose features. The example tasks our method can be extended to include pedestrian movement analysis, pedestrian action recognition, and tracking pedestrians’ views.

Fig. 7 illustrates the distribution of absolute errors and the cumulation of absolute errors per angle. The maximum and minimum of the absolute errors are $85.70$ and $0.33$, respectively, and the number of samples decreases with the increment of the absolute error. Next, the cumulation of errors distributes uniformly over angles despite a few peaks, such as the peaks at $100^{\circ}$ and $150^{\circ}$. The uniform distribution demonstrates that our method recognizes orientations without a bias.

Table IV summarizes the ablation study result (prediction of the intention at $1.5$ sec given $0.5$ sec context). Among the three models (FFNN, GRU and transformer), the GRU model performed the best, achieving $91.28$% accuracy given the state information and $60.40$% accuracy without the state information. With the state information given, all the models performed satisfactorily and the performance gaps between models were marginal. This indicates the importance of the current state information for predicting the future intention of pedestrians. The state information allowed the models to predict future intentions with a small number of model parameters.

However, there seemed to exist a performance limit that models could achieve even when the state information was given. The limit might have arisen from the data discrepancy between training and test sets. Since we had divided the dataset by the scenes rather than mixing the data and splitting by percentage, the train set would not enable models to learn a few features necessary for the perfect performance. We can overcome this limit by collecting more data, which we will conduct as future work.

Next, increasing the model size did not necessarily lead to performance enhancement. The best performing model configurations ($N_{layer}$, $N_{hidden}$) were ($4$, $128$), ($3$, $128$) and ($2$, $32$) for FFNN, GRU and transformer, respectively. At some points, models stopped improving the performance, although we increased the number of model parameters. In other words, making models wider and deeper could result in performance improvement until the performance saturation points. As we collect more data, we would be able to experiment with wider and deeper models and investigate the effect of the model size on the performance more accurately.

Fig. 8 illustrates the effect of the context lengths on the performance of predicting future intentions at various time steps. First of all, injecting more extended contexts did not improve the accuracy of the intention prediction task. In contrast, more extended contexts tended to degrade performance. We presume the short-term context plays a crucial role while the long-term context rapidly becomes obsolete in the intention prediction task. In the context of the transportation systems, the fact that the prediction of pedestrians’ intention requires only short-term contexts could benefit the design of safety systems.

Furthermore, the multi-task learning setting did not consistently improve performance (the second row in Fig. 8). In the case of GRU, the multi-task learning setting helped the models to enhance the performance with the state information given, but the performance decreased in other cases. We surmise two rationales for this phenomenon: 1) predictions of intentions at different future time steps are independent to some degree, which hinders the extraction of common features or 2) the collected data was not enough to take advantage of the multi-task learning setting.