Skeletal Video Anomaly Detection using Deep Learning: Survey, Challenges and Future Directions

In the reconstruction approaches, generally, an autoencoder (AE) or its variant model is trained on the skeleton information of only normal human activities. During training, the model learns to reconstruct the samples representing normal activities with low reconstruction error. Hence, when the model encounters an anomalous sample at test time, it is expected to give high reconstruction error.

Gatt et al. [22] used Long Short-Term Memory (LSTM) and 1-Dimensional Convolution (1DConv)-based AE models to detect abnormal human activities, including, but not limited to falls, using skeletons estimated from videos of a publicly available dataset. Temuroglu et al. [23] proposed a skeleton trajectory representation that handled occlusions and an AE framework for pedestrian abnormal behaviour detection. The pedestrian video dataset used in this work was collected by the authors, where the training dataset was composed of normal walking, and the test dataset was composed of normal and drunk walking. The pose skeletons were treated to handle occlusions using the proposed representation and combined into a sequence to train an AE. They compared the results of occlusion-aware skeleton keypoints input with keypoints without occlusion flags, keypoint image heatmaps and raw pedestrian image inputs. The authors used average of recall and specificity to evaluate the models due to the unbalanced dataset and found that occlusion-aware input achieved the highest results. Suzuki et al. [24] trained a Convolutional AE (CAE) on good gross motor movements in children and detected poor limb motion as an anomaly. Motion time-series images [52] were obtained from skeletons estimated from the videos of kindergarten children participants. The motion time-series images were fed as input to a CAE, which was trained on only the normal data. The difference between the input and reconstructed pixels was used to localize the poor body movements in anomalous frames. Jiang et al. [25] presented a message passing Gated Recurrent Unit (GRU) encoder-decoder network to detect and localize the anomalous pedestrian behaviours in videos captured at the grade crossing. The field-collected dataset consisted of over 50 hours of video recordings at two selected grade crossings with different camera angles. The skeletons were estimated and decomposed into global and local components before being fed as input to the encoder-decoder network. The localization of the anomalous pedestrians within a frame was done by identifying the skeletons with reconstruction error higher than the empirical threshold. They manually removed wrongly detected false skeletons as they claim that the wrong detection issue was observed at only one grade crossing. However, an approach of manual removal of false skeletons is impractical in many real world applications where the data is very large, making the need of an automated false skeleton identification and removal step imperative. In their following work [26], the authors improved the performance of detecting abnormal pedestrian behaviors at grade crossings using a generative adversarial network (GAN)-based framework. Two LSTM-based branches within the generator were used to analyze both local and global motion patterns simultaneously, reconstructing the corresponding inputs in the temporal domain. The discriminator was a fully connected neural network and produced a score representing the likelihood of inputs being an anomaly. Fan et al. [27] proposed an anomaly detection framework which consisted of two pairs of generator and discriminator. The generators were trained to reconstruct the normal video frames and the corresponding skeletons, respectively. The discriminators were trained to distinguish the original and reconstructed video frames and the original and reconstructed skeletons, respectively. The video frames and corresponding extracted skeletons served as input to the framework during training; however, at test time, decision was made based on only reconstruction error of video frames.

In prediction approaches, a network is generally trained to learn the normal human behaviour by predicting the skeletons at the next time step(s) using the skeletons representing normal human actions at past time steps. During testing, the test samples with high prediction errors are flagged as anomalies as the network is trained to predict only the skeletons representing normal actions.

Rodrigues et al. [28] suggested that abnormal human activities can take place at different timescales, and the methods that operate at a fixed timescale (frame-based or video-clip-based) are not enough to capture the wide range of anomalies occurring with different time duration. They proposed a multi-timescale 1DConv encoder-decoder network where the intermediate layers were responsible to generate future and past predictions corresponding to different timescales. The network was trained to make predictions on normal activity skeletons input. The prediction errors from all timescales were combined to get an anomaly score to detect abnormal activities. Luo et al. [16] proposed a spatio-temporal Graph Convolutional Network (GCN)-based prediction method for skeleton-based video anomaly detection. The body joints were estimated and built into skeleton graphs, where the body joints formed the nodes of the graph. The spatial edges connected different joints of a skeleton, and temporal edges connected the same joints across time. A fully connected layer was used at the end of the network to predict future skeletons. Zeng et al. [29] proposed a hierarchical spatio-temporal GCN, where high-level representations encoded the trajectories of people and the interactions among multiple identities while low-level skeleton graph representations encoded the local body posture of each person. The method was proposed to detect anomalous human behaviours in both sparse and dense scenes. The inputs were organized into spatio-temporal skeleton graphs whose nodes were human body joints from multiple frames and fed to the network. The network was trained on the input skeleton graph representations of normal activities. Optical flow fields and size of skeleton bounding boxes were used to determine sparse and dense scenes. For dense scenes with crowds, higher weights were assigned to high-level representations while for sparse scenes, the weights of low-level graph representations were increased. During testing, the prediction errors from different branches were weighted and combined to obtain the final anomaly score. Fan et al. [30] proposed a GRU feed-forward network that was trained to predict the next skeleton using past skeleton sequences and a loss function that incorporated the range and speed of the predicted skeletons. Pang et al. [31] proposed a skeleton transformer to predict future pose components in video frames and considered error between predicted pose components and corresponding expected values as anomaly score. They applied a multi-head self-attention module to capture long-range dependencies between arbitrary pairwise pose components and the temporal convolutional layer to concentrate on local temporal information. Huang et al. [32] proposed a spatio-temporal graph transformer to encode the hierarchical graph embeddings of human skeletons for jointly modeling the interactions between individuals and the correlations among body joints within a single individual. Input to the transformer was provided as global and local graphs. Each node in the global graph encoded the speed of an individual as well as the relative position and interaction relations between individuals. Each local graph encoded the pose of an individual.

In this section, we discuss the existing methods that utilize a combination of different learning approaches, namely, reconstruction and prediction approaches, and reconstruction and clustering approaches.

This section discusses the methods that leveraged a pre-trained deep learning model to encode latent features from the input skeletons and used approaches such as, clustering and multivariate gaussian distribution, in conjunction for detecting human action-based anomalies in videos.

Yang et al. [43] proposed a two-stream fusion method to detect anomalies pertaining to body movement and object positions. YOLOv3 [59] was used to detect people and objects in the video frames. Subsequently, skeletons were estimated from the video frames and passed as input to a spatio-temporal GCN, followed by a clustering-based fully connected layer to generate anomaly scores for skeletons. The information pertaining to the bounding box coordinates and confidence score of the detected objects was used to generate object anomaly scores. Finally, the skeleton and object normality scores were combined to generate the final anomaly score for a frame. Nanjun et al. [45] used the skeleton features estimated from the videos for pedestrian anomaly detection using an iterative self-training strategy. The training set consisted of unlabelled normal and anomalous video sequences. The skeletons were decomposed into global and local components, which were fed as input to an unsupervised anomaly detector, iForest [60], to yield the pseudo anomalous and normal skeleton sets. The pseudo sets were used to train an anomaly scoring module, consisting of a spatial GCN and fully connected layers with a single output unit. As part of the self-training strategy, new anomaly scores were generated using previously trained anomaly scoring module to update the membership of skeleton samples in the skeleton sets. The scoring module was then retrained using updated skeleton sets, until the best scoring model was obtained. However, the paper doesn’t discuss the criteria to decide the best scoring model. Tani and Shibata [46] proposed a framework for training a frame-wise Adaptive GCN (AGCN) for action recognition using single frame skeletons and used the features extracted from the AGCN to train an anomaly detection model. As part of the proposed framework, a pretrained action recognition model [61] was used to identify the frames with large temporal attention in the Kinetics-skeleton dataset [62] as the action frames to train the AGCN. Further, the trained AGCN was used to extract features from the normal behaviour skeletons identified in the ShanghaiTech Campus dataset [17] to model a multivariate gaussian distribution. During testing, the Mahalanobis distance was used to calculate the anomaly score under the multivariate gaussian distribution. Sato et al. [47] proposed a user prompt-guided zero-shot learning framework for the detection of abnormal human behaviour events. A multilayer perceptron feature extractor was pretrained on large-scale action recognition datasets [63, 64] using contrastive learning between the skeleton features and the text embeddings extracted from action class names. The distribution of skeleton features of the normal actions was modeled during training while freezing the weights of feature extractor. During inference, the anomaly score was computed using distribution and the text prompts of an unseen action. Javed et al. [44] proposed a unified framework for learning suitable frames of interest to cut down on redundant data and a two-stream feature block with a hyper-gated fusion model to take advantage of skeleton graph and video frame features. Soft assignments were later processed through a clustering layer, where probabilities were assigned to the instances and a normality score was calculated using the Dirichlet Process Mixture model [65].

ShanghaiTech [55] and CUHK Avenue [66] were the most frequently used video datasets to evaluate the performance of the skeletal video anomaly detection methods. The ShanghaiTech dataset has videos of people walking along a sidewalk of the ShanghaiTech university and anomalous frames contain bikers, skateboarders and people fighting. It has 330 training videos and 107 test videos. However, not all the anomalous activities are related to humans. A subset of the ShanghaiTech dataset that contained anomalous activities only related to humans was termed as HR ShanghaiTech and was used in many papers. The CUHK Avenue dataset consists of short video clips looking at the side of a building with pedestrians walking by it. Concrete columns that are part of the building cause some occlusion. The dataset contains 16 training videos and 21 testing videos. The anomalous events comprise of actions such as “throwing papers”, “throwing bag”, “child skipping”, “wrong direction” and “bag on grass”. Similarly, a subset of the CUHK Avenue dataset containing anomalous activities only related to humans, called HR Avenue, has been used to evaluate the methods. Other video datasets that have been used include UTD-MHAD [69], UMN [68], UCSD Pedestrian[6], IITB-Corridor [28], UCF-Crime[67], HR Crime[7], NTU-RGB+D[56], RWF-2000[70] and Kinetics-250[57]. Table III presents a summary of the characteristics of these datasets and Figure 1 presents one normal and one anomalous frame from these datasets. Among the datasets used in the reviewed papers, some of the datasets were originally not meant for but instead adopted for the task of video anomaly detection. Hence, we only provide details for the datasets that were originally meant for video anomaly detection in Table III and Figure 1. From the type of anomalies present in these datasets, it can be inferred that the existing skeletal video anomaly detection methods have been evaluated mostly on individual human action-based anomalies. Hence, it is not clear how well can they detect anomalies that involve interactions among multiple individuals or interactions among people and objects.

Most of the papers (27 out of 29), detected anomalous human actions for multiple people in the video scene. Other two papers detected irregular body postures and poor body movements in children, respectively, for single person in the video scene. The usual approach was to estimate the skeletons for the people in the scene using a pose estimation algorithm, and calculate anomaly scores for each of the skeletons. The maximum anomaly score among all the skeletons within a frame was used to identify the anomalous frames. A single video frame could contain multiple people, among which not all of them were performing anomalous actions. Hence, taking the maximum anomaly score of all the skeletons helped to nullify the effect of people with normal actions on the final decision for the frame. Further, calculating anomaly scores for individual skeletons helped to localize the source of anomaly within a frame.

The definition of anomalous human behaviours can differ across various applications. While most of the existing papers focused on detecting anomalous human behaviours in general, five papers focused on detecting anomalous behaviours for specific applications, including drunk walking [23], poor body movements in children [24], abnormal pedestrian behaviours at grade crossings [25, 26] and crime-based anomalies [7]. Moreover, the nature of anomalous behaviours can vary depending upon various factors, such as span of time, crowded scenes, and specific action-based anomalies. Some papers identified and addressed the need to detect specific types of anomalies, namely, multi-timescale anomalies occurring over different time duration [28], anomalies in both sparse and crowded scenes [29], fine and coarse-grained anomalies [38] and body movement and object position anomalies [43].

Alphapose [71] and Openpose [72] were the most common choice of pose estimation algorithm for extraction of skeletons for the people in the scene. Other pose estimation methods that have been used were Posenet[73], PPN[74] and HRNet[75]. However, in general, the papers did not provide any rationale behind their choice of the pose estimation algorithm.

The type of models used in the papers can broadly be divided into two types, sequence-based and graph-based models. The sequence-based models that have been used include 1DConv-AE, LSTM-AE, GRU, and Transformer. These models treated skeleton keypoints for individual people across multiple frames as time series input. The graph-based models that have been used involve GCAE and GCN. The graph-based models received spatio-temporal skeleton graphs for individual people as input. The spatio-temporal graphs were constructed by considering body joints as the nodes of the graph. The spatial edges connected different joints of a skeleton, and temporal edges connected the same joints across time.

The choice of a suitable threshold for anomaly detection can vary across different applications as most applications come with different costs for false alarms and missed anomalies [76, 77]. As such, having a metric capable of evaluating the performance of anomaly detection methods across diverse application scenarios, or equivalently, across a wide array of decision thresholds is highly desirable. The Area Under Curve (AUC) of Receiver Operating Characteristic (ROC) curve computes the fraction of detected anomalies, averaged over the full range of decision thresholds. It is the standard evaluation measure used in anomaly detection [76] and also the most common metric used to evaluate the performance among the existing skeletal video anomaly detection methods. The highest AUC(ROC) values reported for the commonly used ShanghaiTech [55] and CUHK Avenue [66] datasets across different methods in Table I and II were 0.83 and 0.92, respectively. A direct comparison may not be possible due to the difference in the experimental setup and train-test splits across the reviewed methods; however, it gives some confidence on the viability of these approaches for skeletal video anomaly detection. Other performance evaluation metrics include F score, accuracy, Equal Error Rate (EER) and AUC of Precision-Recall (PR) Curve. EER signifies the percentage of misclassified frames when the false positive rate equals to the miss rate on the ROC curve. While AUC(ROC) can provide a good estimate of the classifier’s performance over different thresholds, it can be misleading in case the data is imbalanced [78]. In anomaly detection scenario, it is common to have imbalance in the test data, as the anomalous behaviours occur infrequently, particularly in many medical applications [79, 80]. The AUC(PR) value provides a good estimate of the classifier’s performance on imbalanced datasets [78]; however, only one of the papers used AUC(PR) as an evaluation metric.

In general, the efficiency of the skeletal video anomaly detection algorithms depends upon the accuracy of the skeletons estimated by the pose-estimation algorithm. If the pose estimation algorithm misses certain joints or produces artifacts in the scene, then it can increase the number of false alarms. There are various challenges associated with estimating skeletons from video frames [81]: (i) complex body configuration causing self-occlusions and complex poses, (ii) diverse appearance, including clothing, and (iii) complex environment with occlusion from other people in the scene, various viewing angles, distance from camera and truncation of parts in the camera view. This can lead to a poor approximation of skeletons and can negatively impact the performance of the anomaly detection algorithms. Further, there is an associated high cost of powerful hardware required for extracting skeletons using deep learning methods. Methods have been proposed to address some of these challenges [82, 83]; however, extracting skeletons in complex environments remains a difficult problem. The two most commonly used pose estimation algorithms in the reviewed papers are Openpose [72] and Alphapose [71]. Multi-person pose estimation can be categorized into top-down and bottom-up methods [81]. Top-down methods [71, 84] usually employ human detectors to obtain bounding boxes for humans in the input frame and then utilize existing single-person pose estimators to predict body joints. This method highly depends upon the precision of human detection algorithms, and the run-time is proportional to the number of persons in the input frame. Bottom-up methods [72] directly approximate all the body joints of all the humans in the input frame and assemble them into individual skeletons. However, the grouping of joints in a complex scene is a challenging task. Openpose is a bottom-up method and Alphapose is a top-down method. Figure 2 presents the skeleton output of openpose and alphapose on different dataset frames. Some of the existing methods manually remove inaccurate and false skeletons [17, 25] to train the model, which is impractical in many real-world applications where the amount of available data is very large. There is a need for an automated false skeleton identification and removal step when estimating skeletons from videos.

The anomalous human behaviours of interest and their difficulty of detection can vary depending upon the definition of anomaly, application, time span of the anomalous actions, and presence of single/multiple people in the scenes. For example, in the case of driver anomaly detection application, the anomalous behaviours can include talking on the phone, dozing off or drinking [14]. The anomalous actions can span over different time lengths, ranging from few seconds to hours or days, e.g., jumping and falls [85] are short-term anomalies, while loitering and social isolation [86] are long-term events. More focus is needed on developing methods that can identify both short and long-term anomalies. Sparse scene anomalies can be described as anomalies in scenes with less number of humans, while dense scene anomalies can be described as anomalies in crowded scenes with a large number of humans [29]. It is comparatively difficult to identify anomalous behaviours in dense scenes than sparse scenes due to tracking multiple people and finding their individual anomaly scores [17]. Thus, there is a need to develop methods that can effectively identify both sparse and dense scene anomalies. With the development of algorithms for handling different types of anomalies, there is a need for datasets composed of the specific type of anomalies to ensure efficient training and evaluation. This can be handled by either having separate datasets for specific types of anomalies or general datasets with a distribution of multiple types of anomalies.

The skeletons collected using Microsoft Kinect (depth) camera has been used in the past studies [87, 88]. However, the defunct production of the Microsoft Kinect camera [89] has led to hardware constraints in the further development of skeletal anomaly detection approaches. Other commercial products include Vicon [90] with optical sensors and TheCaptury [91] with multiple cameras. But they function in very constrained environments or require special markers on the human body. New cameras, such as ‘Sentinare 2’ from AltumView [92], circumvent such hardware requirements by directly processing videos on regular RGB cameras and transmitting skeletons information in real-time.

The existing approaches for skeletal video anomaly detection involve spatio-temporal skeleton graphs [16] or temporal sequences [17], which are constructed by tracking an individual across multiple frames. However, this is challenging in scenarios where there are multiple people within a scene. The entry and exit of people in the scene, overlapping of people during movement and presence of occluding objects make tracking people across frames a very challenging task.

There can be deployment issues in these methods because the choice of threshold is not clear. In the absence of any validation set (containing both normal and unseen anomalies) in an anomaly detection setting, it is very hard to fine-tune an operating threshold using just the training data (comprising of normal activities only). To handle these situations, outliers within the normal activities can be used as a proxy for unseen anomalies [85, 93]; however, inappropriate choices can lead to increased false alarms or missed alarms. Domain expertise can be utilized to adjust a threshold, which may not be available in many cases.

There is a need to address the challenges associated with the granularity and the decision-making time of the skeletal video anomaly detection methods for real-time applications. The existing methods mostly output decisions on a frame level, which becomes an issue when the input to the method is a real-time continuous video stream at multiple frames per second. This can lead to alarms going off multiple times a second, which can be counter-productive. One solution is for the methods to make decisions on a time-window basis, each window of length of a specified duration. However, this brings in the question about the optimal length of each decision window. A short window is impractical as it can lead to frequent and repetitive alarms, while a long window can lead to missed alarms, and delayed response and intervention. Domain knowledge can be used to make a decision about the length of decision windows.