Real-World Graph Convolution Networks (RW-GCNs) for Action Recognition in Smart Video Surveillance

Noisy Input: The first primary constraint of this new sub-domain is real-world data. Skeleton-based action recognition is highly dependent on human pose data in the form of key points. Generally, works published in the field assume perfect human pose data is provided. This pristine data is often obtained through time consuming manual annotation or the use of expensive, highly accurate sensors in a laboratory environment. In real-world deployment, pose data needs to be generated on the fly, often by a dedicated machine learning algorithm, resulting in imperfect data. This is referred to as noisy data, and in this paper we assume this noise is generated by a DNN-based multi-person pose estimator. We define noise in a similar manner as in the work of Yu et al. [67].

We adopt their definition of spatial noise; that is human keypoint joints that should be in the frame but are not provided. However, we expand upon their definition of temporal noise, separating it into two distinct categories. The first category is frame-level temporal noise. This occurs when an individual’s data is completely missing from a frame, whether due to failure to detect a person or by misidentifying them in a multi-person environment. The second category is receptive field level temporal noise. This is an implicit noise, defined as a limitation of a network’s capacity in being able to view the entire duration of action clip.

Latency Constraint: The second major constraint for real-world skeleton-based action recognition is latency. While some existing works focus on the computational complexity of skeleton-based action recognition, many ignore the number of frames required to perform recognition of a single action. Typically, the amount of frames needed to classify a single action (i.e. clip size or $T$) of existing works is 300. If we are to assume a neural pose estimator running at $FPS_{in}=30$ (a non-trivial achievement depending on the hardware), then a 300 frame clip corresponds to a 10-second delay. This is prohibitively expensive for applications requiring real-time responses, such as patient monitoring, public safety, and general video surveillance.

Edge Computing Constraint: Lastly, an edge computing paradigm becomes a necessity due to the additional privacy and scalability considerations for real-world skeleton action recognition. Most existing video surveillance implementations stream image data from a local camera on-the-edge to cloud servers where the bulk of the computation would be performed. This invalidates user privacy and limits scalability. Privacy is essential toward enabling smart edge video applications to the point of other works dedicating computationally expensive deep neural networks [57] to insure user privacy. In our work, low-power edge devices near the sensors are utilized for computing both the pose estimation and action recognition. By working directly on the video stream at the edge, the only data passed onto the network is pre-processed and contains no personally identifiable information.

This inherently addresses the growing concerns of streaming personal data across the cloud. Additionally, the use of edge devices to handle the bulk of processing positively impacts the scalability of the system. In a cloud computing based system, scalability is limited by the processing power of the cloud server and the throughput of the network. Additional cameras lead to more requests for the server, which can lead to either costly server upgrades or an increase in system-wide latency due to the inability to meet system computation demand. When the system implementation is too costly it can limit the proliferation of smart video applications on-the-edge as well as the democratization of AI as a whole. Cutting back on costly cloud servers with cheaper local edge computation solves this issue, and has been used in existing deep neural network service platforms [34, 35]. However, edge solutions are often tightly resource constrained, which results in other implementations utilizing hybrid edge/cloud solutions [38], or optimizing the DNN for performance [45, 40].

In order to address latency constraints on resource constrained edge systems, we define two new metrics for analyzing and investigating the real-world deployment capability of action recognition algorithms. The first is Actions per Second $(ApS)$.

This metric conveys how much work the model is able to complete in a single second (i.e, throughput). Where $T$ is the clip size, $W$ is the window size, and $CpS_{in}$ is clips per second or the rate a full clip is fed to the network. This metric is needed to compare latency constrained works such as ours to unconstrained works, as they do differing amounts of work in a single clip. While the theoretical input throughput ($FPS_{T}$), accepted by the model is important for fast moving applications such as self-driving cars, the domain of video surveillance has much more lax requirements ( 30 $FPS$ for human motion). Instead, the second metric of Action Product Delay ($APD$), is much more predictive of the real-world deployment performance in domains such as video surveillance.

$APD$ is the product between the inverse of the input frame rate $FPS_{in}$ and the window size of the network $W$. This metric conveys how much delay is needed to process a single action. In applications such as monitoring patients or hazardous working environments, the response time of detecting an incident (e.g. falling, tripping) is critical and its lower bound is the $APD$.

In a real-world setting, skeleton-based action recognition will be part of a larger end-to-end vision pipeline. Fig. 1 presents a typical vision pipeline for smart video surveillance applications. The figure is inspired by existing vision pipeline frameworks as presented in [36] and [23]. Shown in Fig. 1, these frameworks generally consist of three conceptual stages. We also included an additional stage which consists of our RW-GCN network. A detailed description of RW-GCN can be found in Section 4.2. The first is the detection stage. A multi-person human pose estimator is used to detect person keypoints and generate bounding boxes from a single frame of the input video. Next, in the extraction stage, person image crops are generated using the bounding boxes from the previous stage. These person crops are fed through a feature extractor network that generates an embedded feature representation for each person image. These bounding boxes are also used to calculate the temporal locality of each individual in the frame compared with previous frames. In the matching stage, a matching algorithm measures the euclidean appearance similarity between features from the current frame and those seen previously and, taking into account the previously calculated temporal locality, determines what IDs to assign to which person.

This end-to-end vision pipeline is ran entirely on an edge node directly near the camera, with non-sensitive information aggregated from multiple nodes to an edge server, as done in [36].

Spatial-Noise: Skeleton-based action recognition requires human pose information, in the form of a person keypoint skeleton, as an input. In closed-world action recognition, these skeletons are assumed to be flawless, perfect representations of the actors in the scene. However, in the real world, skeletons provided to the network by a human pose estimator will often be flawed. Even state-of-the-art methods are not perfect and will prove to be incorrect on occasion by either misdetecting a keypoint or failing to detect it all together. These types of errors may be inevitable on the whole, but they are even further exacerbated when the application environment differs from the dataset with which the pose estimator was trained, which will almost always be the case in the real world. For systems where hardware resources must be considered, or where real-time restraints exist, using the large, highly accurate state-of-the-art models may not be an option. In these cases a trade-off must be made, using lighter weight pose estimators and often sacrificing accuracy in order to meet application requirements, which can further increase the frequency of these errors.

These missing and misdetected keypoints create a form of spatial noise in the system, as seen in Fig. 2. This noise can be highly problematic when propagated to action recognition. The actor in the top half of Fig. 2 is leaning down to place an item in the bucket. This is clear to the human eye. However, the keypoint skeleton that is used for action recognition has prominent errors in this scene, particularly concerning the actors right arm. In addition to being generally inconsistent in its detection, in the far right image in the figure, the actors right hand is falsely detected all the way at his ankles. If an action recognition system is designed to assume the data it receives is perfect, this kind of error can easily confuse the network and lead to an incorrect action classification. As such, this type of noise needs to be accounted for in real-world action recognition.

To balance between accuracy and real-time constraints, we use EfficientHRNet [37] for acquiring human pose skeletons.

Temporal Noise: Another category of noise common to real-world action recognition is temporal noise. Temporal noise can occur in one of two ways. The first is when the skeleton for an actor is missing in one or more frames of a video segment. Generally this is either because the skeleton was not detected by the pose estimator, or because the actor’s skeleton was not correctly matched to the actor for this frame. In both cases, this leads to the action recognition network having no skeleton information for the actor when such information should be present. This is not an issue in closed-world action recognition where perfect skeleton and ID data is provided, but in real-world action detection, where imperfect algorithms are used to generate keypoints and IDs, this type error is quite common.

The second form of temporal noise occurs when the edge system does not feed all the frames of the action into the network at the same time. While this is not an issue for closed-world action recognition, in the real world this can often encounter in applications where the latency constraints dictate that only a small number of frames can be processed at once.

Fig. 2 illustrates an example of the first category of temporal noise; missing keypoint skeletons. While the actor in the bottom half of Fig. 2 is correctly identified in the first frame, the middle two frames have greyed out keypoint skeletons, indicating that they were not correctly matched to the actor. This is an error caused by the re-identification algorithm. Further, in the last frame, no skeleton was detected at all; a failure of the neural pose estimator. With only a single frame worth of pose data to go on, the action recognition network will have no concept of the actor’s motion and position through time. Intuitively, this will make action recognition for the scene incredibly challenging, and such instances must be accounted for when designing action recognition for the real world.

We opt to use a similar Re-ID methodology as in [36], a real-world solution that does not rely on facial recognition.

The theoretical foundations for our attentive feedback augmentations are based on the information bottleneck theory of deep neural networks, proposed by Tishby et al. [58]

While this theory was initially contested by the work of Saxe et al. [48], the theory was then re-validated by the work of Noshad et al. [41] stating that there was an issue in the calculation of mutual information due to poor mutual information estimators being used, with Nosahd et al. proposing their own Mutual information estimator. Additionally, further attempts have been made to verify the information bottleneck on more complicated architectures [9]. Despite the difficulties in directly applying the information bottleneck to deep learning, other works have shown clear empirical improvements in generalization when developing solutions based on the foundations of the information bottleneck [19, 43]. Our work takes the same approach, and develops architecture level solutions to real-world real-time action recognition using principles inspired from the information bottleneck theory of deep neural networks.

The information bottleneck theory starts with assuming the deep neural network can be represented as a Markov chain:

Each layer’s output is represented by a random variable in the Markov chain. $X$ for example is the input feature, $T_{1}$ is layer $1$’s hidden representation, while $T_{z}$ is layer $Z$’s hidden representation and Y is the output random variable.

The information bottleneck when applied to deep learning states that learning is equivalent to minimizing the following Lagrangian:

We can interpret equation (4) as learning having two distinct and separate phases, by minimizing $-I(T;Y)$, which we can treat as maximizing the inverse, and minimizing $I(X;T)$. The fitting stage is the first and is shown in equation (5):

The fitting phase is regulated by $\beta$, and learns informative features by maximizing the entropy of hidden features $H(T)$, which makes the hidden representation more informative. Additionally, the fitting phase minimizes $H(T|Y)$, which minimizes the information a hidden representation has when information on the label is already known. This can intuitively be seen as removing features irrelevant to the hidden representation.

The second phase of learning is the compression phase as shown in equation (6):

The compression phase minimizes $H(X)$, the entropy on the source data, while maximizing the the amount of information a hidden representation contains about the input, $H(X|T)$. This leads to a model that realizes less on the input information. Our work hypothesizes that utilizing solutions with properties that assist either the fitting or the compression learning phases can lead to robust algorithmic solutions similar to Jeon et al. [19].

The first architecture level solution RW-GCNs employs is based on self-attention. The work of Bloem et al. [2] states that attention allows networks to learn and model data-dependent invariance via capturing the minimum sufficient statistics of the data. Understanding that the information bottleneck can be seen as a framework for finding the minimal sufficient statistics [53], we hypothesise that attention enables models to better capture the minimum sufficient statistics of the dataset more efficiently, which directly helps the model with respect to the the information bottleneck framework. This is further supported by the perspective that attention can operate as a form of pre-training compression [2]. We additionally highlight the fact that early stopping (training only until validation accuracy stops increasing) is often employed in training deep neural networks. Other works have noticed that early stopping often (but not always) stops training before the compression phase, suggesting that the compression phase can lead to overfitting [43, 63]. Knowing that, the compression phase can be interpreted as learning which information to focus on. We believe that self-attention is a flexible enough inductive bias (due to the ability to imitate and extend the inductive bias of convolution [8, 1]), to enable a non-overfitting compression phase.

Feedback can also assist model generalization, not through the compression phase but rather the fitting phase. In fact, the main premise of RW-GCNs is to maintain good representations or maximize $H(T)$ despite the presence of receptive field level temporal noise (induced by latency constraints). With that premise in mind, we look at the fitting phase, max $I(T;Y)$, as a classical channel coding problem where our network is both the encoder and the channel itself and channel capacity is the mutual information $I(T;Y)$. We theorize that if feedback can assist channel encoding in classical communication channels, then it can assist the deep neural networks though improving the fitting phase of neural network training.

It is known that when feedback is applied to a traditional communication channel with uncorrelated Gaussian noise channel capacity does not increase [12]. However, when the noise is correlated with the input (i.e. a channel with memory) feedback can lead to a small improvement in channel capacity. Alternatively, feedback can significantly reduce the complexity of the channel encoder and decoder [6] (i.e. the neural network), and through this reduced complexity the average block-length required to approach capacity in a traditional communication channel is reduced. We believe this is equivalent to reducing the amount of features needed to be seen by a deep neural network in a single time step. We relate this to how convolution neural network layers can be seen having neurons with a limited receptive field (i.e. less spatial features), and the network depth as mechanism that imitates feedback’s ability to achieve reasonable accuracy with reduced complexity. It is known that depth can imitate feedback due to the fact that deep neural networks often have high amount of inter-layer weight sharing [3, 47].

Feedback can additionally provide an additive gain in the capacity of memoryless Multi-Access Channels (MACs), where multiple signals are passed into the same channel [15]. This can be seen as the neural network dealing with multiple features. The capacity improvement is achieved by feedback’s ability to increase the cooperation of the signals, enabling the neural network to better jointly optimize multiple competing features. When only partial information of the multiple signals is desired, a multiplicative gain in channel capacity is achieved [56]. We identify semantic feedback as a mechanism that attempts to maximize $H(T)$, or equivalently increase the information of feature representations by exploiting the high level semantic information (such as deep neural network features), as seen in CliqueNets [66]. We believe this high level semantic information works through feedback’s ability to reduce the complexity of the encoder (i.e. the network) which allows us to work with a smaller temporal receptive field.

Additionally, we identify control feedback as a mechanism that enables the extraction of statistics relevant to specific random variables but irrelevant to others. This type of feedback focuses on minimizing $H(T|Y)$, which reduces the redundancy of hidden features when some information about the target classification is known. This can be seen as an analog to feedback’s ability to coordinate multiple signals in a traditional multi-access communication channel. Additionally, It is known that the information bottleneck leads to models with alternating formulations when conditionally optimizing to fit multiple random variables as seen in [5]. It is potentially possible to learn conditional models that focus on different output random variables depending on the feedback. Intuitively, if we assume a deep neural network model with the classes of tree, dog, and person, the best model would not expose the high-level features of a tree to the classification of the human class. Control feedback potentially can enable this conditional information flow.

In order to address the domain constraints discussed in Section 3, we augment existing ST-GCNs with attentive feedback, creating Real-World Graph Convolutional Networks (RW-GCNs). In all variations of RW-GCNs we assume a latency constraint in the form of a window size $W$. When $W$ is equivalent to the total number of frames in an action clip $T$, RW-GCN functions as a general ST-GCN. For the baseline implementation of RW-GCN, a clip of $T$ frames is broken down into $W$ windows of $\frac{T}{W}$ frames, and an action classification is produced for each of those windows. RW-GCN adopts a sliding window operation with a stride equivalent to $\frac{T}{W}$.

This limited window size leads to a limited temporal receptive field. To overcome this, RW-GCN propagates information from previous windows into the network while processing the current window. To this end, we explore two specialized forms of feedback, (1) semantic feedback and (2) control feedback for RW-GCNs. Each formulation works with the past feature as an input (in addition to current features) and implements two specialized forms of attention (semantic attention and channel attention). In the following, we discuss each in details.

For the latency constrained accuracy experiments, we focus on the NTU-RGB-D-120 [30] dataset. The NTU-RGB-D-120 dataset is the current largest Pose RGB+D human action recognition dataset. This dataset contains 120 different indoor action classes and is the current de facto standard for evaluating skeleton action recognition works. Each sample clip has a possible maximum of 300 frames at 30 FPS. Additionally, this dataset has a maximum of 2 people in a single frame concurrently. The dataset is composed of 106 unique subjects in combination with 32 possible setups. Overall it contains more than 114,000 video samples and 8 million frames. The original NTU-RGB-D-120 dataset has a pose format consisting of 25 nodes in 3D coordinate space $(x,y,z)$. We ignore the depth dimension and reduce the number of nodes to 18, in order to be compatible with end-to-end systems such as REVAMP²T.

The specific hyperparameters of this training utilized a batch size of $N=32$. The baseline RW-GCN models are trained with a learning rate decay of $10^{-1}$ at epoch 30 and again at 60. For this dataset, we only pay attention to the first skeleton sequence even if there may be two actors in a scene. The RW-GCN-SF variation employs a learning rate restart at the 40th epoch and trains for 45 more epochs, for a total of 85 epochs. All consensus models were trained for 60 epochs. We do not test RW-GCN-CFs with this dataset. When validating our results we use the cross-setup and cross-subject validation strategies [30].

The first experimental results we analyze are the accuracy curves of training a baseline RW-GCN consensus model compared to a RW-GCN-SF, which utilizes semantic feedback. Due to the larger number of epochs used to train the feedback augmented networks, it is necessary to decouple the contributions of the training hyperparameters from the contributions of our attentive feedback augmentations. Fig. 5 shows when the training of the consensus model is extended to a similar number of epochs it stagnates at its existing accuracy. These learning curves are done with a learning rate restart at 50 and 90 epochs and without the growing training strategy. We find that when training with growing feedback augmented attention, RW-GCNs can reach similar accuracies, in fewer epochs, avoiding overfitting.

We go on to compare the accuracy of latency-constrained RW-GCNs to multiple existing works previously mentioned in Section 2. This does not include works with extra modalities such as RGB patches. Table 2 shows us that just by utilizing COCO formatted keypoints RW-GCN is able to achieve $87.7\%$ Cross-Subject Accuracy, a $16.4\%$ increase over the regular NTU-RGB-D skeleton baseline. This is done with no latency constraint and is only $1.6\%$ away from the previous SotA work DSTA-net [52]. We believe this increase in performance is due to the dimensional reduction of the input. By reducing the nodes from 25 to 18 and removing the depth channel, we reduce the dimensionality of the input by $48\%$. These dimensions can intuitively be explained as redundant and allow the model to focus on learning rather than dimension reduction and feature selection.

The Cross-Subject results go on to further show that our consensus-based RW-GCN with a latency constraint of $W=30$ performs at $70.6\%$, more than $17\%$ less than the SotA. This is done with $3\times$ fewer frames and therefore has $3\times$ less delay between action classifications. In fact, assuming this work would be based on the 30 FPS input of the NTU-RGB-D-120 dataset, the SotA (and all existing works other than FGCN [65]) would have an action product delay equating to a 10 second delay, which can be prohibitively expensive for real-world applications such as video surveillance. RW-GCN has a 1 second delay, which we classify as sub-real-time. When utilizing semantic feedback, RW-GCN-SF achieves $86.4\%$ accuracy, only $1.3\%$ away from the previous SotA. This significantly reduces the difference between the SotA while satisfying sub-real-time latency constraints, which enables emerging edge applications. When latency constraints are relaxed to allow a 3.33 second delay, we become the new SotA (with respect to Cross-Subject Validation) at 94.16% accuracy. This shows that not only does our method satisfy the sub-domain of Real-World Skeleton Action Recognition, but we make contributions to skeleton-based action recognition as a whole. When looking at Cross-Setup validation, we see our model does not define a new SotA but is only $.4\%$ less than the SotA. This discrepancy between Cross-Subject and Cross-Setup results can potentially be explained by two factors. The first is the lack of depth. Because of our reliance on COCO keypoints, we experience more domain shift between the training set and the validation set due to the differing viewpoints (and lack of features to inform the model of the sample viewpoints). The second factor is the temporal variance across subjects, which also causes a difference between validation and training set sample distributions. Our work has the potential to adapt to temporal variation more flexibly due to our attention feedback augmentations, and as such can learn to be invariant to individual subject temporal variances, minimizing the distribution shift between the training set and the validation set. This explanation is slightly contested by the baseline ST-GCN’s better performance with Cross-Setup validation. This suggests that lack of depth is only an issue in the presence of temporal receptive field noise.

We believe this accuracy is achievable due to the semantic attention block’s ability to encode spatial pairwise feature interaction when the temporal information is highly redundant or irrelevant (i.e. the frames within the window are minimally affected by receptive field noise), with sparse temporal feature interactions needed to encode the representation. As such, we believe our model encodes more discriminate spatial features than existing works when temporal noise is low, leading to our high accuracy. We additionally analyze other works’ ability to match our performance with respect to latency constraints, shown in Table 3. For this analysis we do not train the existing methods, but rather validate them with basic consensus feedback. We find that without training these works are unable to perform in the real-world domain as the best of them only achieved $10.12\%$.

Our last latency constraint analysis on the NTU-RGB-D-120 dataset is an ablation study on generalization improvements of attentive semantic feedback augmentations to ST-GCNs over simple consensus-based RW-GCNs. This ablation allows us to see semantic feedback performing two modes of operation. In the first latency constraint of $W=30$, semantic feedback performs its intended functionality in allowing the model to retain past features through attentive pairwise feature interactions. The latency constraint of $W=30$ can potentially lack temporal redundancy and implicitly enforce a sparse sampling strategy similar to FGCNs [65]. As a result, the model retains its ability to incorporate long-range temporal feature interactions and increases accuracy only $1.3\%$ off from its baseline unconstrained ST-GCN implementation, as seen in Table 2.

The mode of operation of the ST-GCN begins to shift when constrained by a window of $W=60$. The features it records are more temporally redundant, and the model naturally has the capacity to store longer-range temporal interactions. As such, the model improves the unconstrained baseline of $+2.3\%$. We believe that this window constraint is balancing improving its spatial discriminative power with maintaining the temporal feature retaining capabilities of the lingering implicit receptive field level temporal noise. Due to the complexity of these two tasks, the accuracy improvement over the consensus model is only $6.4\%$ as opposed to the $16.4\%$ improvement in the $W=30$ test. Lastly, with a constraint of $W=100$, the semantic attention block can maintain a minimum amount of temporal features representational power and focus on more competently encoding the spatial features. Due to the consensus model’s inability to improve the spatial representational power of the ST-GCN, the consensus model’s accuracy stagnates.

The end-to-end system noise analysis goes beyond the latency constraint analysis and looks at the emergent system noises of a realistic end-to-end edge system. The Northwestern-UCLA dataset [61] is the data set used and is also a Pose-RGB-D human action recognition dataset. It contains 1494 videos. Each clip contains an average of 18.5 frames with a standard deviation of $\pm 13.25$. The dataset has 10 possible action classes that are exceptionally suited to video surveillance applications. We utilize a Cross-View validation strategy employed by [65], where we leave out the last camera view.

In order to truly evaluate the end-to-end system noise applicable to video surveillance applications, we run the RGB images of this dataset though REVAMP²T, with EfficientHRNet-H₀ [37] and the MobileNetV2 [17] feature extractor used in the original REVAMP²T paper [36]. This produces spatial noise in the keypoints due to the neural pose estimator and frame-level temporal noise brought by the Re-ID model, in addition to an extreme latency constraint of $W=10$.

Due to Re-ID varying the number of people detected and the natural temporal variance of the dataset, the input tensor $\mathbf{X}\in\mathbb{R}^{N\times T\times C\times M}$ has widely varying $T$ and $M$ dimensions. We utilize a dynamic batch preprocessing step to deal with this which first pads the $T$ dimension with repeating clips until it is the length of the largest clip in the batch. The $M$ dimension of the tensor represents multiple people. Due to the error prorogation of Re-ID into action recognition, we potentially have single-person clips displaced throughout the $M$ dimension in a disjoint and irregular manner. To solve this the dynamic batching will adaptively sum the input to a smaller $M=\lambda$. If $\lambda=1$, we call this the ID agnostic scenario where we are no longer modeling the error propagation from Re-ID. In the case of a sample having $M<\lambda$ we zero pad the tensor until $M=\lambda$.

For the training, all consensus models were trained for 30 epochs initially then go onto 60 epochs with a standard learning rate decay, while the feedback models are added on top of the consensus model’s weights at epoch 30 and are trained for an additional 40 epochs. Due to the temporal noise injected by dynamic batching, we utilize a batch size of $8$. The validation batch size is set to 1 to minimize the temporal noise brought by dynamic batching. This set of experiments additionally utilizes a learnable multiplicative gate attached to each FB-Augmented attention, initialized at zero. This enables the feedback augmented models to be integrated into the consensus models smoothly.

Table 4 shows the accuracy results over noisy-input on the NW-UCLA action dataset. It also compares our approach against FGCN [65], an existing work that also incorporates feedback, as the SotA. We surpass SotA by $1.3\%$ in the absence of both spatial keypoint noise and Re-ID noise. Furthermore, the results show that our RW-GCN-SF is able to contend with the FGCN architecture despite incorporating a latency constraint of $W=10$, noisy keypoints from EfficientHRNet-H0, and frame-level temporal noise due to the REVAMP²T Re-ID. We report $90.4\%$ accuracy, only $4.9\%$ away from FGCN’s SotA. Additionally, Yang et al. [65] reports their accuracy with what we would consider a latency constraint of 64 frames (as opposed to our 10 frames), from which they focus on randomly sampling.

We believe that our ability to achieve over $90\%$ accuracy despite the aforementioned noise is critical to real-world skeleton action recognition and smart edge video surveillance applications. This is further reinforced by our results in Table 3, which demonstrate the inability of existing works to adapt to the receptive field level noise of latency constraints. Additionally, we go on to analyze the difference in accuracy between control feedback and semantic feedback models. Table 2 highlights that control feedback performs worse than even the baseline consensus model, and fails to work with semantic feedback. Our primary hypothesis is that control feedback failed to perform as effectively as semantic feedback due to RW-GCN-SF modeling pairwise bi-linear feature interactions, while the RW-GCN-CF variant only modeled linear interactions. This is further exacerbated by the lack of temporal redundancy in the features due to the extreme latency constraint. Furthermore, when analyzing temporal noise injected by dynamic batching (by setting the validation batch $N>1$) control feedback methods achieved $84\%$, as opposed to semantic feedback’s $83.1\%$. Our logical conjecture is that the irregular replication of features, through the dynamic batching process, imitates a noisy recurrent feedback where the same features are pushed through the same network weights repeatedly, and that acts as a form of information processing. This form of information processing is more akin to how traditional recurrent neural networks processed temporal information. It could be said our RW-GCNs exist in a family of models where the reliance of network depth and recurrent processing differs for each variation. The AGC-LSTM [54] model would be a variation that is the opposite of our model with a heavy reliance on recurrence and attention, but little reliance on network depth. Control feedback may be more reliable than semantic feedback for models such as AGC-LSTMS, as it controls information propagation.

The last analysis of the Northwestern-UCLA dataset [61] is an ablation study on the effect of Re-ID error propagation. We test RW-GCN architectures in two scenarios. The first is an ID agnostic scenario where the RW-GCN attempts to artificially remove temporal noise by summing across the $M$ dimension of the input to rejoin the displaced temporal features into one single $1\times T\times V\times C$ tensor. While this would theoretically test poorly on crowded scenes with multiple people performing separate actions, the Northwestern-UCLA dataset [61] has very few such samples. We see our best performance with the RW-GCN-SF work achieving $90.4\%$ action recognition accuracy. In the second scenario, we analyze real-world action recognition with respect to Re-ID noise. We vary the hyperparameters that weigh the temporal IoU filter against the appearance feature distance. We find that RW-GCN-SF is the most resilient at $71.8\%$ accuracy. An interesting aspect of our results shows that the appearance centric Noise 1 is overall more inhibiting to RW-GCNs. While this may be just an aspect of the dataset, we believe that this merits future exploration.

Overall, we conclude that the zero-padding of the input feature along the $M$ dimension results in too much noise, which brings down the accuracy of all models in the real-world scenario. It was expected that IoU based Re-ID would outperform appearance-focused Re-ID, however, marginal improvements are achieved from the top network in the appearance-based noise test.

The cross-analysis of our on-the-edge real-time performance metrics is shown in Table 6. The results were measured on a real-world edge device, the Nvidia Jetson Xavier NX. In order to fairly evaluate real-time performance, two metrics were introduced in Section 3; Actions per Second ($ApS$), calculated by equation (1), and Action Product Delay ($APD$), calculated by equation (2). It is important to note that $ApS$ is device specific, while $APD$ is end-to-end system specific.

Our model intuitively supports higher $ApS$ as it produces multiple action classifications in a clip. While some may disparage the necessity of multiple outputs per clip when considering a real-world scenario, it is important to acknowledge that the speed at which actions are performed is both unknown and data dependent. Additionally, action recognition models are typically not aware of the input frame rate (unless trained to be [14], which most works are not). This leads to the necessity of supporting larger $ApS$ than needed in ideal training scenarios, and provides future potential to train a model to be agnostic to varying input frame rates (which, when left unchecked, could lead to accuracy degradation). RW-GCN reports the highest performance at $41.7$ $ApS$, which is $1.6\times$ higher than existing works. It is seen that the overhead of semantic feedback in terms of $ApS$ and parameters is minimal. As mentioned in Section 4 semantic feedback was designed to be lightweight, only working on 32 channels. Shift-GCN is the most parameter efficient model as it was explicitly designed for lightweight and efficient inference, by utilizing data movement across time steps. This model design is not mutually exclusive with RW-GCNs, and in the future can be explored.

It is important to note that the $ApS$ of RW-GCN models varies wildly with respect to frame size. This is due to the execution bias of GPU-based systems. As we decrease window size, the data-dependent nature of RW-GCNs serializes and losses performance efficiency. On the other end of the spectrum, when the window size is increased we end up with the same problem as other works due to needing more frames to compute a single action. The $APD$ calculation is dependent on EfficientHRNet’s H₀ FPS of 22.95 [37]. RW-GCN achieves the lowest $APD$ of $0.4$ seconds with our extreme latency result derived from the Northwestern-UCLA dataset [61].

Lastly, we analyse the scalability of RW-GCNs on lightweight and inexpensive edge devices. We focus on the Nvidia Jetson Nano as opposed to the Nvidia Xavier NX, which was previously used for the results shown in Table 6. The Jetson Nano is nearly $10\times$ less expensive than the Xavier NX as of the writing of this paper. The price is an important factor when determining the scalability with respect to system costs, and in fact democratizing AI through affordability. However, the Nano is much more tightly resource constrained, with only a 128-core Maxwell GPU compared to the Xavier NX’s 384-core Volta GPU and 48 Tensor cores. This is in addition to the greater RAM bandwidth ($\approx 2\times$), and greater memory capacity ($\approx 4\times$). We use $ApS$ to measure how our compute maps to such a constrained environment. We calculate $ApS$ with a clip size of 30 and a window size of 10. We see a $2.67\times$ drop in $ApS$ with only two people in a single scene. This deficit increases to $7.6\times$ less $ApS$ when scaled to 8 people. We believe this to be more than adequate for real time action detection, and in fact believe that further trade offs of $ApS$ are exploitable for supporting scenes with larger amounts of people. Theoretically, we can sequentially execute across the number of people similar to our sliding window solution to large temporal frames. This could potentially allow us to process scenes with 36 people with an ApS of $1$. Future directions can explore further optimization through either offloading deeper layers to a server similar to [39]. By choosing to offload deep layers privacy is preserved. Another direction is utilizing input adaptive compute, similar to the work down by Fang et al. [13].