VegaEdge: Edge AI Confluence Anomaly Detection for Real-Time Highway IoT-Applications

Average Displacement Error (ADE) based Anomaly Detection is a method of computing the average error from the predicted trajectory to assess the accuracy of trajectory predictions for vehicles and comparing it against a threshold ($T^{ADE}_{anomaly}$) as per the application. It computes the average Euclidean distance between predicted trajectories ($\hat{F}$) and actual trajectories (${P}$) overall predicted time steps ($T_{pred}$) and subjects (${n}$) in a scene. The ADE in predicted trajectory from last $T_{past}$ seconds is compared with the desired ADE threshold, $T^{ADE}_{anomaly}$ as:

where ${t=1,...,T_{in}}$ is time and, ${v\in{1,2,...,n}}$ representing the index of the vehicle. By setting a threshold, a criterion is set to identify anomalous trajectories. Value exceeding the threshold indicates a significant disparity between the predicted and ground truth trajectories, resulting from unexpected driving behavior, going off the lane, etc. As PishguVe is designed to predict the normal trajectory of the ego vehicle, the ADE for any vehicle that exceeds the threshold is marked as an anomaly.

Angle-based Anomaly Detection calculates the angle between the predicted future trajectory vector, $\hat{F}_{v}$ and the actual trajectory vector, $P_{v}$ of the ego vehicle and compares it with a threshold according to the application. Given the x and y coordinates of $\hat{F}_{v}$ and $P_{v}$ for past few seconds $t_{past}$, we can compute the direction vectors:

, here $x_{v,f}^{0}$ and $x_{v,f}^{0}$ are the position of vehicle a frame before the start of prediction. The angle between these direction vectors $D_{P_{v}}$ and $D_{\hat{F}_{v}}$ is compared with the threshold, $T^{Angle}_{anomaly}$ as:

where $D_{P_{v}}\cdot D_{\hat{F}_{v}}$ is the dot product of $D_{P_{v}}$ and $D_{\hat{F}_{v}}$, and $\|D_{P_{v}}\|$ and $\|D_{\hat{F}_{v}}\|$ are their respective magnitudes.

For efficient and rapid vehicle detection for edge-integrated IoT devices, we opt for the YOLOv8l [30] model and trained it on the BDD100k vehicle dataset [31]. Our decision was influenced by the system’s overall performance, such as latency, accuracy, and memory requirements.

Model size and performance are critical in edge deployments, particularly for IoT devices. YOLOv8l addresses this by being 35.9% smaller than YOLOv8x [30], making it ideal for resource-limited embedded-IoT devices. Despite its reduced size, its mean Average Precision (mAP) is a competitive 52.9%, only 1% less than the 53.9% of YOLOv8x.

Multi-object tracking (MOT) is fundamentally concerned with the identities of objects within video sequences. ByteTrack [32] uses an innovative association method that considers every detection box. Detection boxes with lower scores are processed by comparing their similarities with existing tracklets to accurately identify true objects while filtering out unwanted detections.

Within this context, the VegaEdge employs the ByteTrack algorithm, renowned for its efficient and robust tracking capabilities. ByteTrack’s architecture uses deep association techniques, ensuring consistent tracking across frames, even for challenges like occlusions and complex interactions. Its performance ia shown in Table I on datasets like BDD100K and MOT20. Furthermore, ByteTrack boasts an impressive running speed without compromising on accuracy, making it an ideal choice for real-time applications such as VegaEdge. As discussed in [32], ByteTrack achieves metrics of 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on the MOT17 test set, all while maintaining a 30 FPS running speed.

The output from detection and tracking is rigorously cleaned to ensure precise input, as the quality of the input directly dictates the accuracy of trajectory forecasts in the next step. To obtain precise vehicular trajectories, we’ve performed targeted data filtration and smoothing:

1. 1.

   Class-specific Inclusion: To eliminate potential noise from extraneous vehicular types, we selectively retain only cars, buses, and trucks.

2. 2.

   Unidirectional Movement: To focus on the flow of incoming vehicles, we exclude the vehicles operating in non-targeted directions, thereby standardizing the directional flow and reducing complexity.

3. 3.

   Temporal Presence Validation: Vehicles with transient appearances can introduce data anomalies. This validation process sets a minimum frame threshold, below which vehicular entries are deemed non-contributory and are subsequently removed.

4. 4.

   Trajectory Continuity: Despite thorough validation in previous steps, some trajectories may have missing frames. We fill such gaps through interpolation techniques, ensuring continuous and complete trajectories.

In summary, the data cleansing and validation processes outlined above are crucial in ensuring the integrity and precision of vehicular trajectories used by VegaEdge’s downstream tasks. By emphasizing class-specific inclusion, standardizing directional flow, validating temporal presence, and ensuring trajectory continuity, we lay the foundation for subsequent trajectory forecasts. This approach mitigates potential inaccuracies and fortifies our framework’s reliability, positioning it to deliver consistent and high-quality results in real-world applications for which VegaEdge will be used.

We focus on highway-centric performance in the framework of VegaEdge’s IoT applications. To meet this need, VegaEdge integrates the SotA PishguVe[15] trajectory prediction algorithm on highway datasets, ensuring fast and accurate results without straining the device.

PishguVe was selected for its ability to make predictions at the pixel level, as shown in Table I, its proven track record of state-of-the-art accuracy on multiple datasets [33, 34, 15], and its efficient memory footprint. Table I, CHD-HA and CHD-EL represents CHD-High Angle and CHD-Eye-level trajectories from CHD dataset respectively. Built on a fusion of graph isomorphism, convolutional neural networks, and attention mechanisms, PishguVe [15] can forecast future vehicle positions with a model size of only 133K parameters. The input to PishguVe is past trajectories of vehicles are represented by a set of absolute coordinates, denoted as $P_{v}$, and a set of relative coordinates, denoted as $\Delta P_{v}$. The absolute coordinates are defined as $P_{v}={(x_{v,p}^{t},y_{v,p}^{t})}$, where ${t=1,...,T_{in}}$, ${v\in{1,2,...,n}}$ representing the index of the vehicle and ${x_{v,p}^{t}}$ and ${y_{v,p}^{t}}$ are x and y coordinates of the center of bounding box of vehicle ${v}$ at time ${t}$ for past trajectory denoted by $p$. The predicted future trajectories are shown as $\hat{F}_{v}={(x_{v,f}^{t},y_{v,f}^{t})}$, here ${t=(T_{in}+1),...,T_{pred}}$, $f$ denotes future trajectory, and ${v\in{1,2,...,n}}$, are generated as a set of coordinates for each vehicle in the image.

VegaEdge uses the trajectory prediction-based anomaly detection approach discussed in section IV of the paper, utilizing ADE and angle-based anomaly detection techniques. These methods offer a straightforward and efficient approach to detecting anomalies. This streamlined process makes our proposed method well-suited for integration within VegaEdge’s IoT-based framework, which operates on hardware and time-constrained embedded IoT platforms. This efficiency allows for quick anomaly detection, enhancing the overall performance and responsiveness of the system. The performance of both approaches on two different datasets is demonstrated in the upcoming section.

VegaEdge’s performance was evaluated across multiple platforms to understand its versatility and efficiency. Our testing platforms comprised a server with an Intel Xeon Silver 4114 CPU, Nvidia’s V100 GPU, and Nvidia’s Jetson Orin and Xavier NX boards. We chose the server setup as a reference point to contrast the performance metrics of the Jetson boards. These Jetson boards are designed for real-world tasks and are notable for their power efficiency, with Orin operating at 50W and Xavier NX at just 20W. Their efficiency and AI capabilities position them as ideal candidates for a wide range of IoT applications requiring edge computing.

Transitioning from the hardware evaluation, Table I shows the performance metrics of three algorithms VegaEdge utilizes for crafting its workflow, as shown in Algorithm 1 and Figure 2. In the domain of Object Detection, YOLOv8l achieves an mAP of 52.9 on COCO and 57.14 at mAP50 on BDD100K. It further scored 34.50 at mAP50-95. The Tracking algorithm, ByteTrack performs at a MOTA of 77.8 on the MOT20 dataset. Lastly, in trajectory prediction, the PishguVe algorithm was assessed on three distinctive datasets. It registered a Pixels/ADE of 20.75 and 16.81 on CHD-EL and CHD-HA, respectively, and a commendable m/ADE of 0.77 on NGSIM.

In this section, we evaluate our anomaly detection methodology on adversarial trajectories. These trajectories are derived using the NGSIM dataset with bird’s eye camera-view of the trajectories, following the adversarial attack approach [29], adopted by [26]. Our evaluation encompasses both the ADE and angle-based anomaly detection techniques. Our tests include both adversarial generated trajectories [29] and actual real-world data from the NGSIM test dataset, similar to the study in [38].

The ADE-based anomaly detection method study revealed a distinct pattern regarding the Area Under the Receiver Operating Characteristic curve (AUC-ROC). The ADE-based anomaly detection method consistently performed best at an AUC-ROC of 0.91 for an ADE window of 0.2s, as visualized in Figs. 3 and 3a. The angle-based approach initially outperformed the ADE-based method for short prediction windows, as seen in Fig. 4a, but its efficacy declined with larger windows, evident in Fig. 4b. Such behavior in the angle-based approach may stem from how anomalies are generated by applying constrained perturbations to real-world trajectories, making anomalies challenging to discern over more extended prediction periods.

The Equal Error Rate (EER) plot in Fig. 5 shows the EER value obtained by plotting False Negative and False Positive Rate for ADE and Angle anomaly methods for 1 second of predicted trajectory. Table II presents the EER for two anomaly detection methods on NGSIM adversarial trajectories across different time-step thresholds. The Angle method outperforms the ADE approach at shorter thresholds, such as 0.2s and 0.4s. However, as the time threshold grows, their performance converges. By 5.0 seconds, the EERs are 0.26 for ADE and 0.77 for the Angle method, indicating a faster performance drop for the Angle approach over extended time steps.

In this section, we evaluate our anomaly detection approach on real-world trajectories sourced from CAD consisting of high-angle (CHD-HA) and eye-level (AHD-EL) camera-view of highway vehicles, as introduced in section III. To offer a comprehensive view of the results, the AUC-ROC values for both fine and coarse grain time steps are graphically represented in Fig. 6 for the ADE-based anomaly detection method and in Fig. 7 for the angle-based method.

Expanding on the earlier analysis, Table III provides a breakdown of the Equal Error Rate (EER) performance for various time thresholds, contrasting the ADE and angle-based anomaly detection methods on CAD’s data. It is clear that the Angle approach consistently outperforms the ADE method across all examined time steps. Starting from an EER of 0.48 at the 0.2s mark, the angle method demonstrates a steady improvement, with an EER of 0.12 at the 5s threshold. In contrast, the ADE-based method initiates with an EER of 0.58 at 0.2s and gradually improves to 0.32 at 5s. These metrics show the superior efficacy of the angle-based approach, especially since the prediction windows are elongated, making it a more robust choice for anomaly detection in the context of the CAD.

Intriguingly, the real-world trajectories demonstrate an inverse trend compared to the results of the adversarial anomaly dataset. Specifically, the AUC-ROC values for the ADE anomaly detection method peak at a 4 and 5-second prediction window, recording the area of 0.80 for a 4s window as shown in Fig. 6b. This suggests a higher sensitivity of the ADE method to longer prediction windows when applied to real-world data. Similarly, in Fig. 7b the angle-based method exhibits stellar performance with an AUC-ROC of 0.94 for the same 5s window. Such observations indicate that while synthetic or constrained trajectory datasets may favor short prediction windows, real-world trajectories might inherently contain more distinguishable anomalies in longer prediction intervals.

Comparing the EER values from the real-world CAD trajectories (Table III) with those from the adversarial NGSIM trajectories (Table II), several striking differences emerge. The NGSIM adversarial trajectories exhibit substantially lower EER values in the initial time steps, especially for the angle-based approach. For example, at the 0.2-second mark, the NGSIM data record a notably lower EER of 0.02 for the angle-based method than 0.48 for the CAD dataset.

A potential reason for this marked divergence might be the inherent nature of the datasets. The NGSIM adversarial trajectories, being synthetically generated, likely present more pronounced and discernible anomalies that the detection methods can more readily identify, especially within shorter prediction windows. In contrast, the CAD real-world trajectories, a genuine reflection of real-world driving behaviors, might be embedded with subtler and more intricate anomalies. These nuances could pose more significant challenges in detection, resulting in higher EER values, especially in the shorter time-step intervals. Moreover, the complexities and variances found in real-world driving behaviors could introduce a wider array of anomalies, making distinguishing between normal and anomalous patterns more intricate for the detection methods when applied to the CAD.

In our evaluation, we primarily focus on the performance of VegaEdge on the Jetson platforms. With its superior computational capabilities, the server platform serves as a benchmarking reference to underscore the efficiency and feasibility of deploying VegaEdge in more constrained environments.

In highway work zones, safeguarding workers from oncoming vehicles is a primary concern. Through trajectory prediction combined with anomaly detection, VegaEdge detects trajectories that may pose threats and alerts workers. To achieve this objective, the proposed design must demonstrate real-time performance on edge devices. The efficacy of VegaEdge, particularly when implemented on Jetson Orin, is exemplified in Table V.

Given an error detection window of 0.2s, VegaEdge provides a buffer time of 8.8s, corresponding to a 235m distance from a camera for a vehicle moving at 60 mph. As the detection window widens, the buffer diminishes but remains noteworthy, with 1s, 3s, and 5s windows yielding buffer times of 8s, 6s, and 4s, respectively. A clear trade-off emerges between larger buffer windows and heightened accuracy, as noted in the anomaly detection results for CAD in section VI-B.

In hazardous highway work zones, these buffer times are beneficial and vital. Even minor increments in warning time can significantly alter outcomes. VegaEdge’s ability to grant these buffers, particularly on Jetson platforms, underscores its practicality in real-world scenarios and its role in bolstering worker safety.

In this subsection, we conduct a comprehensive power consumption analysis of the Nvidia Jetson Orin platform with the primary objective of assessing the practical utility and performance of VegaEdge across various power modes. We report the total power consumed across all the channels ($P_{Total}$) calculated using the following equation:

Where $P_{GPU}$ is the total power consumed by the GPU and SOC cores, $P_{CPU}$ is the power consumed by the CPU and CV cores, and $P_{\text{IOs}}$ is the power consumed by the system’s 5V rail for various input and output ports, respectively on one of the power monitors [39]. $P_{\text{AO}}$ stands for power consumed by Always On (AO) power rail on another power monitor [39].

In Table VI, VegaEdge’s power consumption and throughput on the Jetson Orin are evaluated across different power modes for a real-world highway scenario. At the MAXN (40W) setting, the system processes 758 trajectories per second, consuming 18.14W, and reducing the power mode to 30W and 15W results in decreased throughputs of 477 and 214 trajectories per second, respectively, with corresponding reductions in power consumption. Despite the higher power demands compared to typical IoT devices, VegaEdge on Jetson Orin showcases a valuable trade-off between power consumption and high-throughput processing, making it a viable solution for edge applications requiring rapid data processing.