Parallel Detection for Efficient Video Analytics at the Edge

We study the real time object detection at edge based on the following assumption. First, let the incoming video rate of a live video stream be $\lambda$ frames per second (FPS), and let the detection throughput rate of an object detector deployed on an edge device be $\mu$ frames per second. When $\mu\geq\lambda$, it indicates that the incoming video stream at $\lambda$ rate can be processed in real time with no frame dropping. However, in many cases, given the limited computing resources on an edge device, including the AI hardware for running the pre-trained DNN object detection model, the detection processing rate $\mu$ tends to be much smaller than the live video streaming rate $\lambda$. When $\mu<<\lambda$, e.g., $\lambda=30$ FPS but $\mu=3$ FPS, we observe that even with well-trained object detectors (SSD or YOLOv3) deployed on an edge device with an AI hardware such as Intel NCS2 (Neural Compute Stick 2) [8], the actual online object detection at edge may suffer from low frame processing rate, denoted by $\sigma$. An obvious performance optimization goal is to speed up $\sigma$ as much as possible by maximizing resource and computation capacities. Consider the two ends of the spectrum in implementing and optimizing the video frame processing rate $\sigma$ for object detection at edge: (1) if we want to maintain zero frame dropping, then we limit $\sigma$ to the offline detection processing baseline rate $\mu$, i.e., $\sigma=\mu$. When the incoming video stream rate $\lambda$ is much higher than $\mu$, i.e., $\mu<<\lambda$, the online detection will suffer from the low video processing rate ($\sigma$) and hence low throughput in detection output FPS; and (2) if we want to speed up the actual video processing rate $\sigma$ at edge, we would need to develop resource and computation optimizations such that we can offer fast detection processing rate, that is either the same as the live video stream input rate $\lambda$, e.g., $\sigma=\lambda$, or increasing $\sigma$ towards $\lambda$.

Naïve Approach. When the actual processing rate of the detection model on an edge device is much lower than the input video streaming rate $\lambda$, e.g., say $\mu=2.5$ FPS whereas $\lambda=30$ FPS, if the online detection scheduler simply sends the incoming video frames to the AI hardware for object detection as they arrive at $\lambda$ rate, then the online detection executor may end up dropping some video frames randomly at approximately $(\lambda-\mu)$ frames per second. This translates to dropping $(\lambda/\mu-1)$ frames on average for every frame processed by the object detector running on the AI hardware attached to an edge device. This may result in decreasing the mean average precision (mAP) for object detection quality measured over all frames of the input video. Such performance degradation can be further aggravated when the mismatching gap between the input video stream rate $\lambda$ and the detection processing rate at edge is much larger due to a larger number of randomly dropped frames (see experimental results in Section IV).

In this subsection, we use a benchmark video ETH-Sunnyday from the MOT-15 dataset [15], which has a total of 354 frames, and is taken from a moving camera at 14 FPS with a resolution of 640$\times$480. We install YOLOv3 object detection model trained using DarkNet-53 [5] on an edge server with Intel NCS2 [8]. We first resize the input video frame to the input size of the object detection model, that is 416$\times$416$\times$3 for YOLOv3. Then, the online object detection module running YOLOv3 will process each resized video frame by performing perform object detection inference, followed by post-processing, such as non-maximum suppression (NMS) [6, 4, 5], for computing the bounding box, the class label and the object confidence for each detected object. By applying post-training quantization to use 16 bit half-prevision floating point format (FP16), YOLOv3 is converted into the FP16 data type in order to be deployed on Intel NCS2 sticks with a smaller model size of 119MB instead of over 200MB.

Figure 2 shows the object detection results for four consecutive frames when we use the frame processing rate that will incur zero frame dropping as done in the offline detection, i.e., keeping a large buffer that is able to hold the input video frames at $\lambda$ rate, while scheduling one frame at a time to ensure no frame dropping, resulting in very low detection processing rate $\sigma$ with $\sigma<<\lambda$. Even though in this case, the object detector can successfully identify the objects in each frame, the actual video processing rate $\sigma$ is only 2.5 FPS, which is 5.6x slower than the original ETH-Sunnyday live video stream rate of 14 FPS ($\lambda=14$). We also provide a visual comparison of this online detection processing with no frame dropping (i.e., $\mu=2.5$ FPS) with the original input live video stream rate. The visualization shows the shape contrast in https://github.com/git-disl/EVA, our EVA project site.

Figure 3 shows the online object detection processing, which aims to approximate the input video stream rate, for the video ETH-Sunnyday using the same four consecutive frames as shown in Figure 2. Here the live video stream rate of 14 FPS is used to feed video frames to the online object detection executor (recall Figure 1b). In this case, the online object detection executor experiences random frame dropping at the rate of 5 frames on average for each detection processed frame, i.e., $\lceil 14/2.5-1\rceil=5$. The random frame dropping happens when the object detection scheduler is sending incoming frames to the object detection executor at the input video rate of 14 FPS ($\lambda=14$), while its actual detection processing throughput is only 2.5 FPS (i.e., $\sigma=2.5<<\lambda$). By comparing Figure 3 with random dropping of input frames and Figure 2 with zero frame dropping, it is visually clear that in the four consecutive frames (64 to 67) of Figure 3, a few bounding boxes are misaligned, and some buildings are mislabeled as person or bicycle (e.g., those on the left). As a result, the mAP of this online detection executor is reduced from 86.9% to 66.1% (see more detailed experimental results in Section IV). Readers can find a visualization of the comparison on our GitHub project page (https://github.com/git-disl/EVA), showing the negative effects of random frame dropping on both FPS and mAP of the detection outputs through visual comparison of input video stream with the detection output video stream.

Our multi-model parallel detection framework consists of three main functional components, which work in concert to explore the runtime detection parallelism at edge and to speed up the overall frame processing rate for object detection by reducing the performance gap with respect to the live input video stream rate $\sigma$. Figure 4 shows a sketch of the overall workflow for fast online object detection by exploiting multi-model parallelism. The multi-model detection scheduler and the sequence synchronizer are the two core components, which work together to enable parallel detection processing of multiple input video frames at the same time while maintain the temporal order of the detection processed frames according to the original input video stream sequence.

The multi-model parallel detection scheduler will first determine the number of parallel detection models to use for delivering the near real-time frame processing performance, denoted by $\sigma_{P}$, based on the input video stream rate $\lambda$ and the average single model detection processing rate $\mu$. Then the scheduler will create parallel execution thread pool, one for each model. Finally, the scheduler will choose an adequate scheduling algorithm based on the configuration of the edge system and its edge devices. There are several ways to exploit multi-model parallelism in an edge system, and one often complements another. First, we can explore multiple models running on one edge device by installing multiple AI hardware sticks as attachments, such as Intel NSC2 sticks [8]. Intel NSC2 is a hardware accelerator for deploying deep learning at the edge, which is built on the Intel Movidius VPU (Vision Processing Unit) and a dedicated neural compute engine with 4GB memory. It has the USB plus and play feature, enabling its wide deployment on many edge devices for accelerating DNN inference, supported by Intel OpenVINO software framework [12]. Note that the implementation of multi-model on one edge device will focus primarily on optimizing the model execution parallelism for maximizing the overall performance speed-up. Second and alternatively, we can explore multiple edge devices with one AI hardware like NCS2 attached to each device to run an object detection model. Additional performance factors need to be considered for this approach. For example, we need to design an algorithm to find sufficient edge devices within geographical proximity or with fast network connection to the leader edge device which initiates this video detection job. Then we need to define optimal workload balance for achieving overall performance speed-up when multiple nearby edge devices have initiated their own video detection jobs. The third alternative is to consider edge devices with heterogeneous AI hardware attachments, each is capable of running detection models of different size and complexity or different number of detection models. In summary, the first approach serves as a baseline, as it runs multiple detection models in parallel on one edge device attached with a USB 3.0 hub connecting multiple AI hardware sticks. The second alternative uses multiple nearby edge nodes to run multiple detection models instead, with each edge node attached with one AI hardware stick, running one object detection model. The third alternative addresses heterogeneous edge devices by implementing a hybrid of the first two approaches.

In the remaining of this section, we will describe our parallel multi-model detection architecture and algorithms in terms of the first design alternative, focusing on the design decision on the number of parallel detection models (the parameter $n$), the choice of parallel detection scheduling algorithms and the scheduler implementation, and the sequence synchronizer for achieving real time or near real time processing throughput $\sigma_{P}$. We report our experimental evaluation in Section IV, with different edge node configurations and up to seven Intel NCS2 sticks as AI hardware attachments to an edge device.

The first task performed by our parallel object detection scheduler is to determine the number of detection models ($n$) to use in order to achieve the desired online detection performance in terms of both frame processing rate FPS and mAP. The goal is to choose the right setting of the parallel detection parameter $n$ such that it is cost efficient. Too small $n$ may not be sufficient to achieve near real time detection processing throughput in FPS, whereas too large $n$ may result in resource waste when the parallel detection processing capacity $n\times\mu$ is far beyond the input video stream rate $\lambda$, i.e., $n\times\mu>>\lambda$.

There are several factors impacting the choice of $n$, such as the computation capacity of the AI hardware attached to an edge device, the size and complexity of the pre-trained object detection model, the CPU and memory of the edge device, assuming that the number of AI hardware attachments via a USB hub connecting to an edge device is homogeneous. Let $\sigma_{P}$ denote the detection processing rate achieved by parallel execution of the $n$ detection models. One ideal scenario will be when the parallel object detection can scale linearly with the number of parallel models (or edge devices). As a result, the overall frame processing rate $\sigma_{P}=n\mu$, where $\mu$ is the per model detection processing rate, or the average detection processing rate of the $n$ detectors. $\sigma_{P}=\sum_{i}\mu_{i}$ when different models and/or heterogeneous edge devices are used, and each with its specific detection processing rate $\mu_{i}$ ($1\leq i\leq n$). If we want to match or approach the original input video stream rate $\lambda$, we can set $n=\lceil\lambda/\mu\rceil$, which ensures that $\sigma_{P}=n\mu\geq\lambda$ holds. With this approach, we can determine the right $n$ detection models to process the incoming video stream with zero frame dropping, because the parallel execution of $n$ detection models will be sufficient for online object detection in real time and no frame dropping will be in this ideal scenario. Given that the comfort of human perception for street view of walking pedestrians is around 10-30 FPS, one may further relax the ideal scenario and set $n$ in the range of [$\lceil 10/\mu\rceil$, $\lceil\lambda/\mu\rceil$]. From observations made using our first edge system prototype, implemented in C++ on top of OpenVINO [12] and OpenCV [16], we confirm that when the input video stream rate $\lambda$ is higher than 12 FPS, it is effective to choose $n$ from the range of [$\lceil 10/\mu\rceil$, $\lceil\lambda/\mu\rceil$] instead of using the conservative setting of $n\geq\lceil\lambda/\mu\rceil$, if we aim at achieving near real-time object detection performance in FPS with high detection accuracy in mAP.

Recall the benchmark live video ETH-Sunnyday from the MOT-15-dataset [15] taken from a moving camera at 14 FPS (Figure 2 and Figure 3), if the detection processing rate for no-frame dropping ($\mu$) is 2.5 FPS when running YOLOv3, and if we want to determine the parameter $n$, aiming to closely match the input video streaming rate $\lambda=14$, we can achieve the parallel detection processing rate $\sigma_{P}$ by choosing $n$ from the range of ($\lceil 10/2.5\rceil$, $\lceil 14/2.5\rceil$), which is [$4,6$]. We get $\sigma_{P}=10$ FPS when we choose $n=4$ by running four detection models in parallel, achieving the detection processing throughput close to the incoming video stream rate $\lambda$, which is 14 FPS, and mAP of 86.5%. We call this choice of $n$ the near real time parallel detection approach. Alternatively, if we choose the upper bound of the range, which is $n=6$, then we achieve $\sigma_{P}=14.8$ FPS with mAP of 86.9%, indicating that by running 6 detection models in parallel, the detection processing throughput can match or slightly exceed the input video stream rate $\lambda=14$ with high mAP. We refer to this approach as a conservative real time parallel detection approach. Readers may refer to Table IV in Section IV for more details.

The second task performed by the parallel detection scheduler is also a part of the initialization process. It needs to choose a scheduling algorithm as a part of the runtime configuration of the scheduler. Concretely, upon receiving the input video stream at $\lambda$ rate, an edge device relies on its object detection task scheduler to launch $n$ threads to run $n$ object detection models in parallel. The assignment of incoming frames to $n$ detection models is done by a scheduling algorithm. There are three categories of parallel execution scheduling algorithms: Round Robin (RR) or weighted round robin, First Come First Serve (FCFS), and performance aware proportional scheduler.

A round robin scheduler will assign incoming video frames to the $n$ detection models, one at a time, in a pre-defined round robin order at the input stream rate $\lambda$. For example, consider the input video stream $V$, the RR scheduler will dedicate a separate thread to receive the incoming video frames and place them into a shared parallel execution queue $Q$. At the same time, the RR schedule will launch $n$ dedicated threads, each runs one of the $n$ object detection models in parallel. Each thread will fetch one video frame from the shared queue $Q$ and send it to its corresponding object detection model for parallel detection processing. Upon completing one round of input frame to detection model assignment for all $n$ threads, the next round starts. If the current detection model is still busy in processing a previous frame, then the current frame will be dropped, and the detection results from the latest processed frame will be reused as the detection approximation for this dropped frame. The video frame sequence synchronizer will enforce temporal input stream ordering over both the detection processed frames and the randomly dropped frames prior to displaying the live streaming of the detection processed output results via visualization or video streaming player. When the input video stream rate $\lambda$ is much faster than the per-model detection processing rate $\mu$, the round robin scheduler may incur larger amount of random frame dropping. A weighted round robin typically refers to a static resource adaptive scheduler. It assigns a higher weight to those detection models that run much faster, for example, due to high-capacity AI hardware known at edge device configuration, such as a high end GPU card instead of a low end Intel NCS2 stick. Compared to the round robin baseline, the static weighted RR scheduler can further improve the parallel detection throughput while maximizing execution parallelism.

A first come and first serve (FCFS) scheduler will assign incoming video frames to the $n$ detection models one at a time in the first come first serve order. Concretely, it starts by assigning $n$ input frames to the $n$ available detection models. Then it will assign the $(n+1)^{th}$ input frame to the first detection model that becomes available, instead of following a rigid round robin order for parallel detection processing. The FCFS scheduler is particularly suitable for the scenarios in which either the $n$ detection models or the $n$ AI hardware attachments are heterogeneous, and have different runtime performance for object detection processing.

The performance aware proportional scheduler is a dynamic runtime performance aware scheduler. It extends the round robin scheduler by periodically computing and assigning dynamic weights to the $n$ detection models. Concretely, upon assigning the first $n$ input frames to the $n$ detection models, it will start to monitor the execution time of each detection model and periodically compute the weight for each of the $n$ detection models based on the historical statistics on the execution performance of the $n$ models with respect to detection processing efficiency. Unlike the weighted round robin, which assigns a constant weight to each detection model at compile time, the performance-aware proportional scheduler dynamically assigns a weight to each detection model, enabling the parallel detection scheduling decision to take into account multiple dynamic factors that may impact on runtime performance, such as network bandwidth, edge device execution state, runtime state of AI hardware.

Based on the chosen scheduling algorithm, the edge parallel model detection scheduler will schedule the $n$ threads, one for each detection model. The processed video frames will be sorted by the sequence synchronizer based on their original temporal streaming sequence for visualization and video player presentation of the detection processed frames, including detected objects in bounding boxes with class labels and confidence statistics. Note that for first come first serve and proportional scheduling algorithms, the parallel detection model scheduler will assign frames to threads in an opportunistic manner. The sequence synchronizer will be responsible for enforcing the temporal sequencing constraints on the detection processed frames before forwarding them to the visual presentation phase. Although with round-robin, the temporal sequence constraint is followed by the parallel detection scheduler, the sequence synchronizer serves as an additional checkpoint to ensure and enforce the temporal sequence of the output frames, which is important in the presence of random frame dropping.

Three interesting observations are highlighted. (1) Our model parallel approach achieved almost linear scalability with the number of NCS2 sticks. Given one detection model runs on each of the 7 NCS2 sticks, the parallel detection with 7 models significantly improves the detection processing throughput ($\sigma_{P}$) from 2.3 FPS to 16.0 FPS for pre-trained SSD300 model and from 2.5 FPS to 17.3 FPS for pre-trained YOLOv3 detector. This represents 6.96$\times$ speed-up for SSD300 and 6.92$\times$ speed-up for YOLOv3, compared to using a single detection model on one NCS2 stick. (2) Based on our proposed method for determining the detection parallel parameter $n$ in Section III-B, for ETH-Sunnyday with the input video stream rate of 14 FPS, setting $n$ to be in the range of [$4,6$] is sufficient for delivering near real time detection processing throughput. This experiment confirms this method. For $n$ NCS2 sticks, running $n$ YOLOv3 detection models in parallel, by varying $n$ from 4, 5 to 6, we can achieve a detection processing rate of 10.0 FPS, 12.4 FPS or 14.8 FPS respectively with mAP of 86.5% or higher for all three cases. For street view of walking pedestrians, the moving camera capturing at 10 FPS or higher is considered comfortable and near real time for human perception. Hence, choosing $n$ by 4, 5, or 6 are considered acceptable configurations for our multi-model detection parallel approach. (3) By choosing $n=7$, the parallel detection processing rate exceeds the input video stream rate of 14 FPS for both SSD300 and YOLOv3, achieving the same mAP: 74.5% for SSD300 with 7 NCS2 sticks and 86.9% for YOLOv3 as the zero frame dropping baseline. Clearly, adding one additional AI hardware will consume additional computation cost, including battery consumption if edge device is not wired, although the computation time is negligible thanks to parallel execution of all $n$ models.

Figure 5 shows the trend of FPS (y-axis on the left) and mAP (y-axis on the right) for running SSD and YOLOv3 pre-trained models, by varying $n$ from 1 to 7 (x-axis). First, both SSD and YOLOv3 show the linear scalability with respect to the detection processing rate in FPS. Second, for YOLOv3, as $n$ increase from 1 to 4, the multi-model detection parallel approach continues to increase its mAP. As $n$ continues to increase from 4 to 7, the mAP is stabilized at 62.7%. For SSD, the multi-model detection parallel approach always achieves mAP higher than the no-frame dropping baseline of 54.4%, with $n\geq 2$. For SSD300 on ADL-Rundle-6, our multi-model parallel detection with 3 NCS2 sticks has the detection processing rate of 6.9 FPS. This results in a small amount of random frame dropping, which is on average $\lceil 30/6.9-1\rceil=4$ frames for each detection processed frame, compared to random dropping of 13 frames ($\lceil 30/2.3-1\rceil=13$) when using one detection model running on a single NCS2 stick. For YOLOv3, when using 5 NCS2 sticks, the number of randomly dropped video frames decreases significantly from on average 11 frames for each detection processed frame ($\lceil 30/2.5-1\rceil=11$) when using one NCS2 stick to only dropping on average 2 frames ($\lceil 30/12.5-1\rceil=2$) for each detection processed frame. This also explains the reason for the improved mAP for both SSD300 and YOLOv3 since the number of dropped video frames is significantly reduced.

We provide a visualization comparison to show the effectiveness of our multi-model parallel detection approach with respect to the input video stream rates, accessible at https://github.com/git-disl/EVA.

Edge devices are often with wireless connectivity and limited power supply. Therefore, energy efficiency is an important consideration for video analytics in edge systems. Table VI shows the power consumption of the AI hardware attached to the two types of edge servers (fast and slow) used in the experiments reported in this paper, including Nvidia GTX TITAN X (the popular GPU), Intel NCS2, AMD A6-9225 (the slow server), and Intel i7-10700K (the fast edge server). We run the YOLOv3 object detection model on these four types of edge devices and measure the energy consumption and FPS as reported in Table VI. We highlight two interesting observations. First, different AI devices consume different amounts of power. We use the TDP (Thermal Design Power) to indicate the power dissipation of each AI device. For example, using one Intel NCS2 stick only consumes 2 watts. In comparison, using a single GTX TITAN X GPU has the highest energy consumption with 250 watts. When running YOLOv3 on edge server with slow CPU without AI hardware, it consumes 15 watts. In comparison, when running on edge server with fast CPU, it consumes 125 watts. Second, Intel NCS2 achieved the highest energy efficiency in terms of detection FPS per watt. Here we measure detection processing throughput in FPS with zero frame dropping for all four detection execution environments at edge. Although edge server equipped with Titan X GPU achieves 35 FPS, followed by the fast CPU with 13.5 FPS, Intel NCS2 with 2.5 FP. The slow CPU edge server has the slowest detection performance of 0.4 FPS. Based on both TDP (Watts) and FPS for all four settings, we next measure the energy efficiency in terms of detection FPS per watt, which is computed by dividing the detection FPS by the corresponding TDP. The edge device with Intel NCS2 offers the highest detection FPS per watt (1.25) compared to the edge device with GPU (0.14), edge device powered by fast CPU (0.11) and edge device powered by slow CPU (0.03). This further verifies that edge AI hardware such as Intel NCS2 stick is competitive in edge computing with high energy efficiency and suitable for deploying deep neural network models on wireless edge devices.

This set of experiments is designed to measure the impact of using different parallel detection scheduling algorithms. We focus on comparing the baseline round robin (RR) scheduler with the first come first serve (FCFS) scheduler. Table VII reports the experimental results with these two schedulers by running YOLOv3 on the ETH-Sunnyday video. We highlight three interesting observations.

First, consider NCS2 Only scenario, in which homogeneous AI hardware attachments are used with each running YOLOv3 on ETH-Sunnydy test video, the RR and FCFS schedulers show similar performance as expected when performing multi-model parallel detection using only $n$ NCS2 sticks by varying $n$ from 1 to 7, where the detection FPS for both RR and FCFS schedulers ranges from 2.5 to 17.3 FPS.

Second, when we carry out the multi-model parallel detection using heterogeneous edge devices, we expect different performance impact when using the RR schedule compared to using the FCFS scheduler. In the next experiment, we use Intel i7 (Fast CPU) as the edge server with $n$ NCS2 sticks by varying $n$ from 1 to 7, we perform the multi-model parallel detection using both fast CPU and the $n$ NCS2 sticks. We deploy YOLOv3 on CPU via the OpenVINO framework. This set of experiments shows that compared to the RR scheduler, the FCFS scheduling algorithm achieves much higher detection FPS with $n$ NCS2 sticks when varying $n$ from 1 to 7. This is because FCFS is in general more efficient for heterogeneous execution environments. Concretely, the fast CPU has much higher detection processing throughput (13.5 FPS) compared to that of a NCS2 stick (2.5 FPS). FCFS can effectively leverage such performance disparity whereas the RR scheduler fails to take into account such detection performance disparity when scheduling parallel detection execution.

Third, with the Fast CPU as the edge server and 7 NCS2 sticks, the FCFS scheduler can consistently achieve higher throughput performance of 29 FPS, compared to the performance of 20.1 FPS by the RR scheduler under the same runtime setup. We next conduct the same set of experiments by replacing the fast CPU with slow CPU. It is observed that even with the slow CPU with poor detection of 0.4 FPS, using FCFS scheduler, our multi-model parallel detection approach can still deliver slightly better performance compared to using NCS2 only, because the slow CPU with NCS2 solution benefits from one additional model for parallel object detection. However, when using the RR scheduler, the slow CPU with NCS2 will suffer notably because the poor performance of the detection model running on the slow CPU had significantly negative impact on the overall performance of the parallel detection approach due to the large amount of random frame dropping at the slow CPU.

In summary, FCFS scheduling algorithm provides more consistent performance for the multi-model parallel detection approach under both homogeneous and heterogeneous runtime environments. In our first prototype implementation, we set FCSC as the default scheduler.

To install multiple AI hardware sticks to an edge server, one would need a USB hub. In our first prototype, we connect multiple Intel NCS2 sticks to the edge servers via the USB port with a USB hub. Different versions of USB ports have different bandwidths. For instance, the bandwidth for USB 3.0 is 5 Gbps/s, which is much higher than USB 2.0 with about 0.5 Gbps/s [21]. By connecting AI hardware such as an NCS2 stick to an edge server, the computation heavy DNN model for object detection will be hosted on the AI hardware for detection processing. Hence, the edge server will need to transfer the input video data to the attached NCS2 stick to perform the object detection processing. Hence, the connection bandwidth between the edge server and its attached AI hardware is critical. For example, we can connect the multiple Intel NCS2 sticks to the fast edge server via USB 2.0 or USB 3.0, and the two USB ports have different connection bandwidth, which subsequently impacts the detection processing throughput performance in FPS, as shown in Table IX. This set of experiments reports the results by running pre-trained object detection models by SSD300 and YOLOv3 on ADL-Rundle-6 test video. We highlight two interesting observations.

First, for both SSD300 and YOLOv3, edge detection with USB 3.0 achieves much higher performance in FPS than the same edge server with USB 2.0 port regardless whether we run object detection using only one NSC2 stick or using multiple NCS2 sticks. USB 3.0 port outperforms USB 2.0 port in both bandwidth and FPS throughput. Second, the input data size for YOLOv3 is 416$\times$416$\times$3=519,168 which is about 2$\times$ of the input size of SSD300 with 300$\times$300$\times$3=270,000. YOLOv3 has higher bandwidth requirements than SSD300, which also explains that YOLOv3 fails to further improve the detection FPS after #NCS2=5 with USB 2.0.

This set of experiments motivates us to consider the high speed connection offered by 5G or 6G networks. Table VIII compares the network bandwidths with different types of connections. The reported peak bandwidth of 5G network could reach 20 Gbps [24], followed by 10 Gigabit Ethernet [22] and WiFi 6 with 10 Gbps [23]. With guaranteed 10 Gigabit or higher network connectivity, our multi-model parallel detection approach can run effectively on a group of nearby edge nodes, with one edge node running one object detection model. However, with Ethernet [21] or 4G network with 1 Gbps [24] connection, using multiple AI hardware sticks on a single edge server with a USB 3.0 hub represents a more efficient approach in terms of both detection performance and energy consumption.