Edge-Assisted Lightweight Region-of-Interest Extraction and Transmission for Vehicle Perception

High-resolution images and videos are with much more pixels, which provide rich on-road information for safe autonomous driving[13]. The feature maps extracted by CNNs are semantically richer and the correlation position relationships between objects are spatially more accurate. However, there are few lightweight CNNs designed for real-time processing of high resolution images. The datasets used for training CNNs are mostly small size images, such as 224$\times$224 or 640$\times$640, which may not be applicable to vision tasks on 4K or 8K images [5]. High resolution video streams need to be downsampled before being input to CNNs, leading to accuracy drop. The architecture of CNNs could also be scaled up and trained on high-resolution datasets, at the cost of unacceptable latency [17, 20]. It takes about 49 millisecond (ms) for object detection task of a 4K image using YOLOv5x (1280) [19] on an RTX 3090 GPU, which is far from achieving the real-time inference on video streams.

Equal image partitioning is an effective way to improve the prediction accuracy, but it can not work well with limited computation and bandwidth resources[9]. Instead of reducing the amount of data transmitted to the edge, image partitioning even increases the computational burden of prediction by several times (each part of the image requires CNNs for processing), leading to high end-to-end latency. On the other hand, image partitioning without incorporating object distribution information leads to the loss of accuracy for objects at the boundary of the partitions, and the waste of resources in areas where no object exist. Therefore, we consider RoI extraction based on object location algorithm to reduce the large volume of redundant data transmitted, and improve the prediction accuracy without increasing the input size of CNNs.

In order to demonstrate the advantages of image partition, we conduct experiments on a computer with RTX 3090 GPU, and employ YOLOv5x (640) for object detection task on a 4K YouTube video. As shown in Fig. 1, the detection accuracy of our proposed system is significantly higher compared to directly feeding raw image to the CNNs. Fig. 1(b) shows the result of using CAM-based RoI extraction, which can effectively improve the inference accuracy. Hence, in this paper, we investigate low-complexity CAM-based RoI extraction method to achieve fast and high-accuracy vehicle perception.

As shown in Fig. 2, the edge-assisted real-time video analytics system consists of three components: feature extraction and cropping, CAM-based RoI extraction, and adaptive RoI box offloading. In order to reduce the computing consumption of vehicles and the amount of data transferred to the edge, we employ a lightweight feature extractor and a fixed-mode feature partition method for video frame pre-processing. Then, considering the limited on-board computing capacity, a lightweight CAM-based RoI extractor is utilized on each feature cropping to obtain a set of boxes containing the target objects and regardless of the background region. Before offloaded from vehicles to ENs, the RoI boxes are re-selected to remove invalid boxes and appointed to different down-sampling rates based on the current available bandwidth and edge resources to better balance the inference accuracy and transmission cost. In the rest of this section, we will introduce the specific details of the main component.

The CAM has remarkable object location abilities without training on any pixel-level labels [18]. For a given image $\mathbb{X}\in\mathbb{R}^{D\times(H\times W)}$, let $F_{c}(x,y)$ represents the feature of channel $c$ extracted from the last convolutional layer of the lightweight CNN at spatial location $(x,y)$. After global average pooling layer, we can get a set of scalars $\{f_{1},f_{2},...,f_{n}\}$, each of which corresponding to the mean value of a feature map $f_{c}=\sum_{x,y}{F_{c}(x,y)}$. Combined with the weight $\omega_{c}^{i}$, we can obtain the class activation map $M_{i}$ of class $i$:

where $\omega_{c}^{i}$ reflects the importance of feature map in corresponding channel $c$ for class $i$. Moreover, we can get the contribution value of each pixel to the class $i$, $M_{i}(x,y)$. Larger $M_{i}(x,y)$ indicates the area where the object is more likely to appear, thus enabling the object location based on CAM. However, it is revealed in our experiments that CAM is not good at localizing multiple objects, especially when the objects are small and distributed discretely. The missed target objects will lead to a deterioration in the overall accuracy. Therefore, in order to maximize the performance of CAM low-cost object localization, we crop the feature map extracted by the lightweight CNN into five parts $\{F_{c}^{1},F_{c}^{2},...,F_{c}^{5}\}$ and perform CAM on each cropping separately, as shown in Fig. 2(a). We can get the activation value $M_{i}^{j}$ of each part. Compared with employing CAM on the raw image directly, our method conducts feature extraction only once, while the later requires four times more computational cost.

On the one hand, convolution operation does not change the exact spatial mapping relationship between objects. Therefore, convolution followed by cropping the feature map yields the same result, as partitioning the original image first and then extracting the features separately on each part. On the other hand, CAM-based object localization on each feature cropping can help to seize discrete objects scattered in each part of the image. In conclusion, combining feature cropping with CAM for object positioning allows for high accuracy of RoI extraction with as less computational resources as possible.

After employing CAM on each feature map cropping, we can get the heatmap which shows the localization of target objects (Fig. 3(b)). Areas with higher heat values $M_{i}(x,y)$ indicate that the object is more likely to appear. Therefore, by setting a heat value threshold $\sigma_{m}$, we can obtain the mask $\mathbb{A}^{\{M_{i}(x,y)\geq\sigma_{m}\}}$ to extract the RoI, illustrated as Fig. 3(c). A larger threshold will lead to smaller extracted regions, which will result in the omission of key information and further reducing the overall accuracy. On the contrary, if we choose a small threshold, the extracted RoI will contain a lot of redundant background information. The small proportion of the target object in the image will lead to an increase in the difficulty of the visual task, and redundant information will also reduce the efficiency of RoI extraction, resulting in an increase in the amount of transmitted data. Additionally, retaining a small portion of the background area helps improve the accuracy of the visual task.

In short, we choose an empirical threshold to balance the amount of information extracted and the efficiency of region extraction. After region segmentation, we can get a set of RoI boxes as Fig. 3(e), which can not only reduce the amount of data transferred, but also improve inference accuracy.

The upper part of Fig. 3(b) illustrates the poor localization performance of CAM over the whole background area, the RoI boxes extracted from which are most invalid boxes without any target objects. Offloading invalid boxes to the servers not only wastes bandwidth resources but also increases the computational burden at the edge, thus we adopt a RoI box selection algorithm to remove the invalid boxes that do not contain any target objects.

According to our experimental results, approximately 2 to 6 valid RoI boxes will be extracted from the raw frame through our RoI box selection algorithm, with each box ranging in size from 20 KB to 500 KB. On the one hand, due to non-visual range and high mobility of vehicles, the IoV is unstable and thus leads to the packet loss and untimely update of inference results. On the other hand, predicting RoI boxes increases the computational burden of the edge by several times compared to inputting the raw video frame directly which put computation pressure on resource-constraint edge servers. Lower resolution of the boxes leads to low prediction latency of the CNNs deployed on the edge at the cost of accuracy deterioration. Therefore, to mitigate the impact of network fluctuations on computation task offloading and maximize the prediction performance of CNNs at the edge, we propose a adaptive resizing strategy, which determines the resolution of each box by adjusting the down-sampling rate according to the current available bandwidth and computation resources at the edge.

Generally, image resolution can largely affect the inference accuracy and profiling cost of video analytics tasks. For example, the higher resolution of each box, the more accurate inference results obtained by the CNNs, but it leads to longer prediction latency and more GPU utilization. To measure the performance of CNNs on processing the RoI boxes extracted from a video frame, we define a utility function:

where $N$ denotes the total number of valid RoI boxes at time $t$. $a_{r_{i}}^{t}$ and $c_{r_{i}}^{t}$ represent the accuracy and resource consumption in predicting the $i$-th box with down-sampling rate $r_{i}$ respectively. The resource consumption consists of two components, i.e., inference latency and GPU utilization of the edge server.

Our objective is to maximize the utility function under network bandwidth and edge-side resources constraints:

where $B^{t}$ and $G_{\max}^{t}$ denote the current maximum available bandwidth and GPU resources, respectively. Constrain $C_{1}$ ensures that the total data volume transmitted per frame does not exceed the current available bandwidth. And constrain $C_{2}$ represents that the total GPU usage required for the RoI boxes inference is no more than the total available GPU resources at the edge. When the box resolution is smaller than the input size of CNN model, the inference accuracy will be significantly reduced, so we set a lower bound of resolution $r_{\min}$ and constrain $C_{3}$. We discretize the variables by restricting the value of the downsampling rate $r_{i}$ and adopt hill climbing algorithm to solve the optimization problem $\mathcal{P}$.

Platforms. We implement the CAM-based RoI extraction and adaptive boxes offloading on the NVIDIA Jetson Tx2 (256 CUDA cores) which is considered to be the on-board processor with considerable computation resources. And the edge server runs Ubuntu and has one NVIDIA GeForce RTX 3090 GPU (10496 CUDA cores) and Intel Core i9-10900K CPU.

CNN Models and Datasets. To validate the ability of our RoI extraction system in handling computer vision tasks, we choose a common autonomous driving task and employ state-of-the-art CNN models for inference, i.e., the YOLOv5x (640) will be used for object detection on the edge server. And the lightweight feature extractor is chosen as ResNet18. The test datasets are derived from YouTube containing four different video streams generated by in-vehicle cameras. We use the original encoded MP4 format (3840$\times$2160 resolution, 30fps frame rate) as the input.

Baseline. We compare our proposed system with the following video analytics pipelines.

• $\bullet$

  $\mathsf{ELF}$ [9] offloads the low-resolution video frame every three frames to edge servers to handle new-occurred objects. The region proposals can be obtained via an attention-based Long Short-Term Memory (LSTM) prediction network and the previous inference results.

• $\bullet$

  $\mathsf{EdgeDuet}$ [10] partitions the image into tiles equally and offloads the tiles containing small objects to the remote CNN models located on the edge servers, with the aim of improving detection accuracy.

• $\bullet$

  $\mathsf{LEB}$ is a variant of our proposed system, which only performs local CAM-based RoI extraction and boxes selection. The RoI boxes will be fed into the local CNN model for inference. Moreover, in order to ensure real-time inference on the local side, we choose the YOLOv5s (640) model for object detection task to run on TX2.

• $\bullet$

  $\mathsf{Origin}$ directly inputs the raw video frames into the CNN model without image partition.

In this section, we evaluate our proposed video analytics pipeline under four different datasets and varying bandwidth conditions, and compare it with other baselines. We employ the F1 score [14] to measure the object detection performance and YOLOv5x (1280) placed on the server to obtain the ground truth. The performance is evaluated by the mean accuracy and the frame size after processing with different algorithms.

The results are shown in the Fig. 5, where Fig. 5(a) is the comparison of mean accuracy between our proposed system and other baselines under four different on-board datasets. Compared to other algorithms, our proposed system obtains the highest average accuracy under all datasets. Although both $\mathsf{ELF}$ and $\mathsf{EdgeDuet}$ partition the raw image into tiles and handle each tile separately to improve the detection accuracy. However, $\mathsf{ELF}$ obtains region proposals through a prediction network according to the previous inference results of the low-resolution video frame transmitted to the edge at the beginning, which limits the accuracy performance improvement of the whole system. As for $\mathsf{EdgeDuet}$, the equal partitioning scheme can not eliminate background areas very well which further influences the detection accuracy. To make a great improvement in the inference accuracy, our proposed system performs low-cost RoI extraction based on CAM. The result in Fig. 5(a) shows that compared with $\mathsf{ELF}$, our system achieves up to 16% mean accuracy increase.

Figure 5(b) shows the frame size of three consecutive frames, namely, the amount of data transferred under different algorithms. Likewise, our proposed system demonstrates significant advantages in data compression compared with baselines. The frame size after boxes resizing of our proposed system is only about 100 KB (including four valid boxes) of one frame. The data size of the first frame in $\mathsf{ELF}$ is relatively high because the low-resolution frames are transmitted to the edge for processing together with the region proposals. And the data size of $\mathsf{EdgeDuet}$ is related to the frame position in a group of pictures, which is also larger than ours.

Vehicles’ high mobility lead to intermittent IoV connections, and subsequently packet loss. This calls for adaptability to network conditions in system design. Our model shows great advantages in terms of the adaptability to bandwidth. Fig. 6(a) demonstrates that, in the case of poor network conditions, e.g., 20 Mbps, our method can guarantee a mean accuracy of 0.7, while that of $\mathsf{ELF}$ is only 0.45. In Fig. 6(b), when network bandwidth fluctuates, the inference accuracy of other baselines change drastically, while our algorithm ensures stable and high inference accuracy. Thanks to the extremely high data compression rate of RoI extraction process, the proposed algorithm can maintain high accuracy under various network conditions.

Note that, the overall computational overhead of our system consists of 2.94 ms feature extraction, and 2 ms box size selection at the device side. For the edge, different analytics tasks correspond to different processing latency, e.g., target detection takes only 10 ms. Moreover, the size of effective boxes after resize is only about 100 KB, and the time for parallel offloading to the edge is negligible. In summary, our system completion can run in real time.