YOLOBench: Benchmarking Efficient Object Detectors on Embedded Systems

Object detection constitutes a pivotal task in the field of computer vision, entailing the critical process of identifying and localizing objects present within an image. Applications of object detection models include autonomous vehicles, surveillance, robotics, and augmented reality [7]. The central problem of deploying deep learning-based object detection solutions on embedded hardware platforms is the amount of computation, memory, and power required for their inference [18]. This necessitates the development of efficient object detection models specialized for low-footprint hardware devices to achieve an optimal trade-off of accuracy and latency.

For years, the state-of-the-art (SOTA) deep learning approach to object detection has been the series of YOLO architectures [31]. In recent years, remarkable strides have been taken in advancing YOLO-like single-stage object detectors, prioritizing real-time operation while simultaneously striving for higher accuracy and deployability on low-power devices. These advancements have primarily focused on enhancing various components of the detection pipeline. Key areas of improvement include the design of accurate and efficient backbone and neck structures within the network [34], exploration of different detection head designs (e.g. anchor-based [34] vs. anchor-free [11]), utilization of diverse loss functions [22], and implementation of novel training procedures including innovative data augmentation techniques [16]. These collective efforts have continually refined and evolved YOLO-like architectures, enhancing object detection effectiveness and efficiency in real-time scenarios. The differences between consecutive YOLO versions, such as YOLOv5 [14] and YOLOv6 [20], span various pipeline components, making it challenging to isolate their individual contributions. This paper aims to address these challenges by providing a fair comparison of recent YOLO versions under controlled conditions (e.g. same training loop for all models) to demonstrate the impact of the backbone and neck structure of YOLO-based models in embedded inference applications. We also use the collected accuracy and latency data for multiple YOLO-based detector variations to empirically evaluate training-free performance predictors commonly used in neural architecture search [3]. We summarize our contributions as follows:

• •

  We provide a latency-accuracy benchmark of $550$+ YOLO-based object detection models on $4$ different datasets, called YOLOBench. All the models are validated on $4$ different embedded hardware platforms (Intel CPU, ARM CPU, Nvidia GPU, NPU),

• •

  We show that if modern detection heads and training techniques are implemented for the detector training pipeline, multiple backbone and neck variations, including those of older architectures (e.g. YOLOv3 and YOLOv4), can be used to achieve state-of-the-art latency-accuracy trade-off,

• •

  Looking at YOLOBench as a neural architecture search (NAS) space, we demonstrate that, while most of the state-of-the-art zero-cost (training-free) proxies for model accuracy estimation are outperformed by simple baselines such as MAC count, the NWOT estimator [27] can be effectively used to identify potential Pareto-optimal YOLO detectors in a training-free manner,

• •

  We showcase the effectiveness of the NWOT estimator for optimal detector prediction by using it to identify a YOLO-like model with FBNetV3 backbone that outperforms YOLOv8 on the Raspberry Pi 4 ARM CPU.

There has been a tremendous amount of progress in efficient object detection in recent years pushing the accuracy-latency frontier, including architectures like YOLOv7 [34], YOLOv6-3.0 [20], DAMO-YOLO [37], RTMDet [26], RT-DETR [25] and PP-YOLOE [36]. These works oftentimes improve upon state-of-the-art latency-accuracy trade-offs, providing comparisons of several generations of detectors on the COCO dataset. Benchmarks of different model families are also provided by framework developers, such as MMYOLO [5] and Ultralytics [15]. Additionally, there exist third-party benchmarks of several architectures from the YOLO series on server-grade and embedded GPUs as well as specialized accelerators [17, 28, 10, 39]. We identify a few limitations of the existing efficient detector benchmarks that have served as motivation for YOLOBench:

• •

  Comparisons of different YOLO versions are frequently done either by using a proxy metric for the actual latency like MAC count and number of parameters or by reporting latency values on server-grade GPUs, neither of which is directly indicative of latency on embedded devices,

• •

  Accuracy metrics are usually reported on the COCO dataset, which could be considered too large-scale with respect to actual practical use cases,

• •

  Some architecture parameters (like input resolution) are often considered to be fixed in detector benchmarking, while it is known that they serve as important factors in optimal CNN scaling [13],

• •

  Different YOLO variations being compared to one another are typically trained with different training codebases, training techniques (loss functions, data augmentations), and hyperparameter values, making it hard to disentangle the contribution of the training pipeline improvements vs. better architecture design.

To address these issues, we conduct a thorough accuracy and latency benchmarking of state-of-the-art YOLO detector versions in controlled, fixed conditions to study the impact of backbone and neck design proposed by several YOLO model families.

The purpose of the current study is to thoroughly study the impact of the backbone and neck and its parameters (width, depth, input resolution) on the performance of YOLO detectors in terms of their accuracy and latency. For the rest of the factors influencing the accuracy-latency trade-off, such as choice of the detection head, loss function, training pipeline, and hyperparameters, we aim to have a fixed, controlled setup, so that we can isolate the effect of backbone and neck design on model performance. For this reason, we use the anchor-free decoupled detection head of YOLOv8 [31], as well as CIoU and DFL losses for bounding box prediction used in YOLOv8, as they have been shown to produce state-of-the-art results on the COCO dataset. Anchor-free detection in YOLO models has been also shown to provide latency benefits in the end-to-end detection pipelines [25]. Hence, the main source of variation in YOLOBench models is the structure and parameters of the backbone and neck. We also use the same training code and hyperparameters for all models, as set by default in the YOLOv8 training code released by Ultralytics [15], which provides a relatively simple training loop capable of producing SOTA results.

The flow of candidate model generation, pre-selection, and training is shown in Figure 2. First, we generate the full architecture space consisting of about $1000$ models by independently varying the backbone/neck structure, depth factor, width factor, and input resolution (Table 2). For each architecture, we consider its variations trained and tested on $11$ different input resolutions (from $160$x$160$ to $480$x$480$ with a step of $32$) and $12$ variations in depth and width, aside from $4$ usually considered scaling variants ($n$, $s$, $m$ and $l$). The only exception is the YOLOv7 models, for which we only vary the width factor producing $4$ variations of the model. For YOLOv6 models, we use the v3.0 version [20], for which provided $s$, $m$ and $l$ variations actually represent different architectures aside from different depth and width factors (see Table 2). Hence, we consider YOLOv6s, YOLOv6m and YOLOv6l as different model families and generate the same 12 depth-width combinations for each one.

Latency measurements. The actual inference latency for each model might vary significantly depending on the deployment environment and runtime. Therefore, we collect the latency measurements for each of the models by running inference on $4$ different hardware platforms (runtime and inference precisions specified in brackets):

• •

  NVIDIA Jetson Nano GPU (ONNX Runtime, FP32)

• •

  Khadas VIM3 NPU (AML NPU SDK, INT16)

• •

  Raspberry Pi 4 Model B CPU (TFLite with XNNPACK, FP32)

• •

  Intel^® Core™i7-10875H CPU (OpenVINO, FP32)

We did not consider latency measurements for INT8 precision, as depending on the quantization scheme (e.g. per-tensor vs. per-channel) and approach (e.g. post-training quantization vs. quantization-aware training), there can be a varied impact of INT8 quantization on accuracy. Adding INT8 results for both accuracy and latency in YOLOBench is a matter of future work. All latency measurements were performed with a batch size of $1$ averaged over $200$ inference cycles (with $5$ warmup steps). We measured the inference time required to execute the YOLO model graph, without taking bounding box post-processing (e.g. non-maximum suppression) into account. Note that for VIM3 NPU measurements, the bounding box decoding post-processing operations (operations after the last convolutional layers of the network) were also skipped due to the limitations of VIM3 SDK.

Training pipeline. To obtain the accuracy metric values for the models, we consider the following $4$ datasets: (i) PASCAL VOC (20 object categories) [9], (ii) SKU-110k (1 class, retail item detection) [12], (iii) WIDER FACE (1 class, face detection) [38], (iv) COCO (80 object categories) [24]. Our motivation to include several smaller-scale (with respect to COCO) but challenging datasets stems from the fact that for many practical deployment use cases, the number of object categories to detect and the amount of available data might be limited. The metric of interest for all datasets is mAP_50-95. For all selected models, the training procedure starts with pretraining on the COCO dataset (for $300$ epochs, with a batch size of $64$ and $640$x$640$ resolution), afterward the best COCO weights are used as initialization for other datasets, on which we perform fine-tuning for $100$ epochs (batch size of $64$) on all $11$ YOLOBench resolutions and select the best weights (in terms of mAP_50-95 value) for each one. For the COCO dataset, we do not perform fine-tuning on target resolutions, rather we evaluate the model trained on $640$x$640$ images on all target resolutions (to mimic the deployment of pre-trained COCO weights). All other training hyperparameters are set as per default values of the Ultralytics YOLOv8 codebase [15]. Model weights are randomly initialized for all experiments (e.g. no transfer of ImageNet weights for the backbone is performed).

Candidate model pre-selection. In order to reduce the number of training runs on the COCO dataset, we filter out some of the least promising model candidates from the YOLOBench architecture space as an initial step of our benchmarking procedure. To determine the most promising models in terms of the accuracy-latency trade-off, we compute a proxy metric that is well correlated with the final mAP values of the models fine-tuned on the target datasets. A natural choice for such a metric is model performance when trained on a smaller-scale representative dataset. For this purpose, we use the VOC dataset to train all the model candidates from scratch (random initialization) for $100$ epochs and use the resulting mAP_50-95 value as a proxy metric to predict performance on all target datasets (with models fine-tuned on these datasets from COCO pre-trained weights). We observe a good correlation of such a training-based accuracy proxy with final metrics on all considered datasets (even on datasets from other domains, like SKU-110k; see Appendix C). We also examine the performance of training-free accuracy estimators for this task and compare it to mAP of VOC training from scratch (see Section 4.2).

Once we have the accuracy proxy values and latency measurements for all models in the dataset, we determine the models with the best accuracy-latency trade-off (the Pareto frontier models). We use the OApackage software library [8] to determine the Pareto optimal elements in the latency-accuracy space. We define the second Pareto set as the set of models that are Pareto-optimal if the initial Pareto set models are removed (so that the “second best” models in terms of latency-accuracy trade-off become the best). Correspondingly, we define the $N$-th Pareto set.

For our model pre-selection procedure, we consider the models contained in the first and second Pareto fronts (in terms of mAP_50-95 in VOC training from scratch), with latency for each considered hardware platform separately. We merge all the first and second Pareto sets for each HW platform to form the list of promising architectures to be selected for COCO pre-training. After the COCO pre-training phase is finished for a model, variations of that architecture on multiple resolutions are considered in the benchmark.

In this work, we present YOLOBench, a latency-accuracy benchmark of several hundred YOLO-based models on 4 different object detection datasets and 4 different hardware platforms. The accuracy and latency data are collected in a fixed, controlled environment with the only variation in backbone/neck structure and input image resolution of the detectors. We use these data to demonstrate that it is possible to achieve Pareto-optimal results with a range of different backbone structures, including those of the older architectures in the YOLO series, such as YOLOv3 and YOLOv4. We also observe that depth and width scaling precede input resolution scaling in optimal YOLO-based detectors.

Finally, we use YOLOBench to evaluate zero-cost accuracy predictors, and find that, while many of the existing state-of-the-art predictors perform poorly, pre-activation NWOT score can be effectively used to identify Pareto-optimal detectors for specific target devices of interest. We demonstrate that by using NWOT to find a YOLO backbone (FBNetV3-D) that outperforms a state-of-the-art YOLOv8 model when deployed on a Raspberry Pi 4 ARM CPU.