Replication Study and Benchmarking of
Real-Time Object Detection Models

Anchor-based methods employ predefined detection boxes, called anchors, to predict object location and size [7]. Two anchor-based methods families are prominent. Models branching from R-CNN are mainly two-stage detectors: first generating proposed regions with a selective search algorithm before doing the object classification task for the selected regions [1]. Other models, such as YOLOv7, are instead single-stage detectors where object classification and bounding-box regression are done directly, without using pre-generated region proposals [3].

A greater quantity of anchors allows such models to better recognize objects of different shapes and sizes, leading to increased performances [8]. However, anchor-based methods demand various hyperparameters to define the anchor box characteristics, burdening training costs and hyperparameter tuning, and thus representing a strong inductive biais. Additionally, anchor-based performances are conditional on how well anchor boxes align with objects and may be limited with overlapping objects [9].

Anchor-free detectors often utilize feature maps or specific pixel characteristics to predict object locations and sizes. For example, CornerNet detects objects by predicting their geometrical center [10]. This technique removes the need for predefined anchor boxes, thereby enabling the development of lighter and consequently faster models. However, this method relies on more intricate post-processing techniques like thresholding, max pooling, and non-maximum suppression (NMS). Nonetheless, anchor-free detectors require less post-processing time than anchor-based detectors using NMS post-processing and maintain equivalent accuracy [11].

Attention-based architectures, such as Transformers, are originally from the natural language processing (NLP) field but have more recently made breakthroughs in vision tasks [12], due to the advantages of the attention mechanism. Detectors based on transformers remove the need for anchors and feature maps by directly predicting bounding boxes and class labels using the cross-attention mechanism of the transformer with a set of learnt positional embeddings [6]. The transformer encoder uses self-attention to extract information from the image, and the transformer decoder uses cross-attention between this extracted data and the learnt positional embeddings. The decoder’s output embedding is then used by a classification head and a feedforward network to predict class and bounding box location, respectively. It is important to note that such models use a limited number of detection boxes fixed by the number of learnt positional embeddings. As such, models based on this approach can forgo post-processing steps and directly predict unique bounding boxes [6].

The usability of deep neural networks is not only based on accuracy, but also on model size, inference speed and post-processing load required to use the output. To better capture a model’s practicality, the following metrics are measured:

• •

  Mean average precision from 0.5 IoU to 0.95 with 0.05 steps (mAP);

• •

  Average precision at 0.5 IoU (AP50);

• •

  Average precision at 0.75 IoU (AP75);

• •

  Average precision for small objects (APs);

• •

  Average precision for medium objects (APm);

• •

  Average precision for large objects (APl);

• •

  Network size;

• •

  Prediction time for a batch of 1 image (including all required post-processing);

• •

  Prediction time for a batch of 16 image (including all required post-processing); and

• •

  Prediction time for a batch of 32 image (including all required post-processing).

The recorded inference time considers the entire process from input processing to obtaining usable output, ensuring representability of real-world applications. Studying the prediction time across multiple batch sizes is crucial to understand how a model’s performances scales. This is done by estimating amortized inference time, i.e., the average time taken for a batch. This metric could be representative of almost real-time tasks where detection need to be done on a small batch of buffered inputs — i.e., scenarios where timely processing of a few consecutive frames is essential.

Model training was exclusively made using computing resources at the Northern Robotics Laboratory (NorLab) from Université Laval. GPUs (see table 1) were selected in such a way as to offer a range of performance, memory and release date. Model training was made exclusively with the 4 available A6000 GPUs without any multi-GPU training. Training was made within a Docker environment,¹¹1Based on docker image
pytorch/pytorch:2.2.1-cuda12.1-cudnn8-devel using Slurm²²2https://slurm.schedmd.com/overview.html consulted April 19th 2024. for job scheduling. This contributes to standardizing the software stacks while ensuring the possibility to port both training and evaluation to various machines. A unified pipeline simplifies model comparisons on different hardware.

All models listed in section 3.4 were evaluated through this pipeline. Before each evaluation, a few iterations are made to avoid cold starts. First, the precision metrics are evaluated on the validation set of MS COCO 2017 (see 3.3 for more details), followed by a measurement of the total prediction time on all images. This inference time estimation includes any post-processing used by the model, non-maximum suppression for instance. This is done on batch sizes of 1, 16 and 32.

All used code for this reproducibility project is available in the repository at https://github.com/Don767/segdet_mlcr2024 (April 21st 2024).

For all training and testing, only MS COCO 2017,³³3https://cocodataset.org/#home consulted April 19th 2024. was used. This dataset contains 118K images in its training set and 5K images for validation. All images have bounding boxes and masks over 80 object categories. Reported average precisions are computed using the python library pycocotools.⁴⁴4https://pypi.org/project/pycoco/ consulted April 19th 2024.

Four models in real-time object detection were selected for reproduction, based on their performances, but also to be representative of current state-of-the-art approaches.

Model training code uses MMDetection, an open-source object detection toolbox. The rest of this section give additional implementation information for each reproduced model.

We report the performance metrics for pretrained models from MMDetection in appendix 10. Table 8 shows frame-per-second performances of selected pretrained models. Various attention-based models’ inference time (DETR and DINO families) could not be estimated properly, those models require more specific data that our pipeline does not yet fully support. Such models do offer competitive accuracy performances, but, since it is currently difficult for non-industrial users to access hardware similar to the high-end GPUS tested (RTX-4090), common usage of DINO and DETR models (as implemented in MMDetection) for real-time object detection seems impractical because of the memory requirements for transformer based models. For transformer-based models, YOLOf R50 exhibits comparable speed performances to anchor-free and anchor-based architectures.

Table 8 shows improved speed performances for all models when computing resources increase, which is expected. We also provide the achieved mAP on all selected models in table 11.

Additionally, figures 2, 3 and 4⁹⁹9The exact numbers used to generate these graphs are available in appendix 10 reveals that increasing the batch size reduces speed performances for every model studied. More importantly, not all models seem to be affected in the same way by an increased batch size. Indeed, YOLOx and RTMDet family models maintain better speed performances with an increased batch size than other models tested. Broadly speaking, figures 2, 3 and 4 imply that lighter models are less affected by a larger batch size (large models agglomerate to the bottom of the graphs).

On the flip side, the larger models depicted in the same figure tend to showcase higher levels of accuracy. In essence, the figure illustrates a relative trade-off between accuracy and inference time. Models belonging to the YOLOx and RTMDet families appear to strike a better balance between accuracy and speed compared to other models tested, at least within the range of GPUs examined. This observation extends to our RTMDet implementation, where an aggressive pre-filtering strategy results in lighter and more efficient NMS and thresholding procedures. This approach appears to lead to better-amortized latency, as demonstrated in Figures 3 and 4, emphasizing the crucial role of post-processing in latency measurement.

The following considers in more detail the issues linked to the reproducibility of selected benchmark models.

Yolov7 truly shines for out-of-the-box usage. Indeed, code, weights and usage instructions are easily accessible and user-friendly — including both Docker setup instructions and Google Colab premade setup. These instructions also include the necessary explanation to use features within the bag-of-freebies with Yolov7 premade models.

Yolov7, taken as a product, is an excellent model. However, the same cannot be said for developers: details given within the article and the code repository are insufficient for reproducibility or model customization. First, the article is highly dependent on prior knowledge of previous YOLO-family models. For example, it is mentioned that the architecture is based on YOLOv5’s and includes YOLOR’s implicit knowledge, but no details are given on how this is implemented (whether in the article or the code repository). Searching previous YOLO papers and architectures (mainly YOLO3 [21], YOLOR [17] and YOLOv5 [16]) is essential to understand and reproduce YOLOv7 from scratch. Other articles are referenced as the source of some code blocks in the code itself (for example, YOLOv7 uses a SPPCSPN¹⁰¹⁰10Cross Stage Partial Networks Spatial pyramid pooling layer from https://github.com/WongKinYiu/CrossStagePartialNetworks, consulted April 21st 2024. layer) but not mentioned within the article or the usage instructions. Additionally, the article neglects important information necessary for the reproduction of training conditions, including the data augmentation and the loss function used.

Moreover, YOLOv7 sometimes neglects to use conventional PyTorch or TensorFlow code structure in favor of made-from-scratch implementations of corresponding functionalities. For example, YOLOv7 uses a custom data loader for its training, a custom data augmentation pipeline and a custom learning rate scheduler. This wouldn’t be a problem by itself, but the fact is that YOLOv7’s training environment is highly dependent on said custom structure. In that way, it is quite challenging to reuse the same training regiment as YOLOv7 with a custom reimplementation of the model. In the same way, YOLOv7’s pre-trained weights are defined with a custom pickle¹¹¹¹11https://docs.python.org/3/library/pickle.html, consulted April 26th 2024. format instead of pth as commonly used in PyTorch, complexifying its usage outside of YOLOv7’s premade architecture, as it depends on having specific Python modules at specific paths. The complexity of reproduction is further exacerbated by the absence of details in the original article for any such custom feature.

Finally, the included bag-of-freebies is interwoven within the model’s code and acts as a lot of background noise when trying to understand the model itself. Which bag-of-freebies’ functionalities and with what parameters were used to obtain the results presented in the original paper is also mostly left unanswered. In the end, YOLOv7’s training took 8 days.

The MMdetection toolbox offers an elegant, easily extendable modular design. This characteristic makes it a fitting choice to reimplement not only RTMDet, but most of our selected algorithms. Working within the toolbox conventions enabled us to leverage its rich ecosystem of data processing, data augmentation, loss function and post-processing utilities. Especially for RTMDet, as every tool mentioned in the paper was available directly within the repository, which allowed us to focus on the reimplementation of the model itself.

While MMdetection toolbox’s excellent software engineering practices facilitate easy extensibility, they introduce a significant level of abstraction, making it challenging to locate actual implementations. This, combined with some lack of details in the paper, made some parts of the reimplementation more difficult. Notably, the CSPNeXt backbone is not entirely described in the paper. Furthermore, discrepancies were observed between the actual loss function utilized in the repository and the well-described variant presented in the paper. Curiously, the implementation uses equation 1 as its loss function, which is only briefly described as the baseline in the paper. Training took 7 days and happened without incident.

Some important hyperparameters necessary to reproduce ViTDet were absent in the original paper. First, the paper mentions the usage of a learning rate step during training without further details. Searching the official code repository shows a multiplicative step of 0.1 for iterations 163889 to 177546. Additionally, the original paper does describe the SFP architecture, but omits to mention that the intermediary embedding between convolution and deconvolution layers reduces channel width by half. The article does mention a ViT backbone based on Masked Autoencoder [36], but modifications are made to it in the official code repository for ViTDet which are only briefly mentioned by the original authors. Besides these oversights, reproducibility was mostly uneventful.

The training itself was more time-consuming than expected. At the time of writing this paper, it is still not finished and goes at a rate of 14 iterations per day (on a A6000 GPU). At this rate, the total training time will be more than 7 days. Due to time constraints and occasional divergences in training, it became impossible to complete the entire training schedule planned in the article. Nonetheless, we report the inference time and accuracy of our implementation of ViTDet in tables 4 and 5. The results shown are obtained after 47 epochs. Our implementation achieved a mAP of approximately 10% by that point, confirming our network is learning, albeit at a prohibitively slow rate for our time constraints.

DETR is conceptually simple and paired with a well-detailed article. Reproduction using exclusively the original paper is achievable. Notably, the batch size had to be adjusted to 40 instead of 64 due to memory limitations. Otherwise, hyperparameters (see table 7) presented in the article were used directly. However, issues arose at training. After the intended 300 epochs (6 days of training time), the network always predicted the No object label. Looking at the official code, a few omissions within the paper were found. First, the normalization layer within the transformers is in reality placed before the attention mechanism instead of after. The positional encoding added to the image is only applied to the non-padded section of the input. We tried implementing these changes, but the network still consistently predicted the No object label. In fact, the learning rate schedule used in the original article seem too simple to achieve appreciable results. Pairing this with the lack of data augmentation makes it seem like the training process presented is lacking refinement, which hinders the speed at which the network learns. The model still learns after the 300 epochs originally used, so we are hopeful that we could achieve similar performances to the original paper, but with significantly higher training costs. Nonetheless, we report the inference speed of our implementation in table 6, and found it competitive in terms of inference time.