YotoR-You Only Transform One Representation

CNN-based object detectors, which started with Faster R-CNN [20], have become a widely used approach to solving object detection tasks. In some applications, in which real-time processing is needed or there are hardware limitations, lightweight object detectors are needed.

The most widely used real-time object detectors are based on FCOS [21] or the YOLO family of detectors [19] [23]. Progress in this area has been made by improving the backbones, the necks, the detection heads, the loss functions, and the training procedures used. The backbone is the section of the network whose task is to obtain embeddings that encode the input images. In the case of the backbones, the improvements have been led by the use of building blocks inspired by RepVGG [7], CSP [24], ELAN [4], and VoVNet [13]. The task of the neck is to further process the embeddings to make them useful for generating the detections. In the case of the necks, the most successful approaches are inspired by RepVGG [7] and PAN [18]. Finally, the detection heads generate the output detections of the network. In the case of the YOLO family of detectors, there are several heads, each one dedicated to detecting objects of different sizes. Then, a good design of the backbones, necks, and detection heads is critical for obtaining architectures capable of performing real-time detection without sacrificing performance.

The use of multi-task architectures is promising, as they can integrate several information modalities for improving performance on all of the tasks. However, designing architectures able to perform multiple tasks in real time is challenging because using ensembles of per-task networks negatively affects the runtime of the system.

An architecture named YoloR [25] was proposed to solve real-time multitask problems. The system is based on using a single unified architecture for solving several tasks. YoloR is based on modeling two sources of knowledge: explicit knowledge and implicit knowledge. Traditionally, explicit knowledge is strongly tied to the inputs of the network, and it is represented by the first layers of the network, while the dependence of the implicit knowledge on the inputs is lower, and it is encoded by the deeper layers. For achieving multi-task capabilities with a unified architecture, it is proposed that instead the explicit knowledge is modeled by weights shared by all of the input modalities, while the implicit knowledge is modeled as input-independent parameters in deep layers of the network. The weights related to the implicit knowledge can modify the internal network embeddings by applying operations like concatenation, addition, and multiplication. By using this approach, a single network architecture can benefit from being trained for multiple tasks, as the network weights related to the explicit knowledge can condense the information from all of the input modalities at the same time.

While transformers have been used successfully in natural language processing [22], their use in computer vision has been delayed because the amount of computing processing needed squares quadratically with the number of tokens (patches) in the image. The first work which used plain ViT (visual transformers) for classification [8] used patches of size 16x16 as tokens, and the transformer architecture consisted of a sequence of encoders. Despite the ViTs being able to process all of the tokens at the same time, they have no inductive biases like translation invariance, which are intrinsic to the CNNs architectures. Then, the largest ViT model proposed in that work needed over 300M images to achieve a better performance than state-of-the-art CNN models like ResNets trained with BiT [12]. Also, the problems of the quadratic increase in computational computing power by the number of tokens, the large size of the patches, and the lack of multiscale processing cause the plain ViT architecture to be unable to be used for tasks like object detection or image segmentation.

An efficient visual-transformer-based backbone able to replace CNN-based ones, named Swin Transformer, was proposed in [16]. Swin transformers work by using small patches with size $4\times 4$ as tokens and dividing the image into windows. Then, an encoder is applied to all of the tokens contained on each of the windows independently. This strategy enables the computing power requirement to remain low. As the windows are processed independently, their information must be combined in subsequent layers of the network. Then, after the application of encoders on the windows of a layer, the windows of the following layer are shifted. In other words, each window in a layer can integrate information from four windows in the previous layer. Also, after the application of a nonshifted and a shifted window layer, the tokens are merged, which increases the embedding dimension while decreasing the number of tokens. Also, the patch merging strategy enables the architecture to generate feature pyramids which are useful for tasks involving multiscale processing. Then, Swin transformers can be used instead of CNN-based backbones for tasks like object detection, object instance segmentation, and semantic segmentation.

In recent years, there has been a surge in proposals suggesting innovative combinations of architectures featuring Transformers for object detection. Notably, the authors of [27] present a distinctive fusion involving the anchor-free implementation of YOLO, namely YOLOX [10], and the Swin Transformer. An increase of 6.1% in the mAP performance in their private dataset was observed for the detection of obstacles for autonomous driving, setting a good precedent for the combination of the Swin Transformer backbone with the Yolo family of detectors. Our proposal differs from the YOLOX-S models by keeping the YoloV3 anchors on the detection heads and implementing the improvements added by YoloV4 and YoloR like the new BoG and BoS techniques and the implicit knowledge modeling. Also, the proposed method was trained on the MSCOCO dataset to directly compare with the rest of the state-of-the-art.

Another strong contender has been the DETR [3] family of transformer-based object detection architectures, with some variants like the Co-DETR [28] achieving state-of-the-art performance. The DETR works by using an encoder-decoder Transformer over features extracted by a backbone. This allows the model to achieve better accuracy and the ability to handle complex relations between the objects while keeping a simple structure for the architecture.

Both training and evaluation were performed on a V100 GPU with 32 GB and 16 GB respectively. ¹¹1All provided speed evaluations of the implemented models were performed on a similar V100 GPU with 16 GB and no TensorRT, and are therefore not directly comparable with those reported by other groups on COCO val2017 The training parameters used are the same as YoloR as shown in Table 2. The YoloR training schedule was followed and trained for 300 epochs first, then fine-tuned using 125 epochs, and finally trained for an extra 25 epochs using a lower mosaic count.

From the results shown in Table 5, it is clear that none of the YotoR models can match the speed of YoloR. However, all the YotoR models have increased inference speeds compared to their respective Swin backbones. In particular, TP5 is notable for its inference frame rate that more than doubles compared to Swin-T, while BP4 and BB4 increase by just over 80% to Swin B.

Due to significant variation in time measurements across models, we conducted all timing evaluations on a consistent platform: a 16GB V100 GPU, operating at a resolution of $1280\times 1280$, without non-maximum suppression (NMS) or TensorRT acceleration. Each timing measurement represents the mean of three consecutive runs performed sequentially within a single session. Despite our best efforts, the recorded times were approximately half of those reported in the papers for YoloR and Swin Transformer models. This suggests that there is potential to double the speed results through further optimization.

The mAP (mean Average Precision) of the four YotoR models on val2017 is shown in Table 5. TP5 and BP4 models outperform all the baselines, even YoloR P6, which serves as a reference due to its high mAP performance. The only model that does not surpass it is BB4. However, considering that BB4 is built solely from Swin B without using the YoloR P6 head, this result was expected. However, BB4 manages to outperform Swin B in terms of performance, suggesting the possibility of exploring solutions in larger models such as Swin L, which surpasses the performance of YoloR D6.

The advantages of YotoR models become evident when comparing inference time in milliseconds with mAP, as shown in Figure 4. It allows us to analyze the relationship between inference speed and model performance, which is essential because there is typically a trade-off between both. Speed tends to decrease when aiming for higher accuracy, and vice versa. In this figure, it is clear how all the YotoR models outperform their equivalent Swin detectors in both performance and speed, representing a significant improvement for all these models. Although YoloR P6 still has a considerable advantage in inference time, its performance can be surpassed by YotoR TP5 and YotoR BP4, despite using the same head architecture.

Looking at Table 6, it can be observed that the YotoR models TP5 and BP4 outperform the YoloR P6 model in all mAP metrics. This confirms the significant improvement in the performance of these models. These results are publicly available thanks to the CodaLab team, making it easy to compare them with other participants in the COCO competition.

In addition to the traditional metrics, Table 6 includes measurements of FLOPs (Floating-Point Operations) and the number of parameters for each model, which provide interesting insights. YotoR models have fewer FLOPs than YoloR P6, even though they have longer inference times. This could be explained by the nature of floating-point operations in YotoR, which, although less numerous, might be more computationally intensive or less optimized than those in YoloR. This suggests the potential for optimizing the inference time of YotoR models by improving their operations. On the other hand, the number of parameters in YotoR models falls in between Swin and YoloR models, as expected, since they combine elements from both.

These results are promising, especially when considering that the YotoR models were implemented using the most basic versions of YoloR and the Swin Transformer. This opens the door to exploring new combinations between models from the YOLO family and Swin Transformer, either to create a real-time object detection model with Transformers or to advance the state of the art in performance.

In Figure 5, we showcase instances of detections executed by the YotoR BP4 model on the val2017 and testdev datasets. These examples demonstrate the model’s proficiency in effectively detecting objects of varying sizes, even when they overlap within a group.

In Table 7, we present a detailed comparison between the YotoR models and various state-of-the-art object detection models, which encompass different sizes and architectures. Our focus specifically centers on models of similar sizes and structures. This table provides insights into the performance of these models in both the COCO val2017 and testdev datasets.

While it is evident that the YotoR models exhibit lower performance compared to some of the latest and larger models, it is important to note that only the smaller versions of YotoR were implemented, featuring the smallest number of channels in the connections. This choice accounts for their comparatively lower performance compared to YoloR-W6, YOLOv7-W6, and their larger variants. Despite this, the YotoR models still outperform its individual components like YoloR-P6 and Swin Transformer B. It is worth mentioning that the performance surpasses that of the Swin Transformer without HTC++, albeit with certain details not publicly disclosed and some adaptations made to the YoloR head for processing images of varying resolutions.

In essence, the YotoR models appear to strike a commendable balance between performance and speed. It’s noteworthy that this equilibrium could potentially be further optimized by incorporating larger models within the YotoR family, such as YotoR LE5 or YotoR LD5. The implementation of these larger variants holds promise for enhancing the competitiveness of the models, opening up intriguing avenues for future exploration.