FemtoDet: An Object Detection Baseline for Energy Versus Performance Tradeoffs

Top1-Acc (top1 accuracy, for image classification), and AP (average precision, for object detection) are widely used measures to evaluate the performance of CNN. In addition to the commonly used metrics $\mathrm{top1-Acc}$ and $\mathrm{AP}$, we propose the $\mathrm{Power}$ (energy costs) and $\mathrm{mEPT}$ (mean energy versus performance tradeoffs) to comprehensively evaluate the energy costs of the models and their ability to achieve energy versus performance tradeoffs. Specifically, $\mathrm{Power}$ and $\mathrm{mEPT}$ are represented by

where $\mathrm{{T}_{time}}$ indicates the total times required to evaluate ${N}$ images, $\varepsilon(\mathbf{W},{x}_{i})$ represents the energy costs while models evaluate ${i}$-th image ${x}_{i}$ with trained parameters $\mathbf{W}$. $\varepsilon(\hat{\mathbf{W}},{x}_{i})$ is similar to $\varepsilon(\mathbf{W},{x}_{i})$. The difference is that we set a value of 1 for each layer of channels, where the model parameter is $\hat{\mathbf{W}}$. $\varepsilon(\hat{\mathbf{W}},{x}_{i})$ indicates that the model is an empty state. ${P}$ denotes the model performance (top1-Acc for image classification while evaluating activation functions and convolution operators, AP50 for object detection while evaluating necks of detectors in Sec. 3.1.2) in image ${x}_{i}$. For the power metric, the value is expected to be as small as possible. For the mEPT metric, the result is expected as greater as possible.

To identify low-energy components of detectors, we followed [44] to benchmark their energy costs from three types of structures: activation functions, convolution operators and necks of detectors:

Activation Functions. Rectified Linear Unit (ReLU) [30, 24], GELU [17], Swish [10], SiLU [9], and etc. are widely used for object detection because they have fewer parameters and flops. We calculate the energy costs of different activation functions in the same architecture to explore which activation function is more friendly for designing an energy-oriented detector.

Convolution Operators can automatically learn filer weights while CNNs are trained to summarize helpful information, including horizontal, vertical, edged, diagonal, and etc. features from images. Vanilla convolutions (vanConv) [19], depthwise separable convolutions (DSC) [18] and their large kernel size versions [26] are widely used. Ding et al. [8] pointed out that when CNNs are constructed based on enough large convolution kernels, results can be comparable to the transformer [25]. Here, we set up energy consumption comparison experiments with different convolutions (vanCon and DSC) and multi-scale convolution kernel sizes (3$\times$3, and 5$\times$5) on different network structures (ResNet and MobileNetV2). It is important that for each experiment case, the convolution operator type or kernel size would be the only variable.

Necks of Detectors: Feature pyramid networks (FPN) are essential components for two-stage or one-stage detectors. FPN-based necks of detectors fuse multiple low-resolution and high-resolution feature inputs for better representations, leading to a series of studies for designing manually complex fusion methods [22, 23, 41]. However, they only presented favorable results brought by FPN but ignored its energy costs. We form a series of experiments on FemtoDet by using different neck types, e.g. FPN [22], PAN [23], and our proposed SharedNeck (details can be found in Sec. 3.2.1) to study corresponding metrics (including energy costs, parameters, and detection precision on PASCAL VOC) changes.

All related experiments can be found in Sec. 4.1. We can observe that DSC and ReLU are more energy friendly than vanCon and other activation functions. In addition, by comparing with the necks of detectors such as FPN and PAN, the proposed SharedNeck is more suitable for light-weight detectors.

Backbone: The backbone of FemtoDet contains an initial full convolution layer with 8 filters. We use ReLU as the non-linearity and BN as the batch normalization. From the second layer beginning, all the convolution operators use DSC. The reason is that we follow the results of our benchmarks (the results can be seen in Sec. 4.1) to select the energy friendly components. The whole backbone is described in Appendix Table 9; Neck: We customize a neck for FemtoDet to achieve better tradeoffs between energy and performance. In detail, SharedNeck first aligns the scale information of inputs from backbones, and then merges these alignment features with elements addition. Finally, a DSC performs adaptive multi-scale information fusion between merged features. The implementation details of SharedNeck, and how it differs from other necks are shown in Fig. 1; Head and Training Loss: Here, we use the decouple head of YOLOX as the head of our proposed detector with the same training loss as YOLOX.

Optimizing light-weight detectors is a notoriously challenging problem. The reason for this is that limited by their representation, features learned by detectors are diffuse, as shown in Fig. 2 (b). Instance boundary enhancement (IBE) modules are designed to improve the DSC in FemtoDet, thus overcoming the bottleneck of light-weight models’ representation optimization. IBE is similar to modules introduced in [47, 36, 45], except that our modules are designed for convolutional layers that are factorized into depthwise and pointwise layers. We further introduce a dual-normalization mechanism, such that the IBE modules can be used for object detection, while the modules in [47, 36, 45] just working for low-level tasks. Based on the DSC block, the basic block of IBE includes 3$\times$3 depthwise convolution followed by 1$\times$1 convolution. IBE enhances DSC by designing a new local descriptor, semantic projector, and dual-normalization layer. Specifically, the 1$\times$1 local descriptor is a parameter reusing mechanism generated by the linear transformation of the integrating gradient cues around shared depthwise convolution. Therefore, the object boundary information can be found in the local descriptors. Previously, the local descriptor is known as the difference convolution [47, 36, 45] used for low-level tasks. Although the idea is easy to get that using these object boundary information to enhance the noisy feature representation (shown in Fig. 2 (b)) of the above standard operators, such as depthwise convolution, in practice, the original difference convolution architecture cannot serve for high-level semantic tasks. We observe that features obtained by the difference convolution are unaligned with the features obtained by standard convolution (the figure shown in Fig. 4 of the Appendix). On the one hand, the difference convolution captures diverse information about object boundaries by integrating gradient cues around 3$\times$3 or with larger convolution. On the other hand, high-level tasks encourage standard convolution to summarize the abstract semantic information of images. That is the reason that the classical difference convolution cannot serve high-level tasks. To simultaneously solve this problem, we propose a semantic projector and a dual-normalization layer. The semantic projector is the linear transformation of depthwise convolution to transfer operators related to semantic information extraction; The dual-normalization layer consists of two independent batch normalization modules specifically designed for aligning the unaligned features. After that, we incorporate the object boundary cues from the feature addition between the local descriptor and depthwise convolution to guide the model in learning an effective representation of the instance (refined results can be seen in Fig. 2 (c)).

The description of IBE modules is given in Fig. 3 of the Appendix. For the depthwise convolution of kernel size 3, $C_{in}$ denotes the input channel dimension, $C_{out}$ represents the output channel dimension, the weight matrix can be denoted as $\mathbf{W}_{conv}\in\mathbb{R}^{1\times C_{in}\times 3\times 3}$, and the bias is represented as $\mathbf{b}\in\mathbb{R}^{D}$. The local descriptor of kernel size 1 can be generated from the depthwise convolution by reusing parameters. Its weight matric is denoted as $\mathbf{W}_{des}$, which is integrating the gradient cues around $\mathbf{W}_{conv}$:

where $\mathbf{W}_{des}\in\mathbb{R}^{C_{out}\times C_{in}}$, $\theta_{1}\in[0,1],\in\mathbb{R}^{1\times 1}$ is a learnable parameter projection factor. $\mathbf{R}$ represents the $3\times 3$ convolution in $\mathbf{W}_{conv}$, and ${p}_{n}$ is the ${n}$-the weight value. Integrating the gradient cues around $\mathbf{W}_{conv}$ can help the local descriptors capture the object boundary information effectively. The semantic projector is generated from the depthwise convolution, while diverse semantic presentations can be obtained through a learnable linear transformation, shown as follows:

where $\theta_{2}\in[0,1],\mathbb{R}^{1\times 1}$ is another learnable parameter as the projection factor. Subsequently, the IBE modules perform four steps to refine the feature representation under the guidance of object boundary information. 1) inputs ${x}_{in}$ will be convolved with the obtained three convolution operators, and the corresponding results are denoted as ${x}_{22},{x}_{21}$, and ${x}_{23}$, respectively; 2) elements addition are conducted between ${x}_{22}$ and ${x}_{21}$; 3) the dual-normalization layer enables feature distribution normalization on ${x}_{21}$ and ${x}_{23}$; 4) the two outputs from the dual-normalization layer can be added together; As a result, the aforementioned added features are convolved through pointwise convolution as the final output of the IBE module.

In addition, we leverage the homogeneity and additivity of the convolutions to fold IBM modules into the simpler depthwise separable convolution in the inference stage without performance degradation. The details can be seen in Appendix C.

Table 1 shows the energy-related metric results using different activation functions under the same architecture (MobileNetV2 0.25). We observe that the activation functions can significantly improve models without extra parameters (Param). On the other hand, such activation functions incur unacceptable overhead on some metrics which are inconvenient to measure. For example, Swish improves ReLU-based models performance by 4.93$\%$, but produces more than 8.53$\%$ more energy cost. The mEPT metric also demonstrate that ReLU results in the best energy versus performance tradeoffs.

This sub-section compares energy-related metrics while training an image classifier on the ImageNet dataset based on the different convolution operators (vanCon and DSC), and the different kernel sizes (3$\times$3, and 5$\times$5) on the same convolution operator. Experimental results in Table 2 shows that: 1) When CNNs were constructed with the same kernel size, mobilenetv2 generated much lower energy costs than resnet18, as shown in Table 2 of $\mathrm{vanConv}^{\clubsuit}$ and $\mathrm{DSC}^{\clubsuit}$. In other words, DSC is more energy friendly than vanConv while achieving similar performance; 2) When CNNs were constructed with different kernel sizes and the same types of operators, the smaller kernel generated much lower energy costs than the larger kernel, as shown in Table 2 of $\mathrm{vanConv}^{\clubsuit}$ and $\mathrm{vanConv}^{\diamondsuit}$. However, the performance gain from the large kernels is not so impressive.

Here we evaluated the effects of different necks (including FPN, PAN, and SharedNeck) in FemtoDet. Experimental results are shown in Table 3, and some observation are summarized as follow: 1) Although FPN enables FemtoDet to achieve preferable results, it not only consumes more energy but also has a large parameter overhead; 2) Due to the limited representation of light-weight models, the top-down and bottom-up feature fusion of PAN is poor; 3) SharedNeck achieves the best results in the metrics of parameter, energy cost, object detection performance, and factors of mean energy versus performance tradeoffs by performing adaptive feature fusion.

Experimental results are presented in Table 4, which shows light-weight detectors with eight metrics: parameter (Param), energy cost (Power), AP50, factors of mean energy versus performance tradeoffs (mEPT), mAP, mAP of small size objects (mAP-s), mAP of medium size objects (mAP-m), and mAP of large size of objects (mAP-l) respectively. We can see that although YOLOX has a large number of parameters, it is much lower than FemtoDet in many metrics. NanoDet Plus has the parameter comparable to that of FemtoDet, but only outperforming FemtoDet in the metrics of mAP-s. We believe that for such extremely light-weight detectors, their performances on large-scale objects are the most critical. FemtoDet also presented better results than YOLOX and NanoDet Plus on the metrics of mAP-l. Although the metrics ${i.e.}$ mEPT, used to evaluate performance and energy balance, FemtoDet achieves the best balance compared to the other two models.

In addition, we evaluated the inference speed (FPS) and power of trained detectors on Qualcomm Snapdragon 865 CPU platforms. As shown in Tab. 5, our FemtoDet also achieves the smallest energy costs and inference speed on edge devices.

Extremely light-weight detectors like FemtoDet have broad application prospects in surveillance scenarios. Hence, we evaluate the performance of FemtoDet in pedestrian detection using the campus dataset of TJU-DHD, and show the performance of the detectors on the less stringent metric AP20. Table 6 and 7 demonstrate that FemtoDet is competent for pedestrian detection in common surveillance scenarios: 1) FemtoDet has lower energy costs in deployment, which is 29.74$\%$ lower than YOLOX; 2) FemtoDet obtains 76.3$\%$ AP20, while YOLOX with the same parameters only obtain 71.80$\%$ AP20.

We conduct a series of ablation studies on PASCAL VOC to demonstrate the effectiveness and efficiency of our IBE modules and RecWR training strategy. As shown in Table 8: 1) The second and third rows present that by training with 300 and 1200 epochs for FemtoDet*, such a longer epochs do not yield better gains. 2) The fourth and fifth rows present FemtoDet* was trained with RecWR and 300 epochs with the enhancement of IBE, respectively. Results show that independently equipping IBE and RecWR can improve model performance. 3) The six rows demonstrate that jointly using IBE and RecWR enables FemtoDet to achieve the best performance.