GnetDet: Object Detection Optimized
on a 224mW CNN Accelerator Chip at the Speed of 106FPS

GnetFC-v1 [13, 16] is an image classification model specially designed for the CNN accelerator chip. It can be also used for NLP and multi-modal tasks [22, 24, 23]. The motivation of GnetFC-v1 model is to reduce the CPU load of image classification models by replacing the FC (Fully Connected) layers with the 3x3 convolutional layers. This is done by using multiple convolutional layers without padding at the end of the CNN model. For an input image with a resolution of 224x224, the feature map size of the VGG [12] convolutional layers output is 7x7. After three layers of convolutions without padding, the output size will become 1x1. And the number of channels will be used to represent the model scores for all the classes, which can be used to obtain the classification result.

The GnetFC-v1 successfully reduces the CPU load to zero for the FC layers. However, the maximum number of classes is capped by the maximum number of channels at the last layer of the GnetFC-v1 model due to the limited memory on the CNN accelerator chip. For image classification with large number of classes, such as ImageNet [2] with 1000 classes, the GnetFC-v1 model is infeasible to be implemented on the CNN accelerator chip.

We propose the GnetFC-v2 model to address this problem. The main problem of GnetFC-v1 is inefficiently utilizing the entire feature map of each channel on the CNN accelerator chip. In GnetFC-v2, each pixel on the feature map of 7x7 is used as a classification score connected to a softmax layer. This immediately increase the number of classes for the image classification model by 49x. For example, a tensor with 256-channel feature maps can represent up to 12,544 classes, and a 512 channel will have that number doubled, which is more than 25x for the 1000 classes in ImageNet. Furthermore, if a 14x14 feature map is used to represent the classification scores, the number of classes can be further increased by 4x. That means for a 512 channel, the GnetFC-v2 model can support up to 147,968 classes, which is sufficient for the majority of image classification problems with large number of classes.

The proposed GnetFC-v2 model is trained using ImageNet2012 dataset [10] with 1000 classes. After 5 epochs of training, the accuracy on the training dataset is 69.3%, and the accuracy on validation dataset (50,000 images) is 60.0% on the CNN accelerator chip. More training can further improve the accuracy, which will be the future work. When connected to a host CPU through a USB3.0 interface, the speed of the GnetFC-v2 model on the CNN accelerator chip is 11.95ms (83.68FPS), which includes CNN processings on the CNN accelerator chip and I/O (Input/Output). The I/O includes sending the image to the CNN chip and reading the output tensor from the CNN chip.

The GnetFC-v2 fully uses the output feature map for the image classification task. When using the CNN chip for the object detection task, the same idea can be used for designing the output tensors from the CNN chip to represent the desired format, such as bounding boxes, confidence scores, and classification scores.

Object detection models are usually composed of backbone, detection neck and detection head [1]. The CNN accelerator chip has low-power and fast-speed to extract the features of the input image, but it does not directly support architectures with skip connections, such as FPN [5]. Thus it is straightforward to run the backbone on the CNN accelerator, and leave the remaining of the model to be processed by the host CPU. For example, running the SSD [7] backbone on the CNN chip, and running the detection head on the host CPU. However, the CPU load will be high, thus the FPS will be low. This is not a feasible solution on embeded systems with low-end CPUs.

Similar to GnetFC-v2, the GnetDet model optimizes the object detection model on the CNN accelerator chip, and the object detection result is represented by the output tensor from the chip. Thus no more computation is needed for the detection head on the host CPU. The only load for the host CPU is the NMS, the computation of which is not heavy compared to the CNN model.

Similar to the GnetSeg [14] model which is designed for semantic segmentation tasks on the CNN accelerator chip, the GnetDet model supports input image resolution of both 224x224 and 448x448, and supports RGB/YUV/Y format for the input channels.

The proposed GnetDet model is composed of a backbone and a detection head that is similar to [8]. However, there are three major differences. First, no FC layers inside the model. The CNN accelerator chip only supports convolutional operations, and the detection head is directly connected to the backbone through 3x3 convolutions without any FC layers. Second, only 3x3 conv and 2x2 max pooling are used in the GnetDet model. Third, the number of cells is 14x14 instead of 7x7, in order to improve the detection accuracy.

Depending on the detailed features of different chips, the model architecture of the convolutional layers may vary. We designed two different variants of the GnetDet model, namely GnetDet-Large and GnetDet-Small.

In the context of the CNN accelerator chips, the term of sublayer and major layer will be used. A sublayer is a convolutional layer followed by a Relu function. A major layer has multiple sublayers. Different major layers are separated either by a change of number of channels between sublayers, or by a change of the feature map size caused by downsampling.

The image is split into 14x14 cells. For each cell, there are two bounding boxes which uses 2x4 channels for the coordinates. The confidence scores for the two bounding boxes take another 2 channels. Thus, the output tensor is composed of 10+$C$ channels, where $C$ is the number of classes for the object detection task, which is similar to [8].

We used a combination of VOC2007 [3], VOC2012 [4], and MS-COCO2014 [6] datasets in our experiments. The 4,952 images in the VOC2007 validation dataset is used as our validation dataset in the experiment. The 5,011 images in the VOC2007 training dataset, and the 17,125 images in the VOC2012 training and validation dataset are used for training. In addition, the training and validation data in the COCO2014 dataset is also added to our training dataset. Since COCO2014 dataset has 80 classes, so only the data that overlaps with the VOC2017 20 classes are added in the training. A total of 53,891 images in COCO2014 training and validation dataset are added to our training dataset. Thus we have in total 76,027 images in our training dataset.

Fig. 1 shows some examples of the object detection results given by GnetDet model implemented on the CNN accelerator chip.

Table 1 shows the performance of GnetDet model deployed on the CNN accelerator chip connected to various host CPU platforms. We used three different platforms. The first one is a desktop which has i5-730000HQ [email protected] with USB3.0 interfaces. The second one is an RK3399 having a Dual Core A72 and a Quad Core A53, with both USB2.0 and USB3.0 interfaces. The third one is a Raspberry Pi 3B (RBP3) which has a Quad Core A53 CPU and USB2.0 interface only.

For the inference speed, the time for both I/O and CNN processing are counted. The GnetDet-Large_224_Y model running on the accelerator chip connected to an [email protected] with USB3.0 interface has the inference speed of more than 106FPS. On a RBP3 with USB2.0 interface, this model runs at the speed of 66.67FPS.

The comparison of the FPS metric between USB2.0 and USB3.0 interface on RK3399 shows that the I/O is the bottleneck for inference speed. For USB3.0 on RK3399, the speed for inference and I/O is 100.15FPS. While this number drops to 83.33 when using USB2.0. The comparison of the FPS metric between GnetDet-Large_224_YUV and GnetDet-Large_224_Y on [email protected] also proved this. The computation complexity of these two models are very similar, but the data transferred from the host CPU to the CNN chip on the former model is three times as much as the later model. This further proves that the bottleneck for speed is the I/O interface, but not the CPU. If the host processor has an even faster interface, such as PCIe, the speed could be even faster.

Despite the CNN model of the detector can be fully deployed on the chip, the post-processing of NMS and the pre-processings such as resizing the image and sending the resized image to the chip still runs on the host CPU. The parallel pre/post-processings on the host CPU and the GnetDet model on the CNN accelerator chip can further improve the speed of the whole system, which is out of the scope of this paper.

In order to further improve the accuracy, more data can be added for training. We observe significant accuracy improvement on the VOC2007 validation dataset if we add more training data. If only VOC2007-train and VOC2012-trainval dataset are used for training, the detection accuracy on VOC2007-val is 61.18% for 224 YUV input. When we add the COCO2014 data, the accuracy is improved by more than 3% to 64.20%.

The accuracy of GnetDet model can be further improved by increasing the number of predefined anchors and clustering the predefined these anchors as done in YOLOv3 [9], and using data augmentations and various loss functions as listed in YOLOv4 [1], which will not be expanded in this paper.

For the GnetDet model, all the convolutional computations are running on the low-power and high-speed CNN accelerator. The only CPU load is for the NMS block where the computation is not intensive. However, for scenarios where CPU load is very sensitive, GnetDet can be further modified to remove this NMS block. Recent work [26] has successfully built end-to-end object detection model without the NMS block, with negligible accuracy drop.