Hardware Architecture of Embedded Inference Accelerator and Analysis of Algorithms for Depthwise and Large-Kernel Convolutions

The upper part of Figure 1 shows the two modes for regular convolutions and depthwise convolutions supported by the proposed hardware architecture. The main feature of the architectures is the function of hardware mode switching. On the left side, all of the input feature maps for parallel processing are stored in the memory to generate the same numbers of output feature maps. On the right side, only 1 input feature map for parallel processing is stored in the memory to generate multiple output feature maps. When the regular convolution mode is enabled, the intermediate convolution results of multiple input feature maps are accumulated to generate the final convolution results for a single output feature map. When the depthwise convolution mode is enabled, the intermediate convolution results of multiple input feature maps become the final convolution results of multiple output feature maps. In both of the two modes, the hardware architecture can generate the same numbers of convolution results, and the multipliers in the convolution cores are fully utilized. The overhead of the hardware architecture is the buffer to store the data of multiple input feature maps in the depthwise convolution mode. However, large-kernel convolutions can be efficiently handled since the proposed hardware architecture does not require extra costs for intra-kernel parallelism [18].

The lower part of Figure 1 shows an example of the network arhictecture that is focused in this paper. There are 3 layers in this example, the depthwise convolution layer, the activation layer, and the pointwise convolution layer. The depthwise convolution layer may include the filters with kernels larger than or equal to $3\times 3$. Different from the related works, such as MobileNet [5], the depthwise convolution layers also include dilated convolution kernels, which increase the size of the receptive field and increase the accuracy of the inference result for some applications. The combination of dilated convolutions and depthwise convolutions is abbreviated as DDC. The activation layers include functions such as Rectified Linear Unit (ReLU) and quantization functions. The pointwise convolution layer may include the filters with kernel sizes equal to $1\times 1$ but not limited to $1\times 1$. When the kernels size is not $1\times 1$, the pointwise convolutions can also be regarded as regular convolutions.

The proposed hardware architecture is shown in Figure 2, where the black blocks represent the memory sets, and the white blocks represent the logics. The memory sets include the feature map memory and the filter weight memory. The feature map memory and the weight memory are used to store the input/output feature maps and the filter weights, respectively. The logics include the address generator, the convolution layer processing unit (CLPU), and the activation and pooling layer processing unit (APLPU). The address generator receives the information of network architectures and generates the addresses for the feature map memory and the weight memory. The blocks in the spatial domain of the feature maps are processed sequentially.

The CLPU includes $PE_{num}$ convolution cores to compute the results of convolutions of the input feature maps and the input filter weights in parallel. Each convolution core contains $MAC_{PE}$ sets of multipliers and adders to execute MAC operations.

Figure 3 shows a time chart of the hardware architecture, where the operations are executed in pipeline. In this example, we set $IC=2PE_{num}$ for simplicity. Note that $IC$ represents the number of input feature maps, and $PE_{num}$ denotes the number of convolution cores. The same operations can be applied to regular convolutions and depth-wise convolutions with difference kernel sizes.

The upper part of Figure 3 shows an example of regular convolutions. There are $IC$ input feature maps and $OC$ output feature maps in this layer. From cycle 0 to cycle $T$, the 1st input feature map and the corresponding filter weights are transferred. From cycle $T$ to cycle $2T$, the 2nd input feature map and the corresponding filter weights are transferred, and the results of multiplications in the convolution operations based on the 1st input feature maps are computed. From cycle $2T$ to cycle $3T$, the 3rd input feature map and the corresponding filter weights are transferred. The results of multiplications in the convolution operations based on the 2nd input feature maps are computed, and the results of accumulations in the convolution operations based on the 1st input feature maps are computed. From cycle $3T$ to cycle $(IC+2)T$, the results of convolution operations based on the remaining input feature maps are computed. From cycle $(IC+2)T$ to cycle $(IC+3)T$, the results of ReLU based on the convolutions of the 1st to the $IC$-th input feature maps are generated.

The lower part of Figure 3 shows an example of depthwise convolutions. There are $IC$ input feature maps and $OC$ output feature maps in this layer, and the values of $OC$ and $IC$ are the same. From cycle 0 to cycle $T$, the 1st to the $IC/2$-th input feature maps and the corresponding filter weights are transferred. From cycle $T$ to cycle $2T$, the $(IC/2+1)$-th to the $IC$-th input feature maps and the corresponding filter weights are transferred, and the results of multiplications in the convolution operations based on the 1st to the $IC/2$-th input feature maps are computed. From cycle $2T$ to cycle $3T$, the results of multiplications in the convolution operations based on the $(IC/2+1)$-th to the $IC$-th input feature maps are computed, and the results of accumulations in the convolution operations based on the 1st to the $IC/2$-th input feature maps are computed. From cycle $3T$ to cycle $4T$, the results of accumulations in the convolution operations based on the $(IC/2+1)$-th to the $IC$-th input feature maps are computed, and the results of ReLU based on the convolutions of the 1st to the $IC/2$-th input feature maps are generated. From cycle $4T$ to cycle $5T$, the results of ReLU based on the convolutions of the $(IC/2+1)$-th to the $IC$-th input feature maps are generated.

Figure 4 shows the proposed algorithms. There are 4 levels in the nested loop structure. In the 1st level of loops, each convolution layer of the networks is processed sequentially, and the parameters of the corresponding layer are set. The mode for depthwise convolutions and regular convolutions is selected in this level.

When the mode for regular convolutions is selected, the operations in the 2nd level to the 4th level of loops are executed. First, in the 2nd level of loops, each output channel of the layer is processed sequentially. Then, in the 3rd level of loops, each block of the output channel is processed sequentially. Finally, in the 4th level of loops, each input channel is processed sequentially. The filter weights of the $i$-th convolution layer, the $j$-th output channel, the $n$-th input channel are set, and the convolution results of $PE_{num}$ output blocks are computed in parallel.

When the mode for depthwise convolutions is selected, the operations in the 2nd level to the 3rd level of loops are executed. The number of input channels is equal to the number of output channels, and each channel is processed sequentially. The filter weights of the $i$-th convolution layer, the $j$-th output channel, the $j$-th input channel are set, and the convolution results of $PE_{num}$ output blocks are computed in parallel.

Since the number of loops is reduced when the mode for depthwise convolutions is selected, redundant operations, in which some of filter weights are set to zeros, are not required. The common architecture for the two modes makes the hardware achieve higher utilization than the related works [6, 11].

One of the strengths of the proposed hardware architecture is the function to handle the combination of large-kernel convolutions and depthwise convolutions. To analyze the effectiveness of large-kernel convolutions and depthwise convolutions, the experiments for two kinds of applications are performed. The first application is face detection, and the second application is image classification. Two experiments are performed for both of the applications.

In the first experiment, the accuracy of face detection on the WIDER FACE dataset [17] is analyzed. The network of RetinaFace [3] is tested with depthwise convolutions and large-kernel convolutions, and the backbone network is MobileNetV1-0.25 [5].

The WIDER FACE dataset [17] consists of 32,203 images and 393,703 face bounding boxes with a high degree of variability in scale, pose, expression, occlusion and illumination. In order to compare the proposed network architecture with RetinaFace, the WIDER FACE dataset is split into training (40%), validation (10%) and testing (50%) subsets, and three levels of difficulty (i.e. Easy, Medium, and Hard) are defined by incrementally incorporating hard samples. We use the WIDER FACE training subset as the training dataset, and the WIDER FACE validation subset as the testing dataset. The operations in the context module, which is an important structure used to increase the receptive field, include a series of $3\times 3$ convolutions. Since the feature maps are processed with different numbers of $3\times 3$ filters consecutively, the effects are similar to applying large-kernel convolutions (e.g. $5\times 5$, $7\times 7$) to the feature maps. To show the advantage of the functions of the proposed hardware architecture, some of the operations are replaced with depthwise convolutions and dilated filters. To be specific, two cascaded $3\times 3$ convolutions are replaced with a single $3\times 3$ convolution where the dilation rate is 1, and three cascaded $3\times 3$ convolutions are replaced with a single $3\times 3$ convolution where the dilation rate is 2.

The accuracy of RetinaFace [3] and its variations are shown in Table 1, which includes the proposed network architecture, the depthwise and dilated convolution (DDC) layers. Since the accuracy of the network without the context module is not available in the original paper [3], we add an ablation study to verify the effectiveness of the context module. The results show that the context module can increase the accuracy of face detection by about 1% for the Easy category and the Medium category, and by 4% for the Hard category. The proposed hardware can support the quantized RetinaFace and its variations. In the quantized networks, the feature maps and the filter weights are both quantized into 8-bit data. It is shown that the quantized versions have similar accuracy with the floating-point versions.

When the filters in the context module are replaced with the DDC layers and pointwise convolutions, the accuracy decreases by less than 1% for all the categories. The parameter settings are shown in Table 2. We trained the RetinaFace using the SGD optimizer (momentum at 0.9, weight decay at 0.0005, and batch size of 32) on the NVIDIA Titan Xp GPUs with 12GB memory. The learning rate starts from $10^{-3}$, and is divided by 10 at the 190th and at the 220th epoch. The training process terminates at the 250th epoch.

The computational costs and the model sizes of RetinaFace [3] and the proposed work are shown in Table 3. When the operations in the context module are replaced with depthwise convolutions and dilated filters, the model size of the context module decreases from 138 KB to 23 KB, and the computational cost of the context module decreases from 708 MACs per input pixel to 119 MACs per input pixel. By applying depthwise convolutions and dilated filters to the network, the total computational costs decrease by about 30%, and the total model size decreases by about 20%.

In the second experiment, to show the relation between the size of filter kernels and the inference result, the accuracy of image classification on ImageNet [9] is analyzed. Similar to the first experiment, MobileNetV1-0.25 is used for testing. Since the architecture of MobileNetV1 [5] is composed of depthwise convolution layers, it can be used to evaluate the effectiveness of large-kernel convolutions by simply increasing the kernel sizes of the filters. The accuracy of MobileNetV1 and its variations is shown in Table 4. There are two networks with dilated convolutions, where the concept of hybrid dilated convolutions is adopted [14]. When some $3\times 3$ filters in MobileNetV1 are replaced by $5\times 5$ filters, the accuracy is increased by more than 1%. When some of $3\times 3$ filters are replaced by dilated $3\times 3$ filters, the network can also achieve higher accuracy than the original architecture.

The first experiment shows that the combination of depthwise convolutions and dilated convolutions keeps the accuracy while reducing the computational costs, and the second experiment shows that the combination of depthwise convolutions and large-kernel convolutions can increase the accuracy.

In brief, these results show the importance of supporting depthwise convolutions and large-kernel convolutions, including dilated convolutions, with the proposed hardware architecture.

To compare the proposed hardware architecture with the related work, the processing time for regular convolutions and depthwise convolutions is expressed as equations. The processing time (number of cycles) for the regular convolutions is shown as follows.

where $T_{M}$ and $T_{C}$ are the memory access time and the computational time, respectively. The processing time can be expressed as $\max(T_{M},T_{C})$, and the definition of parameters in the equations are shown as follows.

$IC$: the number of input channels.
$OC$: the number of output channels.
$IN\times IM$: the size of input blocks.
$ON\times OM$: the size of output blocks.
$X\times Y$: the size of filters.
$BW_{FM}$: the bandwidth to transfer feature maps.
$BW_{W}$: the bandwidth to transfer filter weights.
$PE_{num}=8$: the number of processing elements.
$MAC_{PE}=64$: the number of parallel MAC operations implemented in 1 processing element.

The memory access time and the computational time for the depthwise convolutions are shown as follows. Similarly, the processing time can be expressed as $\max(T_{M},T_{C})$.

All of the processing elements are used for both regular convolutions and depthwise convolutions.

In the related work [18], the computational costs (the number of cycles) for the convolution operations are also expressed as equations. To compare the efficiency, we set $PE_{num}=64$, and $MAC_{PE}=8$ for the related work. For simplicity, $IC$ and $OC$ are set to the same value for both regular convolutions and depthwise convolutions. The results are shown in Fig. 5, where (a) and (b) refer to the cases when the size of filter kernels is $3\times 3$, and (c) and (d) refer to the cases when the size of filter kernels is $5\times 5$. For regular convolutions, the computational time of the proposed work and the related work is almost the same when the size of filter kernels is $3\times 3$ or $5\times 5$. For depthwise convolutions, the proposed work has shorter computational time than the related work when the size of filter kernels is $3\times 3$ or $5\times 5$. In the proposed architecture, since the filter weights and the feature maps can be transferred in parallel, the memory access time can be shortened so that the operations of dilated convolutions achieve faster speed than the related work. Also, since the proposed architecture can process the filter weights sequentially, the computational time is proportional to the size of filter kernels, and the computational efficiency does not decrease when the size of filter kernels becomes large.

The specifications of the proposed embedded inference accelerator, which is designed to process modern CNNs, are shown in Table 5. The main feature of the proposed architecture is that it can compute depthwise convolutions with large kernels, including the DDC layers mentioned in the previous section.

We use the 28-nm CMOS technology library for the experiments. The resolution of input images is VGA, and the performance is 512 MAC operations per clock cycle for 8-bit feature maps and 8-bit filter weights. As shown in Figure 2, since regular convolutions and depthwise convolutions can be selected according to the type of layers, the hardware architecture can also process RetinaFace (with MobileNetV1-0.25) efficiently. The gate counts of the CLPU and the APLPU are 1.50M and 0.47M, respectively. The processing speed for RetinaFace, which includes the post-processing algorithm, is 150fps. Compared with RetinaFace, it takes only 83% of processing time to compute the inference result of the proposed network architecture, DDC layers. Since the proposed hardware can support depthwise convolutions with large filter kernels, the target networks can be switched according to the requirement of processing time.

Table 6 shows the comparison of hardware specifications with three related works. Since it is difficult to compare the performance of FPGA and ASIC, we focus on the hardware utilization of MAC units for regular convolutions and depthwise convolutions. In the first related work [6], when dealing with depthwise convolutions, the filter kernels are filled with zeros, and the advantage is that both regular convolutions and depthwise convolutions can be computed by using the same hardware architecture. Suppose that the hardware can handle regular convolutions with $T_{n}$ input channels and $T_{m}$ output channels in parallel, the hardware utilization becomes $\frac{100}{T_{m}}\%$ when dealing with depthwise convolutions with $T_{n}$ input channels and $T_{m}$ output channels. In the second related work [11], regular convolutions and depthwise convolutions are handled with different hardware architectures. The advantage is that both regular convolutions and depthwise convolutions can be handled efficiently, but some of the processing elements do not function when either depthwise convolutions or regular convolutions are handled. In the third related work [18], the processing elements can effectively handle regular convolutions and depthwise convolutions, but as discussed in Sec. 3.2, the performance decreases when the kernel size of filters is larger than or equal to $5\times 5$.

The proposed architecture can handle regular convolutions and depthwise convolutions efficiently with large-kernel filters. The overhead to support the function is the buffer for input feature maps, which is mentioned in Sec. 2.1.