A Learning Framework for $n$-bit Quantized Neural Networks toward FPGAs

Since the amount of the model capacity is too large, it is necessary to cut down it to perform CNNs on FPGAs, which is consistent with the purpose of deep compression. In general, deep compression can be divided into three categories, i.e., pruning, Huffman coding, and quantization. The pruning method will simplify the deep neural network by cutting off the network connections with small weights on the normal trained network [18] [19] [20]. The Huffman coding method is an optimal code used for lossless data compression [21] which uses entropy to encode source symbols by variable-length codewords. Han et al. [20] show that 20% - 30% of the network storage will be saved after Huffman coding the non-uniformly distributed values. When considering perform compressed networks on FPGAs, the network after pruning is an asymmetric structure, which is unsuitable for hardware implementation, and the Huffman coding may only be regarded as a post-compression combined with the other two compression methods, so most of the hardware accelerators will focus on the quantization method.

The quantization based method normally employ the low-precision weights, varied from 1 bit to 5 bits, to represent the CNNs [15, 22, 23, 24, 25, 26, 27, 28]. Some studies train QNNs from scratch by estimating the gradient of expected loss based on Straight-Through Estimator [15, 23, 22, 27]. For example, Courbariaux et al. [15] train a classification neural network from scratch with 1-bit weight and activation, which can run 7 times faster than the CNNs. Choi et al. [23] propose a neural quantization scheme called Parameter Clipping Activation, which uses a parameter to find the optimal quantization scale for arbitrary bit width activations. Choi et al. [22] introduce a novel technique called Statistics-Aware Weight Binning, which finds the optimal scaling factor based on statistical characteristics of the distribution of the weights to minimize the quantization error. The QNNs trained by the above quantization methods only accelerate the inference, Zhou et al. [27] propose a DoReFa-Net that can accelerate both training and inference by low bit width weights, activations and gradients respectively. However, these estimator-based methods have a noise compared to the real gradient. Thus these QNNs can’t achieve an ideal classification accuracy, especially on multi-classification datasets such as CIFAR-100.

Some other quantization methods are dedicated to design special strategies to fine-tune QNNs, which will not rely on the backpropagation algorithm and can bypass the problem of gradient vanishing [29, 26, 28]. They can achieve a much better accuracy as they are independent of estimators. For example, Park et al. [29] propose precision highway that has an end-to-end high-precision information flow for ultra-low-precision computation. This linear weight quantization method is based on the assumption that the weight distribution is the Laplace distribution. Recently, Zhou et al. [26] propose an incremental network quantization method, which converts pretrained full-precision CNNs model into a low-precision model where the weights are constrained to the power of two or zero. It has been studies that there will be little loss on the classification accuracies when using 2-5 bits low-precision weight [27, 26]. However, these quantization methods will depend on the pretrained network structure rather than the backpropagation algorithm, which will be difficult to satisfy the network-structure-optimization requirements due to the hardware limitation.

Considering the inference, the CNNs have a highly hierarchical structure of multiple feature maps, whose structure exposes a large amount of parallelism that makes CNNs very suitable for FPGAs implementation. This structure builds on the accumulation of a huge number of convolutions that will consume a huge number of floating-point resources on FPGAs. In addition, the structure of CNNs often contains many convolutional layers. Thus the convolution module with different parameters needs to be executed iteratively during the inference. Frequent execution of data caching and parameter loading will be limited by the bandwidth. Therefore, in many studies, their hardware structures of CNNs are designed mainly for the two bottlenecks of floating-point resources and bandwidth [12, 17, 16, 13].

In terms of optimizing for floating-point resources, Lu et al. [30] design a fast Winograd algorithm, which can decrease the use of floating-point resources on FPGAs and reduces the complexity of convolution dramatically. Simultaneously, they also give the formula for estimating the computational efficiency, which demonstrates that the fast Winograd algorithm is more efficient than conventional convolutional algorithm due to the use of fewer floating-point resources on FPGAs. Meloni et al. [13] present an accelerator configuration for CNNs that reaches more than 97% DSP resource utilization at 150 MHz operating frequency with 16-bit precision. And they show that the floating-point resource utilization is the highest when executing 3$\times$3 filters on FPGAs.

Other studies have focused on optimizing the data scheduling structure to reduce the impact of the bandwidth. For example, Sankaradas et al. [14] implement a vector processing element (VPE) array coprocessor, which can accelerate the CNNs by optimizing the cache between distributed off-chip memory banks and on-chip computing elements on FPGAs. Peemen et al. [12] show that their scheduler prefers to use only convolutional layers without fully connected layers on FPGAs, which can maximize the efficiency of on-chip memories by reducing the impact of the bandwidth bottleneck.

The crucial issue with the above methods is that they usually only consider the bottleneck at a single level and fail to coordinate these two constraints to improve the computational efficiency of the hardware accelerators. In this paper, we reduce the impact of the above two constraints by introducing the QNNs into FPGAs, which provides a new idea to deal with the above two bottlenecks. Since the weights in our QNNs are quantized to the power of two, the quantized weights directly reduce the bandwidth required to load the weights. In addition, the use of the quantized weights can translate the multiplication into shifting in convolution module, which greatly reduces the use of floating-point resources.

The main idea of our $n$-BQ-NN is based on Fig.1, which shows the information loss led by the quantization method with the power of $2$ can be interpreted as the sampling loss caused by non-uniform sampling. In fact, the weights of CNNs with large absolute values will be dominant to the overall classification accuracy of the networks, although these weights with large values only account for a small ratio among all the weights [11, 31]. For an arbitrary probability density function of the weights in a neural network, denoted as $\phi(x)$, we can use the blue curve in Fig.1 to represent $\phi(x)$ which meets $\int_{-1}^{1}\phi(x)dx=1$. In this way, we can calculate the sampling loss $\Phi(x)$ which can be represented as the area between two distributions (red and blue curves) in Fig.1. By calculating, the sampling loss $\Phi(x)$ is represented as the following recursive formula, where $n$ is the quantized bit width of the weights.

It can be seen from the above formula that the sampling loss always decreases as the increasing of quantized bit width of the weights, which indicates that the sampling loss is negatively related to the quantized bit width of weights. However, the bit width is limited and needs to reduce as much as possible in QNNs. Thus, finding the best balance between quantized bit width and the sampling loss is the key to balancing the performance, speed, and resources of QNNs. We define the $\mathcal{L}(n)=2^{1-n}[\phi(2^{1-n})-\phi(2^{-n})]$ as the variation between two sampling losses, $\Phi(n)$ and $\Phi(n-1)$, from Eq.1. Then, we can prove that $\mathcal{L}(4)$ will approach to zero in our quantization method with the power of $2$, which can be ensured by the Theorem 1. Therefore, continuing to increase the quantized bit width of $n$ after $3$ is not helpful to decrease the sampling loss.

From the hardware perspective, the resource consumption of SHIFT operation is much less than multiplication, so our intention is to use the SHIFT operation instead of multiplication. Considering that the shift right operation will make the weights exceed the constraint range of ($-1$,$1$), thus, all SHIFT operations are shift left and every quantized weight is chosen from the entries $(\pm 2^{-0},\pm 2^{-1},\cdots,\pm 2^{-i},0)$, where $\pm 2^{-i}$ indicates its multiplication can be calculated by $<<i$ and $0$ indicates that no operations are required. Our $n$-BQ-NN quantizes the weights to the entries, which are encoded to $n$-bit and suitable for hardware computation. Under such circumstance, the staircase function $\operatorname{staircase}(W)$ can be used to describe our $n$-bit quantized weights as Eq.4 (typically, $n$ is greater than $1$, and $\operatorname{staircase}(W)$ is degraded to $\operatorname{sign}(W)$ if $n$ is equal to $1$), where $W$ are full-precision weights.

Here $i$ is taken from $r-1$ to $0$ in turn, where $r=2^{n-1}-1$ and $\operatorname{sign}(W)$ is the sign function:

In order to facilitate the discussion as follows, we need to define some variables first, where $W^{l}_{jk}$ represents the weight that connects the $k$-th neuron of the $(l-1)$-th layer to the $j$-th neuron of the $l$-th layer, $b^{l}_{j}$ represents the bias of the $j$-th neuron of the $l$-th layer, $z^{l}_{j}$ represents the input of the $j$-th neuron of the $l$-th layer ($z^{l}_{j}=\sum_{k}W^{l}_{jk}a^{l-1}_{k}+b^{l}_{j}$), $a^{l}_{j}$ represents the output of the $j$-th neuron of the $l$-th layer ($a^{l}_{j}=\theta(z^{l}_{j})$), and $\theta$ is activation function.

We also have to add a extra quantized weight so that we can train our $n$-BQ-NN, where the quantized weight is shown as follows,

And the cost function of mini-batch of $m$ samples in our $n$-BQ-NN is,

Where $x$ is the input sample, $y$ is the actual classification, $a^{L}$ is the prediction output, and $L$ is the maximum number of layers in the network.

By defining $\mathcal{T}_{j}^{l}\equiv\frac{\partial C}{\partial z_{j}^{l}}$ as the error produced by the $j$-th neuron of the $l$-th layer, we can use the back-propagation algorithm to calculate the gradient and update the parameters according to the following three steps¹¹1$\odot$ represents the Hadamard product that is used for point-to-point product between matrices or vectors..

1. -

   Calculating the error of the last layer of the network.

   		$\displaystyle\mathcal{T}_{j}^{L}=\frac{\partial C}{\partial z_{j}^{L}}=\frac{\partial C}{\partial a_{j}^{L}}\cdot\frac{\partial a_{j}^{L}}{\partial z_{j}^{L}}$		(8)
   		$\displaystyle{\bm{\mathcal{T}}^{L}=\frac{\partial C}{\partial\textbf{a}^{L}}\odot\frac{\partial\textbf{a}^{L}}{\partial\textbf{z}^{L}}=\nabla_{\textbf{a}}C\odot\theta^{\prime}\left(\textbf{z}^{L}\right)}$

2. -

   Calculating the error of each layer of the network from the back to the front.

   	$\displaystyle\mathcal{T}_{j}^{l}=\frac{\partial C}{\partial z_{j}^{l}}$	$\displaystyle=\sum_{k}\frac{\partial C}{\partial z_{k}^{l+1}}\cdot\frac{\partial z_{k}^{l+1}}{\partial a_{j}^{l}}\cdot\frac{\partial a_{j}^{l}}{\partial z_{j}^{l}}$		(9)
   		$\displaystyle=\sum_{k}\mathcal{T}_{k}^{l+1}\cdot\frac{\partial\left(\hat{W}_{kj}^{l+1}a_{j}^{l}+b_{k}^{l+1}\right)}{\partial a_{j}^{l}}\cdot\theta^{\prime}\left(z_{j}^{l}\right)$
   		$\displaystyle=\sum_{k}\mathcal{T}_{k}^{l+1}\cdot\hat{W}_{kj}^{l+1}\cdot\theta^{\prime}\left(z_{j}^{l}\right)$
   	$\displaystyle\bm{\mathcal{T}}^{l}=$	$\displaystyle\left(\left(\hat{\textbf{W}}^{l+1}\right)^{T}\bm{\mathcal{T}}^{l+1}\right)\odot\theta^{\prime}\left(\textbf{z}^{l}\right)$

3. -

   Calculating the gradient of weight and bias respectively.

   	$\displaystyle g^{b}$	$\displaystyle=\frac{\partial C}{\partial b_{j}^{l}}=\frac{\partial C}{\partial z_{j}^{l}}\cdot\frac{\partial z_{j}^{l}}{\partial b_{j}^{l}}$		(10)
   		$\displaystyle=\mathcal{T}_{j}^{l}\cdot\frac{\partial\left(\hat{W}^{l}_{jk}a_{k}^{l-1}+b_{j}^{l}\right)}{\partial b_{j}^{l}}=\mathcal{T}_{j}^{l}$

   	$\displaystyle g^{W}$	$\displaystyle=\frac{\partial C}{\partial W_{jk}^{l}}=\frac{\partial C}{\partial z_{j}^{l}}\cdot\frac{\partial z_{j}^{l}}{\partial W_{jk}^{l}}=\mathcal{T}_{j}^{l}\cdot\frac{\partial\left(\hat{W}_{jk}^{l}a_{k}^{l-1}+b_{j}^{l}\right)}{\partial W_{jk}^{l}}$		(11)
   		$\displaystyle=\mathcal{T}_{j}^{l}\cdot\frac{\partial\left(\hat{W}_{jk}^{l}a_{k}^{l-1}+b_{j}^{l}\right)}{\partial\hat{W}_{jk}^{l}}\cdot\frac{\partial\hat{W}_{jk}^{l}}{\partial W_{jk}^{l}}$
   		$\displaystyle=\mathcal{T}_{j}^{l}\cdot a^{l-1}_{k}\cdot\frac{\partial\hat{W}_{jk}^{l}}{\partial W_{jk}^{l}}$

In the above process of deriving the entire back-propagation, except for the gradient of weight of the last step, the other steps are well-defined. Based on the Eq.11, the gradient of weight can be calculated as follows,

Where $\frac{\partial\hat{W}_{jk}^{l}}{\partial W_{jk}^{l}}$ is exactly the $\operatorname{staircase}^{\prime}(W_{jk}^{l})$, which is the derivative of $\operatorname{staircase}(W_{jk}^{l})$. And this derivative satisfies the conditions of Dirac Delta Function $\delta(x)$. According to the properties of $\delta(x)$, $\frac{\partial\hat{W}_{jk}^{l}}{\partial W_{jk}^{l}}$ can be calculated as follows,

Substituting $\frac{\partial\hat{W}_{jk}^{l}}{\partial W_{jk}^{l}}=0$ into Eq.12, we discover that model cannot be trained by back-propagation algorithm due to gradient vanishing.

To resolve the above problem, we reconstruct the quantized weight function as Eq.14 to ensure that the weights can be updated by using the back-propagation algorithm as shown in Fig.2, the blue full line, where $\alpha$ is an adjustable parameter in the range of $(0,1)$.

By substituting Eq.14 into Eq.11, we can recalculate the gradient of weight as follows again with $\frac{\partial\tilde{W}_{jk}^{l}}{\partial W_{jk}^{l}}=\alpha+(1-\alpha)\delta(W^{l}_{jk})=\alpha$,

At this point, we have reconstructed the quantized weight function as Eq.14 to solve the gradient vanishing, but the weights cannot be quantized to the entries $(\pm 2^{-0},\pm 2^{-1},\cdots,0)$ directly as Eq.6. However, we can prove that the reconstructed quantized weight function will approximate to the entries after several iterations, which can be ensured by the Theorem 2.

Assumption. Since the algorithm needs to be iterated, our problem needs to be discussed within the framework of the series. We define $W^{l}_{jk}$ as an iteration of $x_{n}$, $\tilde{W}^{l}_{jk}$ is equivalent to $x_{n+1}$, and the value of $a_{j}$ is chosen from $\hat{W}^{l}_{jk}=\operatorname{staircase}(W^{l}_{jk})=(\pm 2^{-0},\pm 2^{-1},\cdots,\pm 2^{-i},0)$.

The design of $\alpha$ in our reconstructed quantized weight function takes three aspects into consideration: First, the designed function must satisfy the Theorem 2. Second, our reconstructed function indicates that the ratio of $1-\alpha:\alpha$ between quantized weights and full-precision weights can be used to adjust the information ratio of between quantized weights and full-precision weights in the training process. Third, on the other hand, $\alpha$ is the slope of our reconstructed function shown as the blue full line in Fig.2, which can be used to change the gradient descent rate of back-propagation based on Eq.15 during the training.

In the initialization of the networks, the initialization modes MSRA and Xavier [32] which will adjust variance based on the number of inputs are prone to converge than the traditional Gaussian distribution initialization mode with fixed variance in DNNs. Inspiring by this fact, we suspect that adjusting the distribution of quantized weights may make it easier for us to train our $n$-BQ-NN. Here, we consider that full-precision networks are prone to converge than quantized networks; thus, we prefer to keep the distribution of quantized weights consistent with full-precision weights’. Comparing the probability density function before quantization $\phi(x)$ (its corresponding expectation and variance are $E(x)$ and $Var(x)$, respectively) and the probability density function after quantization $\operatorname{staircase}(x)$ (as Eq.4), we make their expectation and variance equal respectively so that their distribution is consistent as follows,

The original full-precision probability density function $\phi(x)$ and the value of quantized weight function $\operatorname{staircase}(x)$ are fixed, so we can only adjust the value range of $\operatorname{staircase}(x)$ to meet Eq.18.

In actual training algorithm for $n$-BQ-NN, the Batch Normalization (BN [31]) is added in our $n$-BQ-NN because it is conducive to reduce the overall impact of the weight scale and accelerate the training. Thus, we will derive the back-propagation algorithm for $n$-BQ-NN with BN and give the training algorithm in this subsection.

First, we define four variables of BN, where $\sigma$ represents the variance of all samples of a batch, $\mu$ represents the sample mean, and $\gamma,\beta$ are the scale variation coefficients. Due to the existence of BN, the bias term can be ignored, so the input of the neuron is re-expressed as $z^{l}_{j}=\sum_{k}W^{l}_{jk}a^{l-1}_{k}$, the normalized input of the neuron is $\hat{z}_{j}=\gamma\frac{z_{j}-\mu}{\sigma}+\beta$, and the output of the neuron is $a_{j}=\theta(\hat{z}_{j})$. Then, we can calculate the error and the gradient, based on the discussion of Section III-B, according to the following three steps.

1. -

   Counting the mean and variance of the sample, and calculating the gradient of them.

   		$\displaystyle\mu=\frac{1}{m}\sum_{j=1}^{m}z_{j}$		(19)
   		$\displaystyle\sigma^{2}=\frac{1}{m}\sum_{j=1}^{m}(z_{j}-\mu)^{2}$

   	$\displaystyle\frac{\partial C}{\partial\sigma^{2}}$	$\displaystyle=\sum_{k}\frac{\partial C}{\partial a_{k}}\frac{\partial a_{k}}{\partial\hat{z}_{k}}\frac{\partial\hat{z}_{k}}{\partial\sigma^{2}}$		(20)
   		$\displaystyle=-\frac{1}{2}\gamma\sigma^{-3}\sum_{k}\frac{\partial C}{\partial a_{k}}\theta^{\prime}\left(z_{k}\right)\left(z_{k}-\mu\right)$

   	$\displaystyle\frac{\partial C}{\partial\mu}$	$\displaystyle=\sum_{k}\frac{\partial C}{\partial a_{k}}\frac{\partial a_{k}}{\partial\hat{z}_{k}}\frac{\partial\hat{z}_{k}}{\partial\mu}+\frac{\partial C}{\partial\sigma^{2}}\frac{\partial\sigma^{2}}{\partial\mu}$		(21)
   		$\displaystyle=-\frac{\gamma}{\sigma}\sum_{k}\frac{\partial C}{\partial a_{k}}\theta^{\prime}\left(z_{k}\right)-\frac{2}{m}\frac{\partial C}{\partial\sigma^{2}}\sum_{k}\left(z_{k}-\mu\right)$

2. -

   Calculating the error of the network.

   		$\displaystyle\mathcal{T}_{j}=\frac{\partial C}{\partial z_{j}}=\frac{\partial C}{\partial a_{j}}\frac{\partial a_{j}}{\partial\hat{z}_{j}}\frac{\partial\hat{z}_{j}}{\partial z_{j}}+\frac{\partial C}{\partial\sigma^{2}}\frac{\partial\sigma^{2}}{\partial z_{j}}+\frac{\partial C}{\partial\mu}\frac{\partial\mu}{\partial z_{j}}$		(22)
   		$\displaystyle=\frac{\gamma}{\sigma}\frac{\partial C}{\partial a_{j}}\theta^{\prime}\left(\hat{z}_{j}\right)+\frac{2}{m}\sum_{k}\left(z_{k}-\mu\right)\frac{\partial C}{\partial\sigma^{2}}+\frac{1}{m}\frac{\partial C}{\partial\mu}$

3. -

   Calculating the gradient of weight, $\gamma$, and $\beta$ respectively.

   $$g^{W}=\mathcal{T}_{j}\cdot a_{k}\cdot\frac{\partial\tilde{W}_{jk}}{\partial W_{jk}^{l}}=\alpha\mathcal{T}_{j}\cdot a_{k}$$

   (23)

   $$g^{\gamma}=\sum_{k}\frac{\partial C}{\partial a_{k}}\frac{\partial a_{k}}{\partial\hat{z}_{k}}\frac{\partial\hat{z}_{k}}{\partial\gamma}=\sum_{k}\frac{\partial C}{\partial a_{k}}\theta^{\prime}\left(\hat{z}_{k}\right)\frac{z_{k}-\mu}{\sigma}$$

   (24)

   $$g^{\beta}=\sum_{k}\frac{\partial C}{\partial a_{k}}\frac{\partial a_{k}}{\partial\hat{z}_{k}}\frac{\partial\hat{z}_{k}}{\partial\beta}=\sum_{k}\frac{\partial C}{\partial a_{k}}\theta^{\prime}\left(\hat{z}_{k}\right)$$

   (25)

With the foundation of the above formulas, we can propose our training algorithm for $n$-BQ-NN, as indicated in Algorithm 1. This algorithm covers two learning modes: training from scratch and fine-tuning on the pretrained model, where the first mode means the weights are randomly initialized and the second mode means the weights are initialized by the pretrained full-precision network model. The overall quantization process is illustrated as Fig.3. The code for training algorithm is available ²²2https://github.com/papcjy/n-BQ-NN.

The above discussion is all about the quantization of weights. To take the integrity of our $n$-BQ-NN and the necessity of subsequent ablation experiments into consideration, we need to discuss the quantization of activations in this subsection. Now, let’s put our eyes back on Section III-B. In the case of the quantized activations, the output of the $j$-th neuron of the $l$-th layer can be rewritten as,

Where $\operatorname{staircase}()$ is activation function.

At this point, we have encountered the same problem, the error of network $\mathcal{T}_{j}^{l}$ becomes zero due to the existence of $\frac{\partial\hat{a}_{j}^{l}}{\partial z_{j}^{l}}$, when the Eq.26 is substituted into Eq.8 and Eq.9.

Considering the expectation of $\frac{\partial C}{\partial z_{j}^{L}}$, the error of network has reappeared, which is guaranteed by Theorem 3.

Under the Theorem 3, we can re-express the error of network and quantize the activations in our $n$-BQ-NN by rewriting the Eq.8 and Eq.9 as follows,

The MNIST dataset [35] consists of handwritten digit images with 32$\times$32 pixels, organized into 10 classes (0 to 9). The training and test sets contain 60,000 and 10,000 images respectively. We perform this dataset without data augmentation [36].

The two CIFAR datasets [37] consist of natural color images with 32$\times$32 pixels, respectively 50,000 training and 10,000 test images, and we hold out 5,000 training images as a validation set from the training set. CIFAR-10 (C10) consists of images organized into 10 classes and CIFAR-100 (C100) into 100 classes. We adopt a standard data augmentation scheme (random corner cropping and random flipping) that is widely used for these two datasets [38, 39, 40, 34, 41, 36, 33, 38]. We normalize the images using the channel means and standard deviations in preprocessing.

The SVHN dataset [42] consists of color images of house numbers collected by Google Street View with 32$\times$32 pixels, organized into 10 classes (0 to 9). There are 73,257 images in the training set, 531,131 images for additional training, and 26,032 images in the test set respectively. We divide the pixel values by 255.0 so that they are in the [0,1] range as [43]. Moreover, we do not preprocess the dataset following common practice without data augmentation [44, 39, 34, 36, 45].

We further evaluate our $n$-BQ-NN on ILSVRC2012 [11] image classification dataset that consists of 1.2 million high-resolution natural images where the validation set contains 50k images. These images are organized into 1000 categories of the object for training, which are resized to 224$\times$224 pixels before fed into the network. In the next experiments, we report our single-crop evaluation results using top-1 and top-5 accuracy.

AlexNet: This CNN architecture is the first structure that shows to be successful on ImageNet classification task, which consists of 5 convolutional layers and 2 fully-connected layers [1]. We use AlexNet coupled with BN that contains a total of 61M parameters.

In training, images are resized randomly to 256$\times$256 pixels, and then a random crop of 224$\times$224 is selected for training. We train our $n$-BQ-NN for 50 epochs with batch size of 128/16 based on AlexNet/Vgg-16. We use ADAM optimizer with the learning rate of 1e-4. We replace the Local Contrast Renomalization layer with Batch Normalization layer. At inference, we use the 224$\times$224 center crop for forward-propagation.

The ablation experiments are listed in Table VI. The baseline AlexNet model scores 56.6% top-1 accuracy and 80.2% top-5 accuracy that is reported in [52]. For ablation studies, we strictly control the consistency of variables, including network structure, bit width and quantized layers. The only difference is the quantization method. In experiments of “1-1” v.s. “1-1”, “1-16” v.s. “1-32” and “2-16” v.s. “2-32”, our $n$-BQ-NN achieves 6.9%, 11.8% and 0.7% accuracy improvements respectively. For “3-16” v.s. “32-32”, our $n$-BQ-NN only reduces the accuracy by 0.6%.

Fig.5 shows the block diagram of our system. In the CNN calculation process, the host computer hands off the weights and images to the coprocessor (ZCU102), and collects the predicted classification results. The transmission mode between host computer and ARM CPU can be switched in PCI or UDP. Before the computation, the host computer is responsible for feeding the images and reducing precision. In addition, the ARM CPU needs to complete the calculation of fully-connected layers that is not suitable for FPGA parallel acceleration and the FPGA accelerates the calculation of convolutional layers.

We build the coprocessor with parallel architecture, as shown in Fig.6. The critical part of the coprocessor is SVPE cluster interface that has $M$ SVPE clusters, where each SVPE cluster consists of $N$ SVPE arrays with a size of $k\times k$. The adders are used to compute partial sums of convolutions while the SVPE arrays compute convolutions. The fetch unit is programmed to fetch images and weights from ARM-based processing system (PS), and the load/store unit is used to load or store intermediate calculation results. The AXI-HP port is used to receive or send the data, and the AXI-GP port is used to receive or send the network structure information and the control signal. A key point to note is 16-bit computational accuracy acts on the data buses to save data bandwidth.

The basic design ideas continue the architecture of $n$-BQ-NN, which converts all 16-bit weights as $n$-bit weights to reduce memory usage and increase the parallelization degree. On FPGAs, due to the shortage of DSPs, this has become a major factor limiting the increase in parallelization degree that directly affects the ability to accelerate calculation for CNNs, because the multiplication in the convolution calculation needs to call DSPs. Instead, we implement the multiplication with SHIFT operation that consumes the Look-Up-Table (LUT) arrays on FPGAs, while the resource of LUTs is more abundant than DSPs’ [30]. In general, our $n$-BQ-NN, which consists of 16-bit activations and $n$-bit weights.

Our architecture of convolution computation is characterized by several key attributes compared with VPEs [14]. First, we organize the architecture as arrays of SVPEs, where the SVPE array is an array of 2D convolvers, each of which consists of $k^{2}$ connected SHIFT and ADDER units instead of Multiply Accumulate (MAC) units, shown in Fig.7. The weights and feature maps are loaded into each PE alternately by AXI-HP port. Before each calculation, the weights are buffered to the specified areas (the double orange rectangles in Fig.7), and then the pipeline calculation starts with the enablement of feature maps. Modeling the SVPE and VPE arrays, we compare their resource consumption, parallelization degree and power on FPGAs shown as Table VII, and our SVPE array achieves the average energy consumption of 3.81 W at different $n$ that is less than VPE array of 5.53 W. Second, we reduce banded off-chip data memory and improve the data movement between the SVPE clusters and the off-chip memory by using $n$-bit weights. Third, all convolvers are homogeneous when $k$ is fixed as our primitive operator. We evaluate the improvement of the computational efficiency of $n$-BQ-NN in hardware by the following section.

Since the filter size (3$\times$3) is fixed for our $n$-BQ-NN, resource utilization will be maximized. Here, we can predict the performance of $n$-BQ-NN on FPGAs by developing an analytical model. In the following, we rely on it to compare computational efficiency between traditional implementation and $n$-BQ-NN on FPGAs.

On the hardware, MAC unit, adder and multiplier will consume DSP. In fact, the number of DSPs only depends on the size of filter and parallelization degree [30, 12] as follows,