Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond

Deep neural networks have achieved great success in many applications, such as computer vision [20], speech recognition [10] and Natural Language Processing [1]. In computer vision, the most proposed architecture, Convolutional Neural Network (CNN), has demonstrated state-of-the-art results in many tasks. However, CNN-based recognition systems need large amounts of memory and computational power, which may take up to weeks on a modern multi-GPU server for large datasets such as ImageNet [5]. Hence, they are often unsuitable for smaller devices like embedded electronics, and there is a pressing demand for techniques to optimize models with reduced model size, faster inference and lower power consumption.

Accordingly, a variety of literature has made attention on the reduction of model size through the use of quantization [22, 4, 21], low-rank matrix factorization [6, 16], architecture pruning [7, 8], etc. Quantization is one of the simpler ways to reduce complexity of any model with less precision requirements for weights and activations as well as speeding up the computation.

The methods for quantizing gradients, weights and activations [22, 4, 13, 33, 24, 2], have achieved much closer performance to full precision networks. Whereas, after the previous efficient quantized neural network (QNN) training, there still has floating point operators in model inference particularly due to the float-type numbers in Batch Normalization (BN).

Since (deep) networks are hard trained, the BN operator [15] is usually used. Implementation of the conventional BN requires much computation of mean and variance, involving the sum of squares, square-root and reciprocal operations. These operators require float-type numbers for high precision. Previous attempts to use low precision networks do not use BN layers [30] or keep them in full precision [33]. In addition, Hubara et al. [13] proposed shift-based BN by replacing almost all multiplication with power-of-2 approximation and shift operations, and Banner et al. [2] devised Range BN for the variance normalizing according to the range of input distribution. These modifications to BN for less computational costs are heuristic, confirmed only with experiments. Besides, modern hardware has less deployment cost with fixed-point style or perhaps some only support fixed-points [14]. Hence, previous QNN training methods with float-type BN layers fail to deploy on total fixed-number hardware.

In this paper we develop a direct way to handle the floating numbers in BN after obtaining a benchmark QNNs with previous training methods. We view conventional BN operator combined with several quantization operators used in feedforward step as an affine transformation with only two floating numbers. In order to eliminate the floating numbers in BN, we start by considering the exact substitution of these two floating numbers to as less as possible fixed numbers. We recommend to quantize floating numbers with shared quantized scale and mathematically show that all floating points are able to convert to fixed-point operators in an idealized case. In addition, we also demonstrate the lower bound and upper bound of the least possible quantized scale in our scheme. After given the proposed quantized scale, we give few search attempts to decide remaining integers. By the way, our methods used for model deployment also guarantee the precision in QNNs’ experiments, which can be seen as a supplement of modern QNN training.

Our main contribution is summarized as follows:

• •

  We propose a new method for quantizing BN in model deployment by converting the two floating points affine transformations to a fixed-point operation with shared quantized scale.

• •

  We give theoretical guarantee of our quantization method, including the existence of quantized scale and the magnitude of optimal quantized bit-width. In addition, we accurately search the least possible quantized scale according to quantized bits to support our transformation numerically.

• •

  We conduct our method on CIFAR and ImageNet datasets based on benchmark quantized neural networks, showing little performance degradation if exists.

• •

  Our scenario is restricted not only in BN, but other affine operators with floating numbers in similar quantization structure, such as several variants of BN: $L^{p}$ BN [11], Instance Normalization [28] and Range BN [2].

The remainder of our paper is organized as follows. In Section 2, we formulate the problem and present the motivation of our transformation. In Section 3, we give the equivalence of problem in Section 2 and show the upper bound and lower bound of satisfied solution through several properties. We then give a trivial search algorithm in Section 4, for finding accurate value of the solution and verifying previous results in Section 3. We also briefly discuss the precision loss in practical hardware implementation and validate numerically in common quantized neural networks in Section 5. Finally, we conclude our method in Section 6.

Quantization mainly focuses on compressing parameters in neural networks, such as weights in convolutional and fully-connected layers, activation outputs of hidden layers, and parameters in BN. A common design for QNN is uniform quantization [32, 18], which makes quantized values evenly distributed among possible values, that is,

where $\Delta\in\mathbb{N}$ (the set of positive integers) is quantized scale, which measures the smallest gap between quantized values, and $\lfloor\cdot\rfloor$ means the floor function. The input $x\in\mathbb{R}$ is restricted to predefined interval $[x_{\min},x_{\max}]$ due to limited computing power. Particularly, for $k$-bit uniform quantization, choose $\Delta=2^{k}-1$ for friendly hardware support and simply choose $x_{\min}=0,x_{\max}=1$ for non-symmetric quantizer, $x_{\min}=-\frac{1}{2},x_{\max}=\frac{1}{2}$ for symmetric quantizer.

The weights which should be uniformly quantized may first be mapped into the required input value range through normalized operator [13]

or combined with hyperbolic tangent function [33]

Here the maximum is taken over all weights $x_{j}\in\mathbb{R}$ in the same layer.

Activations are quantized more difficultly than weights due to nondifferentiable optimization incurred by quantizer functions [3, 29]. Several works [21, 24, 33] have addressed this problem using bit-width allocation across layers or a continuous approximation to the quantizer function in the backpropagation step.

A common quantization approach for activations [23, 13, 33] quantizes output of the previous layer after clipping inputs into a limited interval, that is,

where $\mathrm{clip}(x,a,b)\triangleq\min\{\max\{x,a\},b\},\ b>a$, $y_{\min}$ and $y_{\max}$ are usually integers, and $Q(\cdot)$ is some quantizer function defined earlier. For common used ReLU function, $y_{\min}=0$, $y_{\max}=1$ or $y_{\max}=6$ [12, 26] (ReLU6) or some specific positive integers.

Since weights and activations are of the form with constant quantization bits in denominator, feed-forward calculation is able to execute with complete fixed-point operation without regard of the fixed denominator, which achieves hardware acceleration.

Benchmark models employ BN for deep neural network training. The conventional BN operator is two affine transformations of four parameters:

where $\mu$ and $\sigma$ are the mean and standard error estimators of output in the same layer and updated according to training batch statistics. Except conventional BN, several BN variants use similar transformation with different calculation method, such as $L^{p}$ BN [11] with $L^{p}$ norm of feature as $\sigma$, Range BN [2] with maximum gap in feature as $\sigma$, Group Normalization [31] and Instance Normalization [28] with different part of feature for calculating $\mu$ and $\sigma$. Anyhow, all parameters are fixed during inference and deployment.

We emphasize that floating points come from the division operation in BN and its variants, since $\mu$ is the running mean of batch feature in the same layer, $\beta$ and $\gamma$ are updated based on quantized gradients, but the reciprocal of $\sigma$ fails.

Previous state-of-the-art networks such as ResNet [9] and VGG [27] employ BN after convolutional layer or fully-connected layer, followed by an activation function. Let $\mathbb{Z}$ denote the set of integer numbers. Suppose we use $\Delta=A$ and $W$ to quantize activations and weights. Then from quantizer function $Q_{a}$, we can see outputs of previous layers are of the type $N_{i}/A$ with $N_{i}\in\mathbb{Z}$ and quantized weight $M_{i}/W$ with $M_{i}\in\mathbb{Z}$. Therefore, the outputs of convolutional or fully-connected layer are of the type $\sum_{i=1}^{r}\frac{M_{i}}{W}\cdot\frac{N_{i}}{A}+c$, where $r$ is the number related to the calculation of outputs in the next layer, and $c$ is the bias term if exists and may also be quantized. We record this output of the type $\frac{N}{AW}+c$ with $N\in\mathbb{Z}$. Here we do not identify specific corresponding position of parameters if no ambiguity.

After quantized convolutional or fully-connected layer is BN and quantized activation, then an element of the next layer outputs is

Here we exchange the order of $\mathrm{clip}$ and round function because $y_{min}$ and $y_{max}$ are integers in second equality. We also simplify two-affine operator of BN to one-affine operator and replace some terms as follows:

Generally, floating number operation will cause loss of precision, so this modification may lead to some error due to the limited computation precision. We will give experiments in subsequent section briefly.

Without loss of generality, we assume $y_{\min}=0,y_{\max}=1$ clipped from ReLU function. Other cases can be obtained by shifting the clip range by

Then $Y_{\min}=0$ and $Y_{\max}=A$.

An empirical attempt is trying to make all operators only related to fixed-points, which means quantizing floating points $t$ and $b$ to some integers. A simple way is to let floating numbers $t$ and $b$ be integers respectively, which seems difficult to work because input variable $N\in\mathbb{Z}$ may vary in a wide range. A weaker requirement is to quantize $t$ and $b$ similar to quantizer function $Q$ and $Q_{a}$ do, namely approximate $t$ and $b$ by rational numbers:

Since this consideration would add two more numbers and no hardware support in advance if either $B_{2}$ or $T_{2}$ is decided. Another way is to consider whether the surrogates of $t$ and $b$ can share same quantized scale. For example, same prefixed denominator for whole potential floating numbers $t$ and $b$, which is consistent with previous scheme, because

where we set $K=T_{2}B_{2}$, $B=T_{2}B_{1}$, $T=T_{1}B_{2}$. Then $t$ and $b$ share same pre-fixed quantized scale $K\in\mathbb{N}$.

Thus we want to replace the float numbers $t,b$ to integers $T,B,K$ for friendly hardware execution, where $K$ should not changed when $t$ and $b$ vary to support fixed hardware design.

In particular, we are able to examine whether $t,b$ can be directly converted to integers if $K=1$ already satisfies.

After having obtained $K$, we also need to decide the remaining quantized number $T,B$ based on the given $t,b$, which is the second problem we need to tackle.

Particularly, in the conventional $k$-bit quantization setting, we would set $A=2^{k}-1$ for some small $k\in\mathbb{N}$ and give the least bits number of satisfied $K$.

In this section, we analyze Problem 1 and Problem 2 with some propositions while leave detail in Appendix B.

In the previous section we have discussed $K_{n}$ theoretically. In this section, we present how to compute the accurate $K_{n}$ to evaluate the theoretical analysis.

For convenience, we are able to only consider $t\in[0,1)$ and $\lceil t-b\rceil=1$ by Proposition 5. Then $S_{1}=1$, $0\leq S_{i+d}-S_{i}\leq d$, $S_{i}\leq i$, for $i\in\{1,\cdots,n-1\},\ d\in\{1,\cdots,n-i\}$.

For insurance purpose, we save all possible sequences $\{S_{i}\}_{i=1}^{n}$ as a naive idea, then to check every $K$ if the corresponding $T,B$ exist. We generate the sequences of $T,B$ recursively based on the following result and the scope of $t,b$.

Now we turn to find $K_{n}$ in an accurate way. On account of Proposition 3, we start search from $\frac{1}{4}(n-1)^{2}$ when $n$ is larger than $15$. We use brute force and follow proposition below to search $K_{n}$ which satisfies all possible sequences $\{S_{i}\}_{i=1}^{n}$.

The whole process is showed below:

1. 1.

   First, produce candidate sequences. Note that from Proposition 6, we can get all the $\{S_{i}\}_{i=1}^{n}$ recursively.

2. 2.

   Second, record all satisfied sequences $\{S_{i}\}_{i=1}^{n}$. Given a candidate sequence $\{S_{1},\dots,S_{n}\}$, we use Proposition 7 to confirm there exist $t,b$ which can reproduce this sequence.

3. 3.

   Third, based on the obtained sequence $\{S_{i}\}_{i=1}^{n}$, check whether the given $K$ is satisfied through Proposition 4.

Due to the large amount of the possible sequences $\{S_{i}\}_{i=1}^{n}$, we suggest searching a small part of all sequences and running the window successively until all sequences are satisfied. The main pseudo-algorithm and the auxiliary pseudo-algorithms for search exact $K_{n}$ are all shown in Appendix F.

In this section we assess the effect of our BN quantization method through previous trained QNNs on ImageNet [25] (we also try on CIFAR-10 [19] in Appendix H). Our method is applied to model deployment at layer level, which is fitted for various quantized neural networks using the conventional BN or its variants. Besides, we aim at verifying accuracy variation through our modification for friendly hardware support, rather than obtaining high accuracy as previous quantization training methods do.

There are two main concerns we should verify. First, the derivation in Eq. (1) absorbs the parameters of BN into the neighboring convolutional layer or fully-connected layer, which may have precision degradation. We refer to $bt$ means absorbing BN parameters used in such layer though Eq. (1) based on quantized training models. Second, because the original theoretical analysis is based on real numbers rather than floating numbers, we should check whether the theoretical results match with the expression of floating numbers. Though numerical experiments have already examined in Section 4, we display the final accuracy to support our method again. We adopt $BT$ which uses our fixed-points layerwise replacements in Problem 1 based on $bt$ in the following experimental tables.

We use DoReFa-Net [33], one of efficient QNNs to train a quantized model for verification, and denote $QT$ as the accuracy with DoReFa-Net quantization methods.

DoReFa-Net has low bitwidth weights and activations using low bitwidth parameter gradients to accelerate both in training and inference. In order to get more suitable baselines, we only choose quantizing weights and activations while using full-precision gradients in training, and we do not quantize the final fully-connected layer for better training performance. We adopt $xWyA$ to represent $x$-bit quantized weights and $y$-bit quantized activations. We examine $2W4A$, $4W4A$ and $8W8A$ in subsequent experimental tables. We choose $K=64$ and $K=2^{16}$ for $4$-bit and $8$-bit quantized activations according to the discussion in Appendix D.

In this paper we have investigated the problem of quantizing floating points in BN, combined with quantized convolutional or fully-connected weights and activations. Noting that the conventional BN includes two affine transformations, we have made all floating points into one affine operator with only two floating-points. We have accordingly proposed to quantize each floating-point in the converted one-affine operator sharing a quantized scale. We have shown that possible quantized scale is roughly twice of activation bits in theory and given numerical computation schemes of the corresponding substitution. Our approach enjoys the errorless performance in inference using high precision. The strategy we recommended displays efficient model deployment with complete fixed-point operators.

It is worth emphasizing that our quantization scheme is suitable for other affine operators which are also common in deep NNs. Beyond BN as well as the NN context, we believe that our quantization scheme has potential applications in other problems that involve quantization.

a) & b) If the proposed quantization method is verified to be useful in numerous real applications by the engineers in the future, it will produce good impacts on model compression. Hence, previous style of QNNs would pay attention to entire fixed-point effective QNN design with BN. c) The method we proposed leverages all possible outputs and shows convincing results in theory, though our method may fails when the scope of fixed-point doesn’t support the converted large range of $T,B$ that we seldom see this in practice. Besides, floating operation is likely to introduce precision error, which our method in first conversion step would encounter. d) Our method is data irrelevant.