Data-free mixed-precision quantization using novel sensitivity metric

For a general data generation method, we seek to avoid any dependency on a specific operation in a convolutional network. As shown in Figure 1(a), we first forward a noisy image initialized from a uniform distribution in the range between 0 and 255, inclusive. To produce a set of labeled data, we use a set of class-relevant vectors/matrices, which means one-hot vectors per class in the classification task. In the forward pass, the loss is computed as

where ${x_{c}}$ is the trainable input feature, ${v_{c}}$ is the set of class-relevant vectors/matrices, called the golden set, $y(\cdot)$ is the neural network, and $f(\cdot)$ is an activation function (i.e., softmax) used to compute cross-entropy loss ($\mathcal{L}_{CE}$) with the golden set. In addition, we reinforce the loss by maximizing the activation of output of network to enhance the efficiency of data generation process by referring the prior works for model interpretation [20, 21]. The calculated loss is propagated in the backward pass and generates input feature $x_{c}^{\ast}$ for each class, $c\in\{1\ldots NumOfClass\}$. Finally, we have crafted synthetic data for each class after several iterations of forward and backward processes.

Figure 2 demonstrates the reliability of crafting labeled data for each class by showing the top-2 confidences per class of synthetic data. All the generated data are used not only to measure the layer-wise sensitivity of the network but also to obtain the statistics of activations to improve the quality of the post-training quantized network.

The objective of mixed-precision quantization is to allocate appropriate bits to each layer to reduce the cost (e.g., bit operations [22]) of neural network models while suppressing the task loss growth. The sensitivity metric is an important factor for finding the optimal point between the effects of quantization error and cost. First, we would like to measure the effect of quantization error on the loss of a network that has total $L$ layers when weights of $i^{th}$ layer $(1\leq i\leq L)$, $W_{i}$ are quantized through quantization function $Q_{k}(\cdot)$ into $k$-bit. Given input data $x$ and quantized neural network $\hat{y}$, we consider the Euclidean distance between $y(x)$ and $\hat{y}(x)$ as the $\mathcal{L}$, the loss of network output, for sensitivity measurement instead of task loss. We can define the effect of quantization error $Q_{k}(W_{i})-W_{i}$ of $Q_{k}(W_{i})$ on $\mathcal{L}$ as follows:

The effects of the activation’s quantization error on $\mathcal{L}$ is represented as activation gradients of quantized activation by applying the derivation of the formula shown in the previous equations. Subsequently, we calculate the expectation for the effects of the quantized network on $\mathcal{L}$ by using the geometric mean of $\Omega$ of ${m}$-bit quantized activation $Q_{m}(A)$ and quantized weights $Q_{k}(W)$, which is formulated as

The gradients of the converged single-precision neural network are not changed for the same data. To measure the effect of $Q_{k}(W_{i})-W_{i}$ on other connected layers, we observe the gradient perturbations of $W$ and $A$, which are caused by $Q_{k}(W_{i})$. Consequently, we can measure the effect of quantization error on other layers and loss of the network together, i.e., sensitivity of quantization by using $E[|S_{\hat{y}}|]$ when we only quantize activations or weights of a layer for which we would like to measure the sensitivity. Expressing the single-precision as $Q_{\mathrm{FP32}}(\cdot)$, we have

We evaluate our method using the generated data to determine the clipping range of each layer in post-training quantization. Our experiments exploit a simple symmetric quantization approach, which uses the maximum value of activation as the clipping range of each layer. Hence, maximum values are extremely crucial for confining the dynamic range to utilize the bit-width efficiently, preventing a severe accuracy drop in low-bit quantization.

For our proposed method, input images are initialized according to uniform distribution, which follows standardization or normalization. To generate data, we use Adam optimizer to optimize the loss function with a learning rate of 0.04, and we found that $\lambda$=1 works best empirically.

For InceptionV3, input data are initialized according to $\mathcal{U}$(0, 255), which follows standardization with mean and variance by considering that the model requires synthetic data to be converted into the range of [-1, 1] through preprocessing. For VGG19 and ResNet50, input data are initialized according to $\mathcal{U}$(0, 255), which follows normalization with the factor of 255 because they require the range of [0, 1].

Table 1 shows the empirical results for ImageNet, random noise, and ZeroQ [10]. In the experiment, ImageNet data are produced by choosing one image per class from the training data, and we generate 1000 data samples randomly using the existing method and the proposed method. As one can see, our method shows similar or higher performances over existing data-free method, having less than 0.5% differences from the results of ImageNet cases. As shown in VGG19, although the existing research is hard to generalize, the method maintains sound performance regardless of the network structure.

To verify several metrics in weight sensitivity, we measure the task loss by switching floating-point weights in the order of layers with higher sensitivity values, where all weights are quantized to 4-bit, and activations do not have quantization error. We denote this experiment as A32W{32,4}. A{32,4}W32 indicates the evaluation of the metrics in activation sensitivity when switching the 4-bit quantized activation to floating-point, where weights are single-precision. ZeroQ [10] measures the KL divergence and [15] measures the Euclidean distance between the original model and the quantized model. ZeroQ [10] only measures the weight sensitivity. HAWQ-V2 [16] uses the average Hessian trace and L2 norm of quantization perturbation. We use the data generated by the proposed method for all metrics.

Figure 3 shows the results of ResNet50 and InceptionV3 over ImageNet dataset. Our sensitivity metric reliably and quickly lowered the task loss, which means that the proposed sensitivity metric is good at weights and activations that are comparatively more susceptible to quantization. To express the results of Figure 3 quantitatively, we calculate the relative area under the curve of task loss graph considering the result of the proposed metric as 1 and summarize it in Table 2. Our proposed method has the smallest area in all results.

To show the importance of the data in measuring the sensitivity, we evaluate the proposed sensitivity metric over the different datasets of Section 3.1 on the 4-bit quantized network that clipped the activation range using ImageNet dataset. We measure the task loss as in Section 3.2. The proposed dataset is the most similar in sensitivity to the ImageNet dataset, as shown in Table 3. Notably, the proposed data generation method provides reliable statistics similar to the original training dataset. For InceptionV3, preprocessed ImageNet data are in the range of [-2.03, 2.52]. However, the range of the preprocessed data from [10] is measured as [-11.50, 11.21], while that of our data is in [-2.35, 2.36], whose maximum value is almost similar to that of the ImageNet dataset. These similar statistics are seen in ResNet50. The incorrect statistics of the activation data corresponding to the original training dataset implies inaccurate sensitivity measurement [10]. Thus, our generation method can be used for reliable sensitivity measurement.