GranQ: Granular Zero-Shot Quantization with Unified Layer-Channel Awareness

Quantization can reduce the representational range of pre-trained deep neural networks (DNNs) and minimize memory usage. Quantization methods are categorized into PTQ and QAT based on the timing of the range reduction.

PTQ quantizes fully trained DNNs without any further training [17, 3], offering low computational cost and simple implementation. However, because PTQ directly applies quantization to the FP model, it struggles to properly adjust the range during quantization. To overcome this, calibration is used to select optimal quantization ranges by determining scaling factors. In contrast, QAT fine-tunes the Q model with quantization operations integrated into the training process, enabling direct optimization of activations [17]. In particular, it employs the straight-through estimator [42] to estimate gradients.

However, both PTQ and QAT have the limitation of requiring access to data. PTQ needs data for calibration, whereas QAT requires it for fine-tuning. In practice, obtaining training data is often challenging. To address this problem, ZSQ has been proposed, which performs quantization without using original data [41].

As summarized in Table 1, ZSQ was initially introduced by ZeroQ [4], leading to the development of various ZSQ algorithms. These studies have explored diverse approaches to improve quantization performance under data-free settings. Recent studies have focused on generating high-quality synthetic data [25, 32] and developing more effective methods to utilize them [10, 23, 26]. However, these approaches are still limited in preserving activations in low-bit settings. Therefore, a new quantization method is required to effectively minimize activation loss. In this study, we propose a novel approach to address this challenge.

Activation quantization reduces the precision of intermediate activation values in neural networks by converting them into low-bitwidth representations [8]. This technique is often used alongside weight quantization to further compress the model. Among various activation quantization methods, linear quantization is the most widely used. It uses a normalized scaling factor to adjust the activation values, and subsequently converts them into integers within a fixed bit range [24, 15]. The overall process typically involves three stages: forward quantization, scaling parameter computation, and dequantization [17].

Existing activation quantization methods are designed based on layer-wise quantization to minimize computational cost. However, fixed scaling factors struggle to handle varying activations. This issue becomes more severe under low-bitwidth settings in ZSQ environments. The following outlines the primary challenges faced in ZSQ under low-bitwidth settings.
(Problem 1) Coarse-grained activation representation
In conventional activation quantization approaches, the scaling factor $s$ is fixed per layer or channel, mapping all activations to the same range. However, in low-bitwidth settings, this approach struggles to capture fine-grained activation variations from generated data.
(Problem 2) Severe quantization error in certain layers
During quantization, certain layers induce severe quantization errors, leading to significant activation distortions. These distortions substantially differ from the original distribution and can directly impact the performance of the model.

Layer-wise quantization effectively captures the activation distribution of each layer. However, it becomes inefficient when activation distributions within a layer exhibit significant variation along the channel axis. Particularly in ZSQ, as shown in the FP graph in Figure 2(a), the activation distributions can differ substantially across channels even within the same layer. This implies that limited representation bits are forced to capture a wide range of activation values, making fine-grained quantization difficult. It consistently appears across all layers. We analyzed this phenomenon occurring across all layers and provided a summary in Table 2. In Table 2, we present the cosine similarity $\frac{X\cdot Q}{|X|_{2}|Q|_{2}}$, along with the relative error $\frac{|X-Q|_{2}}{|X|_{2}}$. From Table 2, we identify that both layer- and channel-wise quantization fail to preserve the original activation effectively. This indicates that layer- and channel-wise quantization, which independently process activations at the layer or channel level, are not suitable for ZSQ. In contrast, GranQ, which considers both layers and channels simultaneously, achieves 1.34$\times$ higher similarity and 2.94$\times$ lower relative error compared with those of layer-wise quantization, demonstrating its effectiveness in preserving activations.

Furthermore, Figure 3 provides a visual representation of the effect of quantization on activation values across layers. In layer-quantizaiton, although the overall activations generally reflect the FP activations to some extent, certain layers exhibit fine activation values being mapped to discrete values, leading to coarse representations. These coarse representations have been identified as a major factor leading to a decline in the overall expressiveness and inference performance of the model. Further analysis of Figure 3 is provided in the supplementary material, in Figures LABEL:sup_fig3 and LABEL:sup_fig1.

Based on our analysis, we propose GranQ (granular layer-channel quantization), a fine-grained ZSQ approach designed to minimize computational overhead, as shown in Figure 4. GranQ combines layer and channel information using layer-channel scaling and improves efficiency through vectorized quantization.

The experiments were conducted using widely adopted ZSQ evaluation datasets, including CIFAR-10, CIFAR-100 [27], and ImageNet (ILSVRC 2012) [11] validation datasets. For the CIFAR datasets, ResNet-20 [19] was used as the quantization model, whereas ResNet-18 [19], ResNet-50 [19], and MobileNetV2 [38] were employed for ImageNet. All experiments were conducted using the SGD optimizer [37] with a momentum of 0.9 and weight decay of 1e-4. The CIFAR-10 and CIFAR-100 experiments were each conducted for 200 epochs, with batch sizes of 16 and 200, respectively. For ImageNet, we trained for 400 epochs with a batch size of 16. The initial learning rate was set to 1e-4 for CIFAR-10 and CIFAR-100, and 1e-5 for ImageNet, with multi-step learning rate decay applied. The decay steps were set to 100, 200, and 300 epochs for CIFAR, and at 350 and 400 epochs for ImageNet, with a decay rate of 0.1. We compared our method with existing ZSQ methods [41, 10, 36, 5, 1, 14, 32, 23, 26]. For data generation, we followed the AdaDFQ [36] approach based on ACGAN [35]. Further implementation details are provided in the supplementary material, Section LABEL:sec:Details_of_Experimental_Setup. In our ablation study, layer-wise quantization was applied to all layers containing activation functions, while channel-wise quantization was performed per channel at the batch level.

We evaluated the performance of GranQ against SOTA ZSQ methods, with results summarized in Table 3. All comparison experiments were conducted under 3, 4, and 5-bit quantization settings.

CIFAR-10/100. As indicated in Table 3, GranQ consistently achieved the highest accuracy across all bitwidths. For CIFAR-10, it attained accuracies of 94.06% (5-bit), 93.52% (4-bit), and 91.37% (3-bit). For CIFAR-100, the results were 70.05% (5-bit), 68.79% (4-bit), and 62.73% (3-bit). Notably, GranQ outperformed SynQ [26], the previous SOTA, by +5.45% in the CIFAR-100 3-bit setting. This result demonstrates its ability to effectively overcome the limitations of conventional low-bitwidth quantization techniques. Overall, GranQ consistently outperformed existing methods across all bitwidths in both CIFAR-10 and CIFAR-100, with particularly strong improvements in the 3-bit quantization setting. Remarkably, in the CIFAR-10 5-bit setting, GranQ even surpassed the performance of the FP model. These results suggest that we can effectively apply GranQ to small- and medium-scale datasets.

ImageNet. In the ImageNet experiments, GranQ achieved competitive performance across various bitwidths. Specifically, for ResNet-18, it attained top accuracies of 70.39% (4-bit) and 71.31% (5-bit). For ResNet-50, it achieved the highest accuracies of 76.63% (4-bit) and 71.31% (5-bit). Additionally, in the MobileNetV2 setting, GranQ achieved SOTA performance across all bitwidths (3, 4, and 5-bit). In the 3-bit setting, GranQ achieved the second-best accuracies, with 64.41% on ResNet-18 and 70.76% on ResNet-50. These results are -3.77% and -3.23% lower than those of GenQ, respectively. However, GenQ uses a pre-trained diffusion model [22] and prompts during data generation, leading to slower speeds compared with those of typical GAN-based methods [41]. In contrast, GranQ maintains the speed advantage of GAN-based generation while achieving high performance through fine-grained adjustments.

The most common activation quantization approaches are known for their low computational cost and fast processing speeds. However, they often fail to accurately capture activation distributions, resulting in substantial quantization errors (Figure 3). In layer-wise quantization, representing the entire activation range with a single scaling factor makes it difficult to reflect fine-grained distribution changes. Similarly, channel-wise quantization encounters challenges owing to significant range differences in activations across layers for the same channel, leading to increased information loss (Section LABEL:sec:extraanalysis in the supplementary material). These limitations become more pronounced in low-bit settings, where higher quantization errors result in performance degradation.

To overcome these issues, GranQ introduces the layer-channel scaling method, which combines layer- and channel-wise quantization. Unlike conventional methods, GranQ considers the activation distribution of each channel individually while also incorporating layer information, enabling more precise quantization. Furthermore, although conventional quantization relies on scalar operations with a single scaling factor, GranQ utilizes multiple scaling factors and performs vectorized computations. This approach leverages GPU resources efficiently, minimizing computational overhead while achieving effective quantization.

Experimental results demonstrate that GranQ outperforms traditional layer- and channel-wise quantization methods, achieving higher cosine similarity and lower relative error (Table 2). This demonstrates that GranQ accurately preserves the original activation distribution. Additionally, GranQ maintains computational efficiency in terms of latency and effectively balances the trade-off between computational cost and performance (Figure 5 and Table 5). Most notably, GranQ achieves SOTA performance compared with those of existing layer-wise quantization methods (Table 3), highlighting its efficiency and superiority over scalar iteration approaches.

As GranQ aims to precisely preserve activations during quantization, it partially depends on generated data quality. Consequently, studies on data generation remain highly important. The proposed method has the potential to achieve even better performance when applied to high-quality data generation strategies. In future work, we aim to improve the quantization performance by focusing on both the accurate generation of original activations and their effective preservation.