Integer or Floating Point? New Outlooks for
Low-Bit Quantization on Large Language Models

We first delve into hardware efficiency by comparing the hardware cost of INT and FP operators, including adders, multipliers, and multiply-accumulate (MAC) units, across various bit widths. Utilizing Synopsys Design Ware along with TSMC’s 7nm technology and the Synopsys Design Compiler, we are able to accurately determine the area requirements for each type of operator. In this experiment, we establish a target clock frequency of 0.5GHz for all operators. For 8-bit FP operators, we employ the E5M2 format, while E4M3 yields analogous outcomes. It is worth mentioning that, in 8-bit MAC, the multiplier is 8-bit, and the accumulator is 16-bit to prevent overflow, a standard configuration for low-bit MAC. As illustrated in Figure 3, FP multipliers require less area than their INT counterparts, whereas FP adders necessitate more area than INT adders. Concerning MAC units, which function as a fundamental building block for matrix multiplication in DNNs, FP operations typically demand more area than INT operations. However, this disparity narrows as the bit-width decreases. Intriguingly, at 8-bit, the area requirements for FP8 and INT8 MAC units are almost identical. This observation indicates that INT8 and FP8 demonstrate similar hardware costs and inference performance, aligning with the specifications of the H100 GPU.

In this subsection, we compare quantization errors from different formats through statistical analysis of real tensors and layers in LLMs. Quantization primarily targets Linear layers (operators) as they dominate storage and computation. The linear layer is represented as $A_{out}=W*A_{in}$, where $W$ is the weight tensor, $A_{in}$ is the input activation tensor, and $A_{out}$ is the output activation tensor. We analyze quantization errors on all three tensors for a comprehensive understanding. We perform static tensor analysis on $W$, dynamic tensor analysis on $A_{in}$, and layer analysis on $A_{out}$, which is influenced by error accumulation from both inputs.

In our analysis, we use per-channel weight tensor quantization and per-tensor activation quantization, following widely-used approaches in previous quantization research [9]. The FP4 format adopts an E2M1 configuration, and the FP8 format follows an E4M3 configuration.

Given the analysis findings that no single quantization format consistently outperforms the others in all scenarios, we propose the Mixture-of-Formats Quantization (MoFQ) method. The key idea behind MoFQ is to leverage the complementary advantages of integer (INT) and floating-point (FP) formats in a unified framework, thereby maximizing the potential benefits of both formats.

Specifically, MoFQ allows for the selection of the most suitable format on a layer-by-layer basis. As illustrated in Algorithm 1, the algorithm considers the model to be quantized, a flag is_w_only, format candidates (e.g., INT and FP), bit width (e.g., 8 or 4), and error metric as inputs. If is_w_only is set to true, only the weight tensor in the layer will be quantized (i.e., W-only quantization); otherwise, both weights and activations will be quantized (i.e., WA quantization), resulting in a quantized operator that can leverage low-bit hardware matrix multiplication units. We concentrate on layer-wise format selection with the same format for each tensor/layer and the same bit-width across all quantized tensors/layers, as this approach adheres to the simplicity principle, offering a straightforward and efficient implementation with minimal adaptation to the existing system or hardware. Such a straightforward method can achieve satisfactory results, as will be demonstrated in Section 5. Utilizing a finer granularity than the layer level or increasing the bit-width can indeed improve the quantization effect, but it also comes at the cost of increased memory consumption and higher system or hardware complexity. We leave the exploration of this trade-off as future work, as it requires further investigation and analysis.

The selection of the error metric plays a crucial role in MoFQ, as it influences the balance between quantization accuracy and speed. Several metrics are available, including tensor error, layer output error, and model output error, from less precise to more precise. A more precise metric may lead to better quantization results but at the expense of increased computational time. Thus, choosing an appropriate error metric is essential for achieving the desired balance between accuracy and efficiency in the quantization process. It’s worth mentioning that the evaluation metric for determining the superiority of a format is empirical and doesn’t have an absolute standard. For example, a lower tensor error may indicate a higher likelihood of achieving better model accuracy, but it isn’t guaranteed. In MoFQ, we use various selection metrics to find the right balance between quantization accuracy and speed. Our empirical observations suggest that using tensor error or layer output error suffices to guide the format selection for W-only quantization. Meanwhile, for WA quantization, the model output error offers the best format selection choices.

In low-bit quantization, maximizing number representation efficiency is essential for achieving the best possible precision and expressiveness. One optimization opportunity lies in reallocating the special NaN (Not a Number) and Inf (Infinity) values in standard floating-point formats. In IEEE floating-point formats, an exponent field with all bits set to 1 denotes NaN and Inf values. However, during model quantization, these special values serve no practical purpose, resulting in wasted bit combinations that could otherwise be used to represent a broader range of numbers. By reallocating NaN and Inf representations to normalized numbers, we can enhance the precision and expressiveness of the low-bit FP format for improved model quantization.

In practical quantization scenarios, the impact of NaN and Inf redundancy varies depending on the number of bits used for representation. For instance, in an 8-bit floating-point format (FP8), the impact is relatively minor as it can represent 256 numbers, with NaN and Inf occupying only a small portion. On the other hand, FP8 is primarily used for WA quantization, which requires hardware support for matrix multiplication. Therefore, it may not be suitable to modify FP8 due to potential hardware compatibility issues. However, in a 4-bit format (FP4), the impact becomes more significant as it can only represent 16 numbers in total. As FP4 is primarily used for W-only quantization, format modification can be addressed at the software level. As illustrated in Table 1, 4 of these numbers are used to represent NaN and Inf when adhering to the IEEE 754 standard. By reallocating them, we can obtain additional represented numbers, specifically $\pm 4$ and $\pm 6$. Tensor-wise analysis shows that our redesigned FP4 format can lead to about 35% lower quantization errors compared to the IEEE-aligned FP4 format. This improvement makes more layers prefer our re-designed FP4 than INT4 in W-only quantization.

Table 2 compares various quantization methods on LLaMA models, including INT4(GPTQ), FP4(ours), and MoFQ4. INT4(GPTQ) is the SOTA method from the GPTQ paper[8], FP4(ours) is our FP4 format, and MoFQ4 is mixture of FP4(ours) and INT4 formats. Evaluation results show that FP4(ours) and MoFQ4 generally outperform INT4(GPTQ), with MoFQ4 often yielding better results than FP4(ours). However, MoFQ doesn’t consistently surpass FP4, indicating room for further improvement in our MoFQ approach.

In addition to comparing quantization accuracy, we also examine the runtime required for quantizing the models. As shown in Table 3, we observe that the quantization speed of FP4 is more than 77.8 times faster than INT4(GPTQ) (with calibration nsample = 128), while MoFQ4 is over 9.3 times faster compared to INT4(GPTQ). This can be attributed to the fact that GPTQ necessitates real-time updates of the original weight tensors during the quantization process, resulting in considerable computational overhead. In contrast, FP4(ours) and MoFQ4 adopt the naive linear quantization with round-to-nearest (RTN), leading to significantly faster speeds.

In this subsection, we compare the performance of various quantization methods on LLaMA and OPT models. The methods evaluated include INT8, FP8, and MoFQ8, where MoFQ8 represents the mixture of FP8 and INT8 formats in the quantization method.

Table 4 compares various quantization methods on four datasets: WikiText-2, LAMBADA, PIQA, and HellaSwag. FP8 consistently outperforms INT8 across all cases. This can be attributed to the increased challenge of quantizing dynamic activation tensors compared to static weight tensors, as dynamic tensors exhibit more varied distributions and larger outlier values[22]. In Section 3, we determine that FP8 is better suited for quantizing dynamic tensors than INT8. Consequently, despite FP8’s potentially weaker performance on weight tensors compared to INT8, it still manages to achieve lower overall quantization errors for the models.

Crucially, we find that under our MoFQ8 quantization method, the accuracy of the quantized model remains remarkably close to that of the FP16 model, irrespective of the model size or dataset. This suggests that MoFQ8 effectively chooses the most appropriate format (INT8 or FP8) for each layer’s distribution, ultimately achieving W8A8 quantization results with minimal accuracy loss.