Trainable Fixed-Point Quantization for Deep Learning Acceleration on FPGAs

Earlier research in this field primarily focused on integer quantization, where DNNs’ floating-point weights and activations are quantized to low-bitwidth integer values. This allows inference to be executed efficiently through hardware-accelerated integer matrix multiplications.

Typically, quantization functions map a floating-point tensor $r$ to an integer tensor $q$ using the following formula:

Here, $S$ denotes the scaling factor, and $Z$ serves as the zero point. $S$ and $Z$ determines the range of values that can be represented by the quantized number. clip cuts the value outside the range and round operation maps floating-point value to an integer.

After computations of each quantized layer, the output q will be dequantized to floating-point:

Fake quantization. Fake quantization is often used to simulate QAT  (Jacob et al., 2018; Pappalardo, 2023). In this approach, weights and activations are simulated as integers with floating-point representation, and all the multiplications and accumulations occur in FP32 so that the gradient computation is enabled. Nevertheless, HAWQ-V3 (Yao et al., 2021) mentioned that this method results in significant error accumulation when deployed on real hardware, primarily due to the discrepant integer castings on the simulation and the actual hardware.

Scaling factor quantization. Efforts (Jacob et al., 2018; Shawahna et al., 2022; Yao et al., 2021) have also been made to quantize the scaling factor to lower precision, recognizing that it is often directly cast to a fixed-point number during deployment on hardware which exacerbates the software-hardware gap. HAWQ-V3 proposed that we can construct the multiplier consisting of the scaling factors as a dyadic number during training. A dyadic number $b\cdot 2^{-c}$ is essentially a fixed-point number, where $b$ is an integer, and $c$ represents the number of fractional bits. This approach not only replaces the multiplication between integer output and floating-point scaling factor with integer multiplication and bit shift but also helps avoid direct casting errors.

While the computational overhead related to floating-point scaling factor calculations can be negligible compared to matrix multiplication in convolutional layers, element-wise layers themselves only involve one or two floating operations. Therefore, rather than representing a floating-point tensor as a low-bitwidth integer tensor and a floating-point scalar $S$, an alternative approach is to quantize it to a fixed-point tensor, as fixed-point operations are also efficient on hardware  (Shawahna et al., 2022). A floating-point tensor can be quantized to fixed-point with a given total bitwidth $w$ and an integer bitwidth $i$ through Algorithm 1.

Here, round operation rounds the number to nearest integer, removing extra bits in the fractional part, and overflow ensures the number falls into the representation range with $w$ bitwidth, removing extra bits in the integer part.

The fixed-point emulation in our QFX library is encapsulated in regular PyTorch layers inheriting nn.module. The implementation is all through native PyTorch tensor operations, which makes QFX layers easily be plugged into existing PyTorch models. Most importantly, QFX layers are backpropagation compatible. Models with QFX quantization can support normal gradient descent training.

An example flow of using the QFX library is shown in Figure 1. In contrast to the traditional practice where there are different model copies for training and deployment, users can apply the fixed-point casting, addition, or multiplication operations anywhere in a given PyTorch model at training time. Exactly the same model will be deployed, except that the data type casting will be replaced directly by the hardened circuits. Below we introduce how fixed-point casting and basic arithmetic are implemented and how they are assembled into a quantized BatchNorm layer as an example.

Fixed-point casting. Given a floating-point input $x$, the following equation converts it to a fixed-point representation:

For the quantization function, rounding towards the nearest integer is the most common case. On FPGAs, however, various circuits break the tie differently. QFX supports seven quantization modes, the implementation of which is shown in Table 1. It produces the same numerical outputs as the reference AMD Xilinx ap_fixed library (XILINX, 2022). Similarly, all available overflow implementations are shown in Table 2. One may notice that the casting function leveraging PyTorch native operators is inherently differentiable except for floor, ceil, round, and trunc functions. Their gradients are almost zero everywhere. To enable training with QFX, we redefine their gradients as:

which is known as the straight-through estimator (STE) (Bengio et al., 2013). STE is commonly used to approximate gradients for non-differentiable operations (Mishra et al., 2017; Bhalgat et al., 2020; Jung et al., 2019). Users can also define customized gradient functions and integrate them under the QFX training scheme.

Supporting a wide range of commonly used quantization and overflow modes guarantees that the model inference during software training and FPGA deployment are aligned. However, DNN models are typically trained on GPUs or CPUs where rounding to the nearest even number, i.e., RND_CONV in Table 1, is supported by default. This subtle difference, especially in a DNN accelerator (Zhang et al., 2021), will lead to completely different hardware outputs. In order to synchronize the outputs produced by both software quantization emulation and FPGA deployment, the general QFX approach is necessary.

Basic arithmetic. QFX supports a subset of fixed-point element-wise operations that are the most common in DNNs, including addition, subtraction, multiplication, and division. To enable fixed-point arithmetic during model training, we apply the casting function to both the input and output of these operations. For example, the floating-point addition operation:

transforms into the following for fixed-point computation within QFX:

Each cast function has its own word length and fractional bitwidth. Importantly, the basic operations remain differentiable with QFX casting functions. When deploying the model, the casting functions can be seamlessly replaced by circuits, by programming the data type of input and output variables using high-level synthesis (HLS) or hardware programming languages. The other three operations — subtraction, multiplication, and division, follow a similar approach.

Quantized element-wise operations. With the supported basic arithmetic, common element-wise operations, e.g., BatchNorms and residual connections, can be emulated in fixed-point during training. As an example, the floating-point BatchNorm layer was computed by:

where $\beta$ and $\gamma$ are learnable parameters, and $\epsilon$ is a small value for numerical stability. With fixed-point quantization, it becomes:

where $\alpha=\frac{\gamma}{\sqrt{\operatorname{Var}\left(x\right)+\epsilon}}$ and $\eta=\beta-\frac{\gamma\cdot\mathbb{E}\left[x\right]}{\sqrt{\operatorname{Var}\left(x\right)+\epsilon}}$. Note that the casting operation is embedded within qfx.add and qfx.mul. Both input and output can be specified with different word lengths and fractional bitwidths. The QFX library includes pre-implemented fixed-point BatchNorm layers and residual connections, which are commonly used in DNNs. Users can easily customize their models to fixed-point by substituting the four basic operations with QFX.

In Section 3.1, we have introduced the QFX library. In this section, we delve into how QFX can be harnessed to automate the learning of fixed-point data types, thereby eliminating the need for extensive precision tuning in practical applications.

A fixed-point representation can be divided into two parts: the integer and the fraction, separated by the binary point. The number of bits allocated to the integer and fractional parts determines the range and precision of the fixed-point representation. For example, a decimal number $(13.3125)_{d}$ can be accurately represented by a fixed binary number $(1101.0101)_{b}$, with $4$ integer bits and $4$ fractional bits:

Previous work (Loroch et al., 2017; Abadi et al., 2016; Shawahna et al., 2022) in fixed-point quantization primarily focused on fine-tuning PTQ with the binary point position fixed, demanding additional human effort to strike the right balance between range and precision.

In our work, QFX offers support for fixed-point QAT with different configurations for each layer and even for different weights within a single layer. Consequently, automating the binary point position searching is as straightforward as assigning a new parameter to Algorithm 1 with gradient backpropagation enabled. In this case, we are able to improve the accuracy with all quantization effects included during training.

As shown in Algorithm 2, the trainable floating point $I$ is rounded to integer to be fed into the fixed-point casting function of $x$. Notably, gradient backpropagation in torch.clamp is enabled via STE, and QFX also enables gradient backpropagation for rounding. This ensures that the function is differentiable and binary point position remains trainable.

Dedicated DSP units on FPGAs are often considered as a limited resource in contrast to the ample availability of LUTs. Even for highly compressed DNN models, a nontrivial number of DSPs are typically still required for operations using fixed-point arithmetic. By profiling a state-of-the-art BNN accelerator, FracBNN, we find that more than 60% of the DSPs on its Ultra96v2 FPGA deployment are still in use, even if all the matrix multiplications in the model are binarized. This presents a suboptimal scenario for edge devices. To mitigate this and further conserve DSP resources, this section introduces a multiplier-free fixed-point quantization scheme, which incurs zero DSP usage.

Instead of solely quantizing the weights in element-wise layers (e.g., BatchNorms) to fixed-point format, we further quantize them to a K-hot encoding — enforcing it to have only $K$ ”1”s in its binary representation:

where $K_{i}$ is the position of the $i$-th most significant ”1”, $fbit$ is the fractional bits, and bshift shifts $1$ left by $K_{i}-fbit$ bits in binary format. One multiplication between a fixed-point number $x_{f}$ and an integer number $x_{i}$ typically requires DSP units. By imposing the $K$-hot constraint with a small $K$, the multiplication is replaced by at most $K$ additions and bit shifts.

This $K$-hot fixed-point leads to an entirely DSP-free design.

To convert a floating-point input to a k-hot fixed-point number, we first use the Quant casting function in QFX to quantize it to fixed-point with corresponding integer bits $I$. Then we find the $K$ most significant bits to approximate the number in binary representation. $K$ can be flexibly configured based on the accuracy-efficiency trade-off. A smaller $K$ is more efficient but less accurate. Detailed algorithm description can be found in Algorithm 3.

We quantize element-wise operations to fixed-point, e.g., BatchNorm layers, activations functions, and residual connections. Both input activations and layer weights are quantized to target bitwidth, with the binary point position of each quantized number being automatically learned during training. Subsequently, we employ 2-hot quantization in BNNs for all the weight multipliers in BatchNorm layers and activation functions.

To evaluate the performance and resource improvements by QFX, we implement our FracBNN-ImageNet accelerator with 8-bit QFX quantization on an AMD Xilinx Ultra96 v2 FPGA board, which has 360 DSPs, 71k LUTs, 141K FFs and 949 KB BRAMs. We preserve the existing binary and fractional convolution layers in the original FracBNN accelerator. Our efforts focus on applying QFX to the element-wise layers (including BatchNorm, BPRelu, and residual), as well as binary quantization layers, thereby achieving a DSP-free design in these regions without any degradation in throughput.

With the $ap\_fixed$ library provided by HLS, it is straightforward to cast all the weights and activations of different element-wise layers to 8-bit fixed-point numbers upon obtaining the learned data types from the training stage. In contrast, the fixed-point data types used in the original FPGA implementation were determined manually. Additionally, our K-hot quantization technique facilitates DSP-free implementation in BatchNorm and BPRelu layers. In our design, both layers can be represented as:

where $\hat{a}$ represents a two-hot quantized fixed-point parameter with only two 1s. Thus we can easily transform the multiplication into simple bit shift and addition. Besides replacing DSPs with 2-hot MAC units in element-wise layers, similar optimizations apply to binary quantization layers, where scaling factors could be quantized using K-hot encoding.

Figure 2 compares the hardware performance of our 8-bit QFX accelerators against the original FracBNN accelerator. By limiting the bitwidth of fixed-point numbers in both element-wise and binary quantization layers, 8-bit QFX decreases DSP, LUT and FF overheads by 8.9%, 4.7%, and 7.0%, respectively. Moreover, the multiplier-free design can substantially reduce DSP usage from 55.3% to 10.8%, while only increasing 5.8% and 1.1% LUT overhead compared to 8-bit QFX without 2-hot encoding and the original FracBNN design.

Figure 3 further provides a breakdown of the DSP usage. We observe that 8-bit QFX reduces total DSP usage from 231 to 199, while the 2-hot encoding completely eliminates DSP usage in element-wise layers and binary quantization layers. This leaves only 39 DSPs, which are used for memory address calculations.

Notably, compared to simply using the BIND_OP imp=fabric pragma in HLS to avoid instantiating DSPs, our design is much more efficient. As shown in Table 6, using the HLS fabric pragma to remove DSPs leads to 114.6% and 50.2% LUT and FF increases for element-wise layers with 8-bit QFX quantization. In contrast, the multiplier-free technique with 2-hot encoding requires much fewer LUTs and FFs.