Towards Efficient In-memory Computing Hardware for Quantized Neural Networks: State-of-the-art, Open Challenges and Perspectives

Quantizing the neural network implies the quantization of neural network weights, activations, or both. Quantization methods can be divided into uniform and non-uniform, e.g. logarithmic and codebook quantization. Uniform quantization is a simple widely-used method, which divides the quantized interval into equally distributed sub-intervals. Fig. 2 (a) illustrates corresponding quantization equations for both methods, where $w_{q}$ and $w_{c}$ represent quantized and full-precision weights respectively, $\lfloor x\rceil$ rounds $x$ to the nearest integer and $clamp(x,v_{min},v_{max})$ restricts $x$ within the range of $[v_{min},v_{max}]$. In uniform quantization, the dynamic range of the quantized values is smaller than for non-uniform quantization. To improve this, a layer-wise or channel-wise scaling factor is used at a cost of computational complexity. Non-uniform quantization methods, e.g. logarithmic quantization, have a wider dynamic range allowing better representation of the weights. In Fig. 2 (a), 4-bit radix-4 logarithmic quantization is taken as an example of non-uniform quantization, which approximates the absolute value of the weights using $4^{n}$, $n=-3,-2,-1,...3$ with boundaries being the mid-value of the adjacent values $((4^{n}+4^{(n-1)})/2)$.

The other possible quantization method is a codebook quantization, which is useful when the data or weights distribution does not follow linear or logarithmic distributions. One such method uses k-mean clustering to find the quantized values for a codebook [15]. The codebook can also be formed by selecting the most frequent values among the weights in a neural network and be updated during training [16]. Reading the codebook can bring additional computational and hardware overhead to neural network implementation.

An important part of quantization is to determine the clipping range. For neural network weights, the clipping range is static and determined during the training. While, for the activations, it is different, as the inputs change during the inference. Quantization of the activations can be divided into dynamic and static quantization depending on how the clipping of the function is performed. In dynamic quantization, the clipping range is computed dynamically for each activation leading to higher accuracy but more complex computations. In static quantization, the clipping range is pre-calculated [2].

Different parts of the network exhibit different levels of abstraction and can be affected by the quantization differently [17]. Therefore, it is useful to set different quantization parameters, scale factors, and bit-width for different parts of the network. This is defined as mixed-precision quantization (MPQ). MPQ methods are divided into layer-level and fine-grained MPQs (channel/weight-level) [18, 19]. MPQ parameters can be determined by certain rules, e.g. input layers are more sensitive to the quantization than the other layers in a neural network. Also, MPQ policies can be optimized using differentiable optimization [20] or reinforcement learning algorithms [21]. MPQ reduces the model size and computation energy demands while keeping high-performance accuracy. However, hardware support and additional hardware overhead are required, especially when fine-grained MPQs are adopted.

There are two main methods to quantize a neural network: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) (Fig. 2 (b)). In PTQ, the well-trained full-precision model is quantized after the training. In QAT, the neural network models are quantized during the training. PTQ is faster but leads to lower accuracy than QAT. The main problem of QAT is a zero-gradient issue due to the stair-like nature of the quantization functions, therefore, the traditional stochastic gradient descent (SGD) algorithm with backpropagation cannot be directly applied for QNN training.

The issue of zero gradients in QNN training can be solved either by backpropagating approximated gradients or gradually quantizing the network while training (Fig. 2 (c)). The gradient approximation method is called straight-through estimator (STE), where a Jacobian matrix is set to a diagonal matrix with all 1-s in diagonal entries. The models trained using STE can achieve the accuracy compatible with full-precision models [22]. In general, STE-based methods use full-precision gradients to update the neural network.

The other method to solve the zero-gradient problems in QNN training is a gradual quantization and application of ”soft” quantization methods that converge to ”hard” quantization over time. One such method is additive noise annealing (ANA) [23], which injects noise into the quantized variables for the first few iterations of the training. The other method is adding a regularizer to the SGD update [24]. In addition, to quantize the network gradually, the network can be trained with weights and activations followed by the training with quantized weights and finishing with activation quantization [25]. However, all these training methods require the hardware support of full-precision computation, which makes it challenging to implement on low-power edge devices, especially for IMC architectures.

The QNN weights quantization schemes with high bit-precision in a crossbar-based architecture can be either implemented using high-precision devices allowing multi-level storage or combining several fixed-precision n-bit IMC crossbar cells to represent a single weight. IMC devices are already quantized by default and can represent several states. For example, in SRAM, the state of the cell is already quantized to two possible levels. In RRAMs, as the number of possible conductance states is also limited, the devices are quantized to a certain precision. The quantization of the crossbar outputs and activations is defined by ADC and DAC precision [9] and multilevel reading/writing drivers [26, 27]. Moreover, ADC and DAC precision also affects the resolution of neural network weights, e.g. low-precision ADCs cannot capture all the variations of high-precision weights [9].

In the case of combining several n-bit crossbar cells to form a single weight, the different memory cells represent all bits of the weights from the least significant bit (LSB) to the most significant bit (MSB). The calculation of partial sums is performed to compute the final result (the sum of all the outputs from the different cells), which cause the hardware overhead due to the required ADC resolution. ADC necessary resolution is calculated as $ceil(log_{2}((2^{b_{DAC}}-1)\times(2^{b_{w}}-1)\times i))$, where $b_{DAC}$ is a DAC resolution, $i$ is a number of crossbar rows, and $b_{w}$ is the number of bits in weights [9]. As ADCs are responsible for up to 90% of the area and power consumption in the compute element [28], the quantization methods, number of bits in weights and activations, and circuit and architecture design of an IMC core are critical for the hardware efficiency of QNN implementation. Moreover, the partial sums overhead also depends on input slicing and input resolution (activation precision in QNN).

Mapping of QNN architecture to IMC hardware has several specifications depending on the IMC devices and corresponding hardware designs. There are several mapping challenges corresponding to different parts of the design: non-linear distribution of the quantized levels in IMC devices, quantized weights mapping to several crossbar cells, unrolled convolution kernels mapping to IMC crossbars and mapping of large weight matrices to smaller crossbar arrays.

In high-precision devices, e.g. RRAMs, the quantization levels represented by RRAM conductance are often distributed non-uniformly. Therefore, mapping the uniformly distributed quantized weights to such devices can be challenging, which can also cause a degradation in the performance accuracy. In quantized high-precision weights mapped to low-precision crossbar cells, the weights can be mapped to several columns in a single crossbar or to several crossbars. This depends on particular designs and should be optimized according to the implemented hardware and availability of peripheral circuits, e.g. ADCs. In convolution kernels mapping to IMC crossbars to implement CNN, unrolling 3D kernels to vertical columns can cause the circuits overhead, including interconnects and buffers, and dataflow overhead [29]. In the mapping of large weight matrices to smaller crossbar arrays, the hardware overhead of ADCs should be considered. The search for the optimum crossbar size can be formed as an optimization problem to ensure the efficient mapping of neural network architecture to IMC hardware [30].

Binarized neural network (BNN) is a type of QNN relying on 1-bit weights (+1 and -1 in software) and activations [8]. The fabricated binary SRAM-based IMC designs are shown in [31, 32]. The efficient implementation of BNN in IMC hardware is based on XNOR operation, which reduces ADC overhead using 1-bit sense amplifiers [34] for binarized activations. XNOR-based implementation can be 30 times more efficient than sequential row-by-row read-out. Comparing the implementations of binarized SRAM- and RRAM- based networks, RRAM-based implementation is 5.8 times more energy-efficient than 8T SRAM-based design [33]. The other RRAM-based fabricated prototype of BNN with flash ADCs is shown in [35]. The application of flash ADCs is acceptable for BNNs, however for higher-bit designs, flash ADCs can bring significant area and power overhead [8]. A fabricated MRAM-based IMC macro for 1-bit operations is shown in [36].

The QNN designs with ternary weights imply the quantized values represented by -1, 0, and +1. Ternary weights networks demonstrate higher accuracy than BNNs and better sparsity, which allows skipping the operations related to zero weights [39]. Ternary networks can be implemented using RRAM-based crossbar arrays, where positive and negative weights are represented by two crossbar cells, and a subtractor is used to calculate the final value of the weight, which allows getting zero-weights [60]. The implementation of a ternarized accelerator using SRAMs is also possible [37]. In [37], the network is ternarized introducing a mask. This design aims for reconfigurability for different types of neural network architectures. [39] demonstrates STT-MRAM based ternary neural network implementation with sparse and fast addition. The ternary weights in FeFET-based architecture are illustrated in [40]. [38] illustrates the ternary XNOR network with RRAM devices.

The most common IMC architectures for fixed-point computations are based on SRAM and RRAM memories. As SRAM is a mature memory technology, there are a lot of fabricated SRAM-based IMC architectures shown in [41, 4, 42]. The SRAM implementation of both inference and training accelerator is illustrated in [43]. The fabricated RRAM-based IMC macros designs are shown in [48, 49]. In [48], on-chip learning is considered. In [49], on-chip fine-tuning is demonstrated. The RRAM-based IMC accelerator level designs with several levels of architecture hierarchy are shown in [50, 51, 28, 61]. These QNN designs include the datapath, routing, and arrangement of the network nodes are considered in addition to controlling and computation peripherals [8].

The general trend to move towards general-purpose implementations in QNN accelerators rather than architecture-specific designs leads to the development of configurable bit-precision in IMC architectures. Several works are implementing configurable macros for IMC accelerators to be able to vary the bit precision of the weights [44, 45, 46, 47]. Some recent RRAM-based IMC architectures can also support configurable bit-operations [52, 53]. Apart from SRAM and RRAM, the other IMC devices are less popular for IMC hardware. However, there are several architectures based on PCMs [54, 55], FeFETs [56], DRAMs [57, 58], and flash memories [59].

Table I shows the summary QNN hardware designs for IMC. Fig. 3 illustrates the comparison of power and area efficiency of different IMC hardware designs depending on the weights precision for the architectures in Table I. It is important to note that ADC precision affects power efficiency significantly. In [32], the design with 1-bit ADC is 30 times more efficient than for 5-bit ADC. The precision of weights also affects power efficiency. In [44], the power efficiency drops from 40.2 TOPS/W to 0.6 TOPS/W for 1-bit and 8-bit weights accordingly. The power efficiency of the IMC macro and the power efficiency of the whole system including several macros connected with peripheral circuits has a significant difference, e.g. in [45] IMC macro is 5 times more efficient than the overall system. Power efficiency is also affected by the technology node. For example, for a 4-bit neural network, 7nm SRAM-based IMC architecture [42] is more than 100 times more efficient than 55nm SRAM-based architecture [44].

Despite the recent development of IMC inference hardware designs, there are still challenges that should be addressed to move from IMC accelerators developed in research laboratories to efficient commercial solutions for low-power IMC-based QNN hardware. They include the hardware overhead in crossbar macros, consideration of architecture hierarchy and related challenges, and hardware non-idealities affecting performance accuracy.

Even though IMC crossbar architecture can be highly efficient for MVM operations, the hardware overhead caused by peripheral and control circuits can minimize the benefits of even small low-power non-volatile memories in the design. In crossbar macros, the hardware overhead comes from ADC and partial sums, when high-precision weights split into low-precision crossbar cells [9]. ADC design improvements, implementation of low-power converters, reducing the number of ADCs and ADC resolution, and relying on approximate computing can benefit IMC hardware designs [8].

An efficient IMC hierarchy design is also important. The interconnection of processing elements, computation blocks, and tiles also affects design efficiency. The architecture hierarchy in SRAM-based IMC designs is explored more than in non-volatile memory-based designs. Data movement between the layers, requirements for additional storage, and circuits interconnection in each level of the IMC architecture hierarchy should still be improved further. Moreover, several solutions have been explored to move from a traditional monolithic chip design to 2.5D integration or chiplet-based designs [7].

IMC designs based on non-volatile devices are prone to non-idealities. Even though quantized and binarized architectures are affected less by noise and device variations [63], device-to-device and cycle-to-cycle variability cause errors propagating through the network and affecting the performance accuracy. In addition, the immaturity of non-volatile memories leads to device fault issues and conductance drift with time also reducing performance accuracy [8]. These issues along with the device endurance should still be addressed at the device and material level. Moreover, the 3D stacking capability of IMC devices and crossbar cells should also be developed, as 3D integration is useful to decrease the length of interconnect wires to increase the chip density and reduce IR drop.

One of the main problems of QNN training is the requirement of full-precision gradients for weight updates. This leads to the complexity of neural network training on low-power hardware and IMC architectures that do not support full-precision operations. Several techniques to reduce the number of full-precision weights, e.g. mixing the full-precision weights with quantized weights for training [64]. However, the hardware support for full-precision operation is still necessary in this case. Therefore, the algorithms for QNN training based only on quantized values still need to be developed. Moreover, it is important to adapt these algorithms for IMC hardware relying on hardware-friendly operations, and include IMC hardware considerations, e.g. ADC precision considerations, partial sums, and IMC hardware non-idealities.

Neural network training on a chip can also be limited by the device non-idealities. If IMC hardware non-idealities are not the main concern in SRAM-based architectures, RRAM-based IMC architectures suffer from endurance issues and variabilities. The endurance of $10^{8}-10^{9}$ cycles is required to implement on-chip training, while most of the current RRAM devices are limited to $10^{6}$ cycles [9]. The other issue related to on-chip training implementation of non-volatile IMC devices, e.g. RRAMs and PCMs, is a high write latency. The write latency of RRAMs and PCMs usually reaches 100-150ns, and the write latency of SRAMs can be less than 1ns [65]. Non-linearity in the switching behavior of some non-volatile IMC devices, such as RRAMs, also requires additional hardware overhead to verify and correct the programmed conductance values for on-chip training [8].

On-chip learning and backpropagation circuits create an overhead for additional memory and registers to store intermediate outputs for gradient calculation, control circuits and flexible peripheral circuits for a crossbar array. Moreover, neural network training still requires higher bit precision than inference, which also leads to higher resolution ADCs. The development of efficient and flexible hardware support for on-chip training is an open challenge.

In addition, the QNN IMC hardware should support state-of-the-art software-based training techniques. For example, normalization fusion of neural network layers [66], where batch normalization layers are merged into the previous fully-connected or convolutional layers. Implementation of such a method on IMC hardware might bring accuracy degradation if the trained neural network is directly quantized and deployed on IMC accelerators. Overall, it is important to transfer the software-based methods for QNN to IMC hardware.

Recently, mixed-precision quantization has become an essential part of IMC hardware design, as it is more effective than fixed-precision quantization [67]. The IMC hardware support for mixed-precision designs is an open challenge. This implies that IMC macros should support a different number of bits for weights and activations, and have the ability to be reconfigured for different precision schemes, which also leads to hardware overhead related to computational blocks and control circuits. Moreover, the hardware under-utilization problem should be considered in such architectures, and reconfiguration schemes for effective hardware resource utilization in IMC macros are required for mixed-signal designs.

Most of the hardware for IMC architectures is designed for specific applications and specific QNN types. However, the overall trend in IMC designs is to move towards general-purpose chips [62]. To make a step towards general-purpose functionality, apart from the support of reconfigurable precision, IMC designs should support different workloads, different directions of data movement, quantization schemes, data types, and both spatial and temporal domain computations. The distribution of the workloads in different types of neural networks varies, e.g. ResNet workloads and data distribution for pattern recognition are different from LSTM workloads for language processing. Moreover, there is also a variation of the workloads within a neural network for different layers, e.g. convolution layers and fully-connected layers process the data differently and IMC hardware should support reconfigurability to be programmed accordingly. The issue of different workloads leads to unbalanced execution time and resource utilization in different layers [62].

Reconfigurability requires the hardware support of flexible interconnects. The support of different types of quantization schemes is important, as different applications rely on different data distributions. Moreover, the support of corresponding arithmetic operations and related IMC hardware macros is essential. At the same time, it is important to identify the trade-off between the support of different operations and quantization schemes and related hardware overhead. Overall, to satisfy different tasks, network structures, quantization methods, precision, and latency requirements, IMC QNN architectures should be reconfigurable.

Reconfigurability is also useful for balancing energy consumption and performance accuracy. There is a trade-off between the resources required for the task execution and the performance accuracy in low-power on-edge designs. In general, low-precision architectures usually lead to lower energy consumption but lower accuracy. Flexible precision and reconfigurability of IMC architectures can be used for scenarios when the power source is not available for on-edge devices, so the application can run under constrained power resources at the cost of reduced performance accuracy.

Quantization policies have several parameters to optimize to deploy to IMC hardware, including bit widths, scaling factors, quantization thresholds, and clipping ranges. This problem becomes even more complex for mixed-precision quantization, where different layers can have different quantization policies. Moreover, when deploying on IMC hardware, the performance accuracy can also be affected by IMC hardware non-idealities [68]. Therefore, it is important to include hardware-related evaluation when optimizing quantization policies, as it can affect hardware performance accuracy. It is difficult to find the optimum policies manually, especially when the size of the network increase. To automate the search for optimum quantization policies, it can be formulated as Hardware-Aware Neural Architecture Search (HW-NAS) problem. Such an approach is presented in [67], where optimum bit widths for weights and precision for ADC for IMC architectures are searched using a reinforcement learning approach. CMQ framework [69] also searches for optimum quantization threshold and bit width using a differentiable search approach. Gibbon framework [70] searches for optimum bit width and ADC precision along with neural network architecture and crossbar-related hardware parameters using an evolutionary algorithm-based approach. The further development of such frameworks is useful, as there are no unified frameworks considering the joint optimization of neural network architecture parameters, hardware parameters, and quantization policies. Moreover, the existing frameworks are designed for specific architectures and specific problems and require modifications for general use. Therefore, the development of a unified joint optimization framework for HW-NAS including quantization policies optimization is an open challenge.

Software-Hardware Co-design connects a software-based design of a neural network to hardware its implementation and implies the optimization of all levels of design from the devices and circuit macros to architectures and algorithms [62]. Currently available frameworks for IMC architectures, e.g. PUMAsim [28] or NeuroSim [71], focus on a specific IMC design, and do not support different quantization techniques, especially mixed-precision quantization, and have limited support of various IMC devices. There is still a lack of a universal EDA toolchain for large-scale implementations supporting various IMC hardware designs [62], including different types of neural networks, IMC architectures with various IMC devices and related non-idealities, various macros, and different designs of interconnects for the architecture blocks. Software-Hardware Co-design frameworks should include more QNN features, support different quantization methods, and include more mapping techniques. Overall, the efficient automated compiler for mapping QNN to an IMC-based hardware implementation supporting a wide range of IMC designs is still an open challenge.

While IMC architectures are efficient for certain operations, e.g. MVMs, they cannot fully replace the other types of hardware. Therefore, it is important to integrate IMC architectures with traditional hardware designs and different types of hardware. In particular, this approach will also benefit QNN training on a chip, where floating point operations are still necessary and require different hardware or additional DSP blocks. The efficient integration of various types of IMC hardware together is also necessary. There has been a successful attempt to combine an RRAM-based IMC accelerator with conventional SRAM-based memories and embedded processor [72]. The efficient integration of non-volatile memories with traditional memories, e.g. SRAM, is important to overcome external memory requirements and latency issues. Large-scale integration of relatively novel IMC architectures with traditional hardware designs can bring a lot of benefits in terms of architecture efficiency and is the next step in IMC hardware development.