Containing Analog Data Deluge at Edge through Frequency-Domain Compression in Collaborative Compute-in-Memory Networks

The Walsh-Hadamard Transform (WHT) is akin to the Fast Fourier Transform (FFT) as both can transmute convolutions in the time or spatial domain into multiplications in the frequency domain. A distinguishing feature of WHT is that its transform matrix consists solely of binary values (-1 and 1), eliminating the need for multiplications, thereby enhancing efficiency.

Given $X,Y\in\mathbf{R}^{m}$ as the vector in the time-domain and WHT-domain, respectively, where m is an integer power of 2 ($2^{k},k\in\mathbf{N}$), the WHT can be expressed as:

Here, $W_{k}$ is a $2^{k}\times 2^{k}$ Walsh matrix. The Hadamard matrix $H_{k}$ for WHT of $X$ is constructed as follows:

The Hadamard matrix is then rearranged to increase the sign change order, resulting in the Walsh matrix. This matrix exhibits the unique property of orthogonality, where every row of the matrix is orthogonal to each other, with the dot product of any two rows being zero. This property makes the Walsh matrix particularly advantageous in a wide range of signal and image processing applications [30].

However, WHT poses a computational challenge when the dimension of the input vector is not a power of two. To address this, a technique called blockwise Walsh-Hadamard transforms (BWHT) was introduced [31]. BWHT divides the transform matrix into multiple blocks, each sized to an integer power of two, significantly reducing the worst-case size of operating tensors and mitigating excessive zero-padding.

Frequency-domain transformations, such as BWHT, can be incorporated into deep learning architectures for model compression. For instance, in MobileNetV2, which uses $1\times 1$ convolutions in its bottleneck layers to reduce computational complexity, BWHT can replace these convolution layers, achieving similar accuracy with fewer parameters. Unlike $1\times 1$ convolution layers, BWHT-based binary layers use fixed Walsh-Hadamard matrices, eliminating trainable parameters. Instead, a soft-thresholding activation function with a trainable parameter $T$ can be used to selectively attend to important frequency bands. The activation function $S_{T}$ for this frequency-domain compression is given as

Similarly, in ResNet20, $1\times 1$ convolutions can be replaced with BWHT layers. These transformations maintain matching accuracy while achieving significant compression on benchmark datasets such as CIFAR-10, CIFAR-100, and ImageNet [31]. However, BWHT transforms also increase the necessary computations for deep networks. The subsequent section discusses how micro-architectures and circuits can enhance the computational efficiency of BWHT-based tensor transformations.

In Fig. 2, we propose a design that leverages analog computations for frequency domain processing of neural networks to address the data deluge. The design’s crossbar combines six transistors (6T) NMOS-based cells for analog-domain frequency transformation. The corresponding cells for ‘-1’ and ‘1’ entries in the Walsh-Hadamard transform matrix are shown to the figure’s right. The crossbar combines these cells according to the elements in the transform matrix. Since the transform matrix is parameter-free, computing cells in the proposed design are simpler by being based only on NMOS for a lower area than conventional 6T or 8T SRAM-based compute-in-memory designs. Additionally, processing cells in the proposed crossbar are stitchable along rows and columns to enable perfect parallelism and extreme throughput, mitigating the data deluge further.

The operation of the crossbar comprises four steps, which are marked in Fig. 2. These steps are designed to minimize data movement and thus address the data deluge. The steps include precharging, independent local computations, row-wise summing, and single-bit output generation.

Fig. 3 shows the signal flow diagram for the above four steps. The 16 nm predictive technology models (PTM) simulation results are shown using the low standby power (LSTP) library in [33] and operating the system at 4 GHz clock frequency and VDD = 0.85V. Row-merge and column-merge signals in the design are boosted at 1.25V to avoid source degeneration. Unlike comparable compute-in-memory designs such as [12], which place product computations on bit lines, in our design, these computations are placed on local nodes in parallel at all array cells. This improves parallelism, energy efficiency, and performance by computing on significantly less capacitive local nodes than bit lines in traditional designs, thereby further addressing the data deluge.

Fig. 4 presents the high-level operation flow of our scheme for processing multi-bit digital input vectors and generating the corresponding multi-bit digital output vector using the analog crossbar illustrated in Fig. 2. This scheme is designed to address the data deluge by eliminating the need for ADC and DAC operations. The scheme utilizes bitplane-wise processing of the multi-bit digital input vector and is trained to operate effectively with extreme quantization. In the figure, the input vector’s elements with the same significance bits are grouped and processed in a single step using the scheme described in Fig. 2, which spans two clock cycles. The analog charge-represented output is computed along the row-wise charge sum lines and thresholded to generate the corresponding digital bits. This extreme quantization approach is applied to the computed MAC output, eliminating the need for ADCs. With multiple input bitplanes, labeled as - - - in the figure, the corresponding output bitplanes are concatenated to form the final multi-bit output vector.

Due to the extreme quantization applied to the analog output in the above scheme, the resulting digital output vector only approximates the true output vector under transformation. However, unlike a standard DNN weight matrix, the transformation matrix used in our frequency domain processing is parameter-free. This characteristic enables training the system to effectively mitigate the impact of the approximation while allowing for a significantly simpler implementation without needing ADCs or DACs, thereby further addressing the data deluge. The following training methodology achieves this.

Consider the frequency-domain processing of an input vector $\mathbf{x_{i}}$. In Fig. 4, we process $\mathbf{x_{i}}$ by transforming it to the frequency domain, followed by parameterized thresholding, and then reverting the output to the spatial domain. Consider a DNN with $n$ layers that chain the above sequence of operations as $\mathbf{x_{i+1}}=F_{0}(S_{T,i}(F_{0}(\mathbf{x_{i}})))$. Here, $F_{0}()$ is a parameter-free approximate frequency transformation as followed in our scheme in Fig. 4. $S_{T,i}()$ is a parameterized thresholding function at the i^th layer whose parameters $T_{i}$ are learned from the training data.

To further mitigate the data deluge, we introduce an early termination mechanism in our design. This mechanism is based on the observation that the contribution of higher bitplanes in the input vector to the final output is significantly less than the lower bitplanes due to the binary representation of numbers. Therefore, we can terminate the computation early if the higher bitplanes do not significantly affect the final output, thus saving energy and reducing data movement.

We illustrate the early termination mechanism in Fig. 6. The mechanism involves a thresholding operation that compares the partial sum of the output bitplanes with a predefined threshold. If the partial sum is less than the threshold, the computation is terminated early, and the remaining higher bitplanes are ignored. This mechanism reduces the number of computations and data movement, thereby mitigating the data deluge. The threshold for early termination is a design parameter that can be tuned based on the accuracy-energy trade-off. A lower threshold leads to more frequent early terminations, saving more energy but potentially reducing the accuracy, and vice versa. Therefore, the threshold should be carefully chosen to balance the trade-off between energy efficiency and accuracy.

Our proposed techniques for analog domain processing of frequency operations, including bitplane-wise input vector processing, ADC/DAC-free operations, and early termination mechanism, effectively address the data deluge by reducing data movement and computational complexity. These techniques also maintain high accuracy and energy efficiency, making them promising for future low-precision computations in deep neural networks. Fig. 7 shows our simulation results supporting the claim.

Fig. 1(b) illustrates the implementation of memory-integrated ADC. We specifically focus on our methods and results for eight-transistor (8T) compute-in-SRAM arrays, widely utilized in numerous platforms. Unlike 6T compute-in-SRAM, 8T cells are less prone to bit disturbances due to separate write and inference ports, making them more suitable for technology scaling. However, our suggestions for memory-integrated ADC apply to other memory types, including 10-T compute-in-SRAM/eDRAM and non-volatile memory crossbars.

In the proposed model, two adjacent CiM arrays work together for in-memory digitization, as depicted in Figure 8(a). When the left array calculates the input-weight scalar product, the right array performs SRAM-integrated digitization on the produced analog-mode multiply-average (MAV) outputs. Both arrays then switch their operating modes. Each array consists of 8-T cells, combining standard 6-T SRAM cells with a two-transistor weight-input product port shown in the figure. Memory cells for input-weight products are accessed using three control signals.

For in-memory digitization of charge-domain product-sum computed in the left array, column lines (CLs) in the right array realize the unit capacitors of a capacitive DAC formed within the memory array. A precharge transistor array is integrated with the column lines to generate the reference voltages. The first reference voltage is generated by summing the charges of all column lines. The developed MAV voltage in the left CiM array is compared to the first reference voltage to determine the most significant bit of the digitized output. The next precharge state of memory-immersed capacitive DAC is determined, and the precharge and comparison cycles continue until the MAV voltage has been digitized to necessary precision.

Using neighboring CiM arrays for the first reference voltage generation for in-memory digitization offers several key advantages. First, various non-idealities in analog-mode MAV computation become common-mode due to using an identical array for the first reference voltage generation. Thus, the non-idealities only minimally impact the accuracy of digitization. Second, collaborative digitization minimizes peripheral overheads. Only an analog comparator and simple modification in the precharge array are sufficient to realize a successive approximation search.

Compared to traditional CiM approaches with a dedicated ADC at each array, our scheme’s interleaving of scalar product computation and digitization cycles affects the achievable throughput. However, with simplified low-area peripherals, more CiM arrays can be accommodated than prior works employing dedicated ADCs. Therefore, our scheme compensates for the overall throughput at the system level by operating many parallel CiM arrays. This improved area efficiency of CiM arrays in our scheme minimizes the necessary exchanges from off-chip DRAMs to on-chip structures in mapping large DNN layers, a significant energy overhead in conventional techniques.

In addition to the nearest neighbor networking in Figure 8, more complex CiM networks can also be orchestrated for more time-efficient collaborative digitization in Flash and/or hybrid SAR + Flash mode. Figure 9 shows an example networking scheme where Array-1 couples with three memory arrays to the right for collaborative digitization in Flash mode. Here, the three right arrays simultaneously generate the respective reference voltages for the Flash mode of digitization and to determine the first two most significant bits in one comparison cycle.

The hybrid scheme for data conversion further benefits from exploiting the statistics of the multiply average (MAV) computed by the CiM. In many CiM schemes, the computed MAV is not necessarily uniformly distributed. For example, in bit-plane-wise CiM processing, DACs are avoided by processing a one-bit input plane in one time step. This results in a skewed distribution of MAV, Fig. 10(a). The skewed distribution of MAV can be leveraged by implementing an asymmetric binary search for digitization, Fig. 10(b). Under the asymmetric search, the average number of comparisons reduces, thus proportionally reducing the energy and latency for the operation, Fig. 10(c). The proposed hybrid digitization scheme further exploits the asymmetric search, which can be accelerated by Flash digitization mode.

A 65 nm CMOS test chip characterized the proposed SRAM-integrated ADC. The fabricated chip’s micrograph and measurement setup are shown in Figures 11(a, b). Four compute-in-SRAM arrays of size 16x32 were implemented. The coupling of CiM arrays can also be programmed to realize hybrid Flash-SAR ADC operations, such as obtaining the two most significant bits in Flash mode and the remaining in SAR.

Figure 11(c) shows the transient waveforms of different control signals and comparator outputs, showing a hybrid Flash + SAR ADC operation. Flash mode is activated in the first comparison cycle where CiM arrays generate the corresponding reference voltages, and the first two bits of MAV digitization are extracted. Subsequently, the operation switches to SAR mode, where the remaining digitization bits are obtained by engaging one array alone with another. In the last four cycles, other arrays become free to similarly operate on a proximal CiM array to digitize MAV in SAR mode. Figure 12(a) shows the measured staircase plot of input voltage to output codes and the comparison to an ideal staircase, demonstrating near-ideal performance. This suggests that the proposed SRAM-integrated ADC operates effectively, with the hybrid Flash + SAR ADC operation providing a flexible and efficient approach to digitization within the memory array.

Fig. 13 and Table I show the design space exploration of pro- posed memory-immersed ADC compared to other ADC styles. In Fig. 13(a), leveraging in-memory structures for capacitive DAC formation, the proposed in-memory ADC is more area efficient than Flash and SAR styles. Significantly, Flash ADC’s size increases exponentially with increasing bit precision. In Fig. 13(b), SAR ADC’s latency increases with bit precision while Flash ADC can maintain a consistent latency but at the cost of the increasing area as shown in Fig. 13(a). A hybrid data conversion in the proposed in-memory provides a middle ground, i.e., lower latency than in SAR ADC. Figs. 13(c-d) show the impact of supply voltage and frequency scaling on in-memory ADC’s power and accuracy for MNIST character recognition.

In conclusion, the proposed method of integrating compute-in-memory (CiM) arrays for collaborative digitization offers a promising approach to handling the data deluge in modern computing systems. By leveraging the unique properties of 8T compute-in-SRAM arrays, the method provides a scalable and efficient solution for in-memory digitization. The hybrid SRAM-integrated Flash and SAR ADC operation further enhances time efficiency, while the exploitation of MAV statistics contributes to the ADC’s time efficiency. The successful implementation and characterization of the proposed SRAM-integrated ADC on a 65 nm CMOS test chip further validate the effectiveness of this approach.