Exploring the Sparsity-Quantization Interplay on a Novel Hybrid SNN Event-Driven Architecture

The spiking neuron model used in this work is a leaky integrate-and-fire (LIF) neuron [8], whose characteristics are shown in Equations 1 and 2.

Equation 1 describes how the neuron’s membrane potential ($u_{j}[t]$) evolves over time. The decay factor ($\beta$), ranging between $0$ and $1$, controls how much the previous potential $u_{j}[t]$ affects the current potential $u_{j}[t+1]$. A higher $\beta$ value implies less decay, enabling the neuron to retain more of its previous state, resulting in sparser behavior. Incoming weighted spikes from other neurons ($w_{ij}\cdot s_{i}[t]$) increase the potential, while a threshold-based self-decay term ($s_{j}[t]\cdot\theta$) reduces it. Equation 2 determines when the neuron fires. If the membrane potential exceeds the threshold ($\theta$), the neuron generates a spike ($s_{j}[t]=1$). As such a lower $\theta$ value reduces the potential required for firing, thereby increasing the neuron’s firing frequency.

We employ Quantization-Aware Training (QAT) [9] to quantize model weights and biases into integers, with the quantization error incorporated into the loss function during training. This enables the model to adapt to the quantization constraints, enhancing its performance when operating with quantized data. During evaluation, weights and biases are fully quantized, while neuronal parameters remain in floating-point due to the current lack of support for LIF neuron quantization. The accumulated membrane data undergoes dequantization back to floating-point for accurate spiking operations.

Figure 2 depicts the block diagram of the dense core (dc), comprising three key components: a PE array, control units, and activation units. The PE array features a fixed column of 27 processing elements (PEs) and employs a weight stationary (WS) dataflow. We chose 27 PEs because of the WS requirements, specifically to handle $3$ input channels and $3\times 3$ filters. This choice minimizes memory footprint requirements, as the input layer’s three channels lead to fewer weights compared to input or output activations. Since we employ tiling in the output channel, we parameterize the number of rows, meaning that each row handles one output feature map. Each row then sequentially moves onto the next feature map until all output channels are processed. As a result, partial sums flow horizontally (left to right) while input image pixels flow vertically (top to down) in a systolic manner.

Each PE contains a Multiply and Accumulate (MAC) unit, along with a register for weight storage. All 27 PEs work in parallel to process input channels and generate a single output membrane potential per cycle. To maintain the systolic pipeline flow, we utilize independent registers/FFs to act as “shift registers” to provide a fixed delay for each of the 27 inputs in the array. These registers are controlled by the Staggering Routine within the Control unit. The right-most PE’s shift register depth dictates the array’s pipeline depth.

The Control unit is implemented as a state machine and orchestrates three main tasks:

• •

  Data management: It feeds image data into the PE array. The Address Generation Routine calculates read indices for the 27 top PEs, accessing image buffers (implemented with on-chip flip-flops for their small storage footprint). This enables row-major image storage while facilitating parallel weight access by all 27 PEs. The Staggering Routine manages ”shift registers” that control the data flow from the image buffers.

• •

  Activation synchronization: It starts/stops the Activ unit using the EN signal. Once the PE pipeline is filled (i.e., the first membrane potential accumulation is produced), the Control unit triggers the activation phase (responsible for spike train generation) with the signal EN.

• •

  Output channel tiling: The Control unit coordinates the tiling of output channels, meaning that each row in the systolic array strides through the output channels. When the rows of the systolic array transition to the next output feature map computation, it resets the partial sums and membrane potentials within the PE Array and Activ units using the rst signal. Finally, the unit sets the layer avail signal high to indicate the completion of layer processing.

The accumulated potential from each row flows into the Activ unit, where it performs two core functions. First, it modifies the incoming potential by adding the filter bias value, applying the leakage factor ($\beta$) to mimic the leaky behavior, and then performs thresholding. If the membrane potential exceeds the threshold ($\theta$), the unit subtracts the threshold value from the membrane potential and sets the associated spike to $1$; otherwise, the spike is $0$. Secondly, after processing an entire output feature map, the Activ unit writes the generated spike train to the Block RAM (BRAM), which serves as the intermediate storage between layers. Spike trains in BRAM follow a timestep-major order (see Figure 2) with the spike trains for consecutive timesteps stored in contiguous addresses. For example, if the layer consists of $N$ output channels and $T$ timesteps, then $N\times T$ locations in total are required for the layer’s spike train storage. This layout is consistent for both dense and sparse cores.

We adapt conventional convolution (CONV) and fully connected (FC) layers for event-driven processing by dividing their operation into spike train compression and accumulation phases. The sparse core design, illustrated in Figure 3, implements these phases and consists of two primary components: an Event Control Unit (ECU) and Neural Cores (NC). Within the ECU, the control routine is implemented as a state machine and has several key functions.

With the $(row,col)$ pairs provided, the Accum routine within each NC performs the accumulation phase of the LIF neuron. It reads the membrane potential value from the BRAM, updates them with corresponding coefficient weights, and writes back the result to the BRAM. Similar to the dense core, the sparse cores also support quantization. Also, note that both the Address Generation and Accum routines are fully pipelined and can update one neuron per cycle. Filter weights for CONV layers are stored in BRAM and LUTRAM, while larger fully-connected (FC) weights use Ultra RAMs (URAMs) for their higher density and energy efficiency. After processing all input feature maps, the control unit enables the NC’s Activ. routine to initiate the LIF neuron’s spiking phase.

The architecture unrolls the output channels by a factor of $N$, defined as a top-level parameter, to determine the number of NC instances. Each NC instance strides through the output channels by $N$. For instance, an NC instance with index $2$ and $N=8$ will process OFMs with indices 1, 9, 17, etc. The architecture also features max-pooling on spikes, which aligns better with SNN temporal dynamics and minimizes information loss during downsampling, compared to other pooling methods [10]. Implementation on binary feature maps only requires sliding an OR gate over an $N\times N$ input area, where $N$ is the downsampling ratio.

For efficient resource usage, we only use the FPGA’s on-chip memory (BRAM, URAM, LUTRAM) for storing model parameters and spike trains, avoiding the external DDR memory. Early layers with smaller convolutions use LUTRAM, which is more efficient than flip-flops for small RAMs due to its flexibility, despite being less abundant than flip-flops in Xilinx UltraScale+ devices.

Our strategy for aggressive on-chip storage, while beneficial, comes with a trade-off in power consumption. Since a large portion of this storage remains inactive during layer execution (until NC reaches the next output channel), the associated weight data is not used. To address this, we implement clock gating for our memory units (right side of Figure 3). The memory is partitioned into two regions, with the most significant bit (MSB) of the address line controlling the active region for both reads and writes. An AND gate controls the clock signal to the memory unit, ensuring that only the active region receives clock cycles. While this on-chip storage strategy works well for quantized models like int4, it poses limitations for fp32 models. Specifically, the high LUT usage for weight storage prevents us from scaling beyond the custom VGG9 architecture or deploying larger models.

Due to the lack of native quantization support, the design incorporates floating-point computation capabilities in both dense and sparse cores to de-quantize weights and biases retrieved from memory. We utilize a resource-efficient shift-and-add approach instead of DSP blocks to perform multiplications by constants [11], improving overall efficiency.

We implemented our proposed hardware using SystemVerilog and synthesized on a Xilinx Virtex® UltraScale+™ XCVU13P FPGA. For evaluation, we use the CIFAR-10, CIFAR-100, and Street View House Numbers (SVHN) datasets with a modified VGG9 network trained using snnTorch [12]. The network, trained using surrogate gradients [13], has a structure: 64C3-112C3-MP2-192C3-216C3-MP2-480C3-504C3-560C3-MP2-1064-$P$. $X$C$Y$ denotes $X$ filters of size $Y\times Y$ and MP$Z$ denotes $Z\times Z$ maxpooling. $P$ represents the number of neurons (i.e., population) in the network’s output layer. Prior work [14] has shown that using a population of neurons in the output layer can lead to better accuracy, even with fewer time steps. That is, we can make the output layer larger instead of needing more time steps to achieve good results. Through experimentation, we found that the following $P$ values provided the best accuracy with the minimum possible spike train length: $P=1000$ for CIFAR10 and SVHN datasets, and $P=5000$ for CIFAR100. We selected the network architecture to ensure compatibility with the Virtex UltraScale+ platform, following prior work [6]. We tuned the leaky neuron hyperparameters ($\beta=0.15$ and $\theta=0.5$) and used layer-wise batch normalization to prevent overfitting.

Table I summarizes the implementation results of our proposed hardware for CIFAR100 (in perf²), which are representative of the trends observedfor CIFAR10 and SVHN. We compare the results of full-precision (fp32) with its 4-bit integer counterpart (int4). We empirically determined that a $(1,28,12,54,16,72,70,19,4)$ configuration (neural cores allocated per layer) yields the most balanced execution profile (layer overheads: 0.9%, 13.4%, 13.6%, 13.8%, 12.8%, 12.3%, 12.9%, 15.6%, 4.8%). The int4 implementation demonstrates significant resource savings compared to fp32, using $8\times$ fewer LUTs, primarily due to efficient LUTRAM usage for the CONV_1_2 weight data. Additionally, int4 uses $3.4\times$ fewer BRAMs/URAMs due to the minimum 8-bit width configuration of BRAM primitives for CONV layers. Overall, the fp32 and int4 designs occupy $24\%$ and $34\%$ of the FPGA’s LUT resources, respectively. Importantly, the fp32 hardware consumes $2.82\times$ more power than the int4 configuration, highlighting the power efficiency benefits of quantization.

Our energy analysis comparing int4 and fp32 (Figure 4) reveals significant energy savings benefits of quantization across CIFAR10, CIFAR100, and SVHN datasets. We calculate the energy expenditure per image by summing the energy per layer. For CIFAR10 and CIFAR100, int4 reduces the average energy by $3.4\times$ and $1.7\times$, respectively, across all configurations. The majority of these savings (roughly $3\times$ for CIFAR10 and $1.5\times$ for CIFAR100) stem directly from int4’s inherent power advantage. Additionally, the fp32 designs exhibit a higher spike count ($1.1\times$ and $1.15\times$ for CIFAR10 and CIFAR100, respectively), contributing a further $10-15\%$ to the energy difference. For SVHN, the scaling trend differs slightly due to the resource allocation dominance of the CONV_1_2 layer (with more allocated neural cores), leading to a smaller energy gap between fp32 and int4. The perf⁴ configuration of the quantized hardware achieves a $28\%$ energy reduction compared to its LW counterpart, contrasting with the $52\%$ reduction in the full-precision design. Since the full-precision hardware utilizes LUTRAM for this layer, increasing the NC count has a less pronounced impact on power compared to the quantized hardware. Overall, quantization yields a significant $4\times$ reduction in latency and notable energy savings across all datasets.

Table II depicts our comparison of direct coding and rate coding on CIFAR10 using the quantized LW configuration. Because the rate-coded network receives spikes as inputs, it only needs sparse cores, while the direct-coded network needs our hybrid architecture. As such, although both networks run on our hardware architecture, for a fair comparison, we turned off the dense core for the rate-coded network to prevent unnecessary power consumption. The Total Spikes represents the spikes across all timesteps. With only 2 timesteps, direct coding yielded much higher sparsity (2.6$\times$ fewer spikes) and achieved $10\%$ higher accuracy than rate coding at 25 timesteps. Further increasing the timesteps plateaued the accuracy for both schemes¹¹1To enable reproducibility of our accuracy and implementation, our training and hardware code are available at https://github.com/githubofaliyev/SNN-DSE/tree/DATE25. Importantly, the direct-coded network consumed $26.4\times$ less energy than its rate-coded counterpart, contrary to prior work [3].

Table III compares our proposed hybrid architecture’s dynamic power and throughput to two recent related works: an event-driven design with quantization support [15] and a resource-efficient approach [7]. We focus on dynamic power and throughput as these metrics were reported in prior works.

Compared to [15], we achieve more than 2$\times$ the throughput for SVHN and CIFAR10, with a power increase of $2.2\times$ and $1.8\times$, respectively (Table III). In addition to using a lower power board, [15] obtained lower power measurements by subtracting the board’s baseline power consumption from the accelerator’s total runtime power. Additionally, we outperform their work in inference accuracy ($4\%$ and $8\%$ higher for SVHN and CIFAR10, respectively). Compared to [7], our perf⁴ configuration on CIFAR100 achieves $51\times$ throughput improvement and $2\times$ power savings, with only a $3.1\%$ accuracy decrease. This work is the most relevant comparison to ours due to the use of similar implementation platforms.