HYPER-SNN: Towards Energy-efficient Quantized Deep Spiking Neural Networks for Hyperspectral Image Classification

An SNN consists of a network of neurons that communicate via a sequence of spikes modulated by synaptic weights. The activity of pre-synaptic neurons modulates the membrane potential of postsynaptic neurons, generating an action potential or spike when the membrane potential crosses a firing threshold. The spiking dynamics of a neuron are generally modeled using either the Integrate-and-Fire (IF) [40] or Leaky-Integrate-and-Fire (LIF) model [41]. Both IF and LIF neurons accumulate the input current into their respective states or membrane potentials. The difference between the two models is that the membrane potential of a IF neuron does not change during the time period between successive input spikes while the LIF neuronal membrane potential leaks at a constant rate. In this work, we use the LIF model to convert ANNs trained with ReLU activations, to SNNs, because the leak term provides a tunable control knob that can reduce inference latency and spiking activity.The IF model can be characterized by the following differential equation

Recent research on training supervised deep SNNs can be broadly divided into three categories: 1) ANN-to-SNN conversion-based training, 2) Spike timing dependent backpropagation (STDB), and 3) Hybrid training.

Uniform quantization transforms a weight element $w\in[w_{min},w_{max}]$ to a range $[-2^{b-1},2^{b-1}-1]$ where $b$ is the bit-width of the quantized integer representation. There are primarily two choices for the above transformation, known as affine and scale quantization. Detailed descriptions of these two types of quantization can be found in [50].

Our key motivation for SNN weight quantization is the hardware acceleration of inference using energy-efficient integer or fixed-point computational units implemented as crossbar array based processing-in-memory (PIM) accelerators. Note that the six transistor SRAM array based in-memory computing requires low-precision weights for multiply-and-accumulate (MAC) operations due to low density of the bit-cells. Previous research [51, 52] have proposed post-training SNN quantization tailored towards unsupervised learning, which may not scale to complex vision tasks without requiring high-precision ($\geq{8}$ bits). In contrast, in this work, we propose quantization-aware training, where the weights are fake quantized (see [50]) in the forward path computations, while the gradients and weight updates are calculated using the full precision weights. There are several choices for sharing quantization parameters among the tensor elements in a SNN. We refer to this choice as quantization granularity. We employ per-tensor (or per-layer) granularity where the same quantization parameters are shared by all elements in the tensor, because this reduces the computational cost compared to other granularity choices with no impact on model accuracy. Activations are similarly quantized, but only in the SNN input layer, since they are binary spikes in the remaining layers.

To evaluate the compute cost, let us consider a 3-D convolutional layer $l$, the dominant layer in HSI classification models, that performs a tensor operation $O_{l}=X_{l}\circledast W_{l}$ where $X_{l}\in\mathbb{R}^{H_{l}^{i}\times{W_{l}^{i}}\times{C_{l}^{i}}\times{D_{l}^{i}}}$ is the input activation tensor, $W_{l}\in\mathbb{R}^{k_{l}^{x}\times{k_{l}^{y}}\times{k_{l}^{z}}\times{C_{l}^{i}}\times{C_{l}^{o}}}$ and $O^{l}\in\mathbb{R}^{H_{l}^{o}\times{W_{l}^{o}}\times{C_{l}^{o}}\times{D_{l}^{o}}}$ is the output activation tensor, where $H_{l}^{i}$, $W_{l}^{i}$, $C_{l}^{i}$, $D_{l}^{i}$ are the input height, width, channel size, and spectral size, respectively. Similarly, $H_{l}^{o}$, $W_{l}^{o}$, $C_{l}^{o}$ and $D_{l}^{o}$ are the output height, width, number of filters and the output spectral size, respectively, and $k_{l}^{x}$, $k_{l}^{y}$, $k_{l}^{z}$ represent the filter size in the three spatial dimensions. The result of the real-valued operation $O_{l}=X_{l}\circledast W_{l}$ can be approximated with quantized tensors $X_{l}^{Q}$ and $W_{l}^{Q}$, by first dequantizing them producing $\hat{X_{l}}$ and $\hat{W_{l}}$ respectively, and then performing the convolution. Note that both $X_{l}^{Q}$ and $W_{l}^{Q}$ have similar dimensions as $X_{l}$ and $W_{l}$ respectively. Assuming the tensors are scale-quantized per layer,

where $s_{s}^{X}$ and $s_{s}^{W}$ are scalar values for scale quantization representing the levels of the input and weight tensor respectively. Hence, scale quantization results in an integer convolution, followed by a point-wise floating-point multiplication for each output element. Given that a typical convolution operation involves a few hundred MAC operations (accumulate for binary spike inputs) to compute an output element, a single floating-point operation for the scaling shown in Eq. 4 is a negligible computational cost. Note that $X_{l}$ only needs to be quantized if $l$ is the input layer. In all other cases, $X_{l}^{Q}=X_{l}$ and $s_{s}^{X}=1$.

Although both affine and scale quantization enable the use of low-precision arithmetic, affine quantization results in higher computationally expensive inference as shown below.

where $z_{a}^{X}$ and $z_{a}^{W}$ are tensors of sizes equal to that of $X_{l}^{Q}$ and $W_{l}^{Q}$ respectively that consist of repeated elements of the scalar zero-values of the input activation and weight tensor respectively. On the other hand, $s_{a}^{X}$ and $s_{a}^{W}$ are the corresponding scale values. The first term in the numerator of Eq. III-A is the integer convolution operation similar to the one performed in scale quantization shown in Eq. 4. The second term contains integer weights and zero-points, which can be computed offline, and adds an element-wise addition during inference. The third term, however, involves the quantized activation $X_{l}^{Q}$, which cannot be computed offline. This extra computation, depending on the implementation, can introduce considerable overhead, reducing or even eliminating the throughput and energy advantage that low precision PIM accelerators offer over floating-point MAC units. Hence, we use scale quantization during inference.

Note, however, that our experiments detailed in Section V show that using scale quantization during SNN training degrades the test accuracy significantly. Hence, we propose that training should use affine quantization of both the weights and input layer activations. Note that for a integer math unit or PIM accelerator, we do not necessarily need to quantize the SNN membrane potentials which are obtained as results of the accumulate operations of the weight elements. This is because the membrane potentials only need to be compared with the threshold voltage once for each time step, which consumes negligible energy, and can be performed using high precision fixed-point comparators (in the periphery of the crossbar array for PIM accelerators). However, quantizing the potentials can reduce the data movement cost as discussed in Section VI-B.

In this subsection, we derive the expressions to compute the gradients of the parameters at all layers for our training framework. Our framework, which is illustrated in Fig. 2, incorporates the quantization methodology described above into the STDB technique used to train SNNs [32], where the spatial and temporal credit assignment is performed by unrolling the network in time and employing BPTT.
Output Layer: The neuron model in the output layer $L$ only accumulates the incoming inputs without any leakage, does not generate an output spike, and is described by

where $N$ is the number of output labels, $\mbox{\boldmath$u$}_{L}$ is a vector containing the membrane potential of $N$ output neurons, $\mbox{\boldmath$\hat{w}$}_{L}$ is the fake quantized weight matrix connecting the last two layers ($L$ and $L{-}1$), and $\mbox{\boldmath$o$}_{L-1}$ is a vector containing the spike signals from layer $(L{-}1)$. The loss function is defined on $\mbox{\boldmath$u$}_{L}$ at the last time step $T$. We employ the cross-entropy loss and compute the softmax of $\mbox{\boldmath$u$}_{L}^{T}$. The output of the network is passed through a softmax layer that outputs a probability distribution. The loss function $\mathcal{L}$ is defined as the cross-entropy between the true output ($y$) and the SNN’s predicted distribution ($p$).

The derivative of the loss function with respect to the membrane potential of the neurons in the final layer is described by $\frac{\partial\mathcal{L}}{\partial\mbox{\boldmath$u$}_{L}^{T}}=(\mbox{\boldmath${p}$}-\mbox{\boldmath$y$})$, where $p$ and $y$ are vectors containing the softmax and one-hot encoded values of the true label respectively. To compute the gradient at the current time step, the membrane potential at the previous step is considered as an input quantity [32]. With the weights being fake quantized, gradient descent updates the network parameters $\mbox{\boldmath$w$}_{L}$ of the output layer as

where $\eta$ is the learning rate (LR). Note that the derivative of the fake quantization function of the weights ($\frac{\partial\mbox{\boldmath$\hat{w}$}_{L}}{\partial\mbox{\boldmath$w$}_{L}}$) is undefined at the step boundaries and zero everywhere, as shown in Fig. 3(a). Our training framework addresses this challenge by using the Straight-through Estimator (STE) [53], which approximates the derivative to be equal to 1 for inputs in the range $[w_{min},w_{max}]$ as shown in Fig. 3(b), where $w_{min}$ and $w_{max}$ are the minimum and maximum values of the weights in a particular layer. Note that $w_{min}$ and $w_{max}$ are updated at the end of every mini-batch to ensure all the weights lie between $w_{min}$ and $w_{max}$ during the forward and backward computations in each training iteration. Hence, we use $\frac{\partial\mbox{\boldmath$\hat{w}$}_{L}}{\partial\mbox{\boldmath$w$}_{L}}\approx 1$ to compute the loss gradients in Eq. 9.
Hidden layers: The neurons in the hidden convolutional and fully-connected layers are defined by the quantized LIF model as

where $\lambda_{l}$ and $v_{l}$ represent the leak and threshold potential for all neurons in layer $l$. All neurons in a layer possess the same leak and threshold value. This reduces the number of trainable parameters and we did not observe any significant improvement by assigning individual threshold/leak to each neuron. Given that the threshold is same for all neurons in a particular layer, it may seem redundant to train both the weights and threshold together. However, we observe that the number of time steps required to obtain the state-of-the-art classification accuracy decreases with this joint optimization. We hypothesize that this is because the optimizer is able to reach an improved local minimum when both parameters are tunable. The weight update in Q-STDB is calculated as

where $\frac{\partial\mbox{\boldmath$\hat{w}$}_{l}}{\partial\mbox{\boldmath$w$}_{l}}$ and $\frac{\partial\mbox{\boldmath$z$}_{l}^{t}}{\partial\mbox{\boldmath$o$}_{l}^{t}}$ are the two discontinuous gradients. We calculate the former using STE described above, while the latter is approximated using surrogate gradient [48] shown below.

Note that $\gamma$ is a hyperparameter denoting the maximum value of the gradient. The threshold and leak update is computed similarly using BPTT [32].

We used three publicly available datasets, namely Indian Pines, Pavia University, and Salinas scene. A brief description follows for each one [56].

Indian Pines: The Indian Pines (IP) dataset consists of $145{\times}145$ spatial pixels and $220$ spectral bands in a range of $400-2500$ nm. It was captured using the AVIRIS sensor over North-Western Indiana, USA, with a ground sample distance (GSD) of $20$ m and has $16$ vegetation classes.
Pavia University: The Pavia University (PU) dataset consists of hyperspectral images with $610{\times}340$ pixels in the spatial dimension, and $103$ spectral bands, ranging from $430$ to $860$ nm in wavelength. It was captured with the ROSIS sensor with GSD of $1.3$ m over the University of Pavia, Italy. It has a total of 9 urban land-cover classes.
Salinas Scene: The Salinas Scene (SA) dataset contains images with $512{\times}217$ spatial dimension and $224$ spectral bands in the wavelength range of $360$ to $2500$ nm. The $20$ water absorbing spectral bands have been discarded. It was captured with the AVIRIS sensor over Salinas Valley, California with a GSD of $3.7$ m. In total $16$ classes are present in this dataset.

For preprocessing, images in all the data sets are normalized to have a zero mean and unit variance. For our experiments, all the samples are randomly divided into two disjoint training and test sets. The limited 40% samples are used for training and the remaining 60% for performance evaluation.

We have used the Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient evaluation measures to evaluate the HSI classification performance for our proposed architectures, similar to [23]. Here, OA represents the number of correctly classified samples out of the total test samples. AA represents the average of class-wise classification accuracies, and Kappa is a statistical metric used to assess the mutual agreement between the ground truth map and classification map. Column-$2$ in Table II shows the ANN accuracies, column-$3$ shows the accuracy after ANN-SNN conversion with $50$ timesteps. Column-$4$ shows the accuracy when we perform our proposed training without quantization, while columns 5 to 7 shows the SNN test accuracies obtained with Q-STDB for different bit precisions (4 to 6 bits) of the weights. We observe that for all the datasets, SNNs trained with 6-bit weights result in $5.33\times$ reduction in bit-precision compared to full-precision (32-bit) models and perform almost at par with the full precision ANNs on both the architectures. 4-bit weights do not incur significant accuracy drop as well, and can be used for applications demanding high energy-efficiency and low latency. Fig. 5 shows the confusion matrix for the HSI classification performance of the ANN and proposed SNN over the IP dataset for both the architectures. Although the membrane potentials do not need to be quantized as described in Section III, we observed that the model accuracy does not drop significantly even if we quantize them, and hence, the SNN results shown in Table II correspond to 6-bit membrane potentials. Moreover, quantized membrane potentials can reduce the data movement cost as discussed in Section VI-B.

The performance of our ANNs and SNNs trained via Q-STDB are compared with the current state-of-the-art ANNs used for HSI classification in Table III. Note that mere porting the ANN architectures used in [23, 26] to SNNs, and performing 6-bit Q-STDB results in significant accuracy drop, and hence, shows the efficacy of our proposed architectures.

To model energy consumption, we assume a generated SNN spike consumes a fixed amount of energy [43]. Based on this assumption, earlier works [42, 31] have adopted the average spiking activity (also known as average spike count) of an SNN layer $l$, denoted ${\zeta}^{l}$, as a measure of compute-energy of the model. In particular, ${\zeta}^{l}$ is computed as the ratio of the total spike count in $T$ steps over all the neurons of the layer $l$ to the total number of neurons in that layer. Thus lower the spiking activity, the better the energy efficiency.

Fig. 6 shows the average number of spikes for each layer with Q-STDB when evaluated for $200$ samples from the IP testset for the CNN-3D architecture. Let the average be denoted by $\zeta_{l}$ which is computed by summing all the spikes in a layer over $5$ time steps and dividing by the number of neurons in that layer. For example, the average spike count of the $3^{rd}$ convolutional layer of the SNN is $0.568$, which implies that over a $5$ timestep period each neuron in that layer spikes $0.568$ times on average over all input samples.

Let us assume a 3-D convolutional layer $l$ having weight tensor $\textbf{W}^{l}\in\mathbb{R}^{k_{l}^{x}\times k_{l}^{y}\times k_{l}^{z}\times C_{l}^{i}\times C_{l}^{o}}$ that operates on an input activation tensor $\textbf{I}^{l}\in\mathbb{R}^{H_{l}^{i}\times W_{l}^{i}\times C_{l}^{i}\times D_{l}^{i}}$, where the notations are similar to the one used in Section III. We now quantify the energy consumed to produce the corresponding output activation tensor $\textbf{O}^{l}\in\mathbb{R}^{H_{l}^{o}\times W_{l}^{o}\times C_{l}^{o}\times D_{l}^{o}}$ for an ANN and SNN, respectively. Our model can be extended to fully-connected layers with $f_{l}^{i}$ and $f_{l}^{o}$ as the number of input and output features respectively, and to 2-D convolutional layers, by shrinking a dimension of the feature maps. In particular, for an ANN, the total number of FLOPS for layer $l$, denoted $F_{l}^{ANN}$, is shown in row $1$ of Table IV [58, 59]. The formula can be easily adjusted for an SNN in which the number of FLOPs at layer $l$ is a function of the average spiking activity at the layer $(\zeta_{l})$ denoted as $F_{l}^{SNN}$ in Table IV. Thus, as the activation output gets sparser, the compute energy decreases.

Fig. 7 illustrates the compute energy consumption and FLOPs for full precision ANN and 6-bit quantized SNN models of the two proposed architectures while classifying the IP, PU, and SS datasets, where the energy is normalized to that of an equivalent ANN. We also consider $6$-bit ANN models to compare the energy-efficiency of low-precision ANNs and SNNs. As seen in Fig. 7, 6-bit ANN models are $12.5\times$ energy-efficient compared to 32-bit ANN models due to the similar factor of improvement in MAC energy (see Table V). Note that we can achieve the HSI test accuracies shown in Table II with quantized ANNs as well.

The FLOPs for SNNs obtained by our proposed training framework is smaller than that for an ANN with similar number of parameters due to low spiking activity. Moreover, because the ACs consume significantly less energy than MACs for all bit precisions, SNNs are significantly more compute efficient. In particular, for CNN-3D on IP, our proposed SNN consumes $\mathord{\sim}199.3\times$ and $\mathord{\sim}15.9\times$ less compute energy than an iso-architecture full-precision and 6-bit ANN with similar parameters respectively. The improvements become ${\sim}560.6\times$ and ${\sim}44.8\times$ respectively on averaging across the two network architectures and three datasets. Note that we did not consider the memory access energy in our evaluation because it is dependent on the underlying system architecture. In general, SNNs incur significant data movement because both the membrane potentials and weights need to be fetched from the on-chip memory. Q-STDB addresses the memory cost by reducing their bit precisions by $5.33\times$ (see Section V-C) compared to full-precision models. Moreover, there have been many proposals to reduce the memory cost by data buffering [63], computing in non-volatile crossbar memory arrays [64], and data reuse with energy-efficient dataflows [65]. All these techniques can be complemented with Q-STDB to further decrease the memory cost.