Exploring the Connection Between Binary and Spiking Neural Networks

Our BNN implementation follows the XNOR-Net proposal in Ref. [6]. While the feedforward dot-product is performed using binary values, BNNs maintain proxy full-precision weights for gradient calculation. To formalize, the dot-product computation between the full-precision weights and inputs is simplified in a BNN as follows:

where, $\alpha$ is a non-binary scaling factor determined by the L1-norm of the full-precision proxies [6]. Straight-Through Estimator (STE) with gradient-clipping to ($-1,+1$) range is used during the training process [6]. Note that the above formulation reduces both weights and neuron activations to $-1,+1$ values. Although a non-binary scaling factor is introduced per layer, yet the number of non-binary operations due to the scaling factor is significantly low.

SNN training can be mainly divided into three categories: ANN²²2ANN refers to standard non-spiking networks, Analog Neural Networks [33], where the neuron state representations are analog or full-precision in nature, instead of binary spikes.-SNN conversion, backpropagation through time from scratch and unsupervised training through Spike-Timing Dependent Plasticity [34]. Since ANN-SNN conversion relies on standard backpropagation training techniques, it has been possible to scale SNN training using such conversion methods to large-scale problems [15]. ANN-SNN conversion is driven by the observation that an Integrate-Fire spiking neuron is functionally equivalent to a Rectified Linear ANN neural transfer function. The functionality of an Integrate-Fire (IF) spiking neuron can be described by the temporal dynamics of a state variable, $v_{mem}$, that accumulates incoming spikes and fires an output spike whenever the membrane potential crosses a threshold, $v_{th}$.

Considering $\mathbb{E}[\mathbb{X}(t)]$ to be the input firing rate (total spike count over a given number of time-steps), the output spiking rate of the neuron is given by $\mathbb{E}[\mathbb{Y}(t)]=\frac{w.\mathbb{E}[\mathbb{X}(t)]}{v_{th}}$ (considering the neuron being driven by a single input $\mathbb{X}(t)$ and a positive synaptic weight $w$). In case the synaptic weight is negative, the neuron firing rate would be zero since the neuron membrane potential would be unable to cross the threshold. This is in direct correspondence to the Rectified Linear functionality and is described by an example in Fig. 1. An ANN trained with ReLU neurons can therefore be transformed to an SNN with IF spiking neurons with minimal accuracy loss. The sparsity of binary neuron spiking behavior can be exploited for event-driven inference resulting in significant power savings [15].

Our B-SNN is trained by using BNN training techniques described earlier. However, we utilize analog ReLU neurons instead of binary neurons. Conceptually, the network structure is analogous to Binary-Weight Networks (BWNs) introduced in Ref. [6]. However, we also include additional constraints like bias-less neural units and no batch-normalization layers in the network structure [15]. This is due to the fact that including bias and batch-normalization can potentially result in huge accuracy loss during the conversion process [16]. Much of the success of training BNNs can be attributed to Batch-Normalization. Hence, it is not trivial to train such highly-constrained ANNs with binary weights and without Batch-Normalization aiding the training process. Additional constraints like the choice of pooling mechanism, spiking neuron reset mechanism are discussed in details in the next section. This work is aimed at performing an extensive empirical analysis to substantiate the feasibility of achieving high-accuracy and low-latency B-SNNs.

Note that the threshold of each network layer is an additional hyper-parameter introduced in the SNN model and serves as an important trade-off factor between SNN latency and accuracy. Due to the neuron reset mechanism, the SNN neurons are characterized by a discontinuity at the reset time-instants. If the threshold is too low, the membrane potential accumulations would be always higher than the threshold causing the neuron to continuously fire. On the other hand, too high thresholds result in increased latency for neurons to fire. In this work, we normalize the neuron thresholds to the maximum ANN activation (recorded by passing the training set once after the ANN has been trained) [16]. Other thresholding schemes can be also applied [15] to minimize the conversion accuracy loss further.

Considering that the SNN is operated for $N$ time-steps, the network converges to a Binary-Weight Network as $N\rightarrow\infty$. However, for a finite number of timesteps, we can consider the network to be a discretized ANN, where the weights are binary but the neuron activations are represented by $B=log_{2}{N}$ number of bits. However, since the neuron states are represented by $0$ and $1$ values, B-SNNs are event-driven, thereby resulting in power consumption only when triggered, i.e. on receiving a spike from the previous layer. Hence, while the representative bit-precision can be $\sim 7$ bits for networks simulated over 100 timesteps, the network’s computational power does not scale-up corresponding to a multi-bit neuron model. This is explained in Fig. 2(a)-(e). The left-panel depicts a bit-cell for an “In-Memory” Resistive Random Access Memory (RRAM) based BNN hardware accelerator [35]. The RRAM can be programmed to either a high resistive state (HRS) or a low resistive state (LRS). The RRAM states and input conditions for $+1,-1$ are tabulated in Fig. 2 and shows the correspondence to the binary dot-product computation. Note that two rows per input are used due to the differential nature $+1,-1$ of the neuron inputs. Hence, irrespective of the value of the input, one of the rows of the array will be active resulting in power consumption. Fig. 2(b) depicts the same array for the B-SNN scenario. Since, in a B-SNN, the neuron outputs are $0$ and $1$, we can use just one row per bit-cell, thereby reducing the array area by $50\%$. Note that a dummy column will be required for referencing purposes of sense amplifiers interfaced with the array [35]. Additionally, the neuron circuits interfaced with the array need to accumulate the dot-product evaluation over time. Such an accumulation process can be accomplished using digital accumulators [36] or non-volatile memory technologies [37, 38]. Note that energy expended due to this accumulation process is minimal in contrast to the overall crossbar power consumption [39]. However, the input to the next layer will be a binary spike, thereby enabling us to utilize the “In-Memory” computing block as the core hardware primitive. It is worth noting here that the power-consumption involved in accessing the rows of the array occurs only on a spike event, thereby resulting in event-driven operation.

We evaluate our proposal on two popular, publicly available datasets, namely the CIFAR-$100$ [40] and large-scale ImageNet [7] dataset. CIFAR-$100$ dataset contains $100$ classes with $60,000$ $32\times 32$ colored images where $50,000$ images were used for training and $10,000$ images were used for testing. The more challenging ImageNet $2012$ dataset contains $1000$ classes of images of various objects. The dataset contains $1.28$ million training images and $50,000$ validation images. Randomly cropped $224\times 224$ pixel regions were used for the ImageNet dataset. All empirical analysis and optimizations were performed on the CIFAR-$100$ dataset and the resultant conclusions and settings were used for the final ImageNet simulation. All experiments are run in PyTorch framework using two GPUs. For both datasets, the image pixels were normalized to have zero mean and unit variance. Other standard pre-processing techniques used in this work can be found at (Link). It is worth mentioning here that while evaluations in this paper are based on static datasets, SNNs are inherently suited for spatio-temporal datasets generated from event-driven sensors [41, 42] and such sensor-algorithm co-design is currently an active research area [43].

Our network architecture follows a standard VGG-$16$ model. We purposefully chose the VGG architecture since many of the inefficiencies and accuracy degradation effects of BNNs are not reflected in shallower models like AlexNet or already-compact models like ResNet. However, we observed that VGG XNOR-Nets could not be trained successfully with $3$ fully connected layers at the end. Hence, to reduce the training complexity, we considered a modified VGG-$15$ structure with one less linear layer. Note that only top-1 accuracies are reported in the paper.

As mentioned earlier, we used ANN-SNN conversion technique to generate our B-SNN. While ANN-SNN conversion is currently the most scalable technique to train SNNs, it suffers from high inference latency. However, recent work has shown SNNs trained directly through backpropagation are characterized by much lower latency than networks obtained through ANN-SNN conversion, albeit for simpler datasets and shallower networks [44]. Due to the fact that such training schemes are computationally much more exhaustive, a follow-up work has explored a hybrid training approach comprising ANN-SNN conversion followed by backpropagation-through-time fine-tuning to scale the latency reduction effect to deeper networks [45]. However, as we show in this work, the full design space of ANN-SNN conversion has not been fully explored. Prior work on ANN-SNN conversion [15] has mainly considered conversion techniques optimizing accuracy, thereby incurring high latency. In this work, we show that there exists extremely simple control knobs (both at design time and at run time) that can be also used to reduce inference latency drastically in ANN-SNN conversion methods without compromising on the accuracy or involving computationally expensive training/fine-tuning approaches. Since our SNN training optimizations are equally valid for full-precision networks, we report accuracies for full-precision models along with their binary counterparts in order to compare against prior art.

Our ANNs were trained with constraints of no bias and batch-normalization layers in accordance with previous work [15]. A dropout layer was inserted after every ReLU layer (except those followed by a pooling layer) to aid the regularization process in absence of batch-normalization. Our XNOR and B-SNN networks do not binarize the first and last layers as in previous BNN implementations. We apply the pixel intensities directly as input to the spiking networks instead of an artificial Poisson spike train [16]. Once the ANN is trained, it is converted to an iso-architecture SNN by replacing the ReLUs with IF spiking neuron nodes. The SNN weights are normalized by using a randomly sampled subset of images from the training set and recording the maximum ANN activities. Note that normalization based on SNN activities can be used to further reduce the ANN-SNN accuracy gap [15]. The SNN implementation is done using a modified version of the mini-batch processing enabled SNN simulation framework [46] in BindsNET [47], a PyTorch based package.

In order to train the B-SNN, we first trained a constrained-BWN, as mentioned previously. ADAM optimizer is used with an initial learning rate of $5e-4$ and a batch size of 128. Lower learning rates for training binary nets have proven to be also effective in a recent study [48]. The learning rate is subsequently halved every $30$ epochs for a duration of $200$ epochs. The weight decay starts from $5e-4$ and is then set to $0$ after 30 epochs similar to XNOR-Net training implementations [6]. As shown in Fig. 3, we find that the final validation accuracy improvement for the constrained-BWN is minimal over an iso-architecture XNOR-Net. This is primarily due to the constrained nature of models suitable for ANN-SNN conversion coupled with weight binarization.

However, previous work has indicated that careful weight initialization is crucial for training networks without batch-normalization [15]. Drawing inspiration from that observation, we performed a hybrid training approach, where a constrained full-precision model was first trained and then subsequently binarized with respect to the weights. The resultant constrained-BWNs exhibited accuracies close to original full-precision accuracies, as shown in Fig. 3. A similar hybrid training approach was also recently observed to speed up the training process for normal BNNs [49]. Note that the full precision networks are trained for $200$ epochs with a batch size of $256$, an initial learning rate of $5e-2$, weight decay of $1e-4$ and SGD optimizer with a momentum of $0.9$. The learning rate was divided by $10$ at $81$ and $122$ epochs. The trained full-precision models are also used for substantiating the benefits of the SNN optimization control knobs discussed next.

The full-precision VGG-$15$ model is trained on ImageNet dataset for $100$ epochs with a batch size of $128$, a learning rate of $1e-2$, a weight decay of $1e-4$ and the SGD optimizer with a momentum of $0.9$. Note that the learning rate was divided by $10$ at $30$, $60$ and $90$ epochs similar to that of CIFAR-$100$ training. The final top-1 accuracy of the full-precision ANN was $69.05\%$.

Similarly, we binarized the network from the pre-trained ANN using the hybrid methodology described previously and we also observed a drastic increase in B-SNN accuracy (in contrast to training the model from scratch) similar to Fig 3. The initial parameters used for the Adam optimizer were learning rate of $5e-4$, weight decay of $5e-4$ (and $0$ after 30 epochs), and beta values (the decay rates of the exponential moving averages) of ($0.0$, $0.999$). Note that we observed proper setting of the beta values to be crucial for higher accuracy of the B-SNN training, as suggested in a recent work [49]. We achieved $65.4\%$ top-1 accuracy for the constrained BWN model after 40 epochs of training (the binarization phase after full-precision training).

Optimization settings derived from the previous CIFAR-$100$ experiments were applied to our ImageNet analysis, namely, the pooling architecture, neural node type and relaxing the threshold balancing ($99.9\%$ percentile was used by recording maximum ANN activations for a subset of $80$ images from the training set). The top-1 SNN (ANN) accuracy was $66.56\%$ ($69.05\%$) for the full-precision model and $62.71\%$ ($65.4\%$) for the binarized model respectively. The accuracy versus timesteps variation for the two models are depicted in Fig. 12(a)-(b). The binary SNN model achieves near full-precision accuracy on the ImageNet dataset as well with $5.09$ Normalized #OPS count. Note that the latency and #OPS count can be further reduced by early exit. We did not include the early exit optimization in order to achieve a fair comparison with previous works. A summary of our results on the ImageNet dataset and results from other competing approaches are shown in Table II. Apart from the B-SNN proposal, our simple optimization procedures involving standard non-spiking network based training is able to achieve extremely low-latency deep SNNs.