Rethinking the role of normalization and residual blocks for spiking neural networks

SNN consists of spiking neurons that model the behavior of biological neurons and handle the firing timing of the spikes. Owing to the differences in approximations, several spiking neuron models have been proposed, such as the integrate-fire (IF) [16], leaky-integrate-and-fire (LIF) [17], Izhikevich [18], and Hodgkin-Huxley model [19]. In this study, we adopt the spike response model (SRM) [20] to deal with the refractory period (Section 3).

The refractory period is an essential function of biological neurons to suppress the spike firing. Spike firing occurs when the neuron’s membrane potential exceeds the firing threshold. From a biological perspective, the membrane potential is calculated using PSP, representing the electrical signals converted from the chemical signals. These behaviors are represented within the chemical synapse model shown in Figure 1 (a) [21]. SRM was implemented to approximate this synaptic model better than IF/LIF neurons, which are widely used in SNNs.

It is well-known that SNNs are difficult to train due to non-differential spike trains. Researchers are working on this problem, and their solutions can be divided into two approaches: first, the ANN-SNN conversion [22, 24, 25, 23], and second, the usage of the surrogate gradient [7, 8, 6, 5, 9]. The ANN-SNN conversion method uses the trained ANN parameters of SNN. The sophisticated and state-of-the-art ANN model can be reused through this method. However, this conversion approach requires many time-steps during inference and increase the power consumption. In contrast, the surrogate gradient is used to directly train SNNs by approximating the gradient. The surrogate gradient approach was adopted since the model obtained by surrogate gradient requires far fewer inference time-steps than the ANN-SNN conversion model [26].

One of the techniques that have contributed to the success of ANNs is Batch Normalization (BN) [27]. BN is used to reduce the internal covariate shift, leading to a smooth landscape [28] while corresponding to the homeostatic plasticity mechanism of BNNs [29]. Using a mini-batch, BN computes the sample mean and standard deviation (STD). Meanwhile, several variants have been proposed to compute the sample mean and STD, such as Layer Normalization (LN) [30], Instance Normalization (IN) [31], and Group Normalization (GN) [32]. In particular, LN is effective at stabilizing the hidden state dynamics in recurrent neural networks for time-series processing [30].

Several normalization methods have also been proposed in the field of SNNs, such as threshold-dependent BN (tdBN) [33] and BN through time (BNTT) [34]. Thus, tdBN incorporates the firing threshold into BN, whereas BNTT computes BN at each time step. Furthermore, some studies used BN as is [35]. These studies applied the normalization to the membrane potential. In contrast, our method was applied to PSP, as shown in Figure 2 (b), to simplify the normalization form (Section 4).

We adopt SRM as a spiking neuron model [20]. SRM model is based on combining the effects of the incoming spike arriving at the spiking neuron. It also has a function to the spike firing when the membrane potential $u(t)(t=1,2,\cdots,T)$ reaches the firing threshold. Figures 1 (b) and (c) indicate the behavior of this model. The equations are given as follows:

where $w_{i,j}$ is the synaptic weight from the presynaptic neuron $j$ to the postsynaptic neuron $i$. $s_{j}(t)$ is the spike train inputted from the presynaptic neuron $j$, $s_{i}(t)$ is the output spike train of the postsynaptic neuron $i$, $\ast$ is a temporal convolution operator, and $\theta$ is a threshold used to control the spike generation. $f_{s}$ is the Heaviside step function, which fires the spike when the membrane potential $u_{i}(t)$ exceeds the firing threshold $\theta$. In addition, $\varepsilon(\cdot)$ and $\nu(\cdot)$ are the spike response and refractory kernels formulated using the exponential function as follows:

where $\tau_{s}$ and $\tau_{r}$ are the time constants of spike response and refractory kernels, respectively. Note that $\varepsilon\ast s_{j}(t)$ represents the PSP.

By using Equations (1) and (2), the SNNs with multi-layers can be described as follows:

where $\bm{a}^{(l)}(t)\in\mathbb{R}_{\geq 0}^{\mathrm{C}\times\mathrm{W}\times\mathrm{H}}$ and $\bm{s}^{(l)}(t)\in\{0,1\}^{\mathrm{C}\times\mathrm{W}\times\mathrm{H}}$ are the PSP and input spike tensor of time step $t$; $\mathrm{C}$ is the number of channels; and $\mathrm{W}$ and $\mathrm{H}$ are the width and height of the input spike tensor, respectively. Since $\bm{a}^{(l)}(t)$ does not take a value less than zero, we consider an excitatory neuron. Furthermore, $\bm{W}^{(l)}\in\mathbb{R}^{M}$ is the weight matrix representing the synaptic strengths between the spiking neurons in $l$ and $l+1$ layers; $M$ is the number of neurons of $l+1$-th layer.

Deep neural network are essential to recognize complex input patterns. In particular, ResNet is widely used in ANNs [15, 36], and its use in SNNs is expanding.

The ResNet’s networks are divided into the pre-activation and post-activation residual blocks as follows (Figure 3):

where $\bm{h}^{(k)}$ and $\bm{h}^{(k+1)}$ are the input and output in the $k+1$ block, respectively. $G$ represents the residual function, corresponding to “Conv-Func-Conv” and “Func-Conv-Func-Conv” in Figure 3; $F$ represents the Func layer (“Spike-PSP-Norm”). Note that the refractory period is used in $F$. In the experimental section, we compare these blocks and show that deep SNNs can be trained using the pre-activated residual blocks. This result shows that identity mapping is an essential tool to train the deep SNNs, similar to ANNs [36].

We use SLAYER [6] as one of the surrogate gradient algorithms to tarin the SNN with multi-layers. In SLAYER, the derivative of the spike activation function $f_{s}$ of the $l+1$ layer is approximated as follows (Figure 1 (d)):

where $\alpha$ and $\beta$ are hyperparameters, and $\bm{\theta}\in\mathbb{R}^{M}$ is the firing threshold. SLAYER can be used to train SRM as described in [6].

We evaluated PSP-BN and PSP-LN on the spatio-temporal event and static image datasets. We used N-MNIST [12] and F-MNIST [13]. N/F-MNISTs are widely used datasets containing 60K training and 10K test samples with 10 classes. Each size is $34\times 34\times 30000$ events (N-MNIST), and $28\times 28$ pixels (F-MNIST). We partitioned the 60K data using 54K and 6K as our training and validation data, respectively. We also resized the F-MNIST image from $28\times 28$ to $34\times 34$ to achieve higher accuracy.

We evaluated the performance of several spiking convolutional neural network models, such as six-convolutional layers on both datasets and 14-convolutional layers on N/F-MNIST. We also used more deep models, such as ResNet-106 on N-MNIST and F-MNIST, respectively.

We used hyperparameters shown in Table 1 in all experiments and implemented by PyTorch. We used the default initialization of PyTorch and showed the best accuracies of all models. All experiments were conducted using a single Tesla V100 GPU. In addition to this computational resource limitation, we randomly sampled 6K of the training data for both datasets to train in each epoch since SLAYER requires a significant amount of time to train.

We first evaluate the effectiveness of our normalizations. Table 2 presents the accuracies of PSP-BN and PSP-LN and other approaches. Note that we set our normalization before the convolution as described in Section 4, which is different from the position proposed in previous studies [35, 34, 33]. This table illustrates that PSP-BN and PSP-LN achieve high accuracies on both datasets compared to the other approaches.

We also investigate the effect of the proposed method on the firing rate. Figures 6 and 7 show the firing rates of each method. As shown in Figure 6, our normalized models can suppress the firing rate in most layers compared to the unnormalized model. Furthermore, from Figure 7 and Table 2 show that our normalized models can simultaneously achieve high classification accuracy and low firing rate compared to other normalizations. These results verifies the effectiveness of our normalizations.

Some authors pointed out that the tuning of hyperparameter $\tau_{s}$ is essential to achieve higher accuracy [41]. Therefore, to investigate the effect of $\tau_{s}$, we compare the accuracy with respect to changing $\tau_{s}$. Figure 8 shows F-MNIST’s accuracies of each method for $\tau_{s}=\{1,5,10\}$. As shown in this figure, the models with PSP-BN/LN maintain the accuracy whereas others deteriorate the accuracies. This result shows that the model becomes robust to $\tau_{s}$ using our normalizations.

In addition to $\tau_{s}$, we show the influence of $\lambda$. Figure 9 shows the performance when changing $\lambda$. As shown in this figure, the classification performance of PSP-BN becomes worse for a sufficient small value of $\lambda$. This result verifies the plausibility of our discussion in Section 4.

Finally, we evaluate the performance of SNNs using the residual blocks. Table 3 shows the performance of SNNs using the pre-activation and post-activation residual blocks. As shown in this table, the accuracy is substantially improved using the pre-activation residual blocks. This result shows that the post-activation employed in previous studies without refractory period [33, 5, 11] is unsuitable for SNNs with a refractory period. Thus, while ensuring the biological plausibility, due to the refractory period, we can obtain deep SNNs beyond 100 layers using our normalizations and pre-activation residual blocks.