MAP-SNN: Mapping Spike Activities with Multiplicity, Adaptability, and Plasticity into Bio-Plausible Spiking Neural Networks

BP-based SNNs simulate the spike activities by discrete time-iteration. However, the discreteness causes considerable problems. Within recursive time-iterations, $\Delta t$ needs to be determined as the minimum iterative step length for simulation. The mismatch between the continuous and discrete spike activities is illustrated in Figure 1. For distinction, we define BP-based SNNs with binary digits transmission as Single-Spike Pattern (SSP). With SSP, modeling spike activities becomes problematic when iterative step length $\Delta t$ goes larger. Since models with SSP represent spike activities as binary sequences, only one spike activity can be handled per iterative step. Under this circumstance, the temporal feature is restricted with lost spikes. Therefore, in discrete time-iteration, the proper selection of the iterative step lengths is always an indispensable part of SSP. Inspired by spike multiplicity, we propose a Multiple-Spike Pattern (MSP) to ease the problem by complementing spike activities into each iterative step, as shown in Figure 1.

As the compensation for the difference between neurons’ natural behaviors under such discrete iteration, MSP utilizes integers as the intensity of neural spike activities, allowing multiple spikes transmission in an iterative step. The comparison between the SSP and the MSP is shown in Figure 3(a) and Figure 3(b). By replenishing the spike number within iterative steps, the multiple-spike pattern avoids the spike loss problem and implements potentially higher expressiveness of spike activities during the time iteration.

The MSP proposed in this work can be equivalently converted into the SSP by reducing the time scale correspondingly. One example is shown in Figure 3(c), where two patterns result in the same spike activities during a specific time interval. Furthermore, under the equivalence, the multiple-spike pattern implements the same neuron activity with more leisurely $\Delta t$ selection, which allows the model to achieve high stability for arbitrary iterative step lengths.

Spike-frequency adaptation (SFA) is a biological neural phenomenon, describing a neuron fires with a frequency that reduces over time when stimulated with constant input. The phenomenon occurs in both vertebrates and invertebrates, in peripheral and central neurons, and plays an essential role in neural information processing Benda and Herz (2003). The SFA mechanism leads to non-linearity in spike activities and enriches the temporal feature for a single spike. Specifically, Adibi et al. Adibi et al. (2013) suggest that the SFA mechanism in real neurons like whisker sensory cortexes helps improve the information capacity of a single spike defined by the average mutual information (MI). Therefore, this work adopts the SFA mechanism with MSP for higher efficiency in spike transmission.

By unifying the accumulation activity and spike activity, the iterative LIF model with SFA is defined as:

Here $v(t)$ is the neuron’s membrane potential. $u(t)$ is the consumed membrane potential that produces multiple spike activities. $\tilde{I}(t)$ is the normalized pre-synaptic input current. $\tau_{decay}$ is the decay factor describing the leaky activity of spiking neurons.

Here $V_{threshold}$ is the threshold basis of neuron, $n^{*}$ is the estimated intensity of spike activity, $s$ is the integer number of output spike activities, $q$ is the inhibition coefficient that controls the temporary raising of the threshold, making the intensity of spike activity drop exponentially.

As shown in Figure 4, when the membrane potential $v$ increases, the intensity of spike activity gradually deviates from linearity, showing adaptability to the current input. In this case, the total number of spike activities decreases, and each spike activity brings more features, potentially saving computation operations with less spike transmission while maintaining high performance.

In order to apply backpropagation, we assign a particular pseudo derivative as follows:

This pseudo-derivative provides a unit vector for gradient descent without complicated computations.

In biological neural networks with electric synapses, a spike is considered to be generated in a soma, transmitted through the axon to the synapse, and converted as an electric current into the connecting neuron’s dendrites. For describing spike transmissions through electric synapses, Gerstner Gerstner and Kistler (2002) proposes a Spike Response Model (SRM)  to transform spike activities into current signals flowing into post-synaptic dendrites, defined as:

Here $s(t)$ is the spike activities, $o(t)$ is the spike response signal transmitted from axon terminal to dendrite over time, $K(t)$ is the spike response kernel relating current intensities with spike activities.

SRM provides richer temporal information for the network by allowing the varying effect of certain spike activity. However, the constant parameters of response kernel $K(t)$ are widely pre-defined as a ”ground truth” before training, which limits the potential diversity and plasticity for SRM. Shrestha and Orchard Shrestha and Orchard (2018) first considered the plasticity of SRM by setting response delay as learnable parameters, which unsurprisingly improved the performance. As shown in Figure 5, this work further frees up the shape parameters for better plasticity, allowing shape parameters $a$, $b$, and delay parameter $delay$ to be learnable during training. In this case, the plasticity of spike activity allows each neuron to learn different temporal features, improving the complexity and fitting ability of the model.

As shown in Figure 5, SRMs used for spike activities are going up to the peak and subsequently decreasing towards zero. Therefore, it is possible to ignore the long-time spike response to help reduce computational complexity. This work applies one-dimensional convolution operation in computing SRM by defining a valid time window $t_{k}$ to make the SRM more compatible in nowadays deep learning frameworks. The convolution operation follows as:

Here $C_{n}$ is the one-dimensional convolutional kernel of spike responses, modeled with three variables $a$, $b$ and $decay$. $t_{k}$ is the time-window constant that describes the necessary scope of spike response, defined as Eq.8 with the minimal iterative step length $\Delta t$ and convolution kernel size.

In spiking neurons, dendrites receive signals from pre-synaptic axon terminals and integrate all currents together for the neural soma. In this work, we set up a negative filter with $Swish$ function Ramachandran et al. (2018) to shield negative integrated input currents to improve the stability of membrane potential in the LIF model.

The selection of minimal iterative step lengths $\Delta t$ influences model performance in the discrete iterative models. For the sake of completeness of the analysis, we analyze this instability in the ablation experiments by building control experiments under MLP architecture with different iterative step lengths, as shown in Figure 6(a) and Figure 6(b). The experiments are based on N-MNIST and SHD, respectively, where the unified network structure is 34$\times$34-200-10 on N-MNIST and 700-400-10 on SHD. With the additional properties, the error rates of the model have been significantly improved. Compared with benchmark SSP, our MAP-SNN with complementary MSP and ConvSyn reduce error rates by $(1.0\%,2.8\%)$ on N-MNIST, and $(19\%,41\%$) on SHD, which demonstrates the reliability of proposed methods. Furthermore, the model trained with MSP keeps almost constant error rates across different $\Delta t$, supporting that multiplicity alleviates the discretization problem and improves the model stability on time-iteration with arbitrary steps.

To demonstrate the effectiveness of SFA in spike reduction, we establish a set of controlled experiments on the SHD dataset with the 700-400-10 MLP structure. Figure 6(c) shows the error rates and spike numbers in the training process of models in both SFA mode and Linear mode. The experimental results show that SFA effectively suppresses spike activities by $1.48\times$ times while slightly improving model accuracy by $1.52\%$. In this case, the reduced signal transmissions helpfully decrease the amount of computation in synapses, which is significant to save the power consumption of neuromorphic hardware based on spike transmissions.

To highlight the importance of plasticity for feature extraction, we set up a control experiment to compare the trainable ConvSyn with the untrainable SRM. As shown in Figure 6(d), the experiment is set on the SHD with the 700-400-10 MLP structure, showing the changes of model error rate and loss during the training epoch. The experimental results show that the plasticity allows the model to converge faster and reduces the error rate by $(4.2\%,15.6\%)$ during epoch $[10,80]$, demonstrating the advantage of ConvSyn in temporal feature extraction. We conclude that plasticity helps shorten the training process of models and improve the model’s performance.