Context Gating in Spiking Neural Networks: Achieving Lifelong Learning through Integration of Local and Global Plasticity

We model the above human behavior as three-layer SNN, in which the main part of the network is used to receive tree picture stimuli and output results, while the contextual signals are encoded by one-hot labels and then input into the network to affect the network’s output judgment.

The proposed CG-SNN framework utilizes iterative training to extract and maintain context information for various tasks by alternating between global and local plasticity. The local plasticity is applied to enforce the connection between the context inputs and relevant hidden units. Because the learning of context for each task is essential for multiple-task lifelong learning. Meanwhile, the global plasticity is preserved to update global connection weights, in order to maintain the generalization capability of SNN for a certain task. During the whole training process, dynamic iterative training between global and local plasticity is adopted to balance context learning and generalization learning. Assuming the training epochs of local plasticity and global plasticity are $I_{n}$ and $I_{m}$, respectively. We adjust these two parameters by grid search method for single-spike and multi-spike CG-SNN. Through the cross-scale plasticity from local to global, task-related context information could be consolidated by modifying the connection between task neurons and hidden neurons.

Based on different spike-firing methods, we develop CG-SNN based on two different kinds of SNN, multi-spike SNN trained with the surrogate gradient and single-spike SNN with direct backpropagation. Both SNN use stochastic gradient descent (SGD) algorithms to train on blocked or interleaved data. Once a supervised training step is finished, the network will follow a step (some trials more than one) of local plastic learning.

In the network of single-spike CG-SNN, each spike neuron is permitted to fire spike only once. The non-Leaky IF neuron model is utilized for its simplicity and efficiency [25]. The membrane potential of each postsynaptic neuron $j$ is obtained after integrating the contributions of all the weighted presynaptic neurons of $N_{I}$:

$$V_{j}(t)=\sum_{i=1}^{N_{I}}g(t-t_{i})w_{ij}(1-exp(-(t-t_{i}))),$$

(1)

where $g(x)$ denotes the Heaviside function, which equals zero for negative arguments of $x$, otherwise equals to be one for positive ones. $t_{i}$ indicates the concrete firing time of presynaptic neuron $i$. Thus, the presynaptic neuron $i$ would transmit the corresponding postsynaptic potential to neuron $j$ only if it satisfies $t_{i}<t$; otherwise, that potential vanishes. The postsynaptic neuron $j$ emits a spike when its membrane potential crosses the threshold $V_{th}$ which is set to 1. Due to each neuron being permitted to fire only once, the spike firing times $t_{j}$ is transformed to z-domain by $z_{j}=exp(t_{j})$ to simplify implementation. Meanwhile, the casual set $C_{j}=\{i:t_{i}<t_{j}\}$ is defined as the collections of the presynaptic spikes that determine the time point at which the postsynaptic neuron fires the first spike. Since then, the firing times of each neuron can be computed sequentially over the feedforward process through the above formulations.

In the experimental process, we find that directly using the single-spike structure in [25] is easy to make the network ignore the relatively weak signal (that is, the darker pixels) in the source visual stimulus. Therefore, we apply an MLP structure at the first layer, which is used to extract the features of the source image. The hidden layer then encodes the extracted features (the value is limited by implementing a sigmoid function) together with the contextual signal according to their signal strength for forward propagation.

Besides, to strengthen the difference in STDP’s influence on different synapses, we try to encode the stronger contextual signal with the same intensity as the strongest signals on the hidden layer during Hebbian updating. We also adjust the number of weight updates of global plasticity and local plasticity in a round in order to coordinate the strength between them.

Context gates by STDP. The local plasticity of STDP is a reasonably well-established physiological mechanism of activity-drive synaptic regulation based on the local adjacent neurons. And it has been observed extensively in vitro for more than a decade [26], and it is believed to play an important role in neural activity [27].

$$\Delta W_{ij}^{l}=\begin{cases}A_{+}exp(\frac{t_{j}-t_{i}}{\tau^{+}}),\quad if\ t_{j}>t_{i},\\ A_{-}exp(\frac{t_{i}-t_{j}}{\tau^{-}}),\quad if\ t_{j}<t_{i},\\ \end{cases}$$

(2)

where $A_{+},A_{-}$ and $\tau^{+},\tau^{-}$ indicate the magnitudes and time constants, respectively. Thus, during the local plasticity iteration, the $w_{ij}^{cg}$ is updated by $w_{ij}^{cg}=w_{ij}^{cg}+\lambda_{local}*\Delta W_{ij}^{l}*M_{cg}$. $M_{cg}$ filters the context gating weight connection. During the global plasticity iteration, since the modeled behavior is a binary classification task, we use a simplified cross-entropy to calculate the loss and then update the global weight connection $W_{ij}^{g}$. Besides, we multiply the loss value by the reward value as the participants would receive the numerical reward after their decision in cognitive experiments.

$$L_{single}=-r\times\frac{exp(-o_{0})}{exp(-o_{0})+exp(-o_{1})},$$

(3)

where $r$ is the corresponding numerical rewards of the trials, $o_{0}$, $o_{1}$ is the spike latency of last layers.

The global training of the multi-spike CG-SNN model is implemented based on spatiotemporal backpropagation (STBP) algorithm in [28, 29, 30, 31]. The iterative integrate-and-fire (IF) can be employed to express neuronal dynamics both in the spatial domain and temporal domain [32]. According to the integrated presynaptic neuron input and the membrane potential at $t-1$, the membrane potential of postsynaptic neuron $j$ at $t$ can be computed as:

$$u_{i}(t+1,n)=u_{i}(t,n)+x_{i}(t+1,n)+b_{i},$$

(4)

$$x_{i}^{t+1,n}=\sum_{j=1}^{l(n-1)}w_{ij}^{n}o_{j}^{t+1,n-1},$$

(5)

$$o_{i}^{t+1,n}=g(u_{i}^{t+1,n}-V_{th}),$$

(6)

The Heaviside function of $g(x)$ is to guarantee that the neuron emits spike once its membrane potential crosses over the threshold $V_{th}$ (also set to be $1.0$). The leaky IF neuron model also adapts to CG-SNN, where the leaky term with time constant $\tau$ is introduced when computing the above integration term.

Then, the surrogate gradient function of $g_{(}x)$ is approximated based on the derivative of spike activities of $g^{\prime}(x)=\frac{\alpha}{2(1+(\frac{\pi}{2}\alpha x)^{2})}$, to solve the non-differential problem of discrete spikes firing behavior, where $\alpha$ is set to be 2.0. Based on the above formulation and the gradient computing in [28, 33], the update of synaptic weights can be obtained by the gradient descent rules [34, 35].

During experiments, we find that the effect of STDP learning rules in the multi-spike model was very meaningless. We infer that this result might be due to the fact that the positive contextual signal is powerful and is encoded to a relatively high frequency in the multi-spike network. Thus, the local plasticity learned by STDP becomes unstable, which makes STDP rules difficult to assist the network to distinguish functional neurons. Therefore, we don’t apply STDP rule to multi-spike model as the local plasticity rule.

Oja rules. Inspired by [2], we use Oja’s rule [36], which can regularize the weight based on Hebbian learning, to constrain the parameters:

$$w_{ij}^{l}=w_{ij}^{l}+\eta_{hebb}y(x-w_{ij}^{l}y),$$

(7)

where $\eta_{hebb}$ is the learning rate of the Hebbian update. The hyper parameter $\eta_{hebb}$ is obtained through automatic parameter adjustment using Python package of Ray. It is found to be 0.084 when searching from $1e{-4}$ to $1e{-1}$. $x$ is the inputs and $y$ is the linear hidden units output connected to the inputs $x$ via weight matrix $w_{ij}^{l}$.

Sluggish neurons. According to the ordinary cognition and the conclusion from experiments of human volunteers [24], humans can learn tasks better after blocked training than interleaved training, while the artificial neural network is usually on the contrary. Inspired by [2], we apply the “sluggish” neurons to model the human’s learning bias under interleaved training. That is, when learning in interleaved mode, the relation between the context cues and stimuli of the former samples may interfere with the judgment of the current sample. Therefore, sluggish neurons is used to carry information from previous samples over to the current sample, by applying an exponentially moving average to input information of task units. That is, the task-related context signals should updated by $(1-\alpha)x_{t}+\alpha x_{t-1}^{EMA}$ when training all other samples ($t>1$) after the first sample ($t=1$).

Through the above description of single-spike and multi-spike CG-SNN, the context gating in the input dimension could imitate the cognitive control mechanism of PFC. The proposed model has higher task-specific neural representation capability because of the extra temporal dynamics and better biological plausibility than artificial neural networks.

We first explore how STDP may take effect during context-dependent task learning. As illustrated in Fig. 2 (A), as expected, we note that single-spike SNN is also affected by catastrophic forgetting, and the learning performance of the first task almost immediately drops down. In contrast, when applying STDP learning rule, the decline rate of the first task is restrained to a very slow level, and the performance of the second task can still rapidly increase until convergence to perfect performance. The accuracy of the first task finally stays at about $\sim$75%. In addition, the model with multiple STDP updating seems to have stronger resistance to catastrophic forgetting while its results are relatively unstable. We conduct the ANOVA analysis on the slope during the later half epochs to compare multi-stdp and stdp. The p-value is 0.00596 ($<$0.05) and we could conclude that multi-stdp decelerates slower than stdp.

However, although the memory decline decelerates under the effect of STDP, the accuracy curve under the first task becomes even more unstable than the vanilla network. Then we attempt to figure it out by comparing the average absolute values of the weights related to the context signal and other weights in the hidden layer of the three models. As shown in Fig. 2 (C) (left top), under the influence of STDP learning rule, the absolute value of context-related weight becomes very large. Large weights related to context signal enable the $1_{st}$ task’s memory to retain better (these large weights would be more difficult to change much) during training the successive $2_{nd}$ task. Besides, after training under the blocked data, the time difference of hidden layer neurons’ spike between two tasks is very close (mostly lower than 0.05 ms; the largest is only $\sim$ 0.3 ms) and evenly distributed, which makes the choice matrix of the first task almost replaced by that of the second one. In contrast, the hidden layer neurons of the model using STDP rules always fire spike under the first task earlier than under the second task, which makes the network distinguish the two independent tasks.

Based on these phenomena, we infer that the relatively large weight magnitudes caused by STDP learning rules enable the network to structure some cognition of the past tasks to a certain extent so that it can make correct choices toward relatively strong stimuli after being trained under the second task. This can also be reflected in Fig. 2 (B) (down), in which the choice matrix is the network’s choice (accept reward) made during variety as a function of the two feature dimensions (only a single dimension among branch and leaf is relevant to each task). And the ”rel” and ”irrel” indicate whether one of these two dimension is related to current task. We can see that the choice matrix of the first task smoothly changes along the areas where the choices are consistent under the two tasks.

Though having a certain effect, the stability and the final performance of the model using STDP learning rules in context-dependent tasks are still not satisfactory. The neural dynamics of the brain are constructed by a large group of neural plasticity mechanisms [37, 38, 39]. The exploration and discovery of single-spike CG-SNN might provide some ideas for related research fields. Based on the problems in the single-spike model, we further explore multi-spike CG-SNN, which can implement a more stable and better model of human cognitive control behavior, and has better biological plausibility.

To validate the effectiveness of multi-spike CG-SNN, we compare it to other lifelong learning methods under the same setting, such as the regularization-based lifelong learning method of the orthogonal weights modification (OWM) learning algorithm in [23]. As shown in Fig. 3, in contrast to the vanilla multi-spike model, the lifelong learning models with the OWM algorithm and our multi-spike CG-SNN perform better on both two tasks, because these models retain context information learning to a certain extent. Besides, the performance of the proposed CG-SNN (90.44$\pm$ 3.9% for task 1, 100% for task 2) is much better than the XdG (81.69$\pm$ 6.28% for task 1, 99.94 $\pm$ 0.17% for task 2), while the XdG model applies more strict constraints on the network connections. It may reveal that excessive human constraint sometimes impacts the network’s ability to retain memory. In addition, CG-SNN also adapts to SNN with LIF neuron models. The CG-SNN with LIF ($\tau=10.0ms$) achieves 88.69% $\pm$ 5.19% and 100% for task1 and task2, which is also higher than other models.

Moreover, comparing the accuracy curve of the OWM learning algorithm, our multi-spike CG-SNN method can retain the learned tasks to a greater extent under the same circumstances. Especially, the accuracy curve of our method has almost stopped falling, while the curve of the OWM algorithm still shows a downward trend in the last several trials. And on the bottom figure in Fig. 3 , we can see the detail of the choices made in two dimensions. The vanilla network trained under blocked training almost ignores the task signal, and the OWM model and our CG-SNN model can keep the choices learned from the first task. Meanwhile, the network trained with our method can keep more details near the category boundary of the first tasks.

We evaluate the effectiveness of our model by comparing with the behavior of the human participants as in [24]. And to access the rationality and the biological plausibility of our method, we compare the validation performance and neural dynamics after blocked or interleaved training among the experimental data, the vanilla spiking network, the model proposed in [2], and our method.

Firstly, as shown in Fig. 4 (A), we can see that participants trained under blocked data perform better than those trained under interleaved data while the vanilla network shows the opposite phenomenon. The model of [2] and our model both appear the similarity to human behavior. And participants’ behaviors show the congruency effect during the training period, which means that they perform better on the congruent trail between two tasks than the incongruent trials. This effect is stronger when training in the interleaved data, as shown in Fig. 4 (A) (bottom). We can see that the model of [2] and our model both reveal such a congruency effect while the vanilla network can’t reproduce such an effect because of the catastrophic forgetting.

Further, we fit the psychometric functions (sigmoid) to compare the choice made by our model and human participants, as shown in Fig. 4 (B) top. Sigmoid fitting is to fit the sigmoid function to the choice made by humans and CG-SNN models, seperately for the relevant and irrelevant feature dimensions. We find that our model and the reference model have relatively steeper slopes for the irrelevant dimension under interleaved training than blocked training as shown in human behavior data. But the vanilla network obtains steeper slopes for the irrelevant dimension under blocked training because the catastrophic forgetting makes the SGD network more easily affected by the irrelevant feature. A similar result also can be seen by fitting the factorised model and the linear model to the choice of human and three networks, as shown in Fig. 4 (B) bottom. Choice fraction is obtained by performing the model-based representation similarity analysis on the networks outputs and behavior. Human participants learned the clear boundary of two tasks/dimensions under blocked data and learned a relatively vague boundary under interleaved data. The reference network and the network with our method can model such characteristics in human behavior while the performance of the vanilla neural network is almost the opposite. Therefore, our framework can effectively model human cognitive behavior mode and achieve the similar effect to the framework proposed by [2].

In addition to modeling human behavior patterns, we also compared the neuron selectivity dynamics in our framework with the ANN model in [2], to prove the biological plausibility of our framework. As shown in Fig. 5 (left), the ANN model in [2] shows poor task-selectivity in their network when dealing with trees pictures. When under blocked training, their network only has a very small proportion of task-selectivity units in the hidden layer ($\sim$10%) while in our network a considerable proportion of units is selective to a specific task ($\sim$25%). The number of the neuron units which are only active to the first task even drops to $\sim 1\%$. Besides, we plot the average neural activity of task-selective neurons in Fig. 5 (right). In our network, the task-selective neurons have a higher spiking frequency than the task-agnostic neurons when facing the corresponding task. The reference network exhibits poor performance on the $1_{st}$ task-selective neuron’s activity. Since neuron’s selectivity reflects the task-specific representation that encodes the relevant dimension of each task under block learning, the better selectivity of CG-SNN implies more powerful resistance to catastrophic forgetting.

We conduct experiments to evaluate the effectiveness of the sluggish neurons. In Fig. 6, when $\alpha=0$ it means the network is the same as the vanilla network. Combining the results of Fig. 6 (top) and 6 (left bottom), we can conclude that as the level of sluggishness increases the performance of the network decreases. At the level of neural dynamic, the proportion of the task-selective hidden layer units, which are active only under specific contextual cues, is also reduced. Meanwhile, the choice matrix of the network under interleaved training becomes more and more like the linear model in Fig. 6 (bottom) as the sluggishness values become larger. On the other hand, the same statistical data of the network trained in blocked data with local plasticity hardly be affected by sluggishness. Therefore, the “sluggish” neurons are reasonable assumptions. Meanwhile, though the best accuracy obtained by the trained network under interleaved data (without sluggish) is relatively high, the largest proportion is still lower than that of the trained network under blocked data.