Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks

The main idea of the conversion method is to establish a relation between the firing rate of SNN and the activation value of ANN. With this relation, we can map the well-trained weights of ANNs to SNNs, so high-performance SNNs can be obtained directly, without the need for additional training. (Rueckauer et al. 2017) reviews the state-of-art and proposes a complete mathematical theory of how to make an approximation of the firing rate of spiking neuron to the equivalent analog activation value. Their derivation had some over-simplifications, which can lead to errors in the conversion process. In this work, we rectified those inaccuracies and made the mathematical results more robust. The corrections introduced in the present work will be important only if the chosen threshold of the spiking neurons is not one.

The spiking neuron we use is the IF neuron, which is one of the simplest spiking neuron models. The IF neuron simply integrates its input until the membrane potential exceeds the voltage threshold and a spike is generated. IF neuron does not have a decay mechanism, and we assume that there is no refractory period, which is more similar to the artificial neuron.

For a SNN with L layers, let $W_{l},b_{l},l\in\{1,...,L\}$ denote the weight and bias of layer $l$. The input current of layer $l$ at time $t$, $z_{l}(t)$ can be computed as:

$$z_{l}(t)=W_{l}\theta_{l-1}(t)+b_{l}$$

(1)

where $\theta_{l-1}(t)$ is a matrix denoting whether the neurons in the layer $l-1$ generate spikes at time $t$:

$$\theta_{l}(t)=\theta(V_{l}(t-1)+z_{l}(t)-V_{thr}),\textrm{with}\ \theta(x)=\begin{cases}1\ x\geq 0\\ 0\ x<0\end{cases}$$

(2)

where $V_{l}(t-1)$ denotes the membrane potential of the spiking neurons in layer l at time $t-1$. $V_{thr}$ is the threshold of the spiking neuron.

Here, our equation (1) is a little different from the one in (Rueckauer et al. 2017), whose equation is $z_{l}(t)=V_{thr}(W_{l}\theta_{l-1}(t)+b_{l})$. If all the input currents are multiplied by the threshold, it is equal to setting the threshold to one. Then it results that whatever threshold we set, the input currents will change the corresponding multiple, which counters the effect of the threshold.

Normally, the typical IF neuron will reset its membrane potential to zero after it generates a spike. However, the message of the voltage surpassing the threshold is missing. (Rueckauer et al. 2017) has proofed that it will introduce a new error and proposes a solution to eliminate it. When the membrane potential of a neuron exceeds the threshold, instead of resetting it to zero, subtracts the voltage of the threshold from the membrane potential. In this work, we also adopt this subtracting mechanism for all the IF neurons:

$$V_{l}(t)=V_{l}(t-1)+z_{l}(t)-V_{thr}\theta_{l}(t)$$

(3)

Equations (1) and (3) illustrate the main mechanism of IF neuron. Based on these two equations, we will attempt to establish the relationship between the firing rate of SNNs and the activation values of ANNs.

When $l>0$, by accumulating equation (3) over the simulation time $t$, we can conclude:

$$\sum_{t^{\prime}=1}^{t}V_{l}(t^{\prime})=\sum_{t^{\prime}=1}^{t}V_{l}(t^{\prime}-1)+\sum_{t^{\prime}=1}^{t}z_{l}(t^{\prime})-\sum_{t^{\prime}=1}^{t}V_{thr}\theta_{l}(t^{\prime})$$

(4)

Inserting equation (1) into equation (4) and dividing the whole equation by $t$, we can conclude:

$$\frac{V_{l}(t)-V_{l}(0)}{t}=\frac{\sum_{t^{\prime}=1}^{t}W_{l}\theta_{l-1}(t^{\prime})-V_{thr}\theta_{l}(t^{\prime})+b_{l}}{t}$$

(5)

where $V_{l}(0)=0$, $\frac{\sum_{t^{\prime}=1}^{t}\theta_{l}(t^{\prime})}{t}=r_{l}(t)$, which is the firing rate in layer $l$ at time $t$. To be simple, time step size is set to 1 unit, so $\frac{\sum_{t^{\prime}=1}^{t}1}{t}=1$. Equation(5) can be simplified as:

$$\frac{V_{l}(t)}{t}=W_{l}r_{l-1}-V_{thr}r_{l}(t)+b_{l}$$

(6)

$$r_{l}(t)=\frac{W_{l}r_{l-1}(t)+b_{l}-\frac{V_{l}(t)}{t}}{V_{thr}}$$

(7)

Let $\frac{V_{l}(t)}{t}$ be $\nabla V_{l}$. This yields:

$$r_{l}(t)=\frac{W_{l}r_{l-1}(t)+b_{l}-\nabla V_{l}}{V_{thr}}$$

(8)

This equation is the core equation, which describes the relation of firing rate between adjacent layers. Similar with ANNs, the the firing rate in layer $l$ is mainly given by the sum of wights multiplied by the firing rate in the previous layer $l-1$ and bias. But the firing rate should also minus the approximation error caused by missing the information carried by the leftover membrane potential at the end of the simulation time. In addition, it is obvious that the firing rate is inversely proportional to the threshold.

When $l=0$, the layer $0$ is the input layer. In order to relate the firing rate of SNN and the activation values of ANN, we make them have the same input, which means $z_{0}=a_{0}$, where $a_{0}$ is the input value of ANN. Note that $a_{0}$ is scaled to $[0,1]$ to have the same range as $z_{0}$, which can be done before or after training the ANN. It yields:

$$V_{0}(t)-V_{0}(0)=\sum_{t^{\prime}=1}^{t}z_{0}(t^{\prime})-\sum_{t^{\prime}=1}^{t}V_{thr}\theta_{0}(t^{\prime})$$

(9)

$$V_{0}(t)=ta_{0}-V_{thr}tr_{0}(t)$$

(10)

$$r_{0}(t)=\frac{a_{0}-\nabla V_{0}}{V_{thr}}$$

(11)

By repeating inserting expression (8) to the previous layer, starting with the first layer equation (11), we conclude:

	$\displaystyle r_{l}(t)=$	$\displaystyle\frac{\prod_{i=1}^{l}W_{i}a_{0}}{V_{thr^{l+1}}}+\frac{b_{l}}{V_{thr}}+\frac{W_{l}b_{l-1}}{V_{thr}^{2}}+...+\frac{\prod_{i=2}^{l}W_{i}b_{1}}{V_{thr}^{l}}$		(12)
	$\displaystyle-$	$\displaystyle\frac{\nabla V_{l}}{V_{thr}}-\frac{W_{l}\nabla V_{l-1}}{V_{thr}^{2}}-...-\frac{\prod_{i=1}^{l}W_{i}\nabla V_{0}}{V_{thr}^{l+1}}$

However, the final equation proposed by (Rueckauer et al. 2017) is:

$$r_{l}(t)=a_{l}-\nabla V_{l}-W_{l}\nabla V_{l-1}-...-\prod_{i=1}^{l}W_{i}\nabla V_{0}$$

(13)

where $a_{l}$ is the activation value of ANN in the layer $l$.
As has been mentioned, in that work the threshold is multiplied unnecessarily. So if we set $V_{thr}$ set to 1 in equation (12), which is also what we do in the experiments, equations (12) and (13) will be very similar. This yields:

	$\displaystyle r_{l}(t)=$	$\displaystyle\prod_{i=1}^{l}W_{i}a_{0}+b_{l}+W_{l}b_{l-1}+...+\prod_{i=2}^{l}W_{i}b_{1}$		(14)
	$\displaystyle-$	$\displaystyle\nabla V_{l}-W_{l}\nabla V_{l-1}-...-\prod_{i=1}^{l}W_{i}\nabla V_{0}$

Though both equations have the same error part, the weight and bias part are different. (Rueckauer et al. 2017) overlooks the activation functions:

	$\displaystyle a_{l}=$	$\displaystyle W_{l}(ReLu(W_{l-1}(...ReLu(W_{1}a_{0}+b_{1}))+b_{l-1}))+b_{l}$		(15)
	$\displaystyle\not=$	$\displaystyle W_{l}(W_{l-1}(...W_{1}a_{0}+b_{1})+b_{l-1})+b_{l}$
	$\displaystyle\not=$	$\displaystyle\prod_{i=1}^{l}W_{i}a_{0}+b_{l}+W_{l}b_{l-1}+...+\prod_{i=2}^{l}W_{i}b_{1}$

So actually, there is no direct relationship between the firing rate of SNN and the activation value of ANN. The firing rate cannot be simply presented as the sum of activation value of ANN and some error terms directly. But the firing rate is definitely an estimation of the activation value of ANN, though without activation functions. Let’s back to equation (8). Though it doesn’t apply ReLU explicitly, it does contain the ReLU mechanism inside. Because a spiking neuron can only generate one spike at one time step, $0\leq\sum_{t}\theta_{l}(t)\leq t$. And $r_{l}(t)=\sum_{t}\frac{\theta_{l}(t)}{t}$. The range of firing rate is $[0,1]$. It guarantees that all the inputs to the next layer are greater or equal to 0, which has the same effect with ReLU.

The error term of equation (8), $\nabla V_{l}$, which is the membrane potential remaining in the neurons at the end of the simulation time, is dependent on the total input current of the previous layer, $W_{l}r_{l-1}(t)+b_{l}$. And it is worth noting that the membrane potential can be negative if the spiking neurons receive negative currents. So, there are four cases for equation (8). To be simple, let $R=\frac{W_{l}r_{l-1}(t)+b_{l}}{V_{thr}},\nabla V=\frac{\nabla V_{l}}{V_{thr}}$.

• •

  $R>0,\nabla V>0$. This is one of the most common cases. The spiking neuron receives positive currents and remains positive membrane potential, which is less than the threshold. $\nabla V=\frac{V_{l}(t)}{tV_{thr}}\in[0,\frac{1}{t})\Rightarrow R-\frac{1}{t}<r_{l}(t)\leq R$

• •

  $R<0,\nabla V>0$. This is the only impossible case. If the total current is negative, the leftover membrane potential cannot be positive.

• •

  $R>0,\nabla V<0$. Though the total current is positive, before the end of the simulation time after the final spike, the spiking neuron starts to receive negative currents, which makes the final membrane potential negative. It is just like that the neuron makes an overdraft of spike. Different from ANNs, the firing rate can be different from the same firing rate as input due to the various sequence of input spikes. In this case, $r_{l}(t)>R$.

• •

  $R<0,\nabla V<0$. This case is also very common. One of the most common examples is that the spiking neuron continues to receive negative currents and generates no spikes during the simulation time. So the total input to the next layer is 0. This is just like the negative activation values of ANNs are reset to 0 by ReLU. However, due to the sequence of input current, spikes can be generated during the simulation time though the total current is negative. For example, at first, the spiking neuron continues to receive positive currents and generate a spike. Then it starts to receive negative currents until the end of the time. And the total input current is negative. So, in this case, the firing rate is unpredictable, $r_{l}(t)\in[0,1)$.

According to the discussion above, the error during the conversion process is caused by missing the part of the information carried by the leftover membrane potential at the end of the simulation time. Contrast with the conclusion in (Rueckauer et al. 2017) that the firing rate of a neuron is slightly lower than the corresponding activation value due to the reduction of the error caused by the membrane potential, actually, the firing rate doesn’t have certain size relation with the corresponding activation value. Increasing the simulation time can alleviate the error. And we observe that the firing rate in the last layer is usually lower than the corresponding activation values. This can be partly explained that because $R>0,\nabla V>0$ is the most common case, the general trend of the firing rate is decreasing. The error accumulated layer by layer and the final firing rate is therefore lower than the activation value.

In this section, We introduce a set of methods to alleviate the error during the conversion process.

Figure 1(a) shows the performance of SNNs with different simulation time on four games. If the simulation time is set to 100, the same frame will be fed into SNN as input for 100 times. The percentiles we choose are the ones with the best performance. From the figure, we can conclude that as the simulation time increases, the conversion rates of all the four games also increase, which is consistent with the conclusion in the Methods section that the error caused by the leftover membrane potential can be alleviated by the longer simulation time, but the effect of diminishing. It is a trade-off between the low error and the power-efficiency.

For the games that not very sensitive to the simulation time, like Breakout and James Bond, even with very short simulation time like 100 time steps, the SNN agents can still have good performance with low conversion rates. These SNN models are more robust. The error always influences the size relationships among the actions with close q-values. So the wrong actions chosen by the SNN agent can still be the sub-optimal action. The agent can still have good performance with the low conversion rate. For the games like Star Gunner and Video Pinball, these games are more sensitive to the simulation time, longer simulation time results in a larger conversion rate and then results in better performance.

Figure 1(b) shows the performance of SNNs with different percentile to do the parameter normalization on four games. The percentile is a very sensitive hyperparameter. Numerical studies indicate that we are unlikely to get good performance when the percentile is less than 99. Thus, we tested the effect of the percentile starting from 99. For all four games, the percentile greatly influences the conversion rate and the conversion rate affects the final score. For games like Breakout and James Bond, the conversion rate continuously improves as the percentile increases. Setting the percentile close to 100 can have good performance. For the other games like Star Gunner and Video Pinball, there is a maximum point of performance as the percentile increases. In these two games, percentile 99.9 gives the maximum value. Once exceeding this point, the performance of these games will drop dramatically. This result shows that tuning the percentile value can greatly improve the performance of the agent on these games.

Table 1 shows the performance of our method on multiple Atari games. Due to the limitation of resources, we choose 17 games to test the method. These are the games on which DQNs have very high performances. The total simulation time is 500 time steps. We choose the best percentile in [99.9, 99.99]. Every game is played 50 times. The average scores of DQNs and converted SNNs are shown in the table. The P-Values of DQNs and SNNs obtained by T-Test are also shown. We conclude that the scores of all SNNs are close to the scores of DQNs. And almost all the games, except Kull and Star Gunner, do not have a significant difference between DQN and SNN. This shows that the SNNs converted by our method have comparable performance to the original DQNs.