Spiking-LEAF: A Learnable Auditory front-end for
Spiking Neural Networks

In Spiking-LEAF, the feature extraction is performed with a 1d-convolution Gabor filter bank along with the PCEN that is tailored for dynamic range compression [23]. The Gabor 1d-convolution filters have been widely used in speech processing [24, 16], and its formulation can be expressed as per:

$$\begin{split}\phi_{n}(t)=e^{i2\pi\eta_{n}t}&\frac{1}{\sqrt{2\pi}\sigma_{n}}e^{-\frac{t^{2}}{2\sigma_{n}^{2}}}\end{split}$$

(1)

where $\eta_{n}$ and $\sigma_{n}$ denote learnable parameters that characterize the center frequency and bandwidth of filter n, respectively. In particular, for input audio with a sampling rate of 16 kHz, there are a total of 40 convolution filters, with a window length of 25ms ranging over $t=-L/2,...,L/2$ ($L=401$ samples), have been employed in Spiking-LEAF. These 1d-convolution filters are applied directly to the audio waveform $x$ to get the time-frequency representation $F$.

Following the neurophysiological process in the peripheral auditory system, the PCEN [16, 23] has been applied subsequently to further compress the dynamic range of the obtained acoustic features:

	$\displaystyle PCEN(F(t,n))=\left(\frac{F(t,n)}{(\varepsilon+M(t,n))^{\alpha_{n}}+\delta_{n}}\right)^{r_{n}}-\delta_{n}^{r_{n}}$		(2)
	$\displaystyle M(t,n)=(1-s)M(t-1,n)+sF(t,n)$		(3)

In Eqs. 2 and 3, $F(t,n)$ represents the time-frequency representation for channel $n$ at time step $t$. $r_{n}$ and $\alpha_{n}$ are coefficients that control the compression rate. The term $M(t,n)$ is the moving average of the time-frequency feature with a smoothing rate of $s$. Meanwhile, $\varepsilon$ and $\delta_{n}$ stands for a positive offset introduced specifically to prevent the occurrence of imaginary numbers in PCEN.

The Leaky Integrate-and-Fire (LIF) neuron model [25], with a single neuronal compartment, has been widely used in brain simulation and neuromorphic computing [3, 4, 5, 7, 8]. The internal operations of a LIF neuron, as illustrated in Fig. 2 (a), can be expressed by the following discrete-time formulation:

		$\displaystyle I[t]=\Sigma_{i}w_{i}S[t-1]+b$		(4)
		$\displaystyle U[t]=\beta*U[t-1]+I[t]-V_{th}S[t-1]$		(5)
		$\displaystyle S[t]=\mathbb{H}(U[t]-V_{th})$		(6)

where $S[t-1]$ represents the input spike at time step $t$. $I[t]$ and $U[t]$ denote the transduced synaptic current and membrane potential, respectively. $\beta$ is the membrane decaying constant that governs the information decaying rate within the LIF neuron. As the Heaviside step function indicated in Eq. 6, once the membrane potential exceeds the firing threshold $V_{th}$, an output spike will be emitted.

Despite its ubiquity and simplicity, the LIF model possesses inherent limitations when it comes to long-term information storage. These limitations arise from two main factors: the exponential leakage of its membrane potential and the resetting mechanism. These factors significantly affect the model’s efficacy in sequential modeling. Motivated by the intricate structure of biological neurons, recent work has developed a two-compartment spiking neuron model, called TC-LIF, to address the limitations of the LIF neuron [26]. The neuronal dynamics of TC-LIF neurons are given as follows:

	$\displaystyle I[t]$	$\displaystyle=\Sigma_{i}w_{i}S[t-1]+b$		(7)
	$\displaystyle\begin{split}U_{d}[t]&=U_{d}[t-1]+\beta_{d}*U_{s}[t-1]+I[t]\\ &\quad-\gamma*S[t-1]\end{split}$			(8)
	$\displaystyle U_{s}[t]$	$\displaystyle=U_{s}[t-1]+\beta_{s}*U_{d}[t-1]-V_{th}S[t-1]$		(9)
	$\displaystyle S[t]$	$\displaystyle=\mathbb{H}(U[t]-V_{th})$		(10)

where $U_{d}[t]$ and $U_{s}[t]$ represent the membrane potential of the dendritic and somatic compartments. The $\beta_{d}$ and $\beta_{s}$ are two learnable parameters that govern the interaction between dendritic and somatic compartments. Facilitated by the synergistic interaction between these two neuronal compartments, TC-LIF can retain both short-term and long-term information which is crucial for effective speech processing [26].

Neuroscience studies reveal that lateral feedback connections are pervasive in the peripheral auditory system, and they play an essential role in adjusting frequency sensitivity of auditory neurons [27]. Inspired by this finding, as depicted in Figure 2 (b), we further incorporate lateral feedback components into the dendritic compartment and somatic compartment of the TC-LIF neuron, represented by $I_{f}[t]$ and $I_{LI}[t]$ respectively. Specifically, each output spike will modulate the neighboring frequency bands with learnable weight matrices $ZeroDiag(W_{f})$ and $ZeroDiag(W_{LI})$, whose diagonal entries are all zeros.

The lateral inhibition feedback of hair cells within the cochlea is found to detect sounds below the thermal noise level and in the presence of noise or masking sounds [28, 29]. Motivated by this finding, we further constrain the weight matrix $W_{LI}\geq 0$ to enforce lateral inhibitory feedback at the somatic compartment, which is responsible for spike generation. This will suppress the activity of neighboring neurons after the spike generation, amplifying the signal of the most activated neuron while suppressing other neurons. This results in a sparse yet informative spike representation of input signals. The neuronal dynamics of the resulting IHC-LIF model can be described as follows:

	$\displaystyle I_{s}[t]$	$\displaystyle=\Sigma_{i}w_{i}S[t-1]+b$		(11)
	$\displaystyle I_{f}[t]$	$\displaystyle=ZeroDiag(W_{f})*S[t-1]$		(12)
	$\displaystyle I_{LI}[t]$	$\displaystyle=ZeroDiag(W_{LI})*S[t-1]$		(13)
	$\displaystyle\begin{split}U_{d}[t]&=U_{d}[t-1]+\beta_{d}*U_{s}[t-1]+I_{s}[t]\\ &\quad-\gamma*S[t-1]+I_{f}[t]\end{split}$			(14)
	$\displaystyle\begin{split}U_{s}[t]&=U_{s}[t-1]+\beta_{s}*U_{d}[t-1]-V_{th}S[t-1])\\ &\quad-I_{LI}[t]\end{split}$			(15)
	$\displaystyle S[t]$	$\displaystyle=\mathbb{H}(U_{s}[t]-V_{th})$		(16)

To further enhance the encoding efficiency, we incorporate a spike rate regularization term $L_{SR}$ into the loss function $L$. It has been applied alongside the classification loss $L_{cls}:L=L_{cls}+\lambda L_{SR}$ where $L_{SR}=ReLU(R-SR)$. Here, $R$ represents the average spike rate per neuron per timestep and $SR$ denotes the expected spike rate. Any spike rate higher than $SR$ will incur a penalty, and $\lambda$ is the penalty coefficient.

Table 1 compares our proposed Spiking-LEAF model with other existing auditory front-ends on both KWS and SI tasks. Our results reveal that the Spiking-LEAF consistently outperforms the SOTA spike encoding methods as well as the fbank features [3], demonstrating a superior feature representation power. In the following section, we validate the effectiveness of key components of Spiking-LEAF: learnable acoustic feature extraction, two-compartment LIF (TC-LIF) neuron model, lateral feedback $I_{f}$, lateral inhibition $I_{LI}$, and firing rate regulation loss $L_{SR}$.

Learnable filter bank and two-compartment neuron. As illustrated in row 1 and row 2 of Table 2, the proposed learnable filter bank achieves substantial enhancement in feature representation when compared to the widely adopted Fbank feature. Notably, further improvements in classification accuracy are observed (see row 3) when replacing LIF neurons with TC-LIF neurons that offer richer neuronal dynamics. However, it is important to acknowledge that this improvement comes at the expense of an elevated firing rate, which has a detrimental effect on the encoding efficiency.

Lateral feedback. Row 4 and row 5 of Table 2 highlight the potential of lateral feedback mechanisms in enhancing classification accuracy, which can be explained by the enhanced frequency sensitivity facilitated by the lateral feedback. Furthermore, the incorporation of lateral feedback is also anticipated to enhance the neuron’s robustness in noisy environments. To substantiate this claim, our model is trained on clean samples and subsequently tested on noisy test samples contaminated with noise from the NOISEX-92 [33] and CHiME-3 [34] datasets. Fig. 3 illustrates the results of this evaluation, demonstrating that both the learnable filter bank and lateral feedback mechanisms contribute to enhanced noise robustness. This observation aligns with prior studies that have elucidated the role of the PCEN in fostering noise robustness [16]. Simultaneously, Fig. 4 showcases how the lateral feedback aids in filtering out unwanted spikes.

Lateral inhibition and spike rate regularization loss. As seen in Fig. 4 (b), when the spike regularization loss and lateral inhibition are not applied, the output spike representation involves a substantial amount of noise during non-speech periods. Introducing lateral inhibition or spike regularization loss alone can not suppress the noise that appeared during such periods (Figs. (b) and (c)). Particularly, introducing the spike regularization loss alone results in a uniform reduction in the output spikes (Fig. 4 (d)). However, this comes along with a notable reduction in accuracy as highlighted in Table 2 row 6. Notably, the combination of lateral inhibition and spike rate regularization (Fig. 4 (e)) can effectively suppress the unwanted spike during non-speech periods, yielding a sparse and yet informative spike representation.