sVAD: A Robust, Low-Power, and Light-Weight
Voice Activity Detection with Spiking Neural Networks

Our study leverages the well-established auditory encoder, SincNet [15], known for its exceptional feature extraction capabilities. In contrast to conventional Convolutional Neural Networks (CNNs) processing raw speech data, SincNet introduces constraints on filter shapes in its first layer, making it highly suitable for our VAD model. It uses parametrized band-pass filters with only two parameters, ensuring interpretability and human intuition in the feature extraction process. By integrating SincNet into our auditory encoder, we enhance feature extraction, improving the performance of our SNN-based VAD model across various noise conditions while maintaining a light-weight configuration.

In general, 1D convolutional layers introduce trainable parameters enhancing encoder adaptability to data characteristics and improving performance through training. Inspired by ConvTasNet [17], we extend this data-driven ability by adding an SNN-based 1D convolutional layer (sConv1D) to SincNet-processed features. This empowers the encoder to optimize auditory feature extraction during training. Additionally, sConv1D outputs spikes, facilitating the transformation of raw audio into spike representation for subsequent SNN-based processing.

To enhance our VAD model robustness, especially in low SNR conditions, we introduce an attention mechanism inspired by the human auditory system’s masking effect. In the encoder module (shown in Fig. 1), we implement this attention mechanism as follows: Initially, we subject the extracted features $Y$ to three layers of SNN-based fully connected (sFC) layers to derive an attention mask $M$. Subsequently, we leverage this attention mask to modulate the extracted features, generating attended features denoted as $\hat{S}$:

$$\hat{S}=Y\odot M$$

(4)

where $\odot$ is the element-wise multiplication. The trainable three-layer sFC block optimizes the attention mask for different acoustic conditions.

The classifier utilizes the encoded features as its input and is trained to perform frame-by-frame decision-making during runtime. To enhance the sequential modeling capacity, we establish the classifier using SNN with recurrent connections, referred to as sRNN as shown in Fig. 1. Unlike the feedforward processing update formula (Eq. (2)), the update equation for recurrent neurons includes an additional term for recurrent connections:

$$I_{i}^{l}[t]=\sum_{j}w_{ij}^{l-1}O_{j}^{l-1}[t-1]+\sum_{i}w_{ii}^{l}O_{i}^{l}[t-1]+b_{i}^{l},$$

(5)

where $w_{ii}^{l}$ denotes the recurrent connection weight of neuron $i$ in layer $l$.

The proposed VAD model’s loss function comprises two components: a classification loss and an attention mask loss. For the classification loss, we employ cross-entropy (CE):

$$\mathcal{L}_{\text{CE}}=-\sum^{2}_{c=1}x_{c}\log(p_{c}).$$

(6)

Here, $[x_{1},x_{2}]$ denotes an one-hot encoding ($[0,1]$ for speech and $[1,0]$ for non-speech), and $p_{c}$ is the Softmax probability that the input belongs to class $c$. To calculate the attention mask loss, we utilize the mean squared error (MSE):

$$\mathcal{L}_{\text{MSE}}=\frac{1}{N}\sum^{N}_{k=1}(\hat{S}_{k}-S_{k})^{2},$$

(7)

where $S$ is the clean speech features, and $\hat{S}$ is the modulated features described in Eq. (4). $N$ denotes the total number of elements in the features and $k$ represents the index of elements. Consequently, the overall loss of our proposed sVAD model is defined as:

$$\mathcal{L}=\mathcal{L}_{\text{CE}}+\lambda\mathcal{L}_{\text{MSE}},$$

(8)

where $\lambda$ is the hyperparameter for balancing both losses.

The stateful nature of spiking neurons, similar to vanilla RNNs, enables SNNs training by unfolding the network across all time steps and employing the Backpropagation Through Time (BPTT) algorithm [18]. However, the non-differentiable spiking activation function, as outlined in Eq. 3, poses a challenge for direct BPTT application. This study addresses it by adopting the surrogate gradient approach [19], introducing a boxcar function as an effective surrogate gradient:

$$\Theta^{\prime}(U_{i}^{l}[t]-\vartheta)\approx\theta^{\prime}(U_{i}^{l}[t]-\vartheta)=\frac{1}{a}sign\left(|U_{i}^{l}[t]-\vartheta|<\frac{a}{2}\right),$$

(9)

where the hyperparameter $a$ controls the permissible range of membrane potentials that allow gradients to pass through.

We evaluate our sVAD model by comparing it with ten baseline systems, including standards-based approaches (e.g., ETSI [21] and G729B [22]), statistical methods (e.g., LTSD [23] and Sohn [24]), machine learning-based solutions (e.g., GMM [20], CLC [25], and CNN [26]), and recent SNN-based VAD models (e.g., Bin e. [14], SNN h1/h1-p [27], and VAD system in HuRAI [3]).

For a comprehensive and equitable comparison, we first assess all SNN-based VAD models, considering parameters count and HTER across different noise levels (low, medium, and high). Results are reported in Table 1. Notably, the VAD in HuRAI employs significantly more parameters, making it impractical for real-world use [4]. To ensure fairness, we re-conduct the VAD in HuRAI by aligning the parameter count with those used in our model. For a comparison with the Bin e. model, which uses only $2.6$K parameters, we create a reduced version, named sVAD-S, with a smaller sRNN layer of just $10$ recurrent spiking neurons and two sFC layers for attention mask estimation. Despite these reductions, our modified model exhibits only a marginal increase in HTER across all three noise levels. In addition, it is worth mentioning that our models use the smallest frame shift (i.e., $15ms$) for the encoder among all SNN-based models (e.g., $20ms$ in [14]), resulting in the lowest latency among them.

Next, we compare our sVAD model with the aforementioned ten baseline systems, as depicted in Fig. 2. The results demonstrate that our sVAD outperforms most models, with the exception of the CNN and CLC models. This can be attributed to the fact that the CNN and CLC models use larger frame sizes and shifts for decision-making (with frame sizes and shifts more than 5 times for CNN and more than 3 times for CLC in comparison to our models), thereby introducing increased latency. Despite this discrepancy, it’s crucial to emphasize that the primary focus of this study is the development of a robust, low-power, and light-weight SNN-based VAD model under noisy conditions rather than striving for the top HTER performance.

Overall, our model stands out among SNN-based VAD models, delivering superior performance, especially in high-noise conditions, while maintaining a competitive parameter count. This achievement signifies the successful development of a robust, low-power, and light-weight SNN-based VAD model under noisy conditions.

To validate the efficacy of our proposed auditory encoder, we conduct an ablation study on high-noise conditions, with results detailed in Table 2. We employ the proposed auditory encoder used in sVAD as the baseline. First, we remove the learnable sConv1D layer from the encoder, resulting in a significant $4.1\%$ increase in HTER. Subsequently, the further elimination of the attention mechanism led to another noteworthy $4.0\%$ increase in HTER. In summary, our ablation study demonstrates compelling evidence for the efficacy of our auditory encoder in significantly improving the robustness of our SNN-based VAD model under noisy conditions.

This section estimates the power consumption of our sVAD models and compares them with that of other VADs. Our estimation is based on the Loihi neuromorphic chip [30] and follows the estimation methodology in [14], encompassing power consumption from synaptic operations and neuron updates. As depicted in Table 3, our sVADs demonstrate lower power consumption, particularly the smaller sVAD-S model. It is worth noting that we only estimate the lower bound based on Loihi, which is different from [28, 2, 29] that have run on ASIC chips.