FILTERBANK LEARNING FOR NOISE-ROBUST SMALL-FOOTPRINT KEYWORD SPOTTING

Over recent years, there has been an increasing interest in developing end-to-end deep learning systems in which the feature extraction process is also optimized towards the task goal [1]. The keyword spotting (KWS) task, which deals with the recognition of a very limited vocabulary in speech signals, is, indeed, not alien to this trend [2, 3, 4]. In KWS, researchers seek to replace solid handcrafted speech features —e.g., log-Mel features and Mel-frequency cepstral coefficients (MFCCs)— by learnable features that are able to yield better KWS performance.

In spite of recent attempts [2, 3, 4], the above goal has not been achieved. In [2], Mittermaier et al. studied the integration of a trainable filterbank —the so-called SincNet [5]— into a convolutional neural network (CNN)-based KWS pipeline. In that system, the cut-off frequencies of a sinc-convolution-based filterbank are trained jointly along with the CNN acoustic model. While a formal comparison between SincNet and handcrafted features is missing in [2], this analysis has recently been done in [4], where Peter et al. demonstrate that MFCCs remain superior for KWS.

In previous work [3], we found no statistically significant differences between using learned filterbanks and log-Mel features for KWS. From this observation, we conjectured that much of the spectral information is redundant when it comes to the recognition of a set of few keywords¹¹1Although in the context of reduced-precision speech features for KWS, this same hypothesis was independently made in [6].. Inspired by this, in this paper, we prove —for the first time, to the best of our knowledge— that filterbank learning outperforms handcrafted speech features for KWS as long as the number of filterbank channels is drastically reduced. While this might lead to certain KWS performance degradation, it can also yield a significant energy consumption and inference time reduction²²2This is as a result of the number of multiplications in the acoustic model directly depending on the size of the input feature matrix., which is of utmost importance when deploying typical always-on KWS on low-resource devices.

In this work, we conduct experiments on a noisy version of the Google Speech Commands Dataset [7, 8] by using a CNN-based KWS system integrating residual connections [9]. Experimental results prompt the following observations:

1. 1.

   Filterbank learning adapts to noise spectral characteristics to offer a higher degree of robustness to noise;

2. 2.

   The use of dropout [10] enhances robustness to noise and generalization capabilities of learned filterbanks;

3. 3.

   Switching from 40-channel log-Mel features to 8-channel learned features yields a relative KWS accuracy loss of only 3.5% while involving a 6.3$\times$ energy consumption reduction;

4. 4.

   Switching from 8-channel log-Mel features to 5-channel learned features allows us for essentially maintaining KWS accuracy while leading to a 2$\times$ energy consumption reduction.

The rest of the paper is organized as follows. Section 2 presents our filterbank learning methodology. The experimental setup is described in Section 3. Results are shown and discussed in Section 4, and Section 5 concludes this work.

A block diagram of the KWS system employed in this work integrating filterbank learning can be seen in Fig. 1. First, the short-time Fourier transform (STFT) $X(t,f)$ of a discrete-time input speech signal $x(m)$, potentially comprising a keyword, is calculated. Note that $t=1,2,...,T$ and $f=1,2,...,F$ indicate, respectively, the time frame and linear frequency bin. Furthermore, $T$ and $F$ denote the total number of time frames and linear frequency bins, respectively, in the input speech signal. We can define a $T\times F$ matrix $\mathbf{X}$ representing $x(m)$ in the linear power spectral domain as follows:

$$\mathbf{X}=\left[\begin{array}[]{ccc}|X(1,1)|^{2}&\cdots&|X(1,F)|^{2}\\ \vdots&\ddots&\vdots\\ |X(T,1)|^{2}&\cdots&|X(T,F)|^{2}\end{array}\right].$$

(1)

Next, the filterbank layer —optionally considering dropout [10] at training time— implements the matrix multiplication

$$\mathbf{Y}=\mathbf{X}\cdot g(\mathbf{W}),$$

(2)

where $\mathbf{W}\in\mathbb{R}^{F\times K}$ is the learnable $K$-channel filterbank matrix, which is optimized jointly along with the acoustic model (i.e., back-end) by backpropagation. Moreover, $g(\cdot)=\max(\cdot,\;0)$ is the rectified linear unit function, which is element-wise applied to guarantee the positivity of the filterbank weights. The result of log-compressing the $T\times K$ filterbank representation $\mathbf{Y}$ as in $\log\left(\max\left(\mathbf{Y},\;\eta=e^{-50}\right)\right)$, where the operators $\log(\cdot)$ and $\max(\cdot)$ are element-wise applied, is input to a batch normalization layer producing the speech features for the acoustic model.

Notice that the only difference between this filterbank learning scheme and the one that we already proposed in [3] is the optional consideration of dropout [10] at training time, the goal of which is to improve robustness and generalization of individual filterbank channels.

For acoustic modeling, we use a deep residual CNN integrating dilated convolutions [9], since in a recent overview article [1] we concluded that state-of-the-art KWS acoustic modeling relies on CNNs incorporating both residual connections and a scheme to seize long time-frequency patterns. This model produces word-level posteriors from one-second long input speech segments $x(m)$. A keyword is spotted every time that its associated posterior probability is the largest among all posteriors.

In this section, we compare the use of log-Mel features with the utilization of the learnable features described in Section 2 with —Learned+D— and without —Learned— dropout. Table 1 shows KWS accuracy results (averaged across 5 experiment repetitions), in percentages, as a function of the number of filterbank channels $K$. Due to space constraints, we only report results for a more relevant selection of $K$ values (i.e., $\{40,10,8,7,5\}$). Statistically significant accuracy improvements are identified by means of two-sample $t$-tests [15] with a significance level of 0.05. In Table 1, statistically significant improvements with respect to log-Mel (Learned) are indicated in boldface (underlined).

Table 1 also shows the number of multiplications of the acoustic model per second of input speech, which highly depends on $K$ due to the use of residual connections. Because the number of multiplications exhibits a strong positive linear relationship with the energy consumption of the KWS system, ($R^{2}=0.9641,p=0.0001$) [16], we use the former as a proxy for the latter in this work.

Consistent with what we determined in [3], from Table 1, we see no statistically significant performance differences between log-Mel and learned features when employing a standard number of filterbank channels (i.e., $K=40$). However, filterbank learning starts to become beneficial when $K<10$, especially when dealing with unseen noise types. In particular, for $K=5$, Learned+D outperforms log-Mel features in all the evaluated noisy conditions. We conjecture that the combination of dropout and a reduced $K$ creates a bottleneck in the filterbank layer (quite similar to the ones observed in autoencoders), thereby providing a better knowledge representation for filterbank learning.

While it is true that KWS performance tends to drop when reducing $K$, filterbank learning cushions this drop while achieving noticeable energy savings. For example, switching from standard 40-channel log-Mel (avg. accuracy of 81.95%) to Learned+D when $K=8$ (avg. accuracy of 79.11%) leads to a relative KWS accuracy loss of only 3.5% while involving a $895\mbox{M}/141\mbox{M}\approx 6.3\times$ energy consumption reduction³³3For comparison, the relative KWS accuracy loss between 40-channel and 8-channel log-Mel (avg. accuracy of 76.75%) is of 6.3%.. Furthermore, switching from 8-channel log-Mel (avg. accuracy of 76.75%) to Learned+D when $K=5$ (avg. accuracy of 76.86%) allows us to essentially maintain KWS performance while yielding a reduction in energy consumption of a factor $141\mbox{M}/71\mbox{M}\approx 2$. To assess at a glance the more robust behavior of filterbank learning to the decreasing of $K$ (for both seen and unseen noises), Fig. 2 plots KWS accuracy (averaged across SNRs) as a function of the number of multiplications of the acoustic model.

For the first time, to the best of our knowledge, we have shown in this study that, when the number of filterbank channels is substantially decreased, filterbank learning is able to outperform handcrafted speech features for KWS. The entailed KWS performance-energy consumption trade-off is further compensated by using dropout for filterbank learning, which might be of particular interest to practitioners who deploy common always-on KWS on low-resource devices.

Future work envisages a comprehensive study on this matter aiming at obtaining deeper insight helping to design superior speech features for robust, small-footprint KWS.