AUC Optimization for Robust Small-footprint Keyword Spotting with Limited Training Data

In this paper, we decompose the KWS task into a non-keyword segments detection subtask and a closed-set classification subtask. Specifically, for a given input sample, we first determine whether it belongs to a predefined keyword set. If so, then we decide which keyword it is. Note that the two subtasks are performed simultaneously in our proposed method.

To formalize the task, suppose there is a dataset $\mathbf{X}=\left\{\left(\mathbf{x}_{n},y_{n}\right)\right\}_{n=1}^{N}$ where $\mathbf{x}_{n}\in\mathbb{R}^{D}$ is a high-dimensional acoustic feature of the $n$-th sample, and $y_{n}\in\{0,1,2,\dots,C\}$ is the ground-truth label of $\mathbf{x}_{n}$. Note that, without loss of generality, we always assume that there are $C+1$ categories with class $0$ representing non-keyword segments, and the other classes $1,2,\dots,C$ representing $C$ keywords respectively.

We aim to train a neural network $f_{\theta}(\cdot):\mathbb{R}^{D}\to\mathbb{R}^{C}$ where $\theta$ is the parameter of the network. It maps the $D$-dimensional input acoustic feature to a $C$-dimensional vector. Each dimension of the vector represents the confidence score of its corresponding keyword. In the test stage, we use $f_{\theta}(\cdot)$ to conduct KWS by the following criterion:

where $[p_{n,c}]_{c=1}^{C}=[p_{n,1},p_{n,2},\dots,p_{n,C}]^{T}$ is the output scores of the neural network $f_{\theta}(\mathbf{x}_{n})$, and $\eta$ is a decision threshold. For simplicity, we denote $\mathbf{p}_{n}=[p_{n,c}]_{c=1}^{C}$ in the remaining of the paper.

Several studies have extended the binary AUC optimization to multi-class problems e.g.  [19, 20]. In this work, we propose a new extension suitable for most multi-class classification tasks and computationally straightforward. The key idea of this extension is to modify the two subsets $\mathbf{S}^{+}$ and $\mathbf{S}^{-}$ in the binary AUC optimization to new forms that satisfy the multi-class AUC optimization problem.

Specifically, for the general KWS problem with more than one keyword, we define the subset of positive examples as

and the subset of negative samples $\mathbf{S}^{-}=\ \mathbf{S}_{1}^{-}\cup\ \mathbf{S}_{2}^{-}$ with

where $p_{n,y_{n}}$ is the score at the $y_{n}$-th position of the vector $\mathbf{p}_{n}$, $\max_{c,c\neq y_{n}}\left([p_{n,c}]_{c=1}^{C}\right)$ is the maximum value of $\mathbf{p}_{n}$ after removing the score at the $y_{n}$-th position of $\mathbf{p}_{n}$, and

represents the set of the output scores of the neural network for the non-keyword segments in $\mathbf{X}$.

Algorithm 1 presents the proposed multi-class AUC loss in detail.

In the test stage, the decision threshold $\eta$ is calculated on a validation set by:

where $N^{\prime}$ is the size of the validation set.

This subsection presents the connection of the proposed multi-class AUC to other loss functions.

In our experiments, two popular keyword spotting datasets, Google Speech Commands version 1 (GSC v1) [16] and version 2 (GSC v2) [17] are used for evaluation. The dataset GSC v1 consists of 65K one-second-long recordings of 30 words from thousands of different speakers. GSC V2 is an augmented version of GSC v1, which contains 105K utterances of 35 words. In addition, both datasets contain several minute-long background noise files. The sampling rates of all signals are 16 kHz in the two datasets.

Both GSC v1 and GSC v2 include a “validation_list” file and a “testing_list” file. We use audio files in the “validation_list” and “testing_list” as validation and testing data, and the other audio files as training data. Following previous works, we apply random time-shift and noise injection to training data. Specifically, we first perform a random time-shift of $t$ milliseconds to each sample, where $t\sim U[-100,100]$. We then add background noise to each sample with a probability of 0.8, where the noise is chosen randomly from the background noises. Note that the random time-shift and noise injection are performed on the fly at each training step. Finally, 40-dimensional Mel-frequency Cepstrum Coefficient (MFCC) features are extracted and stacked over the time-axis with a window length of 25ms and a stride of 10ms.

We use res15 [3] as the backbone network. As shown in Figure 1, it starts with a bias-free convolution layer (Conv) with weight $\textbf{W}\in\mathbb{R}^{m\times r\times n}$, where $m$ and $r$ are the height and width of the convolution kernel respectively, and $n$ is the number of the output channels. Then, it takes the output of the first convolution layer as the input of a chain of residual blocks (Res), followed by a separate non-residual convolution layer. Finally, the output of the network is obtained by an average-pooling layer (Avg-pool). Additionally, a $(d_{w},d_{h})$ convolution dilation is used to increase the receptive field of the network, and a batch normalization layer (BatchNorm) is added after each convolution layer to help train the deep network. The details of the backbone network are listed in Table 1.

The tasks in previous works [3, 5, 7, 8] focus on discriminating the 11 keywords (“yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off”, “stop”, “go”, “silence”) and a non-keyword “unknown”, where “silence” denotes silence segments and “unknown” represents all other words. In their settings, all unknown words used in the test set have been seen by the model in the training stage, which is not consistent with real-world KWS applications.

To meet the real-world KWS applications, in our experiments, we consider the task in [12], where ten unknown words (“zero”, “one”, “two”, “three”, “four”, “five”, “six”, “seven”, “eight”, “nine”) are used for testing only. We evaluate our model by comparing the following metrics with other related works.

• •

  Total acc is the classification accuracy on the test set that contains unseen unknown words, which is more likely to reflect the performance of the KWS model in the real world. Note that unseen unknown words represent the above ten unknown words that are used for testing only.

• •

  Closed acc is the classification accuracy on the test set that does not contain unseen unknown words.

• •

  We also report the F1 score that is extended to multi-class one by “macro” average on the test set that contains unseen unknown words.

In addition, we plot the detection error tradeoff (DET) curve of the non-keyword segments detection subtask to evaluate the KWS models.

Usually, the training samples in each mini-batch is randomly sampled from the whole training set, which results in the proportion of the keywords over non-keywords in each mini-batch vary greatly. We denote this sampling method as random sampler. However, the variable proportion will hinder the convergence of the model training of the proposed method. To overcome this problem, we use a fixed proportion sampler, which keeps the proportion of keywords and non-keywords consistent in each mini-batch.

Each model in our experiments is trained for 60 epochs, using the Adam optimizer [21]. The initial learning rate is set to 0.001 and reduced to 0.0001 after 30 epochs. For the cross entropy loss, we use a mini-batch size of 128 and $L_{2}$ weight decay of $10^{-5}$. We use the same hyperparameters in [12] and [13] for the prototypical loss, AP-FC loss and triplet loss. We use the validation set to select the best model among different epochs and evaluate the effect of the hyperparameter $\delta$.

We evaluate the proposed multi-class AUC loss with the fixed proportion sampler and the random sampler. For the fixed proportion sampler, the number of keywords and non-keywords in each mini-batch is set to 32 and 64, respectively; for the random sampler, the mini-batch size is set to 128, which is the same as the other comparison methods. The hyperparameter $\delta$ is set to 0.3. Following the same training procedure, we evaluate all comparison methods for five independent times, and report the average performance.

Table 2 lists the comparison result between the proposed methods and the four baselines. From the table, we see that both the two variants of the proposed multi-class AUC loss achieve significant improvement in terms of the Total acc and F1 score, and achieve a competitive result with the best referenced method in terms of the Closed acc. We take the result on GSC v1 as an example. Comparing to the cross entropy loss, the multi-class AUC loss with the fixed proportion sampler achieves 30.0% and 25.9% relative improvement in Total acc and F1 score respectively. It also achieves a slightly higher Closed acc than the cross entropy loss. Even when compared with the triplet loss with a complex kNN backend, the proposed method still achieves a relative improvement of 11.1% in Total acc and 9.8% in F1 score while maintaining a similar Closed acc.

To further investigate the effectiveness of the proposed method, we conduct a comparison on GSC v2 using the same settings as that on GSC v1. The experimental results again demonstrate the superiority of our method. In addition, the result on GSC v2 indicates that the training data of GSC v2 is responsible for the substantial improvement in all evaluation metrics, which is consistent with the experimental phenomenon in [17]. However, although both of the two variants of the proposed multi-class AUC loss achieve better results on GSC v2 than that on GSC v1, the improvement with random sampler is more evident than that with the fixed proportion sampler. This may be caused by that the training data of GSC v2 contains more non-keywords than the training data of GSC v1.

From Table 2 we also see that the AP-FC loss with a SVM back-end and the triplet loss with a kNN back-end outperform the prototypical loss and the cross entropy loss. It demonstrates that the metric learning-based methods still require a decision back-end to achieve satisfactory performance. In addition, we plot the DET curves of the non-keyword segments detection subtask in Figure 2. From the figure, we see that these curves are consistent with the results presented in Table 2, and we see that the two variants of the proposed multi-class AUC loss outperform the referenced methods.

This subsection investigates the effect of the hyperparameter $\delta$ on performance. Becaue there are no unseen unknown words in the validation set, here we only use the Closed acc and F1 score as the evaluation metrics. For simplicity, we show the experimental results on GSC v1 only. Note that the experimental phenomenon on the other evaluation dataset are consistent with that on GSC v1. Table 3 lists the result on GSC v1. From the table, one can see that the parameter $\delta$, which controls the margin of the AUC loss, plays an important role on the performance. Both of the two variants of the multi-class AUC loss outperform the cross entropy baseline in the two evaluation metrics when $0.2\leq\delta\leq 0.4$. It is also observed that the results in both the two evaluation metrics first increase and then decrease along with the increase of $\delta$, where the best performance is achieved at $\delta=0.3$.