UniKW-AT: Unified Keyword Spotting and Audio Tagging

Most previous work regarding KWS is focused on decreasing a model’s parameter size and increasing its inference speed [2]. Works such as [3, 4, 5] used convolutional neural networks (CNNs) for this matter, while more recently transformer based models [6, 7] and multi-layer perceptron (MLP) mixers [8, 9] have also received attention from the community.

On the other hand, works for AT generally are performance-oriented, meaning that few works exist that focus on practical implications of models such as inference time or model size. Similar to KWS research, CNNs dominate the research for AT, with notable examples being [10, 11]. Further, transformers have also been recently seen success [12, 13, 14]. However, previously introduced models in literature aimed at AT cannot be used for KWS, since KWS is an always-on feature computed on-device and previous work using CNN/Transformer models require a GPU for fast inference. Thus, our work focuses on the usage of lightweight and fast models.

This paper is structured as follows. Section 2 establishes our proposed UniKW-AT framework. Then Section 3 introduces the experiments and Section 4 displays the results of this work. Lastly, Section 5 discusses the shortcomings of our work and Section 6 concludes the paper.

Our work focuses on using a common MobileNetV2 [20] back-end as our default neural network due to its small parameter size of 2.9 M.

Regarding front-end feature extraction, all audio clips are converted to a 16 kHz sampling rate. We use log-Mel spectrograms (LMS) with 64 bins extracted every 10 ms with a window of 32ms. Batch-wise zero padding to the longest clip within a batch is applied for all experiments. Training runs with a batch size of 64 for at most 500 epochs using Adam optimization [21] with a starting learning rate of 0.001. During training, we use mean average precision (mAP) as our primary validation metric. The top-4 models achieving the highest mAP on the joint held-out validation dataset (GSCV1-valid and AS-balanced) are submitted for evaluation. The neural network back-end is implemented in Pytorch [22].

Assume that $\mathbf{x}$ is an input feature of some length, $\mathbf{y}\in[0,1]^{C+K}$ the corresponding label, $\mathcal{F}(\cdot)$ the trainable neural network, and $\hat{\mathbf{y}}\in[0,1]^{C+K}$ the model prediction, i.e., $\hat{\mathbf{y}}=\mathcal{F}(\mathbf{x})$. UniKW-AT is optimized using the binary cross entropy (BCE) objective defined as:

$$\mathcal{L}_{\text{BCE}}(\mathbf{\hat{y}},\mathbf{y})=\mathbf{y}\log{\hat{\mathbf{y}}}+(\mathbf{1}-\mathbf{y})\log(\mathbf{1}-\hat{\mathbf{y}}),$$

(1)

where $\mathbf{y}$ can have multiple positive labels.

One of the core difficulties in jointly modeling KWS and AT are the differences in their training sample duration. In our experiments, the GSCV1 dataset contains only samples of duration less than $t=1$ s, while the majority of training samples in AS are of length 10 s. A simple merge of both datasets during training would lead to a large amount of padding (for the GSCV1 dataset). We provide an ablation study using the naïve data merging approach in Section 4.4.

For KWS, we use accuracy as the default evaluation metric, while AT is mainly evaluated using mAP. Computing accuracy for the UniKW-AT approach requires the use of post-processing since the model can predict multiple labels at once. The core problem lies in the label ambiguity of speech-related labels since these are present for non-target and target keywords. For example, when presented with a target keyword, our approach will likely predict the tuple (speech, target-keyword), but in the case of a nontarget-keyword, our approach will only predict speech. Thus we compute the single-target output label as:

$$\bar{y}=\begin{cases}\operatorname*{arg\,max}\mathbf{\hat{y}}_{\text{KWS}},&\text{if }\max\mathbf{\hat{y}}_{\text{KWS}}\geq\gamma\\ \operatorname*{arg\,max}\mathbf{\hat{y}_{\text{AT}}},&\text{otherwise}\end{cases}$$

(2)

where $\hat{\mathbf{y}}=\left[\mathbf{\hat{y}}_{\text{AT}},\mathbf{\hat{y}}_{\text{KWS}}\right]\in[0,1]^{C+K}$ is the raw model prediction vector, $\bar{y}$ the predicted label index and $\gamma$ a decision threshold. We optimized this threshold on the GSCV1 held-out validation dataset and obtained $\gamma=0.4$ as our default choice. Finally, when evaluating on AS, we only output $\hat{\mathbf{y}}_{\text{AT}}\in[0,1]^{C}$ as the representative prediction vector.

The results of our proposed UniKW-AT approach can be seen in Table 2. UniKW-AT achieves a comparable performance to other methods in regards to both AT and KWS. Notably, it outperforms the EfficientNet-B2 (15.70%), AST (14.80%), ResNet-50 (31.80%), and CNN14 (27.80%) methods on the balanced AS training set in regards to mAP performance (33.42%) without additional data augmentation or external pretraining. Further, our UniKW-AT method also contains the fewest parameters (2.9 M) within all investigated AT models.

Moreover, we observe that our approach only marginally lacks behind other models in terms of GSCV1 performance, achieving an accuracy of 97.53%, without further data augmentation. The much larger Wav2KWS [23] is the only approach to outperform UniKW-AT with an accuracy of 97.90%, while also containing $\approx$ 80 times more parameters. In addition, due to the benefits of the previously stated BCE training regime (see Section 2), we can remove the output layer’s weights and biases, stripping down the model size to 2.2M, denoted as “Ours-Strip”.

We conclude that our results indicate that UniKW-AT provides excellent performance for both individual tasks.

One of the benefits of the UniKW-AT approach is that it naturally enhances the noise robustness compared to standard KWS models. A core reason for this is the addition of AS into the training regime. Thus, we compare our proposed approach to a naïve CE trained baseline with the addition of AS as a noise source.

The results regarding noise-robustness are provided in Table 3. It can be seen that the naïve baseline performs well on in-domain evaluation, achieving an accuracy of 97.63% and 97.47% on the GSCV1 and GSCV2 evaluation sets, respectively. However, a significant performance drop is seen when evaluating the naïve baseline in out-domain (VL106) and noise (Musan) scenarios, since the baseline has been trained on clean data. When adding AS to the baseline training regime, performance steadily improves on all datasets compared to the baseline. With additional AS data the baseline approach slightly outperforms UniKW-AT on the clean GSCV1 evaluation set, achieving an accuracy of 97.75% against UniKW-AT’s 97.53%. On the other hand, the non-target keyword evaluation on all reported datasets (GSCV2, VL106, Musan) shows that our UniKW-AT method outperforms the baseline.

In order to verify the effectiveness of the proposed approach, we explore its real-world usability. Here we use a private in-house dataset containing samples for our voice assistant “XiaoAi”. We collect 131 h of real-world training samples, split into 144,582 individual utterances with a length $t$ of 3 s. Within these 144,582 training samples, 100,582 (70%) contain the target keyword “XiaoAi Tongxue”. We evaluate our findings on two distinct datasets according to our evaluation framework (see Section 3.3). First, we use our “Common” dataset, containing 93,800 test utterances (80 h) with a balanced split of 50% positives (“XiaoAi Tongxue”) and negative samples. Second, we use a difficult non-target evaluation dataset, denoted as “Difficult”, which contains 3,731 (3 h) samples. The Difficult evaluation set consists of non-target keyword samples similar to the target keywords e.g., “Zhewei Tongxue” and other cases of false-alarms such as child babbling and vocalized music. Here we compare UniKW-AT against an in-house KWS model, denoted as “Base-Inhouse”.

The results are displayed in Table 4. Real-world evaluation results show a similar trend to those in Table 3. The proposed UniKW-AT approach sightly underperforms on the Common evaluation set (99.85% vs. 99.98%), but significantly outperforms the baseline on the Difficult dataset (0% vs. 82.73%). These results show that UniKW-AT can enhance noise robustness in real-world KWS applications. Further performance gains can be achieved by tuning the decision threshold $\gamma$. For example, setting $\gamma=0.01$, one obtains 99.96% and 57.45% on the Common and Difficult test sets, respectively.

In this section we explore the impact of two crucial components of our work: The PSL module and the effect of random cropping on the AT branch. The removal of random cropping, denoted as “$-$ Crop” leads to zero padding samples from the KWS dataset to match the Audioset’s length i.e., 10 s. The removal of PSL uses the original hard labels provided in Audioset. The results of these ablation studies are displayed in Table 5.