Deep Embeddings for Robust User-Based
Amateur Vocal Percussion Classification

We used two publicly available vocal percussion datasets throughout the study: the AVP dataset [11] and the LVT dataset [15]. We contrast some of these datasets' characteristics in Table 1. The AVP dataset contains a total of 9778 boxemes (4873 and 4905 from the personal and fixed subsets respectively) recorded by 28 participants and with four annotated labels: kick drum, snare drum, closed hi-hat, and opened hi-hat. Its training dataset was recorded using the isolated samples strategy. To train our acoustic models, we exclusively used the personal subset (participants vocalising boxemes of their choice), although we also used the fixed subset (participants vocalising common boxemes) to train the sequential module of the baseline speech recognition model (see section 3.3). The LVT dataset contains a total of 841 boxemes recorded by 20 participants with three annotated labels: kick drum, snare drum, and closed hi-hat. Here, the training dataset was recorded using the fixed phrase strategy. Also, we exclusively use its third subset, as the recordings' quality and background noise level are similar to those from the AVP dataset.

We manually expanded the annotations (onsets and instrument labels) of both datasets so as to include the syllabic representation of boxemes. The syllables were composed of a first onset phoneme, usually plosive or fricative, and a second coda phoeneme, usually a vowel, a breath sound, or silence (no coda phoneme). These phonemes were annotated following notation conventions from the International Phonetic Alphabet (IPA). Apart from the original phoneme set, we also elaborated a reduced phoneme set in which several similar-sounding phonemes were put together to form single classes. For this reduced version of phoneme annotations, onset and coda phonemes were grouped as shown in Table 2. We also made the final AVP-LVT dataset with expanded annotations publicly available²²2https://zenodo.org/record/5578744#.Yfpu9PXP30o.

Joining the AVP and LVT datasets, we had a total of 5714 boxemes. As this amount of data was relatively modest for deep embedding modelling, we applied waveform data augmentation to boxemes, specifically random pitch-shifting (semitone range = [-1.5,+1.5]) and time-stretching (stretch factor range = [0.8,1.2]), one after the other in random order. This kind of data augmentation is standard in audio signal processing [28] and it has been proven to improve the accuracy of vocal percussion classification algorithms [14]. We applied ten iterations of random data augmentation in this manner, ending up with a total of 62854 boxemes in the final dataset.

As input to neural networks, we built Mel spectrogram representations from each boxeme using 64 Mel frequency bands and a hop size of 12 milliseconds. We used 48 time steps ($\sim$ 0.56 seconds) so that the final boxeme spectrograms had a final dimension of 64x48 and we explored frame sizes of 23, 46, and 93 milliseconds, ultimately reporting the one that brought the best results in Section 3.1. We post-processed spectrograms with a logarithmic transform ($log(spectogram+0.0001)$) and normalised them to a [0,1] range.

We used the penultimate layers of several CNN classifier models as the final embeddings to evaluate on. The architecture of these CNN models is illustrated in Figure 1. They all had four convolutional blocks with 8, 16, 32, and 64 filters respectively and two fully-connected (FC) linear layers, one connecting the flattened feature maps to the embedding space and another one connecting the embedding space to the labels. Each convolutional block had two convolutional layers with kernels of size 3x3 and stride 1x1, each one followed by a batch normalisation module and a ReLU activation gate. A final max-pooling operator with kernel size 2x2 is applied at the end of each convolutional block so as to progressively downsample the feature maps. The design of the network's architecture and its training routines (see section 3.4) was inspired by best practices in CNN-based research [29, 30].

We explored seven different supervision strategies to train and validate the above mentioned CNN classifiers. These strategies use different types of labels that describe the same input data at different levels of abstraction.

We compare the performance of learnt embeddings with two baseline heuristic feature sets and a speech recognition model. We used the Essentia toolbox [31] to calculate audio descriptors and the Kaldi library [27] to implement the speech recognition model.

A schematic diagram of the training and evaluation processes is provided in figure 2.

We chose four participants from the AVP dataset and four from the LVT dataset to compose the evaluation set. Two women and two men per dataset were selected for evaluation on the basis of acceptable pronunciation and overall representativeness of the dataset. In the end, we had a total of 8 participants in the evaluation set and 40 participants in the train-validation set. We provide the distribution of the instrument, syllable, onset phoneme, and coda phoneme labels in the project's code repository.

We trained our seven CNN models (see sections 3.2.1, 3.2.2, 3.2.3, and 3.2.4) using the train-validation dataset. We set up a 5-fold cross-validation routine in which models were trained and validated for 5 iterations per fold, each with different initialisation parameters. Therefore, we end up with 25 deep embedding models per supervision strategy that encapsulate two main sources of training arbitrariness: the content of train-validation splits and random weight initialisation. We compute the mean and the 95% confidence intervals of the subsequent 25 evaluation performances when reporting final results for a given supervision strategy. The same train-validation arrangement applies to the seven baseline feature selection methods. We trained the models using an Adam optimisation algorithm [35], early stopping if validation loss has not decreased after 10 epochs, and a further regularisation routine that downscales the learning rate if validation loss has not decreased after 5 epochs. In the case of phoneme-level supervision, we explored two settings of loss weights to compute the joint loss value after each training batch. The onset-coda weight percentages for those two settings were 50-50% and 60-40%.

As explained in section 3.1, the 8 participants in the evaluation set have their own train and test subsets. From here on, we refer to these as the ev-train and the ev-test sets respectively to improve readability. During evaluation, a KNN algorithm was trained on the ev-train set of each participant and evaluated on the ev-test, taking the learnt CNN embeddings and the baseline feature sets as input representations. As these embeddings and feature sets are expected to have different information for the KNN algorithm to rely on, they are also expected to make the classifier perform better or worse given the underlying suitability of these feature sets for classification. As KNN algorithms are purely distance-based and non-parametric, our assumption is that a high classification accuracy using them is more likely to translate into a high accuracy using other types of machine learning algorithms. In Section 4, we had the opportunity to briefly test this hypothesis by replacing the KNN classifier with three other popular machine learning classifiers whose results are commented on but not reported.

The evaluation procedure above is applied to all proposed and baseline methods except for the GMM-HMM-based speech recognition model. This model does not extract embeddings, but rather predicts the ev-test labels directly ignoring the ev-train subset (user-agnostic). Also, to alleviate the disparity in the amount of ev-train data in the AVP and the LVT datasets (1,000 vs. 220 augmented data samples approx.), we extract two different amounts of embeddings and selected features: 32 to carry out evaluation on AVP participants and 16 to do it on LVT ones. This means that for each supervision method we end up having 25 feature sets of size 32 to evaluate on AVP data and other 25 of size 16 to do it on LVT data.

We report final results in two modalities: participant-wise, where accuracy scores of single test participants are calculated and averaged, and boxeme-wise, where accuracy scores are calculated for all evaluation boxemes in a participant-agnostic way.

Finally, we used saliency maps [17] to highlight sections of the input that are important for model prediction. These maps can be computed given an input $x$ and the penultimate layer output of a deep learning model $A$ as: $\frac{dA_{i}}{dx}$, where $i$ is the label with respect to which the saliency map is computed. In our case, $A$ is the CNN model conditioned on the original instrument labels. We aggregated the saliency map by taking the absolute value and thresholding it so that the top 10% of the map is set to one and the rest is set to zero. Then we averaged the saliency map over the 25 models.

Final evaluation accuracies for all methods are gathered in Table 3. Best results were obtained using a frame size of 46 ms and, in the case of phoneme-level classification, loss weights of 0.6 and 0.4 for onset and coda phonemes. Also, the best-performing feature selection routine was the one using the original syllable label set.

We can observe in the table that all supervised embedding models except for those supervised with instrument-level classes are consistently superior to baseline approaches, including feature selection approaches. This observation lies in accordance with previous literature on the usefulness of deep learning models as feature extractors for speech utterances [16]. It also highlights the unsuitability of instrument-level classes for embedding learning supervision, which was somewhat expected given that participants have different ways of vocalising drum instruments. Thus, label sets of such (high) level of abstraction are undesirable for embedding learning in our case.

We see that the best performance for both participant-wise and boxeme-wise evaluation metrics is achieved by supervised embedding learning models that used the original syllable-level label set. This difference is notable not only for the high mean accuracy score but also for its 95% confidence intervals, which is the lowest for both participant-wise and boxeme-wise metrics. This means that the informative power of the resulting embeddings, apart from being the most prominent, is also the most robust to the two sources of training arbitrariness that we studied here (see Section 3.4). We carried out experiments using other different classifiers than KNN, namely logistic regression, random forest, and extreme gradient boosted trees. There, we observed that the original syllable-level method still outperforms the rest of the approaches, which further reinforces its suitability for classification. All methods performed similarly on these extra algorithms, both in terms of absolute and relative performances.

Models supervised using original and reduced phoneme-level classes also yielded similar scores to those of the ones with syllable-level supervision, although still lower and generally less robust to training arbitrariness, possibly due to the extra complexity of the multi-task learning approach. The same applied to boxeme-level supervision, which performed slightly worse than phoneme-level supervision, possibly indicating that very low levels of supervision abstraction are relatively counterproductive for vocal percussion classification engines.

Another reason that could explain this lower performance for boxeme-level supervision could be its large amount of output labels ($\sim$150 boxeme types), which complicates validation. In order to tackle this issue, we built and trained an alternative siamese network model [36] with the same architecture as the CNN except for the last fully connected layer, which was removed. This network uses metric learning directly on embeddings to discriminate between same-class boxeme pairs and different-class ones, therefore reducing our large label vector to a ``same-different'' binary auxiliary vector. In the end, its final accuracy was found to be moderately lower than the one relative to the CNN classifier, so we kept the latter's result.

We also notice that the generic (user-agnostic) HMM-based speech recognition model performs worse than any other user-based method. This result evidences the importance of taking user idiosyncrasies into account in amateur vocal percussion classification, making user-based strategies preferable to generic user-agnostic ones. Finally, we observe that accuracies derived using the timbre feature set are higher than all the ones pertaining to baseline feature selection algorithms. This result could indicate an excessive information redundancy of selected features compared to that of the timbre set, which is a more internally cohesive feature set.

A clear limitation of the syllable-level embeddings as inputs to amateur vocal percussion classifiers is that these were learnt using samples recorded with electronic devices in a small room with little background noise, which emulates the typical recording setting of amateur music producers. This is likely to work similarly well in recording studios, where audio quality is higher, but may very well lose a significant part of its accuracy when faced with low-quality recordings or contexts with too much noise. The timbre feature set could be of great help for these situations, as its features are context-agnostic and therefore adapt well to challenging recording scenarios.

Four representative examples³³3See project’s repository for more examples of saliency maps. of these maps are shown in Figure 3. In general, we found that models tended to focus more on frequencies between $1000Hz$ and $2000Hz$ for boxemes associated with the snare drum, closed hi-hat, and opened hi-hat. This region usually coincides with a high-energy one for these boxemes, which often share phonetic representations. It also might indicate a useful thresholding point for the network to tell the boxemes apart. Hence, this could be implying that a higher frequency resolution in this region could potentially improve the accuracy of future amateur vocal percussion classifiers.

The network also appears to attend to silences and regions of lower spectral energy in the case of the kick drum and closed hi-hat. This could mean the absence of energy, especially at the high end of the spectrum, might also be a key feature for the models to differentiate the kick drum and closed hi-hat from the rest of the instruments. The high attention density in silences specifically could also be implicitly suggesting that the duration of boxemes is a key factor to distinguish the kick drum and closed hi-hat from the snare drum and opened hi-hat, as the boxemes associated with these last two instruments tend to be longer.