Music auto-tagging in the long tail: A few-shot approach

We employ transfer learning approaches by fine-tuning four different types of pre-trained embeddings on a commonly used music auto-tagging dataset through a simple linear probe. A linear probe is a linear classifier trained on frozen features extracted from a large pre-trained model. It helps to assess how well these embeddings encode task-relevant information with minimal training. Much of the research in few-shot classification [19, 20, 21, 22, 23, 24] frames the target task as a simple multi-class problem; however, music auto-tagging is inherently a multi-label task [4], i.e., multiple tags can be active for the same input. To align our few-shot classifier with the nature of the task, we employ Binary Cross-Entropy loss, coupled with a fully connected layer followed by a Sigmoid activation in our simple linear probe, similar to Wang et al. [25]. This approach essentially trains a separate logistic regression classifier for each class using a one-vs-rest strategy, thus allowing the concurrent presence of multiple tags in a single sample.

The experimental setup is detailed in Figure 1. To simulate the few-shot setting, we sample $N\times K$ ($N$: number of classes, $K$: number of training samples per class) training data points from the training set. The entire validation set is used for early stopping, and the test set is employed to compute the performance metrics of our few-shot classifiers. We use the mean average precision (mAP) and the mean area under the receiver operating characteristics curve (AUC-ROC) to measure the classifier performance to be comparable with previous work[1, 26]. Both mAP and AUC-ROC are calculated as the mean of the binary scores for each class, averaged across all classes. mAP is the mean of the binary average precision score of each class, focusing on how many of the tagged songs are correctly tagged. AUC-ROC, on the other hand, is the mean of the binary AUC-ROC scores for each class, focusing more on how many of the songs that do not have a specific tag are wrongfully tagged. Using both metrics gives us a comprehensive evaluation of our classifiers.

The MagnaTagATune dataset [27] is one of the most widely used datasets for automatic music tagging systems. It comprises 25,863 audio clips, each lasting 29.1 seconds. The dataset includes 188 tags, ranging from specific instruments like violin or drums to broader musical qualities such as classical or jazz. Most automatic tagging models only utilize the 50 most frequent tags [1], discarding the songs not associated with these tags. In our study, we adopt the same split of the top 50 tags as [1] to ensure our performance metrics are comparable to previous work. Figure 2 shows the distribution of the top 50 tags and their composition in the data split. It is worth noting that even the tag with the least amount of training samples has 286 associated clips. At the same time, the remaining 138 tags that are not in the top 50 tags only have 534 clips in total. The tag with the most training data only has 55 associated clips. This long-tail distribution could also potentially benefit from few-shot approaches.

All pre-trained features are extracted using the whole clip. We have chosen three pre-trained embeddings ––VGGish, OpenL3, and PaSST–– because they are extracted from models trained on large-scale audio datasets and have demonstrated strong performance in previous studies [15, 28, 29]. Each of these embeddings will be introduced in more detail below.

For these embeddings, multiple frames of embeddings are extracted over the 29.1-second audio clips. These frames are then aggregated into the mean and standard deviation of all frames at each position of the embedding, which results in a vector that is twice as long as its original dimension. This ensures that all four embeddings have the same time resolution, allowing for a fair comparison.

We conducted a series of experiments to comprehensively evaluate our few-shot classifier design, with each research question addressed in the subsections below. In this section, we refer to models trained with the pre-trained features using the entire training set as "full linear probe" models, and models trained with $N\times K$ training data points as "few-shot linear probe" models.

For our first experiment, we report in Table 1 the performance metrics of our linear probes LP$\ast$ trained and evaluated on 29.1-second audio excerpts with the same data split as proposed by Won et al. [1]. We also provide two state-of-the-art performance comparison systems on the top 50 tags of the MagnaTagATune dataset: (i) SC-CNN is a 7-layer CNN with a fully connected layer and residual connections, trained on 3.69-second audio excerpts, and (ii) Jukebox is a probing classifier with inputs as embeddings extracted by a pre-trained codified audio language model, trained on 29.1-second audio excerpts [26].

We observe that our linear probing classifiers perform competitively with both the end-to-end model trained specifically for this task and the linear probe with additional language information encoded. This indicates that all the pre-trained embeddings used here contain very relevant information for music tagging and a low-complexity linear classifier is sufficient to achieve results close to the state-of-the-art with such inputs. The PaSST embedding seems to outperform OpenL3 and VGGish embeddings. The better performance of the linear probe classifier with the combined (concatenated) embedding input suggests that our few-shot network learns different aspects from each embedding, and combining them allows these features to complement each other, resulting in higher overall performance metrics.

To investigate the impact of each pre-trained embedding on $\mathrm{LP}_{\mathrm{Combined}}$, we calculated the percentage of the weight magnitude of each single embedding in the weight magnitude of the combined embedding. Figure 3 shows that the PaSST embeddings have the highest impact, followed by OpenL3 and VGGish.

The second experiment investigates the data efficiency of each pre-trained features through observing the impact of the number of training samples per class for a 50-way classifier.

The top plot in Fig. 4 shows the mAP of the linear probing classifiers with different types of pre-trained embeddings as the number of samples per class increases. Unsurprisingly, the mAP increases in most cases with increasing number of training samples, but increasing the number of training samples per class beyond 10 only yields small performance gains. $\mathrm{LP}_{\mathrm{PaSST}}$ tends to consistently show the best performance across all training set sizes, followed by $\mathrm{LP}_{\mathrm{OpenL3}}$ and $\mathrm{LP}_{\mathrm{VGGish}}$. Using combined embeddings as input yields better metrics than any of the single embeddings. Figure 4 bottom shows the weight correlation between the linear probe classifiers trained with different training set sizes and the weights of the full linear probe that was trained with all available training data. Assuming that the latter represents the best-case outcome, the plot shows increasing correlation with increasing number of training samples. $\mathrm{LP}_{\mathrm{PaSST}}$ and $\mathrm{LP}_{\mathrm{Combined}}$, in particular, show a high correlation starting from a low number of training samples, indicating highly efficient training data utilization.

While the results presented above focused on the performance in a 50-way scenario, i.e., a classification into 50 classes, in this experiment we evaluate how the number of classes (or labels) influences the performance. As the number of target classes can vary dependent on the specific task, understanding the impact of this can be crucial in a real-world scenario.

Figure 5 depicts the changes in mAP and AUC-ROC as the number of samples per class increases for classifiers with varying numbers of classes. The metrics shown represent the best performing pre-trained features discussed in the previous subsection. As reported in Exp. 2, performance increases as we increase the number of training samples per class $K$ for the classifiers with the same number of classes $N$ (the results of Exp. 2 can be revisited in the right column of the matrix). On the other hand, as we increase the number of classes while keeping the number of samples per class constant, performance tends to be stable except for very low N’s and K’s. Even though increasing the number of classes adds negative examples for the previous classes in the training set, it seems like simple binary classifiers are not dramatically impacted by this training set imbalance.

It is important to note that all tags in the test set always have the same amount of positive and negative samples in all our experiments because we use the entire test set to calculate performance metrics. If we constrained the test set to include only songs with at least one associated tag, it would inflate the AUC-ROC with increasing class number. This is because AUC-ROC involves measuring the false positive rate (FPR), calculated by dividing the number of false positives by the sum of true negatives and false positives. With the constraint, moving from a lower number of classes to a higher number of classes increases the number of true negatives in the test set, thus exaggerating the AUC-ROC.