An Experimental Comparison Of Multi-view Self-supervised Methods For Music Tagging

We use an in-house dataset, which consists of $\sim$4M full tracks, to pre-train our models. We aim to make our dataset as musically balanced and diverse as possible in order to expand the models’ generalization ability toward unseen data. To do so, we collect tracks that have been used in past curated playlists from the music streaming service Deezer. These playlists are categorized by mood or genre; our dataset thus contains a vast collection of high-quality sounds and songs that span various genres and periods. We acknowledge that these may be biased toward Western and Billboard music. Each piece of audio is resampled at 16 kHz, normalized, and converted to mono. We then compute a log-magnitude mel-spectrogram with 128 frequency dimensions to use as input for each of our models.

In music, multi-view methods rely on the idea that various facets of music, such as songs, albums, or artists, collectively share common features [29], which are sufficiently rich and informative to extract meaningful music-invariant traits. For each self-supervised approach, the input data consists of two four-second audio pairs, which we call anchor and positive. For each song, an anchor is randomly selected. The corresponding positive is then selected from the same song. We force this segment to be at least four seconds away from the anchor but no further than sixteen seconds away in order to ensure both segments are contextually similar. Furthermore, we use online training: every time we see an audio track, the constructed pairs are generated on the fly. All self-supervised models are then trained with batches consisting of 256 pairs for 1000 epochs of 512 steps.

Our self-supervised learning models have a ResNet backbone architecture and contain a total of 2.8 million trainable parameters. Similarly to [30], we incorporate an attention layer that summarises the sequential data into a single, 1024-dimension embedding vector. A projector head is then placed upon the backbone to solve our designated pretext task. This ensures that our generated embeddings are not tailored to the pre-training method; they maintain a broader, more general utility. All the approaches we study in this work use the same head. It is comprised of two blocks, which consist of a linear layer, batch normalization, and a ReLu activation. The first linear layer contains 1024 connected weights, while the final linear 2048. All our approaches are trained using a single GPU.⁴⁴4NVIDIA GeForce GTX TITAN X

Our paper studies how five different self-supervised pretext tasks popular in image processing adapt to the audio domain.

Contrastive learning aims to project similar samples closer together in the embedding space while pushing dissimilar samples apart. We use the normalized temperature-scaled cross-entropy loss [31], with a temperature set to 0.1, in order to do so. In this setting, similarly to [14], for each anchor and positive pair, negative examples consist of the rest of the audio segments in the same batch. The distance between embeddings is computed using a cosine similarity.

BYOL [20] uses a teacher-student approach. The student model is encouraged to match a target embedding representation generated by the teacher. The networks are almost identical and start with the same random initialization. The student, however, needs an extra prediction layer for stability reasons. The distance used to compare embeddings is a mean squared error between the normalized student predictions and target teacher projections. At each training step, the student is updated with the normal gradient, whereas the teacher is updated using an exponential moving average (EMA) of recent student weights. Unlike in this work, in [32], a unique, data-augmented segment of audio is used for BYOL pre-training. We hope to explore whether this type of pre-processing leads to more favorable self-supervised learning in the future.

Clustering [21, 22] groups embeddings based on similarity. It uses a teacher-student approach close to BYOL. However, instead of computing similarity between embeddings, the teacher generates the target class distribution the student has to match via a cross-entropy loss. Both models are initialized identically and have the same architecture. At each training step, the student is updated normally, whereas the teacher is updated with an EMA. As presented in [21], we use centering and sharping to avoid collapsing into a single class and uniform distribution.

The two methods are grounded in the imposition of statistical properties within the embedding space. Barlow-Twins [23] encourages the diagonalization and independence of each embedding dimension. The pretext task’s goal is to ensure that the cross-correlation between the anchor and positive embeddings is close to the identity matrix. This reduces the redundancy between the components of each embedding dimension. VICReg [24, 25] combines variance, invariance, and covariance terms. The invariance term encourages proximity between the anchor and positive embedding vectors by minimizing their mean squared distance. The variance term promotes diversity among embedding vectors within a batch. Finally, the covariance term is similar to Barlow-Twins, but it focuses exclusively on reducing off-diagonal terms, effectively decorrelating the variables within each embedding.

We apply the same procedure across all the downstream tasks and pretext approaches. We use an MLP that operates directly on the average embedding space of each track to find the linear separation between classes. Thus, we optimize $1024\times N$ weights, where $N$ denotes the number of classes for each downstream class. The backbone generating the embeddings remains frozen. We evaluate the effectiveness and generalizability of each pretext approach on a set of five music tagging datasets, which are depicted in Table 1. We use the standard evaluation metrics: area under the Receiver Operating Characteristic curve (ROC) and mean average precision (mAP).

All downstream tasks are first trained using their full training set. We use a batch size of 256 pairs and 25 epochs of 128 steps. We use the last checkpoint of the pretext backbone and the best downstream model checkpoint, in terms of validation loss, for test set evaluation. Both ROC and mAP are computed using all song classifications for the test set; therefore they do not provide insights into the metric values’ robustness to data variation. In order to overcome this limitation, we use bootstrapping. The test dataset is sampled without replacement several times and computes the mAP and ROC for each batch. We sample $50\%$ of each test set 50 times and report the mean and standard deviation of all samples.

Results in Figure  1 reveal that the model pre-trained using contrastive learning consistently achieves superior performance compared to those pre-trained with other pretext tasks on both metrics. This observation is noteworthy, particularly since this is not the case in the image domain, which serves as inspiration for the majority of this work. Furthermore, despite other works using contrastive learning with more advanced architectures and incorporating more intricate probes [13, 33], our simple ResNet and MLP combination delivers comparable performance. In second place, clustering exhibits strong performance, most likely due to its ability to create well-defined groupings within the embedding space. We observe an uncharted collapse mode, where the model utilizes only a subset of clusters to partition the feature space, preventing the use of this pretext task to its full potential. On the contrary, the BYOL method exhibits lower performance, which can be attributed to its distinct design. Predicting the same embedding space appears to be overly restrictive and stringent in this context. The differing performances of VICReg and Barlow Twins are intriguing. While the simple orthogonalization and diagonalization of Barlow Twins is successful for conventional tagging tasks like MSD100 and Jam_Top50, it proves insufficient for separating instruments and mood, where VICReg’s regularization is better suited.

Annotating a music dataset is a time-consuming, error-prone process frequently plagued by inherent ambiguities [34]. As a result, a substantial portion of datasets within the MIR community remain relatively small in scale, which hinders our ability to train deep neural networks on them. In this experiment, our objective is to assess the adaptability of each approach when faced with the constraint of limited data availability. We focus on three datasets: MSD100, MTAT, and Jam_Top50. We randomly sample each dataset’s train set four times for four different percentages ($1\%$, $5\%$, $10\%$, and $20\%$) and train a different MLP for each iteration and percentage using the same specifications as earlier. We then evaluate their performance on the full test set, as shown in Figure 2. It is interesting to note that all approaches demonstrate decent performance with just $1\%$ of the training set and nearly optimal results with just $10\%$. Subsequent performance improvements occur at a slower rate. We also observe that, as seen previously, contrastive learning outperforms all other methods. It is, however, worth mentioning that the performance gap between clustering and contrastive methods is less pronounced when using limited data compared to the full dataset. Nonetheless, all other pre-training methods perform similarly and less well than the two methods mentioned previously.

In our empirical findings, we observed that contrastive learning and Barlow Twins are the most stable methods during training. They both avoid convergence issues. These models also rely on a minimal set of hyperparameters, the scale parameter for the former and diagonal and off-diagonal ratios for the latter. On the other hand, VICReg is highly sensitive to hyperparameter choice since it uses multiple losses. This may explain its notable fluctuations in performance. BYOL exhibits instability. We observe substantial fluctuations in both training and validation loss from one epoch to another. It is highly sensitive to the EMA momentum, where slight value variations have a significant impact on the model weights. It is also worth noting that it requires the most time per backpropagation step compared to other approaches. Clustering demonstrates an even greater reliance on hyperparameter selection, as it encompasses a complex interplay of factors, such as the EMA momentum for teacher and centering updates, temperature settings for the sharping and centering process to avoid collapse and weight decay. Striking the right balance among these elements is delicate and often requires the use of multiple schedulers to achieve effective model optimization. We found that the default values for most hyperparameters proposed in the original works for image processing are not well-suited for audio applications. As a result, we introduced new parameter values to mitigate early training plateaus. It is worth noting that further hyperparameter tuning could lead to improved results, a direction that warrants exploration in future investigations. Our settings can all be visualized in the GitHub repository linked with this publication.