Multi-task Learning with Metadata for Music Mood Classification

Music has the ability to express emotions and induce moods (Kim et al., 2010). The mood of a music piece is said to be the emotion or mood expressed by it. The task of recognizing the mood of a music piece is an active research area in music informatics (Pons et al., 2018; Alonso-Jiménez et al., 2020; Choi et al., 2016; Korzeniowski et al., 2020; Kim et al., 2010) and has multiple applications.

Mood recognition finds key applications in the areas of music discovery and recommendation. Given the rise of music streaming, massive collections of music have surfaced online, and these research areas have taken a center stage (Schedl et al., 2018). The capability to detect the moods of a song is an important requirement for these platforms. To highlight the importance, as a popular music streaming service, we get at least 1% of search queries on our platform related to moods such as “sad songs”. In other words, users intuitively associate moods with music, and it is a common user expectation to find music content with mood keywords.

Recent work has modeled it as a multi-label classification problem with deep neural networks, which has resulted in well-performing models. The idea is to apply models from vision research on audio spectrograms, which could be thought of as visual representations of audio data (Choi et al., 2016; Alonso-Jiménez et al., 2020; Pons and Serra, 2019; Pons et al., 2016; Pons and Serra, 2017). More recently, researchers have used song representations derived from listening data as input features instead of spectrograms for this task (Korzeniowski et al., 2020). They have reported interesting insights that user-song interaction data can be more valuable than audio data for predicting the mood of a song.

Surprisingly, previous work has not given much attention to the metadata of songs such as artist and year, which is readily available and can potentially be helpful. In this paper, we propose a method to leverage this data for improving the performance of the existing models for our task. Specifically, we employ the technique of multi-task learning, along with audio metadata, on convolutional neural networks (CNN) for mood classification (Choi et al., 2016; Alonso-Jiménez et al., 2020; Pons and Serra, 2019), and we find that their performances improve.

Multi-task learning is a training paradigm in which multiple tasks are learned simultaneously by a shared model. The training data of different tasks help in learning internal representations (i.e., the model layers and parameters) that are more informative than the individual models of the different tasks (Zhang et al., 2014; Böck et al., 2019; Hung et al., 2019; Hung and Lerch, 2020). In other words, the knowledge present in the data of one task is also used to improve the other task, and vice-versa. In our case, we employ multi-task learning approach on two tasks, namely predicting the mood and predicting the metadata (artist and year), where the former is our primary task. The input of the model is the audio spectrograms and the output is the labels corresponding to both the tasks. We use the same model (i.e., shared parameters) and train it simultaneously for both the tasks.

Our work investigates the research question of whether we can leverage audio metadata to improve the performance of mood classification models. To this end, we propose a multi-task learning approach in which a shared model is simultaneously trained for mood and metadata prediction tasks with the goal to learn richer representations. We experimentally demonstrate that applying our technique on the existing state of the art CNNs for mood classification improves their performances consistently. We conduct experiments on multiple datasets and report that our approach can lead to improvements in average precision by up to 3.4 points for public datasets and 8.7 points for an internal regional dataset.

The rest of the discussion is organized as follows: Section 2 presents the model architecture. We summarize the datasets and discuss the preprocessing steps in Section 3. In Section 4, we present the experimental setup and results. We discuss related work in Section 5, and finally, we conclude in Section 6.

Our model is a convolutional neural network (CNN) and follows a VGG-like structure (Simonyan and Zisserman, 2015). Convolutional neural networks are usually applied to analyze visual images. The idea of applying these to the audio domain is that spectrograms can be thought of as equivalent visual representations of audio data containing the relevant information. The idea is not new, and significant recent work in music informatics has borrowed network architectures and insights from computer vision research (Alonso-Jiménez et al., 2020; Pons and Serra, 2019; Korzeniowski et al., 2020; Hung and Lerch, 2020; Choi et al., 2016; Pons et al., 2016; Pons and Serra, 2017; Pons et al., 2018; Böck et al., 2019).

Figure 1 presents the architecture of our model. The input of the model is a 2D-tensor of shape $(187\times 96)$. It is supposed to be a mel-spectrogram representation of an audio sample with 187 frames and 96 mel-bands. The conversion of audio data to spectrogram is performed in a preprocessing step discussed in Section 3.1. As per the configuration of the preprocessing step, 187 frames of a spectrogram correspond to about 3-sec long audio sample.

The input passes through five blocks of layers with exactly the same configuration. Each block itself consists of four layers, namely convolution, batch normalization, max pooling, and dropout — in the same order. The convolution layer has its filters of size (3$\times$3), in other words, the filters move in both the axes of input, thus, performing a 2D convolution. The number of filters is 128, the stride is 1, the padding is set to “same”, i.e., producing convolution output with the same size as that of the input, and the non-linearity is set to ReLU. After the convolution layer, batch normalization is applied with default parameters (Keras, [n.d.]). Next, max-pooling operation is applied with the stride of (2$\times$2) and the pool size of (2$\times$2). Finally, a dropout layer is applied with the dropout rate of 0.25.

After these five blocks, a flatten layer is applied to change the shape of the output tensor from 2D to 1D. This 1D vector is now passed to be consumed by two different tasks, namely mood classification and metadata classification. Each task is represented as a dense layer, with its units equal to the number of labels. For instance, it is the number of different moods for the task of mood classification. In these dense layers, the sigmoid activation function is applied to output valid label probabilities that are mutually-inclusive. Thus, it models a multi-label classification task.

The number of trainable parameters in the first five blocks of the model is 591k. The number of parameters in the output layers is dependent on the number of labels in the respective task (as well as the number of units in the preceding layer).

The two different output layers allow us to share the same model simultaneously for the two different tasks — thus, putting together a setup for multi-task learning. The loss function of the entire task is a weighted sum of the losses of the two individual tasks. Formally, we train the model with the following loss function:

Here, $L_{mood}$ and $L_{metadata}$ denote the losses corresponding to mood and metadata prediction tasks, respectively. The symbol $\alpha$ is a hyperparameter, and it is the weightage given to the metadata prediction task (secondary task). Setting $\alpha$ to zero disables multi-task learning, and the model degenerates purely into a mood prediction model. It should be noted that metadata is required only at the time of training, and at the time of prediction, we ignore the activations of the metadata output layer. We discuss the choice of the value of $\alpha$ in Section 4.3.

The model described above works with 3-sec audio segments and outputs probability values for the different mood labels. A song is mostly longer than that, and the inference (at the time of validation and testing) for a song is performed by averaging the results of all the 3-sec non-overlapping segments in a song. For each segment of a song, we run the model and compute the probabilities of the mood labels. The probability of a specific mood label for a song is then computed by averaging the probabilities for this mood label over all the segments of the song.

We report our results on three different datasets. The first two are publicly available, and the third one is a private, regional dataset. Following is a brief summary of the datasets:

MTG-Jamendo dataset (MTG) contains 18,486 songs with multi-label annotations of 59 moods. For each song, artist information and full audio are made available. The audio data is available in MP3 format with 320 kbps bitrate and 44.1 kHz sample rate.

MagnaTagATune dataset (MTT) contains multi-label tag (including moods) annotations of 25,877 audio clips of about 30 sec. The audio clips have been created from 5405 songs provided by the Magnatune label. For each audio clip, over 180 unique tags are available along with artist information. The audio clips are available in MP3 format with 32 kbps bitrate and 16 kHz sample rate.

Internal dataset. Models trained with the above datasets perform poorly for songs of regional languages such as Hindi and Punjabi. To this end, we manually created our in-house dataset spanning 22,015 songs. This dataset is annotated with three moods: happy, sad, and energetic. We specifically focus on these three tags, as these are the most frequently queried mood labels on our platform. In addition to mood labels, artist and year information is available. The audio segments are available in MP3 format with up to 320 kbps bitrate and 44.1 kHz sample rate.

CNN based models have been successfully applied to predict mood from the audio data (Choi et al., 2016; Alonso-Jiménez et al., 2020; Pons and Serra, 2019). These models do not assume any knowledge of the music domain — the spectrograms are simply treated as images and common vision models are applied on them. On the other hand, work on musically motivated architectures (Pons et al., 2016; Pons and Serra, 2017) proposed CNN filters of various shapes to detect temporal and frequency-based features such as BPM, onsets and timbre. Although these models tend to require domain knowledge, but the benefit is that they need fewer parameters and perform competitively with lesser training data (Pons et al., 2018).

Filip Korzeniowski et al. explored mood recognition using input features derived from listening data (Korzeniowski et al., 2020) instead of spectrogram based audio features. They reported that features derived from listening data such as interactions of songs with different users are more helpful in recognizing moods versus not so much available labeled audio data.

It is interesting to note that our work does not make any assumptions about the model architecture and the input types. Our simple idea of using audio metadata in a multi-task learning setting applies to all the aforementioned models and with different kinds of input features. Moreover, the modifications required by multi-task learning only demand metadata at the training time, and it does not disturb inference at all, i.e., it does not require metadata for inference. We validate these ideas through our experiments and demonstrate that supplementing the existing models with multi-task learning (with readily available audio metadata) improves the performance of a given model.

We highlighted the importance of the task of mood recognition from the perspective of a music streaming service. We reported an observation that as a popular music streaming service we get more than 1% of search queries related to moods. Thus, making a strong case for us to understand the content mood category to be able to provide relevant search results and to also provide personalized mood-based playlists to the users.

Towards the goal of improving the mood recognition capability, we investigated a research question of whether we can leverage audio metadata such as artist and year, which is readily available, to improve the performance of mood classification models. To this end, we proposed a multi-task learning approach in which a shared model is simultaneously trained for mood and metadata prediction tasks with the goal to learn richer representations. We experimentally demonstrated that applying our technique on the existing state of the art CNNs for mood classification improved their performances consistently. We conducted experiments on multiple datasets and reported that our approach could lead to improvements in the average precision metric by up to 8.7 points.