HouseX: A Fine-Grained House Music Dataset and Its Potential in the Music Industry

In the past decades, previous researchers annotated a few music genre datasets. Among them, one of the most widely used is called GTZAN [4]. Its audio data includes 10 genres, with 100 audio files each genre, all having a duration of 30 seconds. Besides, they have also provided the converted mel-spectrograms, together with the mean and variance of multiple acoustic features, including the spectral centroid and the root mean square (RMS). Defferrard et al. have built the FMA dataset [5] which includes 8 genres and is also annotated in the style of GTZAN. Other works like the MedleyDB dataset [7] contains classical, rock, world/folk, fusion, jazz, pop, musical Theatre and rap songs. Besides genre labels, it also provides processed stems for its 194 tracks.

However, most music genre datasets, including the ones mentioned in the last paragraph, are annotated with general genres like pop, country, rock, etc. One shortcoming is that models trained on these datasets could hardly distinguish the differences among songs within a general genre. In one mixtape or one live set, the songs usually belong to one or few general genres. For example, all the tracks selected in our dataset would be classified as ”Electronic” in datasets like the FMA dataset [5]. Therefore, using models trained on these datasets, the retrievable information would be extremely confined. In view of such limitation, our work is devoted to building datasets that feature electronic music belonging to the same general genre but differ in their sub-genres.

After our evaluation on its wide usage on music festivals, we have chosen house music as the general genre of our dataset, naming it ”HouseX”. The dataset includes 4 sub-genres of house music, with 40 tracks each. These 160 tracks are all produced by the most popular DJ/producers around the world, including [8, 9].

Though having similar or even identical tempos, these four sub-genres of house music possess substantially different characteristics. Future house tracks include plenty of electronic sounds, and their groove can be very bouncy (note that a major sub-branch of future house is future bounce). Bass house music is also filled with synthesized electronic elements. However, the sounds in bass house are generally dirtier and more distorted. These tracks could be so aggressive that one can hardly discern a clear melody line. Though sometimes bass house could sound similar to future house, future house tracks usually have a bright lead sound while bass house would possess mush more richness and saturation on the low end of the frequency spectrum. Progressive house is known to be uplifting and emotional. Beside the widely used supersaw lead sound, many progressive house producers would use real instruments like the piano, the guitar and strings to make the sounds more natural and organic. Melodic house tracks are characterized by the use of pluck sounds and pitch shifting techniques. These elements produce a happy and rhythmical atmosphere.

We have counted a few statistics of our dataset. The tempo of these $160$ tracks range from $122$ BPM to $130$ BPM, and most of them equal $128$. The total duration of them is around $9$ hours and $8$ minutes. Most house tracks have a time signature of $4$/$4$. Therefore, suppose a track has a tempo of $128$ bpm, then one bar in the track has a duration of $1.875$ seconds. We set the length of the sound clip for each mel-spectrogram as $1.875$ seconds, which are is approximately $1$ bars for every track. In total, we derive around $17480$ mel-spectrograms for classification. Some detailed statistics are presented in Tab. 1.

Additionally, we have annotated the timestamps that indicate the start and the end of the drop of each track, using Label Studio [10]. Since the drop is the climax of a track, with respect to energy and emotion, it is the section where the characteristics of its sub-genre are best exhibited. For example, the metallic lead sound in the drop and heavily distorted bass would characterize a bass house track, while the supersaw lead layered with strings and piano would highlight an emotional melody line of a progressive house track. Therefore, we specifically annotated the start and the end of the drop of each track in our dataset, as shown in Fig. 1. These drops would form a subset of the original dataset. In order to make this subset balanced, we only choose one drop with a length of $16$ bars, from each track. Normally, a track includes two drops. We choose the first one unless it is not long enough. Sometimes, especially for emotional progressive house tracks, the first drop only contains $8$-bars while the length of the second one is doubled. In these cases, we annotate the start and the end of the second drop. Though fairly smaller than the whole set, this subset that only contains drops still yields a considerable amount of mel-spectrograms, as listed in Tab. 1. Another thing to note is that the split of the whole set should be done before the conversion from audio to mel-spectrograms. That is to say, we should split the set of tracks, rather than the set of pictures, into $3$ parts, for training, validation and testing respectively. The reason is that electronic music usually contains many loops, and if we split the set of pictures, for a mel-spectrogram in the test set, similar mel-spectrograms would almost surely appear in the training set, which makes the validation and testing process meaningless.

Rigorously, the classification task is defined as follows. A track of house music is given as a wave file and the goal is to assign a proper sub-genre to slices of the track, from the $4$ sub-genres mentioned in the data collection part. Since our wave files are stereo, for each track, we get an array of shape $(2,T)$ where $T$ is the total number of samples, and the sample rate of the waveform $sr=22050$ (Hz). To get reasonable amount of mel-spectrograms, we set the number of bars per slice as $N_{bar\_per\_slice}=1$. Then, since all tracks in our dataset have a tempo of $128$ BPM (or close to $128$ BPM) and a time signature of $4/4$, we have

, as the number of samples for each slice. Then, without overlapping, we cut the track into $N=\lfloor\frac{T}{L}\rfloor$ slices, each as an array with shape $(2,L)$. Using functions implemented in Librosa [11], we convert these $N$ Numpy arrays into $N$ mel-spectrograms, each with a shape of

, where $n\_mels=128$ and $hop\_length=512$ as default. After saving these mel-spectrograms rendered by the $specshow$ function of Librosa, we can load them as JPEG files, transform them into PyTorch tensors [12] and proceed with the standard image classification workflow. A few saved mel-spectrograms are shown in Fig. 2 as examples.

As shown in the Fig. 3, the pipeline consists of two major components. The first step is the conversion from sliced sound waveforms to mel-spectrograms, and the second procedure is to feed mel-spectrograms through a neural network to get the output scores for these mel-spectrograms. Each output score is a $(4,)$ vector.

In the training phase, we iterate through the training set to minimize the negative log likelihood loss between the output logits and the label tensor. Denoting $\mathcal{D}$ as the dataset and $\mathcal{C}$ as the set of labels, mathematically, we need to minimize the objective

, where $\theta$ represents the parameters of the model, each $(x,\;y)$ is a image-label pair, and $p$ are the predicted probabilities and $\mathbb{I}$ is the indicator function.

In the inference phase, we apply $\arg\max$ to the logits to get the sub-genre that is most probable for a slice of a track. Furthermore, if we need to predict the sub-genre of the entire track, the simplest method is to add a voting classifier on top of the predictions of slices. Other approaches include applying a weighted sum over the logits of slices of one track. The weights can even be set as trainable parameters, since some parts of a track like the intro can sometimes be less significant, while other parts like the drop would be crucial.

We have tried several architectures, including 3 traditional convolutional neural networks [14, 13, 15], 2 lightweight convolutional neural networks [16, 17] and a vanilla vision transformer [2] as backbones for the classification task. For these 5 CNN architectures, we put them before one dropout layer [18] and one linear layer from $\mathbb{R}^{1000}$ to $\mathbb{R}^{1024}$. The input dimension is $1000$ since the ImageNet dataset [6] contains $1000$ categories. Besides, we have also train a $6$-layer vanilla vision transformer from scratch, which is built upon the attention mechanism instead of convolutional kernels. The details of our implementation can be viewed by the link provided in the introduction.

A straightforward application is the automation of certain properties, such as the colors of lights, in the visualization of a mixtape. Normally, we want the real-time colors of the lights to reflect the atmosphere of the currently playing segment of the mixtape. Therefore, visual effects engineers would manually control the colors of lights throughout the whole mixtape. Using our model, the control of colors can be automated, which would bring more flexibility. For example, usually a DJ would fix the track list to be played before a show begins, to make sure the music matches up with the visual effects. Using our approach, the DJ performing onstage would be given more freedom to decide what to play next according to the reaction of the crowds, rather than the predetermined track list. We believe that such AI-assisted visualizer would bring much convenience for artists as a new paradigm.

As a result of our survey on the correspondence between music genres and colors, we have pre-defined a mapping from sub-genres of house music, to triple tuples of colors, which could be modified in practice. Note that the following mapping is not completely the same as the one determined by choosing the top-$3$ colors for each sub-genre according to the survey. We have made slight modification on the colors of progressive house and future house to make the visualization more contrastive.

• •

  Future House: blue, red, purple.

• •

  Bass House: black, white, purple.

• •

  Progressive House: orange, yellow, purple.

• •

  Melodic House: green, blue, yellow.

Then, after building a basic sci-fi environment in Blender [21], we have rendered a demo for house music mixtape visualization, which can be viewed via this link. Note that this small set (mixtape) contains $4$ songs, mixed together in FL Studio 20 [22]. These $4$ songs are future house, bass house, progressive house and melodic house in chronological order. The screenshots of the rendered results are shown in Fig. 8.

Besides, this model can be integrated in fine-grained music recommendation system. Currently, many popular music streaming services have deployed recommendation systems for users to easily look for music that matches their appetite. With our approaches that extract information on the sub-genre level, the accuracy, or precisely, the level of customization of recommendation service could be brought to the next level.