Large Music Recommendation Studies for Small Teams

A first challenge is that for collaborative filtering recommendation (especially for music) training data must be collected as close to the study as possible in order ensure that participant data is known by the model. Real data is also difficult to locate and obtain.

In the domain of music, LastFM continues to provide a useful API for user-song listening event (LE) data collection²²2The LastFM API documentation can be accessed at https://www.last.fm/api, though collection is not always a straightforward process. We collected a base data set from pseudo-randomly selected users to train our model by crawling the public social graph of friends lists. Contacting authors of prior research utilizing data sets which fit our needs proved to be vital in developing a method of data collection. Although their data was not up-to-date, we were able to develop our own collection method after corresponding. This data was topped-up before each subsequent study, and appropriately randomized.

An especially interesting challenge was that we needed to be able to collect data from participants as they connected to the system. This data also needed to be as current as possible.

Our solution to this problem was to use a media-tracking application. LastFM and its associated API also worked well for this purpose. For smaller pools of participants, we helped them register an account and manually monitored it over a collection period of a few weeks. For larger pools of participants we specified in recruitment and consent materials that they must have an existing account containing some minimum number of listening events/plays/scrobbles. Amazon Mechanical Turk specifically does not allow researchers to ask participants to log into any accounts, but because LastFM accounts are publicly accessible by default, data can be accessed with only a username. It is worth noting that in our case, some participants appear to have created new accounts to complete the study without being prompted to do so.

To evaluate a recommendation, participants need to be able to listen to it!

Music previews can typically be accessed without having to authenticate with a music service. We used the Spotify API to obtain 30 second previews with album art in the form of HTML iframes embedded in the page ³³3Details on embedding Spotify music previews can be found at https://developer.spotify.com/documentation/widgets/guides/adding-a-spotify-embed/. The relevant track previews were retrieved by searching for tracks using their exact song and artist names as well as the region a participant was connecting form. We discarded the small portion of tracks that did not return any results; this is likely unavoidable.

We trained two different ML models for our project, one more traditional model based on matrix factorization, and one more modern approach based on neural networks. Writing the necessary code to implement models efficiently is time-consuming and error prone. Additionally, training models on huge data sets seems infeasible due to size and dimensionality.

Using open-source libraries can save time and help alleviate the risk of errors impacting results, though they may mis-implement key algorithmic features, or be difficult to extend. Comparing multiple implementations online can help, but one must ensure to follow any licenses and reference the source.

In order to reduce the size of data, we filtered out irrelevant items and users. By filtering out tracks with 10 or fewer LEs we reduced the number of unique tracks by 82% while only decreasing LE count by 6%. Even after filtering, our Variational Autoencoder (MultVAE) was too large to fit in GPU memory, and so we trained multiple model variations concurrently using CPU’s in order to make up for lost time. Some models may simply be infeasible without access to High Performance Computing (HPC) resources. Other models may simply not be suited to real-world implementations without significant structural changes and/or pre-processing steps. This is simply the reality of evaluating models on real populations.

The size of trained models is too large to fit in memory, especially if a new model is loaded for each server connection.

The size of trained models can be very large even after removing unnecessary data (i.e., neural network optimizer information). Our trained MultVAE model was over 6.5GB in size. Luckily, online cloud computing platforms often offer specific instances with large amounts of dedicated memory at the expense of processing power. These instances are a great fit for running user-studies which will inherently have a low number of concurrent users. As of now, simple hosting services such as Heroku will be infeasible due to the memory requirements [6]. AWS Elastic Beanstalk, however, provides very cost effective solutions in the form of memory optimized EC2 instances  [7].

Even with the low number of concurrent users, there will still be some asynchronous computation required. The ideal, yet complex, solution to this problem is to decouple the longer tasks (recommendation and data collection) from the main server using a separate worker process or even server. Developing this architecture can be time-consuming, expensive, and unnecessary for such small temporary applications. We found success by limiting the server to one Python process, and running data collection and recommendation on separate threads using the built in concurrent.futures library⁴⁴4In practice, we used the Flask-Executor python library to manage our futures: https://pypi.org/project/Flask-Executor/. As live user data collection was input/output bound it did not block the server from handling requests, and as most recommendation and processing tasks utilized multi-core optimized libraries these tasks were also handled relatively quickly. This kind of architecture would certainly not work for large-scale applications, but was ideal for our small user-count study due to its simplicity and efficient use of only one compute node.