Transfer of knowledge among instruments in automatic music transcription

Automatic music transcription (AMT) is a challenging task in the musical information retrieval domain. It is the process of converting music recordings into symbolic representations such as music sheets or MIDI files. The problem is especially visible for polyphonic music, where multiple frequencies affect each other, giving a result which is hard to estimate using simple algorithms for time-frequency analysis. Current works often use datasets, which are hard to obtain and maintain. For instruments without any electronic interface, the possibility of gathering an extensive dataset containing real-world recordings with transcription is limited.

This work focuses on discovering the potential of using a model trained on synthesized data for automatic music transcription. These data are easy to generate and could be produced on demand. The intuition behind this idea is related to the possibility of universally recognizing pitch and rhythm by the human ear - it performs similarly no matter the kind of audio source. Synthesized instruments may contain fewer noises than the audio recorded via microphone. However, a model trained on such data may focus more on frequency analysis tasks, which is a good base for fine-tuning it to real-world data.

During the presented research, we trained the U-net model with BiLinear LSTM on software-synthesized samples containing timbre from different instruments. We improved the model’s performance in the target domain based on real-world recordings that prove that the model trained on synthesized instruments could generalize transcription in note and frame metrics across different datasets after zero-shot transfer. We may use it for quick adaptation to another real-world recording dataset competing with models pre-trained on recordings of the specific instrument.

This chapter focuses on the description of currently used models for transcription and audio processing methods. Later we describe data representation and metrics used in the existing literature. Current research often focuses on developing other models [2], and methods [7] to improve standard automatic music transcription metrics or find a way to perform unsupervised [13] or semi-supervised [6] automatic transcription. It may result in analyzing different timbres and instruments less underlined. The importance of timbre in automatic music transcription was presented by Hernandez-Olivan et al. in [12], where they proved that for different instruments, the estimation of source frequency f0 might be a challenging task. In [4], authors demonstrated that neural network-based music transcription depends on audio input representation because different spectrograms may pass different information. Modern deep learning models offer the possibility to perform transcription for multiple instruments together with track separation [9]. Multi-instrument AMT was considered in [18], and results proved that model capable of performing transcription on one instrument is not working well for another instrument, especially when this knowledge is not passed a priori.

After the significant development of machine learning models capable of resolving many problems and approximating complex solutions, it was clear that it may be used for automatic music transcription. Most currently existing solutions are based on image representation of audio recordings in a time-frequency domain called spectrograms. Two kinds of spectrograms are used as input for existing deep learning models to visualize traits of frequencies present in audio recordings - Mel spectrogram and CQT.

Mel spectrogram is based on the result of time-frequency analysis of sound using Short Time Fourier Transform (STFT) [1] and proper alignment using a particular "mel scale", which makes the distance between two pitches equal as the listener perceives it. It is mainly used to maximize transcription accuracy, as frequencies are the essential input to train the model for AMT. Constant Q Transform (CQT) [3] is another method to achieve images containing information about played frequencies. Its properties fit well to analyze the music and expose musical information like timbre, which may be crucial for this research. We decided to use CQT as the primary spectrogram function in our experiments.

MIDI files often represent labels in datasets, which contain all information needed for a musician or software musical instrument to reproduce a similar sound. Usually, it may be represented as a table, where each row contains four parameters:

• •

  value of pitch - it is a general value used to distinguish between different musical tones,

• •

  value of velocity - this value refers to the speed or intensity with which a note or sound is played; it may be often referred to as a measure of how hard a musician strikes a key on a keyboard or plucks a string on a guitar,

• •

  note start - it is the time when the tone is started to be played by an instrument,

• •

  note end - it is time the tone is ended to be played by an instrument.

Timbre is another essential trait of music recordings. The same pitch played by different instruments may result in a different reception of the same tone. It is determined by combining many factors, including a sound’s harmonic content, sustain, attack and decay, and how different frequencies are emphasized or de-emphasized. It results in problems with transferring this knowledge because models trained on spectrograms for guitar recordings will not learn the same way of extracting pitch as the model trained on piano recordings.

Another problem with analyzing different timbres is related to available datasets. Only a few datasets contain real recordings, especially for instruments not available in electronic forms, like a piano. Audio data synthesis is present in almost every currently used dataset for AMT [11], [8], [19] - not only for the generation of new recordings but also for validation of real-world samples. The enormous dataset containing partially generated data is MAESTRO [11], where authors proposed the "Wave2Midi2Wave" approach and created a set of valuable recordings containing traits of real-world ones. There is also some research where authors try to use a semi-supervised approach for transcription to mitigate the problem of small real-world datasets [6] and create the framework for continual learning of different instruments.

Also, currently used metrics for AMT (defined in [15]) are not always enough to measure all aspects of music transcription. Simonetta et al. in [17] proposed another metric, which was correlated with perceptual measures in the outcome of the automatic music transcription model. Due to the lack of timbre generalization capability in the currently existing model, we aim to find a way to create a model which performs well in a new domain (which is related to different instruments) after zero-shot [14] transfer of weights.

As a base for our experiments, we used part of the model presented as a transcriber in [7] by Cheuck et al. for investigating the spectrogram reconstruction effect. It is composed of U-net [16], BiLSTM and a simple linear layer for classification. We decided to use that model for transcription because of a small number of parameters and relatively good results with the possibility to catch time-series dependencies via bidirectional LSTM. Similar architecture was presented by Hawthorne et al. in [10], but it contained convolutional networks instead of U-net. The exact model configuration is presented in figure 1.

We proposed a pipeline presented in figure 2 to measure the usefulness of the synthesized data. The data in experiments are randomly sampled during training to extract fixed sequence length from larger tracks, so the original distribution of data should not significantly impact training. For synthesized data generation we used training labels from GuitarSet and MAPS datasets. After audio synthesis, we performed training of the model (resulting in a model trained on Synthesized Instruments - MTOSI) and later used it for fine-tuning to MAPS and GuitarSet data. To compare transfer from MTOSI to transfer from the model trained on each dataset a similar process was performed for the model trained on MAPS dataset and fine-tuned on GuiterSet and vice-versa.

The experiments were designed to answer the following research questions:

• •

  RQ1: How does the U-net model trained on synthesized data perform on real-world recordings?

• •

  RQ2: How does knowledge transfer from the U-net model trained on synthesized data impact training on real-world recordings?

• •

  RQ3: Is the U-net model trained on synthesized data a better candidate for fine-tuning on other instruments than the U-net model trained on real-world piano recordings?