Optimal Embedding Calibration for Symbolic Music Similarity

We release a pre-trained auto-regressive Transformer language model for symbolic music. To build the composer-centric labels, we choose the MAESTRO dataset Hawthorne et al. (2019) to train the model since the dataset contains large-scale, high-quality symbolic music data with accurate metadata. However, past works Huang et al. (2019); Simon and Oore (2017) have used a vocabulary of only 308 words (midi events), which is far less than the vocabulary of benchmark NLP language models. Additionally, the music sequence is significantly longer than the typical language model’s maximum context length. To remedy this, we employ a vocabulary aggregation technique, as illustrated in Figure 1. Each midi-event in the vocabulary is treated as a single-length word. The Sentencepiece model Kudo and Richardson (2018) is then used to aggregate and expand the vocabulary by integrating high-frequency single-event (single-length) words to multiple-event (multiple-length) words and adding them to the vocabulary. It will generate a vocabulary that includes both single-event and multiple-event words. The details of the pre-trained model are available in our publicly available code.

Let $\mathbf{h}_{l,t}\in\mathbb{R}^{H},l\in\{1,2,...,L\},t\in\{1,2,...,T\}$ denote the embedding vector of the token on layer $l$ and position $t$, where $L$ denotes the total number of Transformer layers, $T$ denotes the maximum context length and $H$ denotes the embedding size. We propose to investigate all possible combinations of the following three embedding calibration techniques.

Note that the original NATSV algorithm Mu and Viswanath (2018) is calculated in the token embedding space. Conducting principal component analysis (PCA) on all tokens in the MAESTRO corpus will be challenging. We propose, heuristically, that PCA be performed on the sentence embedding space instead.

We propose an automated method for creating labels for the purpose of evaluating music similarity. It is worth noting that the MAESTRO dataset contains precise midi events that record performance data. By intuition, the internal consistency of music composed by the same composer is often more significant than the internal consistency of music composed by different composers. As a result, music sequences by the same composer will be more similar than sequences by different composers in general. Although this is not a perfect similarity annotation and hence cannot be used for fine-tuning, it may be used to evaluate all potential calibration methods fairly. To ensure statistical significance, the number of such labels can be much greater than the human-annotated NLP datasets (as in the cases in Reimers and Gurevych, 2019).

We construct music sequences using a sliding window with a size equal to the maximum context length and a stride half the size of the window. Then we randomly select 100,000 pairs of sequences by the same composer and label them positively, and another 100,000 pairs of sequences by different composers and label them negatively. The similarity between two sequences can be measured using the cosine distance between their sentence embeddings. Then, as an evaluation metric for the calibration, we use the commonly used Spearman’s correlation between the calculated similarity and the constructed labels. Without requiring a human-annotated dataset, we may now search for the optimal combination of embedding calibration.

The main results are shown in Figure 2. “LayerAvg” on x-axis indicates $\tilde{L}$, and different markers indicates $\tilde{K}$. For simplicity of narration, we also refer to $\tilde{L}$ as “LayerAvg value”, and refer to $\tilde{K}$ as “NATSV value”. The left figure column indicates “with standard normalization (SN)”, and the right figure column indicates otherwise. All experiments are statistically significant (with $p$-value being less than $10^{-4}$). Since our model is auto-regressive, the last token will have a full attention range that covers all tokens in the sequence. As a result, we choose the last token embedding as the baseline that corresponds to the [CLS] token embedding baseline, which is marked with LayerAvg being 0 on both figure columns. The other baseline is the points with LayerAvg being 1.

As is shown in Figure 2, the optimal metric is obtained on the left column by setting LayerAvg to 8, with SN, and setting NATSV to zero. Note that the correlation metric achieves its maximum value when LayerAvg equals 8. This demonstrates that while averaging more layers, as demonstrated by Li et al. (2020), is beneficial, there is an optimal point over which the correlation metric degrades.

The performance boost provided by SN is significant. The curve with SN and NATSV set to 0 in the left figure column is substantially better in Figure 2. However, the gain with NASTV is only noticeable in a few spots and is not as large as the gain with SN. For instance, for the curve without SN in the right figure column, NASTV equals 1 is always preferable to 0, whereas NASTV equals 2 is only partially preferable to 1.

Since our model is auto-regressive, which means that the attention span of each token is different than that of the auto-encoder Transformer model, a further question arises: How does the position of the token affect its importance for similarity? We undertake the following experiment to address this question. Given the fact that the current method of calculating sentence embedding takes the average of all positions, which implies that all positions are equally important, we examine two scenarios in which the token’s importance is unequal and proportional to its position.