PIANOTREE VAE: Structured Representation Learning for Polyphonic Music

In the context of deep music generation, polyphony can refer to three types of music: 1) multiple monophonic parts (e.g., a four-part chorus), 2) a single part of a polyphonic instrument (e.g., a piano sonata), and 3) multiple parts of polyphonic instruments (e.g., a symphony).

The first type of polyphonic music can be created by simply extending the number of voices in monophonic music generation with some inter-voice constraints. Some representative systems belonging to this category include DeepBach [16], XiaoIce [17], and Coconet [18]. Music Transformer [19] and the proposed PianoTree VAE both focus on the generation of the second type of polyphony, which is a much more difficult task. Polyphonic pieces under the second definition no longer have a fixed number of “voices” and consist of more complex textures. The third type of polyphony can be regarded as an extension of the second type, and we leave it for future work.

Piano-roll and its variations [20, 21, 7, 22] view polyphonic music as 3-D (one-hot) tensors, in which the first two dimensions denote time and pitch and the third dimension indicates whether the token is an onset, sustain or rest. A common way for deep learning models to encode/decode a piano-roll is to use recurrent layers along the time-axis while the pitch-axis relations are modeled in various ways [20, 21, 23]. Another method is to regard a piano-roll as an image with three channels (onset, sustain and rest) and apply convolutional layers [7, 22].

Through the proposal of PianoTree VAE, we argue that a major way to improve the current deep learning models is to utilize the built-in priors (intrinsic structure) in the musical data. In our work, we primarily use the sparsity and the hierarchical priors.

MIDI-like event sequence is first used in deep music generation in performanceRNN [24] and Multi-track MusicVAE [9], and then broadly applied in transformer-based generation [19, 25, 26]. This direction of work leverages the sparsity of polyphonic data to efficiently flatten polyphonic music into an array of events. The vocabulary size of events usually tripples the vocabulary size of MIDI pitches, including “note-on” and “note-off” events for 128 MIDI pitches, “time shifts”, and so on.

However, the format of MIDI-like events lacks the proper flexibility. A few operations are made difficult due to its very nature. For instance, during addition or deletion of notes, often numerous “time shift” tokens must be merged or split with the “note-on” or “note-off” tokens being changed all-together. This has caused the model being trained inefficient for the potential generation tasks. In addition, this format has a risk of generating illegal sequences, say a “note on” message without a paired “note off” message.

Similarly, we see the note-based approaches [27, 28], in which polyphonic music is represented as a sequence of note tuples, as an alternative to the MIDI-like methods. The representation has resolved the illegal generation problem but still not revealed much of the intrinsic music structure. We argue that our work improves on the note-based approaches by utilizing deeper musical structures implied by the data. (See Section 3.1 for details.)

Recently, we see a trend in using graph neural networks (GNN) [29] to represent polyphonic score [30], in which each vertex represents a note and the edges represent different musical relations. Although the GNN-based model offers sparse representation learning capacity, it is limited by the specification of the graph structure design and it is nontrivial to generalize it for score generations.

We first define a data structure to represent a polyphonic music segment, which contains two components: 1) surface structure, a data format to represent the music observation, and 2) deep structure, a tree structure (containing score, simu_note and note nodes) showing the syntactic construct of the music segment.

Each music segment lasts $T$ time steps with $\frac{1}{4}$ beat as the shortest unit. We further use $K_{t}$, where $1\leq t\leq T$ to denote the number of notes having the same onset $t$. The current model uses $T=32$, i.e., each music segment is 8-beat long.

We use the surface structure of polyphonic music as the model input. The VAE architecture is built upon the deep structure.

We denote the music segment in the proposed surface structure as $x$ and the latent code as $z$, which conforms to a standard Gaussian prior denoted by $p(z)$. The encoder models the approximated posterior $q_{\phi}(z|x)$ in a bottom-up order of the deep structure. First, note embeddings are computed through a linear transform of pitch-duration tuples. Second, the note embeddings (sorted by pitch) are then embedded into simu_note using a bi-directional GRU [33] by concatenating the last hidden states on both ends. With the same method, the simu_note embeddings (sorted by onsets) are summarized into score by another bi-directional GRU. We assume an isotropic Gaussian posterior, whose mean and log standard deviation are computed by a linear mapping of score. Algorithm 1 shows the details.

The decoder models $p_{\theta}(x|z)$ in a top-down order of the deep structure, almost mirroring the encoding process. We use a uni-directional time-axis GRU to decode simu_note, another uni-directional (pitch-axis) GRU to decode note, a fully connected layer to decode pitch attributes, and finally another GRU to decode duration attribute starting from the most significant bit. Algorithm 2 shows the details.

We use the ELBO (evidence lower bound) [34] as our training objective. Formally,

where $\beta$ is a balancing parameter used in $\beta$-VAE [35].

We denote the embedding size of note, simu_note and score as $e_{\text{n}}$, $e_{\text{sn}}$ and $e_{\text{sc}}$; the dimension of latent space as $d_{\text{z}}$; and the hidden dimensions or pitch-axis, time-axis and dur GRUs as $h_{\text{p}}$, $h_{\text{t}}$ and $h_{\text{d}}$ respectively. In this work, we report our result on the following model size: $e_{\text{n}}=128$, $e_{\text{sn}}=h_{\text{p},\text{dec}}=2\times h_{\text{p},\text{enc}}=512$, $e_{\text{sc}}=h_{\text{t},\text{dec}}=2\times h_{\text{t},\text{enc}}=1024$, $h_{\text{d},\text{dec}}=64$ and $d_{z}=512$.

We collect around 5K classical and popular piano pieces from Musicalion²²2Musicalion: https://www.musicalion.com. and the POP909 dataset [36]. We only keep the pieces with $\frac{2}{4}$ and $\frac{4}{4}$ meters and cut them into 8-beat music segments (i.e., each data sample in our experiment contains 32 time steps under sixteenth note resolution). In all, we have 19.8K samples. We randomly split the dataset (at song-level) into training set (90%) and test set (10%). All training samples are further augmented by transposing to all 12 keys.

We train four types of baseline models in total using piano-roll (Section 2.2) and MIDI-like events (Section 2.3) data structures. As a piano-roll can be regarded as either a sequence or a 2-dimensional image, we couple it with three neural encoder-decoder architectures: a recurrent VAE (pr-rnn), a convolutional VAE (pr-cnn), and a fully-connected VAE (pr-fc). For the MIDI-like events, we couple it with a recurrent VAE model (midi-seq). All models share the same latent space dimension ($d_{z}=512$). Specifically,

• •

  The piano-roll recurrent VAE (pr-rnn) model is similar to a 2-bar MusicVAE proposed in [4]. The hidden dimensions of the GRU encoder and decoder are both 1024.

• •

  The piano-roll convolutional VAE (pr-cnn) architecture adopts a convolutional–deconvolutional architecture. The encoder contains 8 convolutional layers with kernel size $3\times 3$. Strided convolution is performed at the 3${}^{\text{rd}}$, 5${}^{\text{th}}$, 7${}^{\text{th}}$ and 8${}^{\text{th}}$ layer with stride size $(2\times 1),(2\times 3),(2\times 2)$ and $(2\times 2)$ respectively. The decoder adopts the deconvolution operations in a reversed order.

• •

  The piano-roll fully-connected VAE (pr-fc) architecture uses a time-distributed 256-dimensional embedding layer, followed by 3 fully-connected layers with the hidden dimensions [1024, 768] for the encoder. The decoder adopts the counter-operations in a reversed order.

• •

  The MIDI-like event recurrent VAE (midi-seq) adopts the recurrent model structure similar to pr-rnn. Here, the event vocabulary contains 128 “note-on”, 128 “note-off” and 32 “time shift” tokens. The embedding size of a single MIDI event is 128. The hidden dimensions of the encoder GRU and decoder GRU are 512 and 1024 respectively.

For all models, we set batch size $=128$ and use Adam optimizer [37] with a learning rate starting from 1e-3 with exponential decay to 1e-5. For PianoTree VAE, we use teacher forcing [38] for decoder time-axis and pitch-axis GRU and for other recurrent-based baselines, we use teacher forcing in the decoders. The teacher forcing rates start from 0.8 and decay to 0.0. PianoTree VAE converges within 6 epochs, and the baseline models converge in approximately 40 to 60 epochs.

The objective evaluation is performed by comparing different models in terms of their reconstruction accuracy of pitch onsets and note duration[39, 40], which are commonly used measurements in music information retrieval tasks. For note duration accuracy, we only consider the notes whose onset and pitch reconstruction is correct. Table 1 summarizes the results where we see that the PianoTree VAE (the 1${}^{\text{st}}$ column) is better than others in terms of F1 score for both criteria.

Figure 4 shows the latent note space by plotting different note embeddings after dimensionality reduction by PCA (with the three largest principal components being reserved). Each colored dot is a note embedding and a total of 1344 samples are displayed; note pitch ranges from C-1 to C-8 and note duration from a sixteenth note to a whole note.

We see that the note embeddings have the desired geometric properties. Figure 4 (a) & (b) show that at a macro level, notes with different pitches are well sorted and form a “helix” in the 3-D space. Figure 4 (c) further shows that at a micro level, 16 different note durations (with the same pitch) form a “fractal parallelogram” due to the binary encoding of duration attributes. One of the advantages of the encoding method is the translation invariance property. For example, the duration difference between the upper left cluster and the lower left cluster is 8 semiquavers, so is the difference between the upper right and lower right cluster. The same property also applies to the four smaller-scale parallelograms.

Figure 5 is a visualization of the latent chord space by plotting different simu_note embeddings under PCA dimensionality reduction. Each colored cluster corresponds to a chord label realized in 343 different ways (we consider all possible pitch combinations within three octaves, with a minimum of 3 notes and a maximum of 9 notes). The duration for all chords is one beat.

The geometric relationships among different chords are consistent and human interpretable. In specific, Figure 5 (a) shows the distribution of 12 different major chords, which are clustered in four different groups. By unfolding the circle in a counterclockwise direction, we can observe the existence of the circle of the fifth. Figure 5 (b) is the visualization of seven C major triad chords: forming a ring in the order of 1-3-5-7-2-4-6 degree in the counterclockwise direction.

Latent space traversal [4, 41, 5] is a popular technique to demonstrate model generalization and the smoothness of the learned latent manifold. When interpolating from one music piece to another in the latent space, new pieces can be generated by mapping the representations back to the signals. If a VAE is well trained, the generated piece will sound natural and form a smooth transition.

To this end, we invite people to subjectively rate the models through a double-blind online survey. During the survey, the subjects first listen to a pair of music, and then listen to 5 versions of interpolation, each generated by a model listed in Table 1. Each version is a randomly selected pair of music segments, and the interpolation is achieved using SLERP[42]. Since the experiment requires careful listening and a long survey could decrease the quality of answers, each subject is asked to rate only 3 pairs of music, i.e., $3\times 5=15$ interpolations in a random order. After listening to the 5 interpolations of each pair, subjects are asked to select two best versions: one in terms of the overall musicality, and the other in terms of the smoothness of transition.

A total of n = 33 subjects (12 females and 21 males) with different music backgrounds have completed the survey. The aggregated result (as in Figure 6) shows that the interpolations generated by our model are better than the ones generated by baselines, in terms of both overall musicality and smoothness of transition. Here, different colors represent different models (with the blue bars being our model and other colors being the baselines), and the height of the bars represent the percentage of votes (on the best candidate).

In this section, we further explore whether the polyphonic representation helps with long-term music generation when coupled with standard downstream sequence prediction models. (Similar tasks have been applied to monophonic music in [43] and [6].)

The generation task is designed in the following way: given 4 measures of piano composition, we predict the next 4 measures using a Transformer decoder (as in [44]). We compare three different music representations: MIDI-like event sequence (Section 2.1), pretrained (decoder) simu_note embeddings, and latent vector z for every 2-measure music segment (without overlap). Here z is the mean of the approximated posterior from the encoder. For all three representations, we use the same Transformer decoder architecture (outputs of dimension = 128, number of layers = 6 and number of heads = 8) with the same training procedure. Only the loss functions are correspondingly adjusted based on different representations: cross entropy loss is applied to midi-event tokens and MSE loss is applied to both simu_note and latent vector z. We use the same datasets mentioned in Section 4.1 and cut the original piano pieces into 8-measure subsequent clips for generation purposes. We still keep 90% for training and 10% for testing.

We then invited people to subjectively rate different music generations through a double-blind online survey (similar to the one in Section 4.6). Subjects are asked to listen to and rate 6 music clips, each of which contains 3 versions of 8-measure generation using different note representations. Subjects are told that the first 4 measures are given and the rest are generated by the machine. For each music clip, subjects rate it based on creativity, naturalness and musicality.

A total of n = 48 subjects (20 females and 28 males) with different music backgrounds have participated in the survey. Figure 7 summarizes the survey results, where the heights of bars represent means of the ratings and the error bars represent the confidence intervals computed via within-subject ANOVA [45]. The result shows that simu_note and latent vector z perform significantly better than the midi-event tokens in terms of all three criteria (p < 0.005).

Besides the aforementioned generation task, we also iteratively feed the generated 4-measure music clips into the model to get longer music compositions. Figure 8 shows a comparison of 16-measure generation results using all three representations. The first 4 bars are selected from the test set, and the subsequent 12 bars are generated by the models. Generally speaking, using simu_note and latent vector z as data representations yields more coherent music compositions. Furthermore, we noticed that long generations using the simu_note representation tend to repeat previous steps in terms of both chords and rhythms, while those generations using the latent vector z usually contain more variations.