Compound Word Transformer: Learning to Compose Full-Song Music
over Dynamic Directed Hypergraphs

To apply neural sequence models such as recurrent neural networks (RNNs) or Transformers (Vaswani et al. 2017) to automatic music composition (a.k.a., symbolic-domain music generation), one has to represent a piece of music as a sequence of tokens drawn from a pre-defined vocabulary (Oore et al. 2018). Unlike the case in text, such a vocabulary usually involves tokens of various types. For example, to represent a musical score, we may need tokens that describe the content of the musical notes (e.g., pitch and duration), their placement along time, the instrument that plays each note, as well as indicators of metrical events such as the beginning of a new beat, bar (measure), or musical phrase (Wu and Yang 2020). We need such a diverse set of tokens as music is multifaceted; a type alone captures only a certain aspect of music (e.g., melody, harmony, rhythm, timbre) and cannot faithfully represent a music piece.

As different types of (musical) tokens may have different properties, modeling the dependency of these tokens might not be the same as modeling words in text. However, to our best knowledge, little work has been done to explicitly account for the heterogeneity of tokens in music. The tokens are mostly treated equally, in the same way as words in text (Huang et al. 2019; Payne 2019; Huang and Yang 2020).

We are therefore motivated to study in this paper whether we can improve sequence modeling of music by highlighting the role of token types. Our first proposal is to customize the prediction heads for tokens of different types. Specifically, using the Transformer as the main architecture of the underlying sequence model, we approach this by using different feed-forward heads for tokens of different types.

Our second proposal is to group consecutive and related tokens in a token sequence into “compound words,” and then perform sequence modeling over the resulting sequence of compound words. This is to capture the co-occurrence relationship of tokens—e.g., to generate a new musical note, we may need at least two consecutive tokens to indicate its pitch and duration; to change the tempo in the middle of a piece of music, we need a token to indicate the target tempo value, and an co-occurring time-related token to indicate the time of the tempo change. Under the proposed compound-word modeling, the individual tokens (e.g., pitch and duration) are still predicted separately with different heads. Yet, instead of predicting them at different time steps, we predict multiple tokens of various types at once in a single time step. The token embeddings of the tokens predicted at the current step are then combined and fed as the input for the next time step. Namely, the self-attention is computed over combined embeddings of individual tokens of a compound word.

From a theoretical point of view, the proposed model can be interpreted as a learner over discrete-time dynamic directed hypergraphs (Kazemi et al. 2020). Here, a graph consists of nodes that each corresponds to a token in our vocabulary. A sequence of tokens can then be viewed as a sequence of edges (each connecting two nodes), or a walk, over this graph. A sequence of compound words, in contrast, can be viewed as a sequence of hyperedges (each connecting multiple nodes) (Feng et al. 2019), over the same graph. We discuss this at greater length later in the paper.

We refer to the proposed representation as the compound word representation, or CP for short. CP can be considered as an extension of existing representations, with the following additional merits. First, it allows for fine-grained, type-specific control over the prediction heads. For example, we can now use different loss functions, sampling policies, and token embedding sizes for different token types.

Second, as a compound word represents multiple tokens at once, it requires much less time steps to generate a music piece using compound words. Namely, the sequence length of the same music piece is much shorter in CP than in existing representations. As the computational complexity of a Transformer is related to the sequence length (Vaswani et al. 2017), this makes training and inference faster, and may facilitate learning the long-range dependency in music.¹¹1For example, we can study whether the proposed model creates music with better “structureness,” or long-term repetitions (Wu and Yang 2020; Jhamtani and Berg-Kirkpatrick 2019) in the future.

Finally, the sequence length in CP is determined by the number of compound words in a sequence, not by the number of individual tokens per compound word. Therefore, it is possible to add new token types (by adding the corresponding feed-forward head) to increase the expressivity of the representation, without increasing the sequence length. This makes it easy to extend to underlying representation, though we do not explore this potential in this work.

For performance study, we consider generating expressive Pop piano music at full-song scale in both the unconditional setting (i.e., from scratch) and conditional setting (i.e., generating the piano arrangement given the lead sheet). This involves modeling fairly long music sequences for up to 10K individual tokens each. We show that, with CP, we are able to train a linear Transformer decoder (Katharopoulos et al. 2020) with music quality similar to that of strong baselines, with faster training and inference time. We provide audio examples and open source the project at a GitHub repo.²²2https://github.com/YatingMusic/compound-word-transformer

Both language and music have principles governing the organization of discrete structural elements (e.g., words or musical notes) into sequences (Patel 2003). As such, the Transformers, which have been firstly shown to work well for text generation (Child et al. 2019; Keskar et al. 2019), have been increasingly applied to music generation in recent years, by treating music pieces as sequences of discrete tokens akin to text words. We list some related papers in Table 1.

Table 1 shows that most existing work adopt a music representation derived from either MIDI-like (Oore et al. 2018) or REMI (Huang and Yang 2020), with possible addition of track- or structure-related tokens. MIDI-like and REMI differ mainly in how the advance of time is represented: the former uses [time_shift] tokens to mark the time interval (in absolute time) between note-related tokens, whereas the latter assumes symbolic timing and uses [bar] and [position] tokens to place tokens on a metrical grid that uniformly divides a bar into a certain number of positions. Neither MIDI-like nor REMI groups the tokens by token types.³³3Upon paper completion, we noticed an early but preliminary attempt of grouping tokens by (Hawthorne et al. 2018b).

Existing work also differs in the length of the attention window (see the methodology section for definition) and vocabulary size (which is data- and task-dependent). To our knowledge, our work represents the first one to consider Pop music modeling at full-song scale (involving 10k tokens per song), and to use the recently-proposed linear Transformer (Katharopoulos et al. 2020) as the model backbone.

We discuss the proposed model from a graph-theoretical point of view below. Given a vocabulary of tokens, we can construct a fully-connected static graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ (Kivelä et al. 2014) comprising nodes $\mathcal{V}=\{1,\dots,M\}$ and edges $\mathcal{E}=\mathcal{V}\times\mathcal{V}$. Each node corresponds to an individual token in our vocabulary. This way, a token sequence $\mathcal{S}_{\text{MIDI-like}}$ or $\mathcal{S}_{\text{REMI}}$ can be viewed as a sequence of edges (each connecting two nodes), or a walk, over this graph.

In CP, the vocabulary (and accordingly the graph) is augmented with a set of special tokens, denoted as $\mathcal{V}^{*}$, that includes for example type-specific [ignore] tokens and family tokens. And, a compound word consists of $K+1$ nodes, one from each of the $K$ types and an additional one from the set of family tokens. A sequence of compound words, namely $\mathcal{S}_{\text{CP}}$, therefore, involves transitions from $K+1$ nodes to another $K+1$ nodes per time step. Such a transition can be viewed as a directed hyperedge (Feng et al. 2019; Jiang et al. 2019), that connects at once $K+1$ source nodes (e.g., $cp_{t-1}$) to $K+1$ target nodes ($cp_{t}$). It is directed because the order of the nodes matters (i.e., from $t-1$ to $t$).

A sequence of compound words also forms a dynamic directed hypergraph (Kazemi et al. 2020): $\{\mathcal{G}_{1},\mathcal{G}_{2},\dots,\mathcal{G}_{T}\}$, where $\mathcal{G}_{t}=(\mathcal{V},\mathcal{E}_{t})$. Starting from an empty graph with no edges, at each time step $t>1$ we add a new directed hyperedge, labeled with the time step $t$, connecting in total $2K+2$ nodes. In practice, we have a [BOS] token (beginning of sequence) and [EOS] token (end of sequence), so the hyperedge at $t=1$ and $t=T$ connects to only $K+2$ nodes.

A neural model for graphs, or a graph neural network (GNN), can be regarded as an encoder-decoder pair (Kazemi et al. 2020; Rossi et al. 2020), where an encoder is a function that maps from a graph $\mathcal{G}$ to node embeddings $\mathbf{z}_{i},i=1\dots M$, and a decoder takes as input one ore more node embeddings and makes a prediction based on these, e.g., node classification or edge prediction. The proposed CP Transformer can therefore be regarded as a learner over dynamic directed hypergraphs, as at each time step $t$ it manages to predict the next hyperedge to be added (i.e., $\widehat{w_{t,k}}$ and $\widehat{f_{t}}$) based on the node embeddings updated from $\mathcal{G}_{<t}=\{\mathcal{G}_{1},\mathcal{G}_{2},\dots,\mathcal{G}_{t-1}\}$, or the collection of input embeddings $\mathbf{x}_{<t}=\{\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{t-1}\}$ marked with positional embeddings (i.e., edge labels on the directed hyperedges).

We note that, while we introduce the proposed methods in the context of music modeling, the idea of compound words is generic and may be applicable to sequences seen in other data domains, when multiple tokens (i.e., a hyperedge) are needed to represent a single event, entity, or object.

To test the effectiveness of the proposed methods, we implement a CP Transformer that learns to generate Pop piano music with human performance characteristics such as expressive variations in velocity (i.e., the force with which a note is played, which is related to loudness) and tempo (Oore et al. 2018; Lerch et al. 2019). We consider Pop piano for its richness and expressivity, and for offering a direct performance comparison with the Pop Music Transformer (Huang and Yang 2020) (see Table 1).

Specifically, we consider both the conditional and unconditional generation tasks. In the former, a lead sheet (i.e., a melody line and an accompanying sequence of chord labels) is given, and the model has to generate a piano performance according to that. In the latter, the model generates a piano performance of full-song length from scratch freely.

We intend to compare CP with REMI in our evaluation. We provide the implementation details below.

The experiments hereafter are conducted in the interest of a resource-constrained scenario, assuming that we only have a single GPU with 11 GB memory and are only willing to train a model for 3 days. We conjecture that this makes sense for most middle-size academic labs worldwide. Yet, to have an idea of the model performance when more resources are available, we include to the evaluation of the conditional task two settings exceeding such a specification.

We firstly compare the efficiency of the models in terms of training time, inference time, and GPU memory usage, under the conditional setting. The average result over the 50 held-out test songs is shown in Table 4.

GPU memory usage. Table 4 shows that both CP$+$linear and REMI$+$XL require $<$11 GB GPU memory for training. Accordingly, in our implementation, we train them (separately) on an NVIDIA RTX 2080 Ti GPU (with 11GB memory). In contrast, REMI$+$linear requires 17 GB GPU memory, so we train it on a TITAN GPU with 24 GB memory.

Training time. We see that REMI-based models require much longer clock time to reach a low training loss. While it takes nearly 7 days for REMI$+$XL to reduce the negative log-likelihood (NLL) of the training data to 0.27, it takes only 0.6 days for CP$+$linear to reach the same NLL. Such a training efficiency is desirable (especially given that it is on a single 2080 Ti GPU), as it makes further extensions and modifications of the model easy and affordable.

Inference time. CP$+$linear is remarkably fast, taking on average $<$30 seconds to complete the conditional generation of a song. As a song in our dataset is about 4 minutes, this is much faster than real time. In contrast, REMI$+$XL and REMI$+$linear are about 3x and 1.7x slower, respectively. CP$+$linear is fast for it generates in total 8 individual tokens (of different types) at once each time step.

Table 4 also compares the efficiency of REMI$+$XL and CP$+$linear under the unconditional setting, for which we generate also 50 songs (from scratch) and report the average inference time. We see that CP$+$linear is even faster here, requiring only $<$20 seconds to create a new song at full-song length. In contrast, REMI$+$XL is on average 7x slower.

Next, we compare the performance of the models in terms of two objective metrics, also under the conditional setting. As the goal is to generate a song given a lead sheet, we can measure whether the generated song has a melody line and chord progression similar to that in the given condition, and take that as a figure of merit. (In contrast, proper objective evaluation of unconditional generation models remains an open issue (Yang and Lerch 2020; Dong et al. 2020; Wu and Yang 2020).) Specifically, we consider:

• •

  Melody matchness. We represent the lead sheet and the correspondingly generated piano both in the REMI format and compute the bar-wise longest common sub-sequence (LCS) of the two resulting sequences $\mathcal{S}_{\text{REMI}}^{\text{LS}}$ and $\widehat{\mathcal{S}}_{\text{REMI}}^{\text{piano}}$. When two notes (each from the two sequences) have the same pitch and close onset time (within the 8-th note), we consider that as a match. We divide the length of the LCS by the number of [pitch] tokens in $\mathcal{S}_{\text{REMI}}^{\text{LS}}$ (i.e., the number of target melody notes) of that bar, and take the average value of such a ratio across all the bars of a song as a simple measure of melody matchness.

• •

  Chord matchness. The chroma vector (Fujishima 1999) represents a short-time fragment of music by the distribution of energy across the 12 pitch classes (C, C#, etc) and offers a simple way to evaluate the harmonic similarity between two fragments. We calculate the segment-wise cosine similarity between the chroma vector representing each chord label of a lead sheet (which would be binary-valued) and the chroma vector of the correspondingly generated piano segment (normalized by the maximum value so it is $\in[0,1]^{12}$), and treat the average value across time as a measure of chord matchenss.

Table 4 shows that the evaluated models all have matchness close to that of the training set, and much higher than that of the random baseline (i.e., the average matchness between a lead sheet and a random song from the test set). This suggests, while CP$+$linear is easier and faster to train than REMI$+$XL, they may generate music of similar quality. We further investigate this through a user study, which directly assesses the perceptual quality of the generated music.

We devise an online questionnaire that solicits anonymous response to the music generated by different models for both the conditional and unconditional settings. For the former, we present excerpts of 32 bars taking from one-third location of the music. For the latter, we present the full songs (i.e., when an [EOS] token is generated).⁸⁸8It turns out that the REMI$+$XL model seldom generates [EOS] tokens even when the music is already quite long (e.g., 8 minutes), so we stop it each time when it has generated 7,680 tokens. Our intention is to investigate whether CP$+$linear and REMI$+$XL indeed generate music of similar perceptual qualities.

The generated music is rendered into audio with a piano synthesizer using a free, non-professional grade sound font. Each batch comprises the result of the evaluated models in random order. A subject has to rate the music for three random batches for each setting separately, in terms of the following aspects on a five-point Likert scale. 1) Fidelity: is the conditionally generated piece similar to the reference, from which the condition lead sheet was taken from? 2) Richness: diversity and interestingness. 3) Humanness: does the piece sound like expressive human performances? 4) Correctness: perceived absence of composing or playing mistakes. 5) Structureness: whether there are structural patterns such as repeating themes or development of musical ideas. 6) Overall. As the music can be long, the questionnaire may take around 30 mins to complete.

Table 5 shows the average result from 18 subjects. We see that REMI$+$XL performs the best in the conditional setting, yet with only moderate performance gap between the models.⁹⁹9In the conditional setting, the global structure of the song to be generated is fairly outlined in the given condition (i.e., the melody). Thus, it seems sufficient for models to learn from short segments. In contrast, CP$+$linear performs (slightly) better consistently across the four metrics in the unconditional setting, suggesting it a powerful alternative to REMI$+$XL.

In this paper, we have presented a new variant of the Transformer that processes multiple consecutive tokens at once at a time step. Each individual token is associated with a token type, which is exploited by the model to customize its input and output modules. The proposed model achieves sequence compression by integrating the embeddings of the tokens, which can be seen as forming a hyperedge over a dynamic graph. We show that the new Transformer works remarkably well for modeling music, creating full-song piano of comparable perceived quality with a competing Transformer-XL based model in much shorter training and inference time.

Research on automatic music generation may infringe copyright laws and may raise concerns regarding the role of human musicians in the future. Cares have to be given regarding the fair use of existing musical material for model training, and the potential concern of “deepfaking” an existing artist’s style in computer-generated music.

We are grateful to our interns at the Taiwan AI Labs, Joshua Chang for developing the symbolic-domain chord recognition algorithm, and Yu-Hua Chen and Hsiao-Tzu Hung for helping organize the PyTorch code. We also thank the anonymous reviewers for their valuable comments.