Melody Structure Transfer Network: Generating Music with Separable Self-Attention

The self-attention can be viewed as a generalization of self-similarity matrix (SSM) [?]. SSM can reflect the self-similarity structure of a music piece, and has been used for music segmentation and motif discovery [?; ?], and music generation [?]. However, SSM can only capture the repetition patterns. According to the composition theory [?], there are more than 10 kinds of techniques e.g. transposition, compression, expansion, mirror for developing motif into melody.

We further identify the connections between self-attention and the sequence structure. The multi-head self-attention mechanism allows modelling of multi-level dependencies from input sequences [?]. Each position in the decoder attends to all positions in the decoder up to (including) that position. This is somewhat similar to the composition process by human, where a prime motif (a short sequence of several notes) is first devised and then developed into a long sequence via repetitions and variations. These repetitions and variations occur at multiple levels (sub-phrases, phrases and sections) and form the multi-level structures. The multi-level dependencies learned by the self-attention mechanism is closely related to the multi-level structures.

Seeing the fundamental connection between self-attention and sequence structure, we argue that the learned dependencies in self-attention mechanism can be used to guide the generation of new music piece, and the realistic structures can be kept during generation by dependency transfer.

Now the problem becomes how to transfer the learned dependencies from the real music to generation. Specifically, the generation of token $s_{t}$ in step $t$ should attend to similar positions as the real one. To achieve this goal, we hope the calculation of self-attention can be expressed as:

$$\begin{array}[]{cl}Z=Attention(Q,K,V)&=Att_{M}\cdot V\\ &=\mathbf{f}(dep,\mathbf{g}(s))\cdot V\end{array}$$

(2)

where $Z$ is the output hidden state of a self-attention layer, and $Att_{M}$ is the attention matrix at this position. $s$ denotes the past tokens in the sequence, $dep$ is the structure dependency related variables for transfer. $\mathbf{f}(\cdot)$, $\mathbf{g}(\cdot)$ are transformations.

We further expect that $dep$ shall meet two requirements: 1) containing rich structure-related information; 2) independent from the past input sequences thus it can be transferred to different generated sequences.

For the first requirement, as the structure is closely related to positions, to force variable $dep$ to learn the structure related information, we make only $dep$ relate to the positions. Both $V$ and $\mathbf{g}(s)$ should be independent from the positions. To satisfy the second requirement, the computation of $dep$ should not include the input tokens.

Note that directly transplanting the self-attention matrix $Att_{M}$ from a training piece to the generation phrase can hardly work well. Because in the original self-attention mechanism, the calculation of attention in each layer depends on the input token at each time step (think back the calculation of query, key and value). Different input sequences will lead to different attention matrices. In this paper, we show how to implement the self-attention mechanism in Eq. 2 by separating the calculation of query, key and value. A structure embedding is introduced to help the learning of variable $dep$.

In the original self-attention mechanism [?], the computations of query, key, value all depend on the position embedding and input token embedding. Here, we separate the calculation of query, key, value, and introduce a structure hidden state $h_{d}$ and note hidden state $h_{x}$. We design two architectures to implement the separable self-attention mechanism in Eq. 2. The designed architectures are shown in Fig. 2. The calculation of the separable self-attention are:

$$\begin{array}[]{ll}q=(h_{d}+\lambda h_{x})W_{q}&k=(h_{d}+\lambda h_{x})W_{k}\\ v_{d}=h_{d}W_{d}&v_{x}=h_{x}W_{x}\\ a_{d}=Att(q,k)v_{d}&a_{x}=Att(q,k)v_{x}\end{array}$$

(3)

$$\begin{array}[]{ll}q_{x}=(h_{d}+\lambda h_{x})W_{q_{x}}&k_{x}=(h_{d}+\lambda h_{x})W_{k_{x}}\\ q_{d}=h_{d}W_{q_{d}}&k_{d}=h_{d}W_{k_{d}}\\ v_{d}=h_{d}W_{v_{d}}&v_{x}=h_{x}W_{v_{x}}\\ a_{d}=Att(q_{d},k_{d})v_{d}&a_{x}=Att(q_{x},k_{x})v_{x}\end{array}$$

(4)

where $Att(q,k)=Softmax\left(\frac{qk}{\sqrt{D_{h}}}\right)$.

Note Eq. 3 and 4 specify the separable self-attention mechanism. They corresponds to the architectures in Fig. 2(b) and (c), respectively (We omit the superscript $l$ for clarity). Different from the original self-attention block where the only hidden state $h^{l}$ encode all the information from input, we introduce separated structure hidden states $h_{d}$ and the note hidden states $h_{x}$. The $v_{x}$ and $v_{d}$ at each block are calculated from the input note hidden states and structure states, respectively. In Fig. 2(b) the attention coefficients for $v_{x}$ and $v_{d}$ are the same, the $q$ and $k$ could be related to both the note states and structure states. In Fig. 2(c) the attention coefficients for $v_{x}$ and $v_{d}$ are different, $q_{d}$ and $k_{d}$ are only related to the structure states, while $q_{x}$ and $k_{x}$ still could be related to both the note states and structure states.

At the first block, the original music transformer architecture [?] takes the addition of note embedding $E_{note}$ and position embedding $E_{pos}$ as input, where $h^{0}=E_{note}+E_{pos}$. We introduce a structure embedding $E_{d}$:

$$h^{0}_{d}=E_{pos}+E_{d},\quad h^{0}_{x}=E_{note}.$$

(5)

Fig. 2(b) and (c) are two exemplar implementations of Eq. 2. Others may design more architectures for Eq. 2.

The structure embedding in Eq 5 is defined as $E_{d}=\{e_{d_{t}}|t=0,1,2,...,T\}$, where $e_{d_{t}}$ is a trainable vector which denotes the embedding at $t^{th}$ time step for a training piece. It is computed as

$$e_{d_{t}}=w_{t}\cdot e_{d}$$

(6)

where $w_{t}$ is the transform vector of size $(n_{state},)$ at $t^{th}$ time step, $e_{d}$ is the structure representation of a training piece. Each music piece in the original training set is assigned a structure representation of size $(n_{state},)$ randomly. $n_{state}$ is the length of hidden state in the transformer architecture.

We augment the music pieces in original training set using the pitch transposition technique [?], which is to transpose the pitches totally by $\{-6,-5,...,5,6\}$ steps. This augmentation changes the tonality of the original music, but not the structure. Thus the augmented versions share the same structure embedding $e_{d}$ with the original ones. The $e_{d}$ for each training piece is learned during training. Then the learned $e_{d}$ can be transferred to the generation stage.

Someone may argue that a conditional music transformer (CMT) [?] may also work for the structure transfer. In this setting, the structure embedding is used as a control code, which is prepended ahead of the music sequence embeddings. This code provides a control over the generation process. However, the control embedding can only affect the global dependencies via its keys at that single position. Therefore, it has difficulty in controlling the subtle dependencies between all the tokens along the sequence and thus lead to unstable structure transfer. We also implement a conditional transformer, as compared in our experiments.

We adopt the frame-based event representation scheme in [?]: time is quantized using uneven sub-division. Unlike other works quantizing uniformly with the sixteenth note, the work [?] designs the uneven subdivision scheme, where each beat is divided into 6 uneven ticks. This scheme allows to represent note sequences including triplets efficiently. In addition, real note names are used for generation of readable sheet music.

We convert the sheet music into note token sequences using the above scheme. Here we introduce two additional tokens, the tonic and mode, to represent the key signature of sheet music. These two tokens are put in front of the note sequences. Example of our representation is shown in Fig. 3.

We test our proposed Melody Structure Transfer Net (MSTN) on two public datasets. The Session dataset [?] and the Wikifonia dataset [?]. The Session dataset consists of monophonic folk melodies in the Scottish and Irish style taken from the Session website [?]. The original Session dataset consists of more than 48K melody pieces with ABC notation format [?]. In this paper, we take the same subset as [?] from the Session dataset. Only melodies with 4/4 time signature are considered, and the pieces which consist of notes less than sixteen note are dropped. This results in approximate 21K melodies. The Wikifonia [?] contains more than 6500 lead sheets in MusicXml format [?]. We also select a subset from these music pieces according to the above standards and finally there are 3500 sheets. The chords track are dropped and only melodies are kept. These two datasets are split to $90\%$ training and $10\%$ validation. The training pieces are augmented with the pitch transposition strategy. All pieces are encoded into the sequence representation as stated in Sec. 3.6.

To demonstrate that MSTNs can transfer the templates’ structure to the generated ones, rather than simply remember the training templates, we adopt several structure related metrics [?]. We also devise several metrics to evaluate the structure similarities and sample diversity. The metrics we used are as follows:

1) Repeat Count (RC), Repeat Duration (RD) and Repeat Onset (RO): Number of repeats corresponding to various lookback values, various durations, and various onset. These three kinds metrics are proposed in [?], where only 16 bars of short music fragments are considered. We improve these measures to capture the repeat pattern across longer term by extending the lookback values (see Fig. 4). These three metrics are calculated for the duration repeats (-D) and duration-interval repeats (-DI) separately, leading to 6 metrics – see computing details in [?].

2) Pitch Distribution (PD) and Duration Distribution (DD): The statistics of various pitches and durations. These two metrics are popular in music generation tasks to compare the pitch and duration distribution similarity between the generated music sets and training sets [?].

3) Rhythm Structure Similarity (RSS) and Interval Structure Similarity (ISS): Here we adopt the self-similarity matrix (SSM), which is commonly used for the music structure analysis [?]. Despite that SSM is often calculated from the music audio, some researches [?] applied it to the sheet music. In this paper we compute the SSM for rhythm and intervals separately at the bar level. For each note in a bar, we can calculate its duration beat length and its interval from the previous note. We represent the rhythm pattern in a bar as list of (start time, duration) tuples, and represent the interval pattern in a bar as list of intervals:

$$\begin{array}[]{ll}Rhy(j)=[(st_{1},d_{1}),(st_{2},d_{2}),...,(st_{m},d_{m})]\\ Intv(j)=[iv_{1},iv_{2},...,iv_{m}]\end{array}$$

(7)

where $m$ is the number of notes in a bar, $st_{l}$ denotes the start beat position of the $l^{th}$ note in the bar, and $d_{l}$ denotes its duration. $iv_{l}=interval(iv_{l-1},iv_{l})$ denotes the staff interval from the previous note to current note. If there is no previous note or the interval is inapplicable (eg. one of the note is ’rest’), the interval value is assigned as $null$.

With the rhythm and interval representation for each bar, we are able to calculate the rhythm and interval similarity between any two bars. And then the SSM for any given music piece can be obtained. We compute the rhythm similarity between two bars as the ratio between the duration of matched tuples and the total duration of a bar, and the interval similarity as the ratio of length of max matching string in two bars to the max length of these two bars:

$${\begin{array}[]{ll}SSM_{Rhy}(i,j)=\frac{\Sigma_{l\in ms}d_{l}}{Dur},ms=\{l|Rhy(i)(l)\in Rhy(j)\}\\ SSM_{Intv}(i,j)=\frac{max\ match\ length(Intv(i),Intv(j))}{maxlength(Intv(i),Intv(j))}\end{array}}$$

(8)

where $i,j$ are indexes of two bars, $Dur$ is the total duration of a bar, and for music pieces of 4/4 time signature, $Dur$ is 4 beat length. Then we can obtain a SSM of size $L*L$ for a music piece of $L$ bars. The SSM is a symmetric matrix as the similarity between bar $i$ and $j$ equals to is counterpart.

After computing the SSM for each music piece, we are able to evaluate the structure similarity between any two music pieces by compare their rhythm and interval SSM. The Rhythm and Interval Structure Similarity are calculated as the root mean square error between their corresponding SSMs.

4) Rhythm Duplicate Rate (RDR) and Interval Duplicate Rate (IDR): It is necessary to check to what extent the model could generate new samples, rather than simply remember the training samples. It is also important to evaluate the diversities between the samples generated from a given template structure. Here we propose the rhythm duplicate rate and the interval duplicate rate to evaluate the diversity between the template and its generated samples ($RDR_{TAB}$ and $IDR_{TAB}$), as well as the diversity between the samples generated from the same template ($RDR_{AB}$ and $IDR_{AB}$). These duplicate rates are computed as the ratio of the number of bars with same rhythm or intervals between two samples to the number of bars in the samples.

We evaluate the two versions of MSTN (i.e. MSTN-C and MSTN-U, corresponding to Fig. 2(b) and (c), respectively. We also evaluate the implemented conditional music transformer (CMT). A baseline music transformer (MT) is also implemented and evaluated on several applicable metrics for clarity. All these models consist of 7 layers of self-attention blocks with 8 heads and 256 hidden states. The learning rate is $2e-5$ with 5 epochs of warm up. The max length is set to 100 bars (2400 time steps). The models are trained for 100 epochs on Wikifonia and 50 epochs on Session dataset. The parameter $\lambda$ in Eq. 3 and Eq. 4 is chosen as $0.1$ after we coarsely browse several values ranging from $0.001$ to $1$.

For evaluation, we generate two samples for each given template, which enables to compute the duplicate rates between samples generated from the same templates. The structure similarities between samples and their templates are calculated separately on the two samples and then averaged. Each of the music pieces in the training set has been taken as template. We generate samples in two modes: 1) free composition mode: trained models generate samples freely, without any given prime; 2) continuation mode: trained models generate subsequent sequences from a given one-bar motif. The metrics are computed on each set of generated samples, and then averaged on the whole dataset.

The statistics of the repeat-related metrics, as well as the pitch and duration distribution are given in Fig. 4. The KL-Divergences on these distributions between the dataset and the generated set from each model are shown in Table 1. The calculation of these KL-divergences is the same as that in [?]. To better understand the performance, we also present the evaluation statistics on the baseline music transformer model (MT). Table 1 shows that the samples generated by CMT and the proposed MSTNs are closer to the dataset than those of the baseline MT. It is reasonable because CMT and MSTNs have more constraints than the baseline MT during the training process. These statistics do not differ much between the samples generated by CMT and MSTNs.

Table 2 shows the statistics (the lower the better) on structure similarities (RSS, ISS) and duplication rates (RDR, IDR). The samples of MSTNs and CMT generated in the free composition mode and the continuation mode are evaluated. In both free composition mode and continuation mode, MSTN-C and MSTN-U could generate samples which have more similar rhythm structures and interval structures to the template melodies than the CMT. This means that the proposed MSTNs can transfer the structure from the template melodies to the generated pieces well, while the CMT fails.

As the CMT samples has weak relationship with templates, their rhythm and interval duplication rates with the templates are also lower than MSTNs samples. But as to the duplication rates between samples, the CMT samples in free composition mode dose not perform better than MSTNs, which means that in free composition mode, the samples’ diversities of proposed models does at least as well as CMTs. In continuation mode, it is also reasonable that MSTNs’ samples are less diverse than CMTs’ because the structure constraints of MSTNs are more tense than CMTs. Given the same motif, MSTNs will try to develop the motif according to the composition techniques of the template melody, which will definitely lead to similar generated samples. As for CMT, the model does not learn the structure embedding well, so it will not develop the motif according to the template composition techniques, and thus lead to diverse samples. So the lower duplication rates of CMT samples in the continuation mode also show that the CMT model can not transfer the structure well. Fig. 5 show the melodies and theirs SSMs. The SSMs in this figure are the addition of the rhythm SSM and the interval SSM. Figure 5(a) shows a template melody. (b) and (d) are melodies generated using MSTN-U and CMT in free composition mode, respectively. (c) and (e) are melodies generated using these two model in continuation mode, where the first bar is the given motif. It is obvious that the SSM of MSTN-U melodies are much similar with the template’s SSM, and SSM of CMT melodies are much different with that of template. We can see from (c) that the given motif is developed in to melody with a composition technique which is much similar to the template, where the bar 9-16 is similar to the bar 1-8, bar 21-24 is similar to bar 17-20, and bar 25-32 is similar to bar 9-16.

These results verify that the proposed MSTNs can transfer the structure of template melodies to generate new samples. The generated samples are of good diversity in the free composition mode, and their diversity will drop if the samples are generated from the same prime. MSTNs perform much better than the CMT. We upload 3000 sets of generated results in music xml format for the Wikifonia dataset as supplement materials. These pieces are generated with MSTN-U model in continuation mode, where the first bar is the given motif. Each folder is named with the corresponding template’s name in the Wikifonia dataset, and for each template two samples are generated.