Can Machines Generate Personalized Music? A Hybrid Favorite-aware Method for User Preference Music Transfer

The term “style transfer” was proposed by [21], who transferred the style of an input image to another specific style. A large body of work has since emerged in this field [22]. Music style transfer applies the principle of image style transfer in which input is transferred to a new style while retaining content-related information. However, different from image style transfer, the representation methods, transfer methods, and transfer types are more varied and complicated in music style transfer.

In terms of representation methods, some researchers have transferred a music style directly onto a raw audio signal [14, 23, 24], which is a sound approach to representation and can be depicted as waveforms and spectrograms. Others have aimed to transfer a music style based on MIDI-format piano-roll [11, 12, 25, 26], a performance control representation method which features symbolic scores. MIDI-format piano-roll can be further depicted as a MIDI-like event matrix and beat-based event sequence. In this paper, audio music is not the scope of our research and we use beat-based event sequence to represent symbolic music since it is denser and contains more music information, such as tempo changes [27], which can be studied in greater depth in music style transfer.

With respect to transfer methods, some scholars have proposed disentangling the content and style features of music [25, 12, 14, 28, 3]. By minimizing content and style feature differences, the content features of new music should be similar to the content music while the style features of new music should be similar to the style music. For instance, Hu et al.[3] attempted to transfer music to be therapeutic based on one piece of a user’s favorite music. Yet “music style” is a notoriously fuzzy term, making it difficult to completely strip style information from input music. Others have tried to directly transfer music from one style to another without differentiating between content and style features [29, 13, 30, 31]. Overall, to avoid disentangling content and style features and improve interpretability, the proposed method provides a new way to transfer the music by altering and fixing some music events of the input symbolic music.

Regarding style transfer, most music style transfer methods focus on transferring music genres [13, 29], music timbres [24, 32], and composition styles [2]. Although these approaches have achieved satisfactory performance, they are generally trained on large music samples and do not demonstrate the favorite-aware capacity to suit a user’s personal tastes. It would therefore be inappropriate to apply these models to UPMT. Recently, Cífka et al.[28] proposed one-shot music style transfer that only requires one piece of music but entails a complicated process to generate parallel training examples for supervised learning; this procedure is time consuming, and the quality of synthetic data is uncertain. Therefore, to better fit the user’s favorite music, we propose the favorite-aware mechanism including one favorite-aware loss function and two knowledge based steps to learn unique user preferences.

The Transformer [33] has achieved satisfactory results in different generation tasks that require maintaining long-range coherence. The unique beat-bar hierarchical structure and long-range repetition of music suggest that the Transformer-based model might lead to better performance in music modeling. Transformer-based methods have recently been applied to music problems [34, 35, 36, 37]. Huang et al.[34] proposed Music Transformer to generate music and were the first to apply Transformer to music generation. They showed that a Transformer-based model can generate qualified music. However, they used MIDI-like representation methods to represent music. This method misses musical features such as tempo changes and fails to measure note duration explicitly, which further influences the performance in terms of rhythm, an important aspect of music. Different Transformer-based approaches [35, 36, 37] were subsequently proposed and have achieved better performance. Two representative methods are the Pop music Transformer (PopMT) [27] and PopMAG [38]. In PopMT, the authors proposed a new representation method called REMI and applied Transformer-XL [20] as the backbone to generate pop music to improve output performance. In PopMAG, the authors presented another novel music representation method called MuMIDI and addressed the challenge of multi-track harmony modeling. To the best of our knowledge, these methods are data-driven and are not designed for music transfer. The proposed approach applies Transformer-based methods to accomplish music style transfer by fixing and modifying certain music events at each time step.

Considering one piece of a user’s favorite music $X$ that is provided by the user, prior knowledge (e.g., the distribution of different music events and unique music patterns) can be extracted from a user’s favorite music $X$, which is denoted as $K(X)$. The goal of UPMT is that, given an arbitrary piece of music $Y$, a method $M$ with parameters $\phi$ can be used to transfer music so that the transferred new music $\hat{Y}$ contains knowledge $K$ of a user’s favorite music $X$ and preserves important information (e.g., music structure and length) from the original music $Y$. This process can be formulated as follows:

To address the aforementioned data limitation problem and enhance model performance, we propose UP-Transformer. This model utilizes music knowledge, including the distribution of music events and SMPI in a user’s favorite music, and adopts a pre-trained Transformer-XL [20] as the backbone model to transfer music. A diagram of the proposed framework appears in Fig 2. The framework includes two phases: a fine-tuning phase and a transfer phase.

Each phase proceeds as follows:

1. 1.

   The user’s favorite music $X$ is segmented into many pieces.

2. 2.

   An up-to-date music representation method called REMI [27] is applied to transfer symbolic music data into a music event sequence, which will be discussed in Section III.C.

3. 3.

   A pre-trained Transformer-XL model is used as the backbone model and will be fine-tuned based on a novel favorite-aware loss function that utilizes distribution knowledge from the user’s favorite music, to be addressed in Section III.D.

4. 4.

   In the transfer phase, two steps (i.e., event selection and event learning) will be adopted along with SMPI from the user’s favorite music and the pre-trained Transformer-XL to transfer input music into a new piece of music and satisfy user preferences; this task will be described in Section III.E.

The novelties of UP-Transformer are twofold. First, we propose a new favorite-aware loss function to fine-tune Transformer-XL based on the distribution of different music events in one piece of a user’s favorite music. Second, two music steps are implemented to regulate the transfer process based on the extracted SMPI.

In this paper, REMI [27] is used to represent MIDI scores in symbolic music. In REMI, there are eight different music events, which are defined based on note, tempo and chord information of symbolic music. Each music event includes one or more classes to reflect different values. For example, Bar (B) indicates the bar line in piano rolls. One note contains four different events, Position (P), Note Velocity (NV), Note On (NO) and Note Duration(ND), respectively. P represents one of the $q$ possible discrete locations in a bar, where $q$ indicates the time resolution adopted to represent a bar. NV is quantized into 32 levels to indicate the perceptual loudness of the note event. A collection of 128 NO events indicates the onset of MIDI pitches from 0 to 127 ($C_{-1}$ to $G_{9}$); that is, there are 128 classes in the Note On event. ND is used to indicate the length of one note, which contains a value ranging from 1 to 64. In addition, Tempo event indicates local tempo changes and is added at every beat. Tempo Class (TC) and Tempo Value (TV) are used to represent local tempo values between 30 and 209 beats per minute (BPM). Chord (Ch) events are also considered in REMI (i.e., 12 chord roots and five chord qualities). Each Tempo Class, Tempo Velocity, and Chord is preceded by a Position event. Figure 3 demonstrates the music tokenization process.

Compared to the MIDI-like event matrix, REMI [27] explicitly defines different events in sequence. REMI thus includes more information, such as tempo changes and chord information, and produces a denser representation. Further, each note is grouped into four consecutive events – P, NV, NO, and ND – which affords us more flexibility in manipulating the sequence.

Previous work [39] has shown that Transformer-based models can benefit from pre-training techniques on large-scale music datasets. Therefore, rather than using CNN-based style transfer techniques from the image domain [40, 41], we begin with a Transformer-XL [20] model with a default attention mechanism that was pre-trained on a pop music dataset. Transformer-XL introduces the notion of recurrence and revises the positional encoding scheme which, in theory, can encode arbitrary long content into a fixed length. In addition, to fully leverage a user’s favorite music and to guide the model to learn specific music events in the case of limited training samples, we propose building a novel favorite-aware loss function that directly learns the music event distribution from a user’s favorite music [27].

After REMI [27], each token is a class (value), and each class belongs to one of the eight music events described in the previous sub-section. Each token therefore correlates with one of these eight music events. In the original Transformer-XL [20], the model is optimized using the cross-entropy loss function, which treats all classes equally and assigns each class the same weight. However, the physical meaning of each music event varies. For instance, notes constitute the most important information in music. In a sequence, the combination of P, NV, NO, and ND represents one note. It is thus reasonable to pay closer attention to note-related events than to other events such as TC and TV.

Inspired by the physical meaning of each music event and due to limited training samples, the favorite-aware loss function gives greater attention to music events that have more effects on a user’s favorite music. In other words, favorite awareness is calculated based on the distribution of events in the user’s favorite music. The fine-tuning procedure is detailed in Algorithm 1. We also conduct experiments to validate this model design.

Formally, let $E=\{e_{B},e_{P},e_{Ch},e_{TC},e_{TV},e_{NV},e_{NO},e_{ND}\}$ be the set of eight music events. We use $e_{k}$ and $c$ to represent an arbitrary music event and its corresponding class, respectively. Here, $e_{k}\in E$ is also a set that contains one or more classes and $c\in e_{k}$.

Given one music piece $X^{l}$ with sequence length $N$, $X^{l}=\{x_{1}^{l},x_{2}^{l},\ldots,x_{N}^{l}\}$ where $x_{n}^{l}=c\in e_{k}$, then Transformer-XL [20] $T_{\theta}(\cdot)$ can predict the next token $\hat{x}^{l}_{i}$ conditioned on the previously generated tokens $x_{t<i}^{l}$, formulated as $\hat{x}^{l}_{i}=T_{\theta}(x_{t<i}^{l})$. The favorite-aware loss $L_{f}$ is defined as

where $N$ is the sequence length, $C$ is the total number of classes, $p_{i,c}$ corresponds to the $c^{\text{th}}$ element of the one-hot encoded label of the ground truth $x_{i}^{l}$, and $q_{i,c}$ is the predicted category probability that observation $i$ is of class $c$. $w_{c}$ is the $c^{\text{th}}$ favorite-aware weight $\textbf{w}=\{w_{1},w_{2},\ldots,w_{C}\}$ and can be further calculated from a user’s favorite music $X$ with the following formula:

where $\tilde{E}\in E$ denotes selected music events, $Count(\cdot)$ is a function that counts the appearance of an input element in the sequence $X$, and $\alpha>0$ is a modifiable parameter. In this way, information that appears more frequently in the user’s favorite music $X$ will have a higher weight. For events that are not selected for transfer, the same weight will be given with a fixed value $\alpha$.

In the transfer phase, an arbitrary input sample $Y=\{y_{1},y_{2},\ldots,y_{N_{y}}\}$ with length $N_{y}$ will be transferred to a new piece of music $\hat{Y}=\{\hat{y}_{1},\hat{y}_{2},\ldots,\hat{y}_{N_{y}}\}$ based on the user’s favorite music $X=\{x_{1},x_{2},\ldots,x_{N_{x}}\}$ with length $N_{x}$. We apply two music steps to increase the model’s explainability and improve its performance.

Specifically, the event selection and event learning steps are included to solve UPMT. The details of these two steps appear in Algorithm 2. First, we define the event selection step as follows: only selected music events $\tilde{E}\in E$ in $Y$ are transferred while other music events remain the same. The event selection step can be executed as follows:

1. 1.

   If $y_{n}\in\tilde{E}$, then $y_{n}$ will be transferred and $\hat{y}_{n}$ will be predicted by implementing the event learning step (lines 5–19).

2. 2.

   If $y_{n}\notin\tilde{E}$, then $\hat{y}_{n}=y_{n}$ (lines 20-22).

For example, if we define $\tilde{E}=e_{NO}$, then only the NO event in $Y$ will be transferred. Other events will remain the same. Event selection fixes music events that we do not want to transfer and only focuses on a specific music event to ensure that the new music follows the structure of the input music, thus preserving information from the original music $Y$.

The event learning step controls the predicted event by forcing it to be similar to the user’s favorite music under certain conditions. Melody is one of the most basic elements in music. Melodic sequences, melodic phrases, or motifs can be considered music patterns and are represented as groups of consecutive notes that make sense together. In music, at least one music pattern always appears repeatedly that makes the melody unique. We define this music pattern as SMP because it is akin to a signature that makes music recognizable. To identify this pattern, we first find all selected music events $\tilde{E}$ in a user’s favorite music $X$ and denote these events as $\tilde{E}^{X}=\{\tilde{e}_{1}^{X},\tilde{e}_{2}^{X},\ldots,\tilde{e}_{z_{X}}^{X}\}$, where $z_{X}$ is the total number of selected music events in $X$. We then extract SMP from $\tilde{E}^{X}$ based on a method called SMP Extractor (Algorithm 3). We define the SMP as $M=\{m_{1},m_{2},\ldots,m_{a}\}$ and $M\subset\tilde{E}^{X}$, where $a$ is the length of SMP that can be changed based on different users’ favorite music.

In addition, rather than applying SMP directly, we calculate the difference and call it SMPI $I=\{i_{1},i_{2},\ldots,i_{a-1}\}$, which is calculated as $I=\{m_{2}-m_{1},m_{3}-m_{2},\ldots,m_{a}-m_{a-1}\}$. For example, if the selected event $\tilde{E}=e_{NO}$ and an SMP $M$ is $\{60,62,62,64,62,62,60,68\}$, then SMPI $I$ is represented as $\{2,0,2,-2,0,-2,8\}$. We use SMPI $I$ instead of SMP $M$ because music tends to have different pitch levels, but in terms of music similarity, relative differences are more representative; humans barely hear the difference if the ratio of two notes’ frequencies (i.e., SMPI) is the same [42]. For instance, an SMP $M=\{72,74,74,76,74,74,72,80\}$ sounds familiar to users compared with SMP $M=\{60,62,62,64,62,62,60,68\}$ where $I$ remains the same.

The event learning step is described in Algorithm 2 (lines 6–19) and can be executed as follows:

1. 1.

   A $flag$ is set to indicate if the event learning step will be executed or not.

2. 2.

   If $flag=0$, then the event learning step will not be executed. The current token $\hat{y}_{n}$ will be predicted by the pre-trained Transformer-XL [20] $T_{\theta_{new}}(\cdot)$; that is, $\hat{y}_{n}=T_{\theta_{new}}(\hat{y}_{1},\ldots,\hat{y}_{n-1})$ (lines 6–7).

   If the difference between the current predicted token and the previous predicted token $\hat{y}_{n}-\hat{y}_{v}=i_{1}$ (the difference matches the first item in SMPI $I$), then $flag$ will be set to 1 to indicate that the event learning step will be executed in the following steps (lines 8–10).

3. 3.

   If $flag=1$, then the event learning step will be executed. Specifically, subsequent tokens that belong to the selected event $\tilde{E}$ will not be predicted by Transformer-XL [20] $T_{\theta_{new}}(\cdot)$; instead, the event will be updated directly based on SMPI $I$ until all information from $I$ has been transferred to the new music $\hat{Y}$ (lines 11–13).

   If all items in $I$ are updated to subsequent tokens, then the flag will return to 0 to indicate that updating is complete (lines 14–17).

4. 4.

   Update previous predicted token (line 19).

The event learning step first extracts SMPI from the user’s favorite music based on prior human knowledge and then regulates the result to be similar to the user’s favorite music, which improves model performance directly.

All users’ favorite music samples were provided by the subjects and downloaded from open sources; more specifically, each subject provided one piece of their favorite music. Twenty-three subjects were recruited. Nineteen pieces of music were included due to overlap between subjects’ favorite music. In the fine-tuning phase, these 19 pieces of music were segmented into the same length. Some basic statistics of the training samples are listed in Table I.

In the transfer phase, seven songs from seven genres (classical, jazz, pop, rock, folk, new age and light music) were used as input music for transfer. Detailed information about users’ favorite music and the input music can be found in the supplementary materials.

In the fine-tuning phase, because the training data were limited, one piece of a user’s favorite music was divided into segments with a length of 128 tokens. In Transformer-XL, we followed the original setting mentioned in [27]. For the favorite-aware loss function, we set $\alpha=0.01$. We fine-tuned the model for 200 epochs or until the total loss was smaller than 0.1. In the transfer phase, we chose the track containing the most note information to complete the transfer task; other tracks remained identical to the input music in order to make the music sound more similar to the original input music. After transfer, all tracks’ information was merged to generate new music.

No existing work is directly applicable to UPMT. To evaluate the proposed method, we designed four models by extending the following related work for comparison:

$\bullet$ PopMT[27]: This approach applies REMI to represent symbolic music data and uses Transformer-XL to generate music.

$\bullet$ PopMAG [38]: This approach applies MuMIDI to represent symbolic music data and uses the Recurrent Transformer encoder and decoder to generate music.

$\bullet$ CycleGAN[13]: This approach extends CycleGAN by introducing additional discriminators to transfer different music genres, which deals with matrix data.

$\bullet$ Therapeutic Transfer Model (TTM) [3]: This approach takes matrix data as input and transfers two pieces of music by extracting the main and secondary features using a convoluted neural network–based model.

In the training phase, we first pre-trained these models based on the original training settings of each. Then, they were fine-tuned on a user’s favorite music based on their pre-training settings (e.g., sequence length, sampling parameters, and so on).

In the transfer phase, PopMT and PopMAG use Transformer-XL as the backbone and were employed to complete transfer tasks by transferring specific music events $\tilde{E}$ and fixing the remaining music events, which we named PopMT_T and PopMAG_T. CycleGAN after fine-tuning was used to directly transfer the input music to fit users’ preferences. In TTM, we implemented the pre-trained feature extraction network used in [3] to extract features. Then, the output music was optimized by assuming that the user‘s favorite music was the “style“ and the input music was the “content”. In addition, because the source code and parameters of PopMAG are not yet public, we tried to reproduce the model as accurately as possible.

Although many evaluation metrics have been proposed in various papers, quantitative evaluation in the music field remains an open question, especially in music style transfer. Following [38], we use the following four metrics to evaluate the transferred result and to compare our method with other models.

$\bullet$ Pitch Class (P): The distribution of pitch classes is computed.

$\bullet$ Note (N): The distribution of notes is computed.

$\bullet$ Duration (D): Duration values are quantized into 32 classes corresponding to 32 duration attributes, and the distribution of classes is computed.

$\bullet$ Inter-Onset Interval (IOI): This element measures the interval between two note onsets. The IOI values are quantized into 32 classes, and the distribution of classes is computed.

Then, the average overlapped area (OA) of the distributions of four factors is calculated to measure the similarity between two pieces of music:

where $OA$ represents the average OA of two distributions, $P^{A}_{i,j}$ denotes the distribution of feature $A$ in the $i$-th bar and the $j$-th track in a ground truth musical piece, and $\hat{P}^{A}_{i,j}$ denotes that in the generated musical piece; $A$ can be any of four factors. It should be noted that we did not measure velocity and chord, which were evaluated in [38], as our focus is a transfer task in which velocity and chord information remained the same.

In addition, melody is one of the most basic and distinctive features used to measure music similarity [43]. However, the aforementioned metrics only assess music event information that does not include melodic information. To the best of our knowledge, existing measures in music style transfer do not measure melodic similarity. Therefore, we designed a new metric, PS, to assess the consecutive music pattern interval overlap between two sequences, which is described in Algorithm 4. The new metric can evaluate melodic similarity between two pieces of music if the target event is Note On. If $PS$ is higher, then the melody of the new music is more similar to the target music, meaning that users will be more familiar with the new music. In addition, if the pattern length $p$ is longer, then the similarity will be lower because it is more difficult to find a match between two sequences.

In this subsection, we first conduct two experiments without implementing two steps to determine 1) the best music event to be transferred and 2) the most appropriate sequence length for transfer. Then, we perform three experiments to validate 3) the effect of using Transformer-XL as the backbone model, 4) the effect of using the favorite-aware loss function in the fine-tuning phase, and 5) the effect of implementing two steps in the transfer phase.

To validate that the proposed metric, PS, can measure melodic similarity and that the result is consistent with human judgment, we referred to the Kendall correlation to quantify the relationship between PS and the qualitative similarity score. This correlation can be used on small data samples and does not make assumptions about the underlying distributions, noise, or nature of relationships [44]. If the music has a high PS value, then the music will have a high qualitative similarity score as PS reflects melodic similarity, and the Kendall correlation will be high. TABLE X shows the Kendall correlation and the p-value when a different music pattern length $p$ is chosen. Results reflect a positive correlation between PS and human judgment in terms of music similarity when $p=2,3,4$. This pattern also indicates that humans can recognize the similarity in two pieces of music even if the music pattern length is quite short. Although the p-value is not significant when $p=5$, it shows a positive relationship (Kendall correlation $=0.22$) between PS and the qualitative similarity score.