Syllable-level lyrics generation from melody exploiting character-level language model

As shown in Figure 1, the Transformer on the right side generates the candidate syllable tokens based on the encoded melody latent representations $M$ and previously generated lyrics. The fine-tuned language model on the left evaluates the probability of the candidates based on the given lyrics generated, which aims to improve the coherence and correctness of the generated lyrics.

As an example shown in Figure 1, the proposed model has generated a sequence of lyrics tokens don’t get any big in previous time steps. In the current time step, the Transformer decoder predicts syllable ger with a probability of 0.3 and predicts syllable ideas with a probability of 0.2. Considering the Transformer is trained on a limited amount of data, it might assign a higher probability to ger because the syllables can construct a word bigger. However, the language model, which is trained on a large amount of corpus, can predict ideas with a higher probability of 0.6 because the sentence don’t get any big ideas is more meaningful in natural language. Then, in the re-ranking stage, token ideas can be assigned the highest probability after weighting the two probabilities. In such a way, the language model can help the Transformer generator predict better lyrics in terms of grammar and meaning.

We focus on exploiting the language model in Figure 1, hoping to improve the coherence and correctness of the lyrics generated by the main model by using the knowledge of a pre-trained character-level language model to re-evaluate the token probabilities during beam search generation. It could improve the results and generated lyrics quality as opposed to using solely the baseline encoder-decoder Transformer model.

The probability that the language model computes would be $P_{l}(x_{i-1},x_{i-2},…,x_{0}|x_{i})$, where $x_{i}$ is the ith syllable of the lyrics. We only start using the language model from the second generation step, ensuring that $x_{0}$ is known. The probability modelled by the Transformer model would be $P_{m}(x_{i}|x_{i-1},x_{i-2},…,x_{0},f_{n},f_{n-1},…,f_{0})$, where $f_{i}$ are the melody features at time $i$.

The total probability for a given token is then:

where $\lambda_{l}+\lambda_{t}=1$ are weights indicating which model we prioritize.

In our work, we fine-tune the pre-trained Google CANINE Clark et al. (2022) model using our dataset. We chose CANINE as it is a widely recognized open-source character-level language model. We use the task of Next Sentence Prediction (NSP), i.e., given a syllable $s$ and a lyric $l$, predicting the probability $P(s|l)$ that $s$ follows $l$. Note that in the case of character-level language models, both $s$ and $l$ are sequences, hence the NSP approach can work well. Fine-tuning is essential since the word distribution of lyrics differs significantly from that of resources typically used in training the language model, such as books or Wikipedia. For instance, lyrics contain the words love, hate, and gotta more frequently, and have more lenient grammar.

We have created the dataset for fine-tuning CANINE based on our lyrics dataset Yu et al. (2021). As each syllable of lyrics with its preceding sequence in our original dataset can be thought of as a data point, we are able to obtain a fine-tuning dataset of a considerable size of over 2 million examples.

An example of constructing data samples can be seen in Table 1. For negative examples (label 0), we select a random syllable from the same lyrics sequence that is not the correct continuation of the input sequence. We believe that using syllables from the same lyrics sequence poses a bigger challenge to the language model compared with selecting from the whole vocabulary since the syllables in the same sequence are more plausible candidates than unrelated ones from the vocabulary.

Since the syllables are separated by blank spaces in the melody-lyrics dataset, the lyrics it generates are different from the correctly formatted language that CANINE is used to. Therefore, to enable the pre-trained CANINE model to learn the blank space distribution, we introduce negative data samples with incorrect spacing, i.e., some without the space like “example” and some with it like “_example”. The more probable variant is selected and used to form the context for the next generation step. This allows us to use the language model for connecting the syllables generated by the Transformer into full words. Specifically, for the first three predictions of the NSP task, we introduce negative examples with incorrect spacing, and in the following predictions, we set an incorrect spacing probability of 60%, to avoid significantly increasing the size of the dataset. Negative examples with random syllables selected as the candidate have the spacing information preserved from the original location of the candidate. For instance, in the example “i know why your mean to me when i call on the”, “_the”, the candidate syllable the has a space in front of it, since this is how it originally appeared in the lyric.

Moreover, in order to improve the robustness of the model and its ability to recover from mistakes, in 40% cases we also include examples where one syllable from the preceding lyrics is randomly switched to a different syllable from the same lyric. For instance, in “i know whytel mean to me when i”, “_call”, the syllable tel has randomly replaced the syllable _your, making it a negative data sample. Since we are aiming to simulate mistakes, we randomly insert a space before the syllable with a probability 50%.

The dataset used for training the model is imbalanced, with a higher proportion of negative examples compared to positive examples. The reason for such construction is that it reflects the real-world scenario, where the model performs a beam search with multiple candidates, out of which only one is expected to be correct. The model is able to perform well despite the imbalances, achieving convergence after 5 epochs of training.

At each beam search step excluding the first, we have $n=\mathrm{beam\ size}$ candidate syllables for each of the $n$ beam sequences with the highest probabilities: $S=s_{1},...,s_{n}$, in total $n\times n$ candidate sequences to consider. The generated candidate syllables are then

At the first beam search step, we start with a single <BOS> (beginning of sentence) special token, and generate the $n$ best candidates for it, which become $s^{0}=s_{1},...,s_{n}$.

Each generated candidate is associated with the probability assigned by the main transformer model $\mathbf{M}\in\mathbb{R}^{n\times n}$. We also compute the fine-tuned language model probabilities for the sequences

The final combined probabilities are then

for $0<i,j\leq n$ at each timestep $t\in T$, where $\lambda_{m}+\lambda_{l}=1$ are weights assigned to the predictions of each model. We then select the $n$ best sequences, and continue the process using them as the new $s^{t+1}=s_{1},...,s_{n}$.

However, this does not take into account the probabilities at previous timesteps. If we consider text generation, the sequence “I am co ming home” might receive a low score, since home is just one of the possible continuations where one can be coming. However, the sequence “the the could ath lete”, despite making less sense, could score higher, this is because having predicted syllable “ath”, the model would be highly confident that the next syllable is “lete”.

To prevent that, a standard technique is to compare the candidates using cumulative probabilities, given by

where $C^{0}_{i,j}=M^{0}_{i,j}$, since we do not engage the language model in the first beam search step.

We trained a melody-to-lyrics Transformer model as a strong baseline and the basis of our methods. To leverage the ability of the language model, we set the weight of the fine-tuned language model to 75%, leaving 25% for the Transformer. Although the use of the language model noticeably slows down the beam search procedure, a complete evaluation on a validation set containing approximately 1000 examples can still be done in less than 3 hours on an A100 GPU. The fine-tuning of the language model was performed using default hyperparameters from the huggingface library Wolf et al. (2019), and lasts less than one day on an A100 GPU, despite the size of the fine-tuning dataset.

Evaluating creative text objectively is an exceedingly challenging task. Sequence evaluation metrics such as ROUGE and BLEU have limited utility when evaluating creative text because they mainly focus on measuring n-gram similarities between generated sequences and reference sequences. When evaluating creative text, it is crucial to understand that the goal is not to replicate a single ground truth reference. In some cases, an outstanding lyric may be unfairly penalized simply because it deviates from the ground truth, despite effectively fitting the melody and showcasing artistic excellence.

To the best of our knowledge, there exists no objective metrics that can comprehensively capture the quality of the generated lyrics. Therefore, we only use the objective metrics as a means to validate the reconstruction ability of the proposed model.

Table 2 shows the evaluation results of our model (Transformer + LM) and the baselines. We selected the recently published semantic dependency network (SDN) as a strong baseline, which already surpassed some methods like LSTM-GAN, SeqGAN, and RelGAN Duan et al. (2023a). We also implemented the original Transformer as another baseline. The BLEU and ROUGE metrics are slightly worse for the proposed model, however, the difference is insignificant enough to judge that our approach stays relatively close to ground truth in terms of the modeled syllable distribution. In the subjective evaluation in the following sections, and in the generated lyrics from Appendix A, we show that objective metrics can be misleading when evaluating models on a creative task. Examples of generated lyrics accompanied by the input melody are shown in Figure 2, which show that the lyrics generated by our model can better capture the characteristics of musical lyrics. More generated lyrics by using the proposed methods compared with the baseline model can be seen in Appendix A.

Due to the above-mentioned limitations of objective metrics, we proposed to evaluate the quality and correctness of generated lyrics via Large Language Models (LLMs), since they are objective and have a vast linguistic knowledge. It should be noted that our method only evaluates the texts of lyrics, without considering how well they fit the given melodies. Although feeding symbolic melodies could potentially strain the capabilities of LLMs, it is an approach worth exploring in future work.

We asked the GPT-3 Brown et al. (2020) to evaluate our generated lyrics. After experimenting with the prompts, we proposed the following prompts to let ChatGPT do the evaluation tasks.

I will send you three sets of generated candidate lyrics for 20-note melodies. I want you to evaluate them in terms of naturality, correctness, coherence (staying on topic), originality, and poetic value. Try to give numerical scores to all three candidate methods of lyric generation. I will send them in separate messages, please evaluate them after the third message. Is it clear?

By clarifying the task by the prompts, we hope to exploit the well-known strong language ability of ChatGPT. The conversation is available online¹¹1https://chat.openai.com/share/46166c1e-5505-4f74-af3d-3627c905b66c.

In addition to the aforementioned evaluation session, we informed ChatGPT that the lyrics are syllable-split, lowercase, and without punctuation. This additional information made ChatGPT more aware of the characteristics of our input beyond natural language. The conversation of the second version evaluation can be seen at ²²2https://chat.openai.com/share/bcfdcac3-b63c-44e2-bb29-c93699eae8f2.

We show the results from both runs in Table 3. In both cases, the proposed method is able to outperform the baseline, and in the second evaluation, it also outperforms the ground truth data. During the first evaluation, the ground truth has the highest values in all the categories, while the proposed method is equal to the baseline in two, and outperforms the baseline in 3 of the categories as indicated in bold. During the second evaluation, the proposed method has the highest values in all of the categories. We argue that by clarifying the characteristics of our text input, ChatGPT focuses more on the correctness and quality of the syllable-level split lyrics, hence giving higher scores on our model. This also verified the effectiveness of our proposed methods with language models.

Subjective evaluation is an important metric for evaluating creative text generation systems, especially for evaluating the fitness between the generated lyrics and input melodies.