MusicMamba: A Dual-Feature Modeling Approach for Generating Chinese Traditional Music with Modal Precision

In melody generation, the condition sequence is typically defined as $x_{1:t}=[x_{1},...,x_{t}]$ and the target sequence as $y_{1:k}=[y_{1},...,y_{k}]$, where $k>t$. The prediction of the $j$-th element in the target sequence can be expressed as $y_{j}|[x_{1},...,x_{t}]\sim{p(y_{j}|x_{1},...,x_{t})}$ where $p(y_{j}|x_{1},...,x_{t})$ represents the conditional probability distribution of $y_{j}$ given the condition sequence. Chinese traditional music often includes various modes. For example, in pentatonic modes, if a note $N_{i}\in\{C,D,E,G,A\}$ serves as the tonic note, then the following notes, if they follow a specific interval relationship, form a mode $M$. Therefore, to generate Chinese music with mode characteristics, the target sequence can consist of multiple modes and transition notes, represented as $y_{1}=(M_{1},f(M_{1}),M_{2},f(M_{2}),...,M_{l},f(M_{l}))$, where $M_{i}$ corresponds to a subsequence of notes within a specific mode. where $M_{i}$ corresponds to a subsequence of notes within a specific mode, and $M_{i}$ represents the transition note sequence following $M_{i}$. The task of generating melodies with Chinese modes can ultimately be formulated as the following autoregressive problem:

$$p(\mathbf{y}|\mathbf{x},M)=\prod_{i=1}^{l}p(M_{i}|C_{i})\cdot p(f(M_{i})|C^{\prime}_{i})\,,$$

(1)

where $M$ is the collection of multiple modes, $C_{i}=(\mathbf{x},\mathbf{y}_{<i})$ and $C^{\prime}_{i}=(\mathbf{x},\mathbf{y}_{\leq i})$. During the step-by-step generation of notes, the corresponding mode sequence $M_{i}$ is generated first, followed by the generation of the transition note sequence $f(M_{i})$ based on the mode sequence.

In generating traditional Chinese music, mode generation is a crucial and complex component. Chinese music often features intricate modal structures, such as pentatonic and heptatonic scales, where the selection and transition of modes are vital to the style and expression of the music. However, existing music representation methods face significant limitations in capturing and generating these modal structures. Although the REMI representation [13] effectively captures rhythm, pitch, and velocity information through events such as bar, position, tempo, and note, it struggles with complex modal structures, particularly when handling the dynamic modes in Chinese music.

To address this issue, we extended REMI by introducing two new events in the REMI-M representation to explicitly describe modes:

• •

  Note type event. Distinguishes between mode notes and transition notes, helping the model to more accurately capture modal information.

• •

  Mode-related events. Include the start, end, and type of mode, enabling REMI-M to explicitly annotate and generate modal changes in the music.

As shown in Figure 2, the original REMI and MIDI-Like encodings result in low mode generation rates, whereas REMI-M demonstrates significant improvements, achieving mode generation rates exceeding $0.8$ across all tested music lengths. These enhancements allow REMI-M to better handle complex modal structures, significantly improving the stylistic consistency and theoretical accuracy in generated music.

In music generation tasks, it is crucial to capture both local melodic details and global musical structure dependencies within long contexts. To achieve this, as shown in Figure 3, we designed a hierarchical feature extraction and integration architecture named the Dual-Feature Modeling Module, which combines the long-range dependency modeling capability of the Mamba Block with the global structure capturing ability of the Transformer Block.

Dual-Feature Modeling Module. In music sequence generation, both melodic details (like note variations and modal transitions) and overall structure (such as phrases and repetition patterns) are crucial. Traditional architectures often struggle to capture these levels of features simultaneously. Let $\mathbf{H}$ represent the feature matrix. The Mamba Block captures melodic details and modal dependencies by computing a dot product between the mode mask and melody tokens, generating the feature representation $\mathbf{H}_{1}$. This provides essential long-range and local information for integration. The Transformer Block primarily models global structural information, processing input melody embeddings with positional encoding to obtain the structural representation $\mathbf{H}_{2}$.

Bidirectional Mamba Fusion Layer. To integrate the outputs of the Mamba Block and Transformer Block, we introduce the Bidirectional Mamba Fusion Layer. This layer simultaneously receives the long-range features $\mathbf{H}_{1}$ generated by the Mamba Block and the global features $\mathbf{H}_{2}$ generated by the Transformer Block. Through a bidirectional scanning mechanism, the forward and backward features are processed separately to obtain $F_{\text{forward}}$ and $F_{\text{backward}}$. Then, self-attention is applied to the forward and backward features to extract key information:

$$F_{1}=\text{Attention}(F_{\text{forward}}),\quad F_{2}=\text{Attention}(F_{\text{backward}})\,,$$

(2)

Next, these two directional features are concatenated to obtain the fused feature $\mathbf{H}_{\text{fusion}}$, and further processed by a linear layer:

$$\text{Output}=\text{Linear}(\mathbf{H}_{\text{fusion}})\,,$$

(3)

The fused feature $\mathbf{H}_{\text{fusion}}$ combines the long-range dependencies and global structure of the melody, providing complete information support for generating complex and coherent music sequences. Finally, the linear layer maps the fused features to the output space, generating the final music sequence.

To evaluate the quality of the music samples generated by the model, we designed a subjective listening test. We recruited 10 music enthusiasts from social networks, each of whom plays at least one musical instrument. Each participant was asked to listen to 10 generated audio samples. They rated the samples based on three criteria: coherence, richness, and style, with scores ranging from 0 to 10. In the subjective evaluation results, MusicMamba outperformed all baseline models in coherence, richness, and style, showing the best overall performance.