Frequency-Temporal Attention Network for Singing Melody Extraction

We choose Combined Frequency and Periodicity (CFP) representation [15] as the input of our model due to its effectiveness and popularity. The CFP representation has three parts: the power-scaled spectrogram, generalized cepstrum (GC) [16, 17] and generalized cepstrum of spectrum (GCoS) [18]. We use 44100 Hz sampling rate, 2,048-sample window size, and 256-sample hop size for computing the short-time-Fourier-transformation (STFT).

The design of frequency-temporal attention module represents the first contribution of this work. The detailed architecture is shown in Fig. 2. Unlike previous works for speech-related tasks, we do not employ classic attention models [19, 20]. Instead we use 1-D convolution to select relevant regions (frequency/time) for this task.

Formally, given the input feature map $\mathbf{S}$ $\in\mathbb{R}^{F\times T\times C^{\prime}}$, an average pooling is firstly applied to the spectrogram for calculating the distribution of magnitudes along the time axis. Unlike 2-D average pooling, we use row average pooling to achieve this. The frequency descriptor $f\in\mathbb{R}^{F\times C^{\prime}}$ can be calculated:

where $s_{ij}$ is the element in the $i$-th row and $j$-th column in $\mathbf{S}$.

Then 1-D convolution is used to dynamically select task-related frequency for singing melody extraction. Since 1-D convolution is good at learning the relationship along the frequency bins or time axis, we choose to employ 1-D convolution to perform frequency-temporal attention. For a frequency descriptor $f$, the process of 1-D convolution can be written as:

where $k^{l}$ is the kernel of 1-D convolution at $l$-th layer and $V^{l}\in\mathbb{R}^{F\times C}$ is the newly generated feature map, * stands for the convolution operation. Finally, we apply a softmax layer to the output of the 1-D convolution layer $V$ to obtain the frequency attention map $\mathbf{A_{f}}\in\mathbb{R}^{F\times C}$:

Similarly, we can also obtain the temporal attention map $\mathbf{A_{t}}\in\mathbb{R}^{T\times C}$ through the same process mentioned above.

Meanwhile, we feed feature map $\mathbf{S}$ into 2-D convolution layers with the kernel size of $(3\times 3)$ and $(5\times 5)$ to generate two new feature maps $\{\mathbf{S_{f}},\mathbf{S_{t}}\}$$\in\mathbb{R}^{F\times T\times C}$. Then we perform matrix multiplications between $\{\mathbf{S_{f}},\mathbf{S_{t}}\}$ and $\{\mathbf{A_{f}},\mathbf{A_{t}}\}$ to obtain the final output $\mathbf{E}=\{\mathbf{E_{f}},\mathbf{E_{t}}\}$:

where $broadcast$ is the process of making matrices with different shapes have compatible shapes for element-wise multiplication.

The design of selective fusion module (SFM) represents the second contribution of this work. Inspired by the idea of selective kernel networks used in computer vision [14], we devise this module to dynamically select spectral and temporal features and fuse them. The detailed architecture is shown in Fig. 3. This module takes three inputs: the feature map $\mathbf{S^{\prime}}\in\mathbb{R}^{F\times T\times C}$ generated from $\mathbf{S}$ via a $(1\times 1)$ convolution, the frequency attention result $\mathbf{E_{f}}$ and the temporal attention result $\mathbf{E_{t}}$. Firstly, an element-wise addition is performed to fuse the three inputs into a new feature map $\mathbf{\Gamma}$ and then a global average pooling (GAP) is performed to obtain global descriptor $g\in\mathbb{R}^{C}$.

After a fully connected (FC) layer for non-linear transformation, three FC layers are used to learn the importance of each channel of the feature maps. The softmax layer is applied to obtain the attention map. After obtaining the attention maps, matrix multiplication is performed between the three inputs and the attention maps to obtain the weighted feature maps. Finally, we fused the three weighted feature maps by element-wise addition operation. The fuse feature map contains rich information selected from the spectral and temporal modes.

The design of melody detection branch (MDB) represents the third contribution of this work. The motivation of this branch is to design a general scheme that does not depend on special structure and can fast predict the presence of a melody. In this branch, we employ four stacked convolution layers to directly perform downsampling on the spectrograms. Specifically, we have the first three convolution layers with the kernel size $(4\times 1)$ and stride size $(4\times 1)$ and the last convolution layer with the kernel size $(5\times 1)$ and stride size $(5\times 1)$. As a result, the output of this branch is $m\in\mathbb{R}^{1\times T}$ stride size of $(5\times 1)$. Following [8], the output of MDB is concatenated with the salience map to make an $(F+1)\times T$ matrix. By including this branch, the voicing recall (VR) rate can be improved.

We randomly choose 60 vocal tracks from the Medley DB dataset. To increase the amount of the training data, we augment the training dataset by copying some of the chosen vocal tracks. Accordingly, there are 98 clips in the training dataset. We select only samples having melody sung by human voice from ADC2004, MIREX 05²²2https://labrosa.ee.columbia.edu/projects/melody and MedleyDB for test sets. As a result, 12 clips in ADC2004, 9 clips in MIREX 05 and 12 clips in MedleyDB are selected. Note that there is no overlap between the training and testing datasets.

To adapt the pitch ranges required in singing melody extraction, following [8] we set hyperparameters in computing the CFP for our model. For vocal melody extraction, the number of frequency bins is set to 320, with 60 bins per octave, and the frequency range is from 31 Hz ($B0$) to 1250 Hz (${D\kern-0.45003pt\leavevmode\resizebox{}{6.14998pt}{\raisebox{0.0pt}{$\sharp$}}}6$). We divide the training clips into fixed-length segments of T = 128 frames, which is 128 milliseconds. Our model is implemented with Keras ³³3https://keras.io. For model update, we choose the binary cross entropy as the loss function. The Adam optimizer is used with the learning rate of 0.0001.

Following the convention in the literature, we use the following metrics for performance evaluation: overall accuracy (OA), raw pitch accuracy (RPA), raw chroma accuracy (RCA), voicing recall (VR) and voicing false alarm (VFA). We use mir_eval library [21] with the default setting to calculate the metrics. For each metric other than VFA, the higher score, the higher performance. In the literature [1], OA is often considered more important than other metrics.

To investigate how much the proposed frequency-temporal attention module contributes to the model, we first remove the frequency attention, and only temporal attention is used to encode the temporal information. As is shown in Table 1, the performances on both datasets are decreased. When focusing on OA, the performance is decreased by 2.4% on ADC2004 and by 2.8% on MIREX 05. We then remove the temporal attention and only keep the frequency attention. The performances on both datasets are decreased by 2.9% and by 7.3%, respectively, on the both datasets.

We then investigate the effectiveness of the selective fusion module. When focusing on OA, the performances of the ablated version are decreased by 2.8% and 5.9% on ADC2004 and MIREX 05, respectively. The results justify the assumption that direct concatenation may hinder the further improvement of the model. Lastly, we investigate the effectiveness of the proposed melody detection branch. The results w.r.t. OA on both datasets are decreased by 1.4% and 3.7%, respectively, on the two datasets. The results demonstrate the effectiveness of the proposed simple design for melody detection. However, when focusing on VFA, the ablated version achieves a better score than the proposed model. We analyzed this phenomenon, and found that the shallow CNNs cannot capture semantic information well and it is prone to predict a non-melody frame with complex harmonic patterns as a melody one.

The performances on the three datasets are listed in Table 2. Three baseline methods are compared in Table 2, including MCDNN [5], SegNet [8] and MD+MR [11]. We carefully tune the hyper-parameters of the three baseline methods to ensure that they reach their peak performances on our training dataset. The proposed model and the three baseline methods are trained on the same dataset. Compared with the baseline methods, the proposed method achieves the highest score in general. The results clearly confirm the effectiveness and robustness of our proposed model. For comparison with other baselines, when focusing on OA, the proposed method outperforms MCDNN by 23.1% in ADC2004, by 11.1% in MIREX 05, and by 10.5% in Medley DB. Since all of the three baselines have melody-detectors, when focusing on VR and VFA, the proposed method achieves the highest score in ADC2004 and MIREX 05 datasets, and comparable result in MedleyDB.

To investigate what types of errors are solved by the proposed model, a case study is performed on two opera songs: “opera_male3.wav” and “opera_male5.wav” in the ADC2004 dataset. We choose SegNet [8] to compare with due to its effectiveness and popularity. As depicted in Fig. 5, we can observe that there are fewer octave errors in diagram (a) and (c) than in diagram (b) and (d). Moreover, from 1000-1200 ms in the diagram (d), we can find some errors that predict the wrong frequency bin near the right one, which are correctly predicted in diagram (c). Through the visualization of the predicted melody contour, we can say that the performance gains of the proposed model can be attributed to solving the octave errors and other errors. However, we can also observe that there seem to be more the melody detection errors (i.e., predicting a non-melody frame as a melody one) than in SegNet [8]. The goal of the melody detection branch is to fast predict the presence of the melody which does not depend on special structure. We leave this for future research topic to design more accurate and fast network for improving the quality of melody detection.