Semi-Supervised Singing Voice Separation with Noisy Self-Training

Our proposed self-training framework for singing voice separation consists of the following steps:

1. 1.

   Train a teacher separator network $\mathcal{M}_{0}$ on a small labeled dataset $\mathcal{D}_{l}$.

2. 2.

   Assign pseudo-labels for the large unlabeled dataset $\mathcal{D}_{u}$ with $\mathcal{M}_{0}$ to obtain the self-labeled dataset $\mathcal{D}_{0}$.

3. 3.

   Filter data samples from $\mathcal{D}_{0}$ to obtain $\mathcal{D}_{f0}$.

4. 4.

   Train a student network $\mathcal{M}_{1}$ with $\mathcal{D}_{l}\cup\mathcal{D}_{f0}$.

This framework can be made iterative by repeating steps 2 to 4, using the student network $\mathcal{M}_{i}$ as the new teacher to obtain a self-labeled dataset $\mathcal{D}_{i+1}$, and training a new student model $\mathcal{M}_{i+1}$. The process stops when there is no performance gain. We illustrate the framework pipeline in Figure 1.

We use the PoCoNet [26] for both teacher and student models. The neural network takes the concatenation of real and imaginary parts of the mixture’s STFT spectrogram as input. The separator estimates the complex ratio masks for each source. The wave-form signal is obtained by applying inverse STFT transform on the estimated spectrograms.

The separator is a fully-convolutional 2D U-Net architecture with DenseNet and attention blocks. Each DenseNet block contains three convolutional layers, each followed by batch normalization and Rectified Linear Unit (ReLU). Convolutional operations are causal only in the time direction but not in the frequency direction. We choose a kernel size of $3\times 3$ and a stride size of 1, and the number of channels increases from 32, 64, 128 to 256. We control the size of the network by varying the number of levels in U-Net and the maximum number of channels. In the attention module, the number of channels is set to 5 and the encoding dimension for key and query is 20. The connections of layers in DenseNet and attention blocks follow [26]. Frequency-positional embeddings are applied to each time-frequency bin of the input spectrogram. For time frame $t$ and frequency bin $f$, the embedding vector is defined as:

where $F$ is the frequency bandwidth and $k=10$ is the dimension of the embedding.

We use MIR-1K [13], ccMixter [14], and the training partition of MUSDB [15] as the labeled dataset for supervised training. The training set contains approximately 11 hours of recordings. We use DAMP [27] as the unlabeled dataset for training the student model. DAMP dataset contains more than 300 hours of vocal and background recordings from karaoke app users. Since these recordings are not professionally produced, there exists bleeding of music in the vocal tracks and bleeding of singing voice in the accompaniment tracks as well; hence, it is not suitable for supervised source separation.

To reduce dimensionality and speed up processing, we downsample each track to 16 kHz and convert it to mono. We further segment the recordings to non-overlapping 30-second segments, and if the segment is less than 30 seconds we zero-pad to the end of the signal. The spectrograms are computed with a 1024-point STFT with a hop size of 256.

For both teacher and student training, we minimize Equation 3 with an Adam optimizer with an initial learning rate of 1e-4, and we decrease the learning rate by half for every 100k iterations until it’s no greater than 1e-6. We set $\lambda_{\text{audio}}=\lambda_{\text{spec}}=1$ in Equation 2 and $\lambda_{\text{voc}}=\lambda_{\text{acc}}=1$ in Equation 3.

To augment the training set, we randomly select a window size of $T=2.5,5,10$ seconds as the input to the model to experiment with the effect of input length, each with a batch size of 4, 2, and 1, respectively. The maximal batch size is chosen under the memory limit. We experiment with different probabilities of applying random mixing with $p=0,0.25,0.5,0.75,1$.

The teacher model is trained on the labeled datasets. Then, it assigns pseudo-labels for the unlabeled dataset. We infer vocal labels using DAMP vocal tracks as input to the teacher model and infer accompaniment labels from DAMP accompaniment tracks. Due to the leakage in these vocal and background tracks, they can be viewed as mixtures where one source is more likely to dominate the other, compared to normal mixtures.

As in previous studies on singing voice separation [1, 2, 4, 7, 19], we measure the signal-to-distortion ratio (SDR) to evaluate the separation performance. Following the SiSec separation campaign [28], we use the 50 songs from the test partition of MUSDB [15] as the test set. We partition each audio track into non-overlapping one-second segments, and we take the median of segment-wise SDR for each song and report the median from all 50 songs. We use the python package museval¹¹1https://sigsep.github.io/sigsep-mus-eval/ to compute SDR.

We select the configuration for the teacher model by experimenting with different input window sizes of training samples and the number of model parameters. Table 1 shows the test SDR for the combinations of input and model size. We first observe that using longer input size improves both vocal and accompaniment SDR. The improvement can be attributed to the attention blocks where longer input context provides more information for separation. Another observation is that larger models do not guarantee performance gain. The largest model (15.4M param) performs significantly better than the smallest one (1.6M) but is slightly worse than the 8.3M version for the probability of random mixing $p=0,0.5$. Using the best combination of input length (10 seconds) and model size (8.3M), we experiment with different probability of applying random mixing. [6] shows that random mixing does not have a positive effect on test SDR, and one possible explanation is that it creates mixtures with somewhat independent sources. Our experiments, however, indicate that random mixing alone significantly improves the results. The best performance is obtained when random mixing is always applied. Our observations are consistent with the argument in [29] that “one-versus-all” separation benefits from mixing independent tracks. Intuitively, mixtures with dependent sources are more difficult to separate. Random mixing makes it easier for the model to learn and to converge faster on the training set. Meanwhile, by mixing up sources from different songs, the training set becomes more diverse and the model has a better ability to generalize at inference time.

In addition, we verify that the DAMP dataset should not be applied directly in supervised source separation tasks by including this dataset along with the other labeled datasets. We experiment with two model sizes (1.6M and 8.3M) using 10-second input without random mixing, and the SDR values degrade sharply for both cases.

Table 2 summarizes the test SDR for student models. As opposed to the teacher model, the 15.4M model has a 0.75 dB SDR gain compared to the 8.3M model. The observation that the larger capacity student model improves the performance is consistent with the findings in [23].

To verify the quality control approach with VADs, we first count for each song the number of “poor-quality frames” as defined in Section 2.1.1 from three different datasets: DAMP, self-labeled DAMP, and MUSDB. From the visualization in Figure 2, the unprocessed DAMP contains the highest percentage of data with a large number of poor-quality frames, the distribution of MUSDB is concentrated in the low count region, while the self-labeled dataset lies in between. This implies that the count of “poor-quality frames” based on the output of VADs is a reasonable indicator of the quality of data samples. The experimental results demonstrate that the proposed data filtering method with VADs further improves the performance. The highest SDR is obtained when only the top quarter of the self-labeled data is included in the training. Incorporating a higher percentage of self-labeled data may provide more diversity but is more likely to include samples with poor quality, thus negatively affecting the model’s performance.

To compare the separation of singing voice with state-of-the-art, we also include models that separate the mixture into four sources. It has been shown in [6] that, these four-source models have similar vocal separation performance compared to two-source models, even though the four-source separation task is more challenging than the two-source counterpart; possibly because of the additional supervision provided by different instrumental sources in the multi-task learning setup. Hence, we include the vocal SDR values of state-of-the-arts for four-source models [10, 11] in our comparison. Our proposed approach, the student model using quality control with VADs, obtains the highest vocal and average SDR among all models, and the vocal separation outperforms others by a significant margin. The accompaniment SDR is higher than the baseline model with mono input [7] but worse than the stereo ones [1, 2]. Stereo input contains more spatial information for accompaniment than vocal since the left and right channel difference for background tracks are at a much larger scale than vocal tracks. Such information may improve the separation of accompaniment.