PoLyScriber: Integrated Fine-tuning of Extractor and Lyrics Transcriber for Polyphonic Music

Lyrics transcription of solo-singing is typically performed by a speech recognition engine, such as Kaldi [40]. Gupta et al. developed a DNN-HMM using an early version of DAMP dataset and reported 29.65% word error rate (WER) with duration-adjusted phonetic lexicon. More recently, Dabike et al. [41] created a cleaner version of DAMP dataset and constructed a factorized Time-Delay Neural Network (TDNN-F) using Kaldi [40] with WER of 19.60%. Demirel et al. [25] further incorporated convolutional and self-attention layers to TDNN-F and achieved WER 14.96%. Our recent work [42] introduced an end-to-end Transformer-based framework along with RNN-based language model and is shown to be competitive with current approaches. This provides possibility and flexibility for lyrics transcription in polyphonic music where we would need a pre-trained solo-singing model to initialize the acoustic model for the integrated fine-tuning process.

Studies show that acoustic models trained on solo-singing data [43, 44, 45, 46, 47, 48, 49, 41] performs poorly on polyphonic music test data [50, 51, 24]. One way to adapt a solo-singing acoustic model towards polyphonic music is through domain adaptation [52]. It is found that an acoustic model adapted on polyphonic music data outperforms that adapted on extracted vocals. This suggests that singing voice extraction or separation introduces unwanted artifacts, that adversely affect the lyrics transcription performance. Despite much progress, the lyrics transcription of polyphonic music remains to be improved. As singing vocals and background music are highly correlated in polyphonic music, resulting in overlapping frequency components, lyrics transcription of polyphonic music is more challenging than that of solo-singing.

Many lyrics transcription systems [20, 24, 25, 17, 18, 53, 54] are based on hybrid modular architecture, that consists of acoustic, lexical and language models [40], each with a different objective, thereby suffering from disjoint training issue. Studies show that end-to-end (E2E) systems bring us many advantages [55, 56, 57, 58, 59, 60] including its simpliciy without the need of elaborate controlling of GMM, HMM and neural network models, its independent capability of lexicon and its flexibility to incorporate other models. E2E model only needs one single neural network with one objective function to optimize for the lyrics transcription task. We reported an E2E transformer-based system for lyrics transcription of polyphonic music [42] that outperforms other approaches in the literature. It consists of a transformer based encoder-decoder framework with a multi-head attention that implicitly learns the global time contextual dependency of the singing vocal utterance, beyond just the current frame. We adopt this singing decoder [42] as our network backbone.

PoLyScriber is an E2E approach utilizing polyphonic data augmentation where vocal extraction model is integrated with the lyrics transcription model and they are trained together solely towards the objective of lyrics transcription. This model uses both realistic and simulated polyphonic music data for E2E training, as shown in Fig. 1 (a) with dotted lines and the symbol (2). Having access to solo-singing data, simulated polyphonic music is created by data augmentation. We conduct polyphonic data augmentation by randomly selecting music and mixing it with solo-singing data with random signal-to-noise ratios (SNR) while training the PoLyScriber. By doing so, we expect to improve the diversity of polyphonic music data and further to be beneficial for model generalization.

If solo-singing or background music data is not available, we explore a transcription-oriented training strategy, PoLyScriber-NoAug, that does not adopt polyphonic data augmentation. PoLyScriber-NoAug has the same architecture as PoLyScriber trained solely towards the objective of lyrics transcription, while the training data used for PoLyScriber-NoAug is only realistic polyphonic music data, as explained in Fig.1 (a) with symbol (1).

On top of PoLyScriber, we also introduce a variant by additionally incorporating a vocal extraction loss along with the transcription loss to supervise the extractor front-end training on the simulated as well as real data as illustrated in Fig. 1(a) with dotted lines and the symbol (3).

The extractor encoder consists of four residual encoder blocks (REBs) where each REB contains two residual convolutional blocks (RCBs) followed by an 2 $\times$ 2 average pooling layer to reduce the feature map size. The real/simulated polyphonic music STFT input $X$ is first encoded as polyphonic representations via the extractor encoder. Each RCB consists of two convolutional layers with kernel sizes 3 $\times$ 3. A residual connection is added between the input and the output of a RCB [37]. A batch normalization and a leaky ReLU non-linearity with a negative slope of 0.01 is applied before each convolutional layer inside each RCB following the pre-act residual network configuration [37].

The two intermediate convolutional blocks (ICBs) then transform the polyphonic representations into a hidden descriptor, where each ICB has the same architecture as the REB except the pooling layer. The hidden descriptor is further fed into the extractor decoder, which is symmetric to those in the extractor encoder and it contains four residual decoder blocks (RDBs). Each RDB consists of a transposed convolutional layer with a kernel size 3 $\times$ 3 and stride 2 $\times$ 2 to upsample feature maps, followed by two RCBs. After the extractor decoder, an additional ICB and a final convolutional layer are applied to generate four outputs (the magnitude of the cIRM $|\hat{M}|$, direct magnitude prediction $|\hat{Q}|$, real part of the cIRM $\hat{P}_{r}$ and the imaginary part of the cIRM $\hat{P}_{i}$). Therefore, the extracted singing vocal STFTs $\hat{S}$ is predicted following [37] as below:

As illustrated in Fig. 1 (b) and Eq. 1, the angle of cIRM $\angle{\hat{M}}$ and the angle of extracted vocal STFTs $\angle{\hat{S}}$ can be obtained from $\hat{P}_{r}$ and $\hat{P}_{i}$, while $|\hat{Q}|$ is residual component to $|\hat{M}|$ for getting extracted vocal magnitude $|\hat{S}|$. After obtaining the STFTs of the extracted vocal, an inverse STFT is applied to obtain the extracted vocal waveform.

The sRes extractor is pre-trained on parallel polyphonic music and clean solo-singing data with a L1-loss that is computed on the waveform domain as shown below:

where the $\mathbf{\hat{s}}(t)$ and $\mathbf{s}(t)$ are the extracted singing vocal waveform and the corresponding target solo-singing waveform with $t$ as the discrete time index, respectively.

The transcriber encoder consists of an acoustic embedding module and twelve identical sub-encoders where each sub-encoder contains a multi-head attention (MHA) [68] and a position-wise feed-forward network (FFN). The extracted vocal acoustic features $F$ for the transcriber are first obtained from the extracted singing vocal audio via the feature extraction block, which first performs downsampling on the audio to a sampling rate of 16KHz and then extracts 80-dim filterbank feature with a window of 25 ms, shifting 10 ms. $F$ is then encoded into the acoustic embedding by the acoustic embedding module using subsampling and positional encoding (PE) [68]. The acoustic embedding block contains two CNN blocks with a kernel size of 3 and a stride size of 2. The sub-encoders then transform the acoustic embedding into a hidden representation $H$. Residual connection [69] and layer normalization [70] are employed inside each of the sub-encoder.

The transcriber decoder consists of a lyrical embedding module and six identical sub-decoders, where each sub-decoder has a masked MHA, a MHA and a FFN. During training, $Y$ represents the lyrical token history that is offset right by one position, however, during run-time inference, it represents the previous predicted token history. $Y$ is first converted to lyrics token embedding via lyrical embedding module, that consists of an embedding layer, and a positional encoding (PE) operation.

The lyrics embedding is fed into the masked MHA that ensures causality, i.e. the predictions for current position only depends on the past positions. The output of the masked MHA and the acoustic encoding $H$ are then fed to the next MHA for capturing the relationship between acoustic information $H$ and textual information from the masked MHA. The residual connection [69] and layer normalization [70] are also employed inside each of the sub-decoder.

A combined CTC and sequence-to-sequence objective is conducted for the transcriber pre-training as shown below:

where $\alpha\in[0,1]$, $R$ is the ground-truth lyrical token sequence. The transcriber decoder are followed by a linear projection and softmax layers, that converts the decoder output $O$ into a posterior probability distribution of the predicted lyrical token sequence $G_{s2s}$. The sequence-to-sequence (S2S) loss is the cross-entropy of ground-truth lyrical token sequence $R$ and its corresponding predicted lyrical token sequence $G_{s2s}$. Also, a linear transform is applied on $H$ to obtain the token posterior distribution $G_{ctc}$. CTC loss is computed between $G_{ctc}$ and $R$ [60].

We explore the task of lyrics transcription on polyphonic music. As shown in Table II, the polyphonic music training dataset, Poly-train, consists of the DALI-train [71] dataset and a NUS proprietary collection. The DALI-train dataset consists of 3,913 English polyphonic audio tracks¹¹1There are a total of 5,358 audio tracks in DALI, but we only have access to 3,913 English audio links.. The dataset is processed into 180,034 lyrics-transcribed audio lines with a total duration of 208.6 hours. The NUS collection dataset consists of 517 popular English songs. We obtain its line-level lyrics boundaries using the state-of-the-art audio-to-lyrics alignment system [18], leading to 26,462 lyrics-transcribed audio lines with a total duration of 27.0 hours.

The Poly-dev, serving as the validation dataset, consists of the DALI-dev dataset of 100 songs from DALI dataset [18], and 70 songs from a NUS proprietary collection. We adopt three widely used test sets – Hansen[72], Jamendo[17], and Mauch[73] to form the Poly-test as shown in Table II. The test datasets are English polyphonic songs, that are manually segmented into line-level audio snippets of an average length of 8.13 seconds, each of which we will refer to as an audio line. In our experiments, we transcribe the lyrics of a song line-by-line following [42].

We study the use of pre-training on solo-singing for lyrics transcription. A curated version [41] of the English solo-singing dataset $\textit{Sing! 300}\times\textit{30}\times\textit{2}$ ²²2The audio files can be accessed from https://ccrma.stanford.edu/damp/ is adopted, and detailed in Table III. A recent study [25] reports the state-of-the-art performance on this dataset, that serves as a good performance reference. As indicated in [41], the training set Solo-train consists of 4,324 songs with 81,092 audio lines. The validation set Solo-dev and the test set Solo-test contain 66 songs and 70 songs with 482 and 480 audio lines respectively. The lyrics of this database are also manually transcribed [41].

We use a standard music separation database, MusDB18 [34], for the singing vocal extractor pre-training and evaluation in PoLyScriber and its variants. MusDB18 is a widely used database for singing voice separation and music source separation tasks, and it contains 150 full-length tracks with separate vocals and accompaniment where 86, 14 and 50 songs are designed for training, validation and testing, respectively. All songs are stereo with a sampling rate of 44.1 kHz, as described in [37].

For the purpose of data augmentation for training the transcriber, PoLyScriber and PoLyScriber-L, we create a simulated polyphonic music dataset. The simulated training set (4,324 songs) is generated by adding music tracks, selected at random from MusDB18[34], to every audio clip in Solo-train data at the time of training. Specifically, for each epoch of training, the solo-singing training set audio clips are mixed with random background accompaniment tracks at a wide range of signal-to-noise ratios (SNRs) sampled randomly between [-10dB, 20dB].

To understand the effects of different extractor architectures on lyrics transcription, we implement two systems - the main extractor is the simplified Residual-UNet (sRes) described in Fig. 1 and Section IV-A, which is one of the top performing systems for singing vocal extraction in Music Demixing Challenge 2021 [67], and another extractor for comparison is the Open-Unmix (UMX) [31] that was the best performing open-source music source separation system in the source separation challenge SiSEC 2018 [33]. We further detail the network architecture of the simplified Residual-UNet (sRes) and UMX below.

For Simplified Residual-UNet Vocal (sRes) Extractor, model compression is applied to reduce the model size and speed up the integrated fine-tuning process. The original Residual-UNet model [37] employs model architecture with a large parameter size of 102 M trainable parameters, which brings difficulties in model deployment due to the expensive and time-consuming computation. To reduce the model size, we apply model compression with several redundant layers removed while ensuring that the performance of vocal extraction does not get affected significantly. Compared to the original Residual UNet [37], in this work, we have removed two REBs, two ICBs and two RDBs where two RCBs are further removed in each REB and each RDB to establish the simplified Residual-UNet. The sRes extractor thus contains four REBs, two ICBs, and four RDBs with only 4.4 M trainable parameters.

Regarding to UMX Vocal Extractor, we utilize “UMX” model from Open-unmix serving as UMX vocal extractor [63] as a comparison system for the front end as it is a widely used open-source singing vocal extraction system. UMX is trained by parallel polyphonic mixture and clean singing vocal audio in MusDB18 dataset [34] using bidirectional Long short-term memory (BLSTM), and it learns to predict the magnitude spectrogram of singing vocal from the magnitude spectrogram of the corresponding mixture inputs (singing vocal+background music). The network consists of a 3-layer BLSTM where each layer has 512 nodes. UMX is optimized in the magnitude spectrogram level using mean square error, and the singing vocal prediction is obtained by applying a mask on the input polyphonic music.

We compare the two extractors briefly. The sRes and UMX extractors are both spectrogram-based approach built on top of the MusDB18 dataset [34] and produces the extracted vocal spectrogram, but the objective function of UMX is computed on spectrogram-level while that of sRes is on time-domain. The idea of obtaining extracted vocal spectrogram are also different. The sRes extractor estimates both phase and magnitude of the cIRM as well as incorporates the additional direct magnitude prediction to compensate the magnitude of cIRMs, while the UMX extractor does not predict the phases for extracted singing vocal that makes it suffer from incorrect phase reconstruction problem [37].

Reference baselines are presented in Table IV and they include direct modeling (DM) as well as two-step pipeline approaches. Specifically for direct modeling method, DM and DM-Aug models are directly trained and validated on polyphonic music data, and they share the same architecture which only includes transformer-based transcriber that is pre-trained by solo-singing data.

There are two kinds of two-step pipeline strategies. First, in the pre-and-pre strategy [41, 49], the extractor and the transcriber are separately pre-trained. During inference, the trained extractor front end is used to get the extracted singing vocal that is then passed to the transcriber, that is trained on clean solo-singing vocals, for the decoding of the polyphonic music development and test sets. The second strategy, pre-and-fine, firstly employs pre-training on the front-end extractor for extracting vocals from polyphonic music, followed by fine-tuning solo-singing based transcriber using extracted vocal data [52]. The solo-singing based transcriber is pre-trained by solo-singing data. Polyphonic music data also goes through the extractor for testing the pre-and-fine model.

To be more specific, we use off-the-shelf SOTA systems for both the modules in our two-step baseline, where pre-and-pre and pre-and-fine approaches are experimented on two pre-trained extractors including simplified Residual-UNet and UMX. The pre-trained “UMX” [63] is utilized to extract singing vocals for the two-step pipelines pre-and-pre UMX and pre-and-fine UMX, and the pre-trained simplified Residual-UNet is employed to extract vocals for pre-and-pre sRes and pre-and-fine sRes models. We note that the audio inputs for the feature extraction block are real/simulated polyphonic music for DM or DM-aug models, and extracted singing vocal for pre-and-pre and pre-and-fine models.

Instead of directly training the transcriber on polyphonic music as in DM-aug, the proposed integrated fine-tuning approach alleviates the background music inference problem by integrating the extractor and the transcriber for training in a single network. Specifically, we conduct pre-training of the simplified Residual-UNet extractor first, and then globally train the pre-trained extractor and transcriber in a single network. The pre-training uses MusDB18 dataset [34] following the default parameter settings in [37], and the integrated fine-tuning of PoLyScriber and its variants (PoLyScriber-NoAug and PoLyScriber-L) uses the polyphonic music data as detailed in Table IV. Specifically, data augmentation is performed for PoLyScriber and PoLyScriber-L models but not for PoLyScriber-NoAug model. The feature extraction block inputs for PoLyScriber and its variants are intermediate extracted singing vocal audio from the extractors.

We use ESPnet [74] with PyTorch back end to build all transcribers, and the interpolation factor $\alpha$ between CTC loss and S2S loss is set to 0.3 for training. Other parameters of the transcriber follow the default setting in published LibriSpeech model (LS online)³³3See the pretrained librispeech model “pytorch large Transformer with specaug (4 GPUs) + Large LSTM LM” from the ESPNET githubhttps://github.com/espnet/espnet/blob/master/egs/librispeech/asr1/RESULTS.md., where attention dim is 512, the number of heads is 8 in MHA and FFN laryer dim is 2048. During integrated fine-tuning, the extractor input are upsampled to 44.1k sampling rate using the training data as indicated in Table IV. All models are trained using the Adam optimizer with a Noam learning rate decay as in [68], 25,000 warmup steps, 2,000,000 batch-bin, and 20 epochs [68]. We follow the default setting in ESPnet [74] to average the best 5 validated model checkpoints on the development set (Solo-dev for pre-and-pre models, librispeech dev dataset for LS model and Poly-dev for the rest models) to obtain the final model as in Table IV. We follow the common joint decoding approach [75, 60], which takes CTC prediction score into account during decoding by setting different CTC weight. During decoding process of different models, the default setting is used where the penalty, beam width and CTC decoding weight are set to 0.0, 10 and 0.3, respectively.

Since singing vocals extracted from polyphonic music are noisy version of clean solo-singing vocals, it is reasonable to initialize the polyphonic lyrics transcriber with a pre-trained solo-singing lyrics transcription model. We developed the pre-trained solo-singing transcription model (SoloM) by first training a speech recognition model (LS) on LibriSpeech dataset [76] following LS online model setting with 80-dim fbank features. We note that the pre-trained LS and SoloM models have the same transformer-based E2E architecture design as in Section IV-B.

We investigate the effect of the integrated fine-tuning by comparing the proposed PoLyScriber and its variants with the two-step approaches. In Table IV, we observe that PoLyScriber-NoAug framework consistently outperforms pre-and-fine sRes and pre-and-pre sRes models. This suggests that our integrated fine-tuning strategy of combining the training process of the extractor indeed yields better results in predicting lyrics compared to the two-step approaches, because both of these modules are optimized towards the common objective of lyrics transcription. Since the extractor in the two-step methods is not optimized towards the goal of lyrics transcription, the output from the extractor may not be a suitable input for the lyrics transcriber, leading to a sub-optimal solution. This is due to the fact that singing vocal extraction front ends are optimized to estimate the target singing vocal, which is different from optimizing for lyrics transcription. Therefore, the E2E PoLyScriber-NoAug, that optimizes the whole network towards lyrics transcription objective, performs better, and is able to address the mismatch problem between the front end and back end in the two-step approaches.

Furthermore, the superiority and effectiveness of the PoLyScriber-NoAug model over pre-and-fine sRes model suggests that the integrated fine-tuning approach is capable of performing effective transcription using only real-world polyphonic music data independently, without needing any parallel singing vocal and polyphonic music data generated from data augmentation.

To study the importance of tackling music interference problem for lyrics transcription, we compare direct modeling (DM) approach with integrated fine-tuning approach. DM method directly trains the transcriber on polyphonic music data without considering the background music as an interference. In Table IV, we observed that PoLyScriber-NoAug outperforms the DM method across all the datasets, which suggests that lyrics transcription can benefit from the integrated fine-tuning process with the extractor to handle the music interference problem.

It is interesting to note that the DM method has a comparable, sometimes even better, performance than the two-step methods. This has also been observed previously in [51, 18]. This may be an indication that background music may not be as harmful as the vocal artifacts introduced by the vocal extractor in lyrics transcription [18, 42]. This observation also indicates that the integrated fine-tuning would finally optimize the extractor into a state where its output will be somewhere in-between absolute background music suppression and complete background music presence, so as to gain from the benefits of both.

Data augmentation is employed by randomly adding music to solo-singing to create simulated polyphonic music data while training the transcriber. We use this data augmentation method in direct modeling and integrated fine-tuning approaches. In Table IV, we can see that DM-Aug outperforms DM for all the three testsets, and PoLyScriber outperforms PoLyScriber-NoAug. This indicates that the proposed singing data augmentation is a simple yet beneficial solution to the problem of lack of diversity in training data, thereby improving model generalization.

Having access to the parallel solo-singing and simulated polyphonic music, we are able to incorporate the extraction loss for the integrated fine-tuning framework. We investigate if the additional extraction loss is helpful to the integrated fine-tuning. In Table IV, we can see that the incorporation of the extraction loss in the PoLyScriber-L model shows a slight performance degradation on hansen and mauch datasets. Similar observations have been made in ASR studies where ASR-oriented optimization is effective for multi-talker speech recognition but an additional separation loss would not bring improvement to the recognition performance [77, 78, 79].

To analyze the relationship between the transcriber and the extractor in the integrated fine-tuning approaches, we present the extracted singing vocals from different systems and the corresponding original polyphonic music, along with their predicted transcriptions at the link ⁴⁴4https://xiaoxue1117.github.io/PaperSample/. Qualitatively, when we listened to the outputs of the extractor after integrated fine-tuning, we found that some background music/noise got added back into the audio and distorted vocal parts in two-step approaches were recovered in integrated fine-tuning approach. Some interesting observations were that the lyrics transcription improved, compared to two-step methods, especially in the chorus sections of the song where multiple voices were present simultaneously. The integrated fine-tuning seemed to have drowned out the other vocals present in the background by adding back some music/noises. This gives evidence that the proposed integrated network adds in some background music/noise and settles somewhere between completely suppressed background music (as in two-step approaches) and completely present background music (as in DM approaches), yielding better results than both. Similar observation are also presented through spectrogram visualization in the following Section VI-E.