WaBERT: A Low-resource End-to-end Model for Spoken Language Understanding and Speech-to-BERT Alignment

Recently, self-supervised learning (SSL) has achieved great success in the fields of speech processing. There are multiple approaches for SSL of speech processing, including wav2vec [15], wav2vec 2.0 [2], HuBERT [9], WavLMs [3], and data2vec [1].

wav2vec [15] pre-trains a network optimized via a noise contrastive binary classification task.

wav2vec 2.0 [2], HuBERT [9], WavLMs [3], and data2vec [1] all employ BERT-like Masked Acoustic Model(MAM) task for model pre-training, but utilize different features as aligned target labels for prediction.

These methods have shown impressive results for speech processing, especially in phoneme classification and automatic speech recognition (ASR). It leverages large amounts of speech data to learn universal speech representations, which can benefit almost all speech downstream tasks by fine-tuning.

However, pre-trained speech models still present some challenges. On the one hand, in higher-level SLU tasks, satisfying performance is still hard to reach. Some researches demonstrate that the pre-trained speech models do not learn significant semantic information [16, 14]. On the other hand, speech data is at a lower abundance and more difficult to obtain compared to text data. For example, the text file size of LibriSpeech ASR corpus is 297MB, and wave2vec employs this dataset to achieve SOTA results on ASR tasks. Meanwhile, for natural language processing (NLP) models, Bert [5] uses BooksCorpus and English Wikipedia as datasets, which are summed up to 16GB, and Roberta [10] is trained with datasets Books Corpus, English Wikipedia, CC-News, OpenWebText and Storie, in 161GB, 542 times bigger than LibriSpeech ASR corpus. The lack of data limits the performance of Speech models.

We propose to overcome these challenges by combining a pre-trained speech model with a pre-trained NLP model.

The rapid development in pre-trained neural language models has significantly improved the performance of many NLP tasks. Models like BERT [5], RoBERTa [10] and DeBERTa [8] all get acceptable results on NLP tasks.

NLP tasks are similar to SLU tasks but different in data format. The input data of NLP tasks are texts, while the input data of SLU tasks are audios. Texts and audios are similar and inter-convertible, and they have enormous common knowledge naturally. Therefore, NLP models have the potential to be applied in SLU tasks. However, the different distribution and different lengths between audios and texts prevent NLP models from participating in SLU tasks directly. Instead, NLP models are applied in SLU in a more indirect and auxiliary way, the spoken language is recognized as texts by ASR, and then NLP models is fine-tuned for downstream SLU tasks [16]. Obviously, this method suffers from errors that occur in the ASR process and loses emotion information by dropping the feature of speech models.

We propose to overcome these challenges by forming an end-to-end model, which combines the pre-trained speech model and the pre-trained NLP model by aligning the outputs of speech and NLP models at word-token level.

Forced alignment (FA), a strategy to produce the bidirectional mapping between the given text and speech sequences, has been widely used in speech processing tasks for decades. Conventionally, FA models are based on hidden Markov model (HMM) [13], such as Prosodylab-Aligner [7] and Montreal forced aligner (MFA) [11]. Also, neural network based FA strategy like NeuFA [https://arxiv.org/abs/2203.16838] and CIF [6, 4] is proposed. Even Connectionist Temporal Classification (CTC) can be generally regarded as an FA strategy.

Nevertheless, when applying these methods to align text and speech modalities instead of text and speech sequences, most of the methods become ineffective. For HMM methods, the hidden states should be discrete, and the number of hidden states should be limited. Due to words or phonemes being discrete, and the number of words or phonemes being limited, text can play a role as hidden states in HMM methods. For aligning text and speech modals, the tensors that needed to be paired are continuous, and the tensors are unlimited in the account. Therefore, HMM models can not apply in modalities alignments. CTC faces the same problem as HMM models.

NeuFA needs ASR and Text-to-Speech (TTS) tasks for joint training, thus, it is not a lightweight construction and not easy for transfer, which makes it not a good solution for modalities alignment

CIF does not require the paired tensor to be discrete or limited in the account, so CIF shows the potential to work in text and speech modals.

We propose to make CIF a better aligner by modifying the aligned token similarity loss.

Data2vec extracts acoustic representation vectors from raw audio, and corresponding linguistic representation vectors can be obtained from BERT. Suppose the acoustic representation vectors as $A=(a_{1},a_{2}...,a_{M})$, and the corresponding linguistic representation vectors from $i_{th}$ NLP model layer as $L_{i}=(l_{1}^{i},l_{2}^{i},...,l_{N}^{i})$, where $M>N$.

Due to the acoustic and linguistic representations being consistent in the time dimension, the outputs of the two modalities should be able to be aligned no matter which model layer’s output is.

To avoid collapse, one modality should be frozen during alignment. For the concern of reducing the information loss of BERT, which contains more information than wave models, the parameters of BERT are fixed. Inspired by the work[4], we use the serial CIF [6] mechanism to achieve the monotonic alignment between the speech and text modalities.

CIF [6] mechanism is a strategy that by acquiring the length the inputs should be resized to, outputs a latent representation that is consistent with the length, also, a predicted length is provided as another output. After applying CIF to the output of wav2vec, acoustic representation vectors as $\hat{A}=(\hat{a}_{1},\hat{a}_{2}...,\hat{a}_{N})$ are obtained. Regarding $L_{i}$ as the learning target of the $\hat{A}$, We calculate the loss between $\hat{A}$ and $L_{i}$. The work[4] uses the equation below to calculate the loss between two modals for alignment:

$$cos(x,y)=\frac{x^{T}y}{\left\|x\right\|\left\|y\right\|}$$

(1)

$$L_{cos}=\sum_{i=0}^{N}(1\mbox{-}cos(\hat{a_{i}},l_{i}))$$

(2)

The similarity between corresponding representations is the only consideration in this loss equation design, and differences between different tokens are ignored. Emphasizing the difference between different representations would make the adjacent tokens easier to distinguish, and make alignment more precise. Therefore, we introduce Info Noise Contrastive Estimation (InfoNCE) to calculate the loss between two modalities for alignment. InfoNCE is defined as below:

$$InfoNCE(X,Y)=\frac{1}{N}\sum_{i=0}^{N}(-log(\frac{exp(\frac{cos(x_{i},y_{i})}{\tau})}{\sum_{j=0}^{N}exp(\frac{cos(x_{i},y_{j})}{\tau})})$$

(3)

And our aligned token similarity loss for calculating the loss between two modalities for alignment is defined as below:

$$heL_{InfoNCE}=\frac{1}{2}InfoNCE(\hat{A},L)+\frac{1}{2}InfoNCE(L,\hat{A})$$

(4)

The distance between the speech representation lengths that should be aligned to and the predicted length is measured by:

$$L_{quantity}=N_{predicted}-N$$

(5)

After aligning the outputs of data2vec with $i_{t}h$ BERT layer, the outputs of data2vec after the CIF aligner, share similar values with the outputs of $i_{t}h$ NLP model. Grafting between models means replacing a part of the model with another model. Data2vec can be grafted on BERT by replacing the first to $i_{t}h$ BERT layers with the whole data2vec model, and the new grafted model is supposed to get the same result at the top of the BERT as the input is paired texts. Therefore, the original vocabulary ID can be predicted directly with BERT fixed, through this frame, and the BERT-like vocabulary id prediction loss is defined as $L_{subword}$.

The loss $L_{t}otal$ for training is defined as:

$$L_{total}=L_{InfoNCE}+L_{quantity}+L_{subword}$$

(6)

We employ LibriSpeech ASR corpus for training, and it contains 960 hours of training data. We train with the adamw optimizer and a batch size of 128, distributed over 8 GPUs. The learning rate is linearly ramped up during the first 4000 iterations to its base value of 0.0005. After this warmup, we decay the learning rate with a linear schedule. The model is trained for 26 epochs because the loss value is stable enough after 26 epochs.

When coming to inference, the first to $i_{th}$ NLP layer is dropped. For fine-tuning, with the BERT fixed, the performance of downstream tasks is still acceptable.

BERT last layer as labels, data2vec are trained over LibriSpeech ASR corpus, with $L_{subword}$ removed. After training of 19 epochs, the loss is stable. The TIMIT dataset is selected as the test dataset. The models are evaluated on the MAE and medians of absolute errors of the predicted left and right boundaries at the word level. Then the comparisons are made with CIF with $L_{cos}$ as the baseline. As shown in Table 1, CIF with $L_{InfoNCE}$ outperforms CIF with $L_{cos}$. The MAE is reduced from 426.44 ms to 126.05 ms, and the medians drop from 386 ms to 77 ms, which means that CIF with $L_{InfoNCE}$ always predicts more accurate boundaries than CIF with $L_{cos}$. This can also be demonstrated by the accuracies at different tolerances (percentage below a cutoff) evaluated and shown in Table 2. The accuracies are improved by 0.2825, 0.5532, 0.2893, and 0.0253 at 50, 100 500, and 1000 ms tolerances for word level.

Figure 3 proves CIF with $L_{InfoNCE}$ is better in other aspects. It shows the heat-map of the similarity between the acoustic representations after alignment and the linguistic representations. And after CIF with $L_{InfoNCE}$, the heat-map shows a valid diagonal, and for CIF with $L_{cos}$, it is still a mess.

Still, an interesting phenomenon is shown is Figure 4, when using $L_{cos}$, the distribution of acoustic representations is similar to linguistic representations, and drops the initial acoustic distribution. However, acoustic representations with $L_{InfoNCE}$ remains more initial acoustic distribution, and still make a good performance on alignment.

SA refers to classifying a given speech segment as having negative, neutral, or positive sentiment. It is a higher-level task since the semantic content is also very important. For example, negative sentiment can be expressed by disparagement, sarcasm, doubt, suspicion, frustration, etc. [12]. We evaluate SA using weighted recall and F1 scores on SLUE benchmark. The results from Table 3 show that, our model outperforms two-stage pipeline approaches and one-stage approaches in dev dataset.