Text-Conditioned Transformer for Automatic Pronunciation Error Detection

A typical workflow for the ASR-based APED is depicted in Fig.1. The training dataset is generally constructed by three parts, the target text to be read, the collected speech, and the canonical pronounced text marked by professional annotators. For example, L2-Arctic dataset[17] manually labels the correct phonemes and mispronunciation error tags about the collected speech. Three annotators who are experienced in transcribing speech samples of native or non-native English speakers participate in the annotating process to ensure the high quality. Based on such a dataset, an ASR model is trained to recognize the canonical phoneme-level text $\mathbf{p}=(p_{1},p_{2},...,p_{n},p_{n+1})$ from the extracted audio features $\mathbf{x}=(x_{1},x_{2},...,x_{m})$. We should note that the described ASR-based APED is general, and can be applied with any ASR systems that can translate the audio features into phonemes. However, as we focus on Transformer, we limit our description on the attention-based training and inference in the following paragraph. For the attention-based models, the cross-entropy loss is used between the predict phonemes $\hat{\mathbf{p}}$ and the canonical phonemes $\mathbf{p}$:

where $p_{n+1}=\langle$EOS$\rangle$.

For the inference stage, the Transformer works quite differently from the training stage. The Transformer uses autoregressive outputting method to recognize the canonical phonemes sequentially. The recognized phonemes string will end with $\langle$EOS$\rangle$. Next, Needleman-Wunscha algorithm[38] is applied to align the recognized sequence $\hat{\mathbf{p}}$ with the target phonemes $\mathbf{t}=(t_{1},t_{2},...,t_{k})$. After the alignment process, the error states $\mathbf{e}=(e_{1},e_{2},...,e_{k})$ with consideration of the target phonemes can be returned to the user. An alignment example is shown in Table 1. We can observe that this sample includes 1 deletion and 2 substitution errors. The mispronounced phonemes whose error states are marked as 1 can be returned to the users.

For better clarification, we summarize the training and the inference stage of the ASR-based model in Table 2. We use a 39-dim Mel frequency cepstral coefficients (MFCC) feature as the encoder input. The start-of-sentence tag ($\langle$SOS$\rangle$) and the right-shifted 1-dim label of the canonical phonemes are concatenated as the decoder input in the training stage. This input is replaced by $\langle$SOS$\rangle$ and a regressively decoded phonemes string in the inference stage. The decoder tries to predict the probability of the next phoneme and $\langle$EOS$\rangle$ for output. There are in total 42 tags for classification, including 39 phonemes and $\langle$SOS$\rangle$ $\langle$EOS$\rangle$ $\langle$PAD$\rangle$.

We should note that there are several lengths defined for the described sequences. First, the attention mechanism is adopted to match the speech features ($length=m$) and the recognized phonemes ($length=n+1$). Next, the alignment operation is applied to find out the error states, whose length is equal to that of the target phonemes ($length=k$). However, such an alignment operation is performed in the inference stage, thus not jointly optimized with the ASR model. Such a dilemma inspires us to integrate the alignment operation or the target text into the training stage.

As shown in Fig.2, for the proposed method, we move the alignment operation into the data preparing stage. We align the canonical phonemes and the target phonemes to obtain where the errors occur in advance.

Next, we directly evaluate the relationship between speech features and the target phonemes. Thus, the network can be viewed as a fusion model. Moreover, the mother language (L1) of the speakers shows to affect the acoustic characteristics when studying a new language (L2) [39, 40]. Meanwhile, the extracted L1 features have also been proved to be helpful in the APED task [41]. Thus, we introduce the accent related auxiliary task to extract the L1 information. As shown in Fig.3, we append an extra classifier after the encoder and use the cross-entropy loss between the predicted accent $\hat{a}$ and the ground truth accent $a$ presented in the dataset:

Since the speech evaluation dataset is scarce, we first obtain a basic acoustic model by training the model on ASR datasets. The training process is similar to conventional ASR-based APED methods discussed in Section 3.1, and the new ASR loss function is,

where $\alpha$ is the weight of the auxiliary accent task.

We further adapt this basic acoustic model to the APED task. A training and inference summary of the proposed model is shown in Table 3. We will discuss the details and the differences between the proposed model and the ASR-based model in the remaining paragraphs.

Firstly, for the auxiliary accent classification task, while the input audio features are sequential, the accent is a 1-dim global attribute. We try to process the sequential data with gated recurrent units (GRU)[42] or a simple GlobalMean. Experiments in Section 4 show that GlobalMean performs a little better. Note that there are 6 kinds of accent including Arabic, Chinese, Hindi, Korean, Spanish, and Vietnamese, for the used dataset in our experiments.

Secondly, the prior target phonemes are used as an extra condition for the decoder input instead of the canonical pronounced phonemes, in both the training and the inference stage. For the audio features $\mathbf{x}$ and a certain target phoneme $t_{i}$ (target phoneme at step $i$), the decoder output is changed to $\hat{e_{i}}$, which indicates the matching degree of the audio features and $t_{i}$. As we use a binary state to judge its goodness, we use the sigmoid activation at the last layer for binary classification in Fig.3.

As the whole process is differentiable, we can directly optimize the loss between the predicted error states $\hat{\mathbf{e}}$ and the ground truth error states $\mathbf{e}$. For now, several classification losses can be used for this model. We first apply a basic binary cross-entropy (BCE) loss between the predicted error states $\hat{\mathbf{e}}=(\hat{e_{0}},\hat{e_{1}},\hat{e_{2}},...,\hat{e_{k}})$ and the ground truth error states $\mathbf{e}=(e_{0},e_{1},e_{2},...,e_{k})$ as the evaluation loss,

where $e_{0}=\langle$SOS$\rangle$. A further discussion about the choice of loss functions is presented in Section 4.5.

However, compared with ASR-based methods, a binary state only concerns about whether the target phoneme is correct or mispronounced. Thus, the model may lose information about the exact phoneme. To fix this, we still require the proposed model to conduct the ASR task with an auxiliary weight of $\beta$, and the whole loss function is,

The canonical phonemes to be recognized are aligned with the target phonemes for the proposed model using the aforementioned Needleman-Wunscha algorithm in Section 3.1 to make these two phoneme strings have equal length $k+1$.

Lastly, we should note that the proposed model has a consistent behavior in the training and inference stage, as shown in Table 3. This characteristic makes the inference in our method faster compared with ASR-based autoregressive Transformers, shown in Section 4.2.

We use Librispeech [45] as the dataset for ASR training. This dataset contains approximately 1000 hours of 16kHz read English speech. It is divided into “clean” (460 hours) and “other” (500 hours) parts based on its recognition difficulty. The “clean” part is further divided into the training set of 100 hours and 360 hours, development set (dev-clean), and test set (test-clean). As the APED task focuses on the phoneme-level error, we first convert the dataset into phoneme-level transcriptions using the Montreal Forced Aligner tool[46]. Next, we train the Transformer on different parts of the trainset for 300 epochs, including train-clean of 100 hours (train-100h), the whole train-clean part (train-460h), and the whole train part (train-960h). We use dev-clean as the validation dataset to choose the best model and test-clean for inference performance comparison. Adam optimizer, with a learning rate of $10^{-3}$, is used. We use a CTC-based ASR model called Jasper5x3 proposed in [47] for comparison. This model is constructed by 5 repeated Jasper blocks, and each Jasper block is constructed by 3 repeated Conv1D sub-blocks. The model parameters of Jasper5x3 are 44M, while the proposed Transformer is 32M. We show the phone error rate (PER) performance in Table 4. As we can see from the table, for different amounts of training resources, the attention-based Transformer structure generally performs better than the CTC-based method on PER. This observation is in accord with the conclusion in [14, 15], as the attention mechanism in Transformer can capture more relevant information compared with the CTC loss which holds the conditional independent assumption.

Next, we conduct the APED task on L2-Arctic dataset [17]. This corpus contains 26,867 utterances with 6 different accents, from 24 nonnative speakers. The 3,599 utterances annotated on phoneme-level are used for the APED task. The trainset, valset, and testset are divided into 8:1:1. For the testset, each sentence contains about 30 target phonemes on average. This suggests that the conventional autoregressive ASR-based models need to forward about 30 times on average to get each decoded phoneme sequentially. On the contrary, the proposed methods only need to forward once. We conduct the latency evaluation on a server with Intel Xeon E5-2680 CPU, and 1 NVIDIA P100 GPU. As shown in Table 5, the proposed method can bring great speedup for the APED inference.

For the APED task, the model should make a good balance of detecting the wrong pronunciations and accepting the correct ones. Thus, $F_{1}$ score is chosen as the main indicator for the performance. As defined in [48], the hierarchical evaluation structure is first divided into correct pronunciations and wrong pronunciations by the canonical pronounced phoneme. Next, depending on whether the predicted error state matches the ground truth label, the outcomes are further divided into true acceptance (TA), false rejection (FR), false acceptance (FA), and true rejection (TR). In other words, T/F suggests whether the prediction of the model is correct for the APED task, and A/R is the decision of the model. Based on this evaluation structure, $F_{1}$ score of the APED system is defined as follows:

To count $TR,TA,FR,FA$ for metrics, the predicted binary error states $(\hat{e_{1}},\hat{e_{2}},...,\hat{e_{k}})$ are firstly filtered by a threshold of $\theta=0.5$ to transform from a continuous float with the range of $(0,1)$ into discrete binary integer $\{0,1\}$,

Next, each outcome is calculated by following equations,

Apart from the conventional classification-related metrics including $F_{1}$ score, accuracy, precision and recall, the false rejection rate (FRR) and the false acceptance rate (FAR) are also of vital importance to the APED task. They are calculated as follows,

We first conduct experiments to explore the auxiliary accent classification task. We start from the model obtained on Librispeech dataset, and train for another 200 epochs, with the learning rate decreased to $10^{-4}$. We find that the GlobalMean method performs a little better than the GRU, as shown in Fig.4. We use $\alpha=0.7$ for the ASR-based Transformer in Eq.3, and lower it to $\alpha=0.1$ in Eq.5 to balance the $l_{asr}$ loss for further experiments about the proposed text-conditioned version.

Next, we adapt this pretrained ASR-based model to the proposed text-conditioned version. We still train the whole model for 200 epochs, with the learning rate of $10^{-4}$. We set the ASR-based model without the auxiliary task for baseline and the proposed methods with ablation for comparison. We find that $\beta=0.3$ generally gives the best performance. The results are shown in Table 6. The performance of the GOP method tested on this dataset[17] and the initial Transformer model pretrained on Librispeech is also reported in this table. First of all, as the initial model is purely trained on Librispeech, which only includes standard pronunciations, its FRR is relative high as it directly treats some unseen accents as wrong pronunciations. When adapted to handle the L2-Arctic dataset, the model has a significant improvement. By further employing the accent auxiliary task, the $F_{1}$ score is increased by nearly 0.1. If we simply use the target text as the condition and change the prediction target to the error states, the basic binary cross-entropy loss can bring a 0.19 improvement in terms of the $F_{1}$ score. We discuss the effect of different loss functions for the proposed method in the next subsection.

First of all, as the $F_{1}$ score is an important metric for the APED task, inspired by [49], we directly utilize the generalized $F_{1}$ score to optimize the predicted error states. To make it differentiable, the sums of probabilities are used instead of counts. We do not apply Eq.9 before calculating Eq.10 - 13. We should also note that Eq.9 is applied only for metrics calculation. For all the loss functions discussed in this subsection, $\hat{e}$ is continuous. As we try to maximize the $F_{1}$ score, the $F_{1}$ evaluation loss of the proposed method is,

Another consideration is that only 14.56% of the labelled phone segments are mispronounced for the L2-Arctic dataset³³3Dataset document at https://psi.engr.tamu.edu/l2-arctic-corpus-docs/, which may cause an unbalance between correct pronunciations and mispronunciations. Thus, we adopt focal loss[50] to mine the hard labels. Formally, if we define $e_{t}$ as:

The focal loss is,

where $\gamma$ modulates how much the well-classified samples are down-weighted. When $\gamma=0$, this loss function is equivalent to Eq.4.

We apply $F_{1}$ loss function and focal loss with different $\gamma$ values to the proposed model. We can see from Table 6 that, when adopting the $F_{1}$ loss function instead of the basic BCE loss, the result can be slightly improved. For the focal loss, we find that a small $\gamma$ value ($\gamma$=0.5 in our experiments) performs the best, and a bigger value will lead to a degraded $F_{1}$ score. Meanwhile, the auxiliary ASR task can boost the performance for all these loss functions. The focal loss version has the highest $F_{1}$ score 0.605 for the default $\theta=0.5$, which is a relative 8.4% improvement over the baseline ASR-based method.

We further analyse the behavior of the proposed method.

For the APED task, we need to make a trade-off between FAR and FRR. Meanwhile, as noted in [51], it is usually more unacceptable to take the correct pronunciations as wrong ones (false reject) than to regard the mispronunciations as correct ones (false acceptance). We can observe from Table 6 that the proposed methods all have a higher FAR and decreased FRR compared with ASR-based models, which suggests our model is behaving in a more acceptable way.

For the actual deployment, as the proficiency level of the target language varies among different students, the trade-off between FAR and FRR should be easy to adjust. Compared with ASR-based models, the proposed method can simply change the threshold $\theta$ to control how strict the APED system is. We further explore the effect of changing $\theta$ for different loss functions. We use a step of 0.1, i.e., $\theta\in[0.1,0.2,...,0.9]$. The metrics are shown in Fig.5. By increasing $\theta$, according to Eq.9, more output is judged as correct, and less output is judged as error. As a result, FAR (and precision) increases, while FRR (and recall) drops. Compared with the F1 loss version, the BCE loss and the focal loss version have a wider range of FAR and FRR when adjusting $\theta$ and can be a better choice for the actual deployment.

Since the proposed method is related to the input text, we break the testset into different parts to show the impact of how much the canonical phonemes differ from the target phonemes, i.e., the pronunciation error rate. A larger pronunciation error rate suggests that the input text information becomes less related to the input speech. We use the focal loss version and the ASR-based baseline for comparison. The result is shown in Table 7. We can observe that the proposed method makes higher relative improvement when the error rate is lower.

Finally, we try to analyze the auxiliary ASR task. For the ASR-based Transformer, on the one hand, the encoder extracts the speech-related features as embeddings; On the other hand, the decoder uses the attention mechanism to query the corresponding weight of each memory for the input text. Thus, the attention mechanism in the decoder will conduct an alignment between the text and the corresponding speech. What will the proposed text-conditioned Transformer do if the input text is not the canonical pronunciation but the target one? We plot the attention map of the proposed method without or with the auxiliary ASR task in Fig.6 to explore its behavior. For simplicity, we call these two models as the simple version and the full version in the following discussion.

As shown in Fig.6, for the simple version, it still tries to align the speech feature with the target phonemes for the shallow layers. As for deeper layers, the alignment between the phonemes and the speech features becomes vague. We conjecture that the training target causes this phenomenon. As for the ASR task, the network has to predict the next phoneme exactly; However, for the APED task, the network just needs to handle the error pattern for each phoneme and outputs a binary state, which is an easier task. Under such a cosy target, the deeper layers may not work hard to do the alignment, but choose to focus on summarizing the error patterns. As pretraining on the ASR task can be viewed as a sequential adaption[52], the pretrained weights perform as a regularization for the APED optimization space. Meanwhile, as suggested in [53], the adapted model does not deviate from the pretrained weights significantly. Thus, based on the pretrained ASR weight, the model still has the ability to distinguish different phonemes and match the input phonemes with the audio features memory. This may be the reason that the simple version can still get a satisfying improvement, as shown in 6. When the model is required to conduct ASR task, namely, the full version, the attention maps appear to be regular, which are similar to those Transformers that are applied for ASR tasks. As a result, the full version generally performs better than the simple version.