A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques

The input of the model is the phoneme sequence converted from prior text and the fbank acoustic feature, and the output of the model is the phoneme sequence corresponding to the audio. Specifically, it is assumed that the audio input is $x=[x_{1},...,x_{t},...,x_{T}]$, where $x_{t}$ represents the feature vector of speech at frame $t$ and $T$ represents the total number of frames of a speech. The $N$-length phoneme sequence input is $s=[s_{1},...,s_{n},...,s_{N}]$, $s_{n}$ is a phoneme at position $n$ in the prior phoneme sequence. Our model contains three modules: the sentence encoder, the audio encoder and decoder with the attention mechanism. The two encoders extract high-level feature representations $h$ from the input speech feature $x$ and sentence-to-phoneme sequences $s$:

$$h^{Q}=\mathrm{Audio\_encoder(}x)$$

(1)

$$h^{K},h^{V}=\mathrm{Sentence\_encoder}(s)$$

(2)

where the $h^{Q}$ and $h^{K}$ are the query and key. In our work, wo adopt CNN-RNN struct as the audio encoder that has been experimented in the previous research [10] and showed a good results, Bi-LSTM as the sentence encoder that has strong sequential modeling ability. Then the attention mechanism takes the $h^{Q}$,$h^{K}$ and $h^{V}$ as input and forms a fixed-length context vector:

$$c=\mathrm{Attention}(h^{Q},h^{K},h^{V})$$

(3)

The workflow of the entire model is shown in Figure 1. we detail the encoders and the attention mechanism in the following subsections.

One difficult of the model our proposed is the imbalance between positive and negative samples in training data. Here, we define a positive sample as a mispronunciation phone, and negative samples as one that is the canonical. Without data augmentation, the model would tend to output the prior phonemes and ignore the learn‘s mispronunciation, i.e., higher accuracy and lower recall, because the prior phoneme sequences were fed into the model, the most of which is negative samples, i.e., the number of negative samples is high and the number of positive samples is less, which leads to the model preferring to output canonical phonemes. A key goal of mispronunciation detection is to detect the wrong phonemes, thus to make our model can learn more features of positive samples, we increase the number of positive samples by the following three easy random replacement data augmentation operations:

We conducte TIMIT [18] and L2-ARCTIC [17] datasets to evaluate the performance of the our model, both corpora are publicly available. The L2-ARCTIC is a non-native English corpus built for CAPT and other tasks, it contains 24 non-native speakers (12 males and 12 females), whose L1 languages include Hindi, Korean, Mandarin, Spanish, Arabic and Vietnamese.

To unify the phone set of these two datasets, we map the 61-phone set in TIMIT and the 48-phone set in L2-Arctic to the same 39-phone set. All TIMIT data are used as training data. for L2-ARCTIC that has been annotated by experts, we select 6 speakers (”NJS”, ”TLV”, ”TNI”, ”TXHC”, ”YKWK”, ”ZHAA”) to build the test set, and the 6 speakers (”MBMPS”, ”THV”, ” SVBI”, ”NCC”, ”YDCK”, ”YBAA”) are used as development data to save our best model, remainding 12 speakers as training data. The detail of data split is shown in Table 1.

In our experiments, the input phoneme sequences were provided by these two corpora. Certainly, we also can produce it by the Montreal forced-aligner [19] as [17]. Our all models are trained using same parameters, such as learning rate, batch size, etc.

In this section, phone recognition evaluation metric employed is the phone error rate(PER), which is computed by aligning the annotated phone sequence and recognized phone sequence by the model using edit distance algorithm. The experimental results of phone recognition are shown in Table 3, which shows that the PER of the free phone recognition using the CNN-RNN-CTC is 27.75%, the result means that the original model on these data does not perform well. After introducing a priori text information, the setence-to-phoneme and the setence-to-character reported on sed-mdd have performances of 16.06% and 20.66% respectively in our proposed model structure, we can find that taking phoneme recognition as the goal and using text phoneme sequence as input is helpful to improve the performance of phoneme recognition. Although we modified the prior phoneme sequence in data augmentation, the performance did not decrease, and randomly selected 10% of the input phoneme sequence and randomly modified it according to the confusion-pairs(CP10%) to achieve the best PER of 15.48%.

The confusion matrices of most frequently misrecognized vowels and consonants by CNN-RNN-CTC and our best MD&D model(Attention-VC10%) are shown in Table 2. Our model significantly obtains better results than CNN-RNN-CTC for all confusable consonants and vowels. In addition, we find that this tendency to confusable after the introducing the prior text still remains(i.e. /$d$/-/$t$/, /$z$/-/$s$/, etc.), and there is no obvious damage to the acoustic information.

Following previous work [10, 15], the hierarchical evaluation structure are used to measure the MD&D system performance. The true accept(TA) and true rejection(TR) indicates predict correctly, while the false accept(FA) and false rejection(FR) indicates predict incorrectly. The true rejection(TR) is divided into correct diagnosis and diagnosis error. Three main evaluation indicators precision, recall and f-measure are calculated by the following formula:

$$\mathrm{Precision=\frac{TR}{TR+FR}}$$

(11)

$$\mathrm{Recall=\frac{TR}{TR+FA}}$$

(12)

$$\mathrm{F-}\mathrm{measure=2\frac{Precision*Recall}{Precision+Recall}}$$

(13)

For the model architecture, it can be found that our proposed phoneme-attention model improves the F-measure from 49.29% to 52.51% when comparing to CNN-RNN-CTC baseline, which is slightly better than character-attention model as well. We also notice a significant decrease in recall due to the large number of phonemes classified as input phonemes after the inclusion of prior text constraints, which is reflected in the TA reaching 93.06%. In the actual application of the MD&D, in order for learners to perceive the system as reliable, the TA needs to maintain a very high performance, which means learner’s many correct pronunciation in the canonicals detected by the machine are correct.

For the data augmentation, it can be seen that we achieved the best result in the experiment 56.08% after using VC(10%) method, which is an absolute improvement of 6.79% compared with the baseline. In addition, all data augmentation experiments maintain high TA and f-measure performance.

Through the analysis of the correct diagnosis and the incorrect diagnosis, we find that the main reason for poor recall performance are the decrease in the number of diagnosed error. What’s more, we show a comparison of the results in detecting the type of mispronunciation as in Table 4, the error diagnosis of the three types is greatly reduced after adding a priori text information, especially the two types of substitution and deletion.