DEEP SEGMENTAL PHONETIC POSTERIOR-GRAMS BASED DISCOVERY OF NON-CATEGORIES IN L2 ENGLISH SPEECH

This phonetic segment classifier consists of convolutional layers, recurrent layers and dense layers, as shown in Fig. 3. This unified architecture [16] takes advantages of the complementarity of Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs) and Deep Neural Networks (DNNs), and it is widely used in Automatic Speech Recognition (ASR) tasks [17, 18, 19]. In this work, we applied 2D CNN upon the input segmental acoustic feature sequences. The output of the final CNN layer is fed to the RNN layer. Since Gated Recurrent Unit (GRU) [20] has a performance on par with Long Short-Term Memory (LSTM) [21] while is more efficient in computation, we adopt GRU layer in our model. The final state of the GRU layer is fed to the FC layers and the softmax layer. Within this structure, CNNs serve as the speaker-independent feature extractor, which is robust to spectral variation. RNNs can model the temporal characteristics within a phone segment. DNNs are used to transform the final hidden features into a space that makes the output easier to classify.

To train this phonetic segment classifier, we need L1 (native) speech to implicitly guide the model to learn what canonical pronunciation of a phone should be. At the same time, since second language (L2) space has some deviations from the native space, we also need L2 speech to adapt the model parameters to fit it well. Hence both L1 and L2 speech corpus are used for training this classifier. In Section 3, the segment-level phone recognition rates are shown under different training corpus setups, and it indicates the importance and correctness of using both L1 and L2 speech to train the model.

The output of the classifier is segmental phonetic posterior-grams, which is a probability vector where each element corresponds to a phone category. The higher probability one element preserves, the more likely the segment belongs to the phone category which the element corresponds to. According to this, for a segment of canonical pronunciation, the classifier will output a vector with a high probability (confidence) on that canonical phone element. However, for the segments of L2 non-categories, the classifier will be ambiguous on more than one element, which results in a probability vector with more than one peak of comparable probabilities. From this point, we can set a threshold and look for the segments where the probability vector has more than one element higher than this threshold. These segments can be regarded as non-categories. After this exploration, we can use an identical name to denote all the segments with the same element set above the threshold. Then we have non-categorical patterns. For example, we can use $l\_n$ to represent the non-categorical segments with an element set of {l, n} above a given threshold. Section 3 gives the perceptual test results on these explored non-categories.

Our experiments are conducted on two speech corpora: 1) the CU-CHLOE-C (Chinese University-Chinese Learners of English - Cantonese speakers) data set [22] as the L2 speech corpus; and 2) the TIMIT data set as the L1 speech corpus. Both L1 and L2 corpus are divided by speakers into training set and evaluation set, as shown in Table 1.

The acoustic feature adopted in this work is 13 dimensional Mel-Frequency Cepstrum Coefficient (MFCC). There are 3 CNN layers with 64 output channels for each. The kernel size is 3 by 3, and no pooling function is used. The GRU layer has 128 hidden units. Its final state is transformed to output by 3 dense layers with 512 hidden units per layer, and the dropout [23] rate is set as 0.2 to prevent over-fitting problems. The implementation software used in this work is Pytorch 0.4.1 [24]. We use the combined training set of L1 and L2 to train the model, and use the evaluation set to test the performance on each corpus. During the training, we split 10% data from training set as the validation set. Adam [25] is used as the optimizer, and the initial learning rate is set as 0.001. The probability threshold used for exploring non-categories is set to 0.4.

Table 2 shows the phone recognition performance of our system trained on different training sets. If only L1 corpus involved in training, the system can have a phone recognition rate of 72.92% and 32.12% on L1 and L2 evaluation set respectively. This implies there is some deviation from L2 space to L1 space. On the counterpart, if only L2 corpus involved in training, the system has a phone recognition rate of 27.24% and 71.02% on L1 and L2 evaluation set respectively. However, if a combined training set of both L1 and L2 is used, the performance on L1 and L2 evaluation set can almost respectively reach the phone recognition rate when merely using the corresponding corpus as the training set. This implies even though there is some deviation between L1 and L2 space, a model trained on a combination training set can project them into a common space and still preserve the recognition performance. This system will be used for non-categories exploration.

To verify the explored non-categories indeed perceptually different from their corresponding categorical phones, we designed perceptual tests on series of non-categorical confusion groups. Each non-categorical confusion group consists of audios from the non-category, and also the corresponding categorical phones. In our case, since the threshold to explore non-categories is set to 0.4, we only consider the non-categories that resemble between two categories. For simplified writing, we mark the non-category that resemble two categories $P_{1}$ and $P_{2}$ as $P_{1}$_$P_{2}$.

In each group, we randomly select 12 audios (6 audios from the non-category, 3 audios from each of two categories). For the audios in each category, we select the segments where the SPPGs have a high probability (confidence) of 0.9 at that phone element.

These perceptual tests are conducted in the same way as [11]. During the tests, the listeners are asked to give their perceptual choice among 4 options after carefully listening each segment with short contexts: 1) More similar to $P_{1}$; 2) More similar to $P_{2}$; 3) Equal similarity to $P_{1}$ and $P_{2}$; 4) Not similar to either $P_{1}$ or $P_{2}$. 25 listeners majoring in linguistics or English were invited to participate in these perceptual tests.

Fig. 4 shows the average proportion of 4 options in three sample non-categorical confusion groups. The complete confusion scores are provided in Supplemental Material A. From these pie charts, we observe that listeners almost have an agreement on native phone categories, and give the corresponding phones as their choices. However, for non-categories, there is no majority perceptual vote among the given categorical phones. This implies these non-categories are indeed perceptually different from the corresponding categories, and bear a resemblance to both of them.

To compare the confusion degree of the explored non-categories in [11] and this work, we display the averaged confusion scores of the non-categories that are discovered in both [11] and this work. Table 3 shows the average proportion of each option in such non-categories. The averaged confusion scores in this work are shown in the first line of each option, while the previous results are shown within parentheses in the second line. The score gap between Option 1 and 2 is worked out and displayed in the bottom part. We observe that for most non-categories, such as [ax_er], [aa_ao] and [d_t], the score gap of the first two options is much smaller in this work. From the average sense, this score gap is decreased from 26.1% to 18.8%. Moreover, the averaged score of Option 3 is also obviously increased from 7.1% to 14.6%. These two observations verify that the non-categories explored in this approach are more accurate with higher confusion degree.

Table 4 shows the explored non-category set compared with that in [11]. There are 9 additional and 10 existing non-categories. To explore the possible reason, the k-means clustering algorithm used in [11] may be subjected to a data imbalance problem, since the non-categorical data, especially for some non-categories (for e.g., r_w), is far less than the categorical data. This imbalance data problem makes it difficult to cluster out those non-categories of a small amount. In contrast to this, this approach does not involve the whole data structure for clustering, so not suffer from this imbalance data problem. However, there are 2 non-categories absent from this explored set. Actually, they are observed, but the amount of these patterns is too small to be investigated.

The sample audios for native phone categories and those explored non-categories can be found in this link: https://anonymousdemos.github.io/ICME2019demos.github.io/.

In this work, the learners' L1 language is Cantonese, which has a different phone inventory compared with English. The Cantonese vowel inventory is generally richer than English, however it lacks the rhotic and many low vowels found in English [26]. The vowel inventories ²²2These vowels are denoted in International Phonetic Alphabet (IPA) form. of Cantonese and English are illustrated in Fig. 5 and Fig. 6 respectively. A comparison of these two inventories shows that the English vowel [ae] ([æ]), [ax] ([@]), [ah] ([2]) and [aa] ([A]) are missing in Cantonese. The lack of these vowels in the learner's L1 is hypothesized to be a source of possible mispronunciation in L2 [27], and L2 learners tend to pronounce these missing vowels from their L1 which are phonetically-similar to them (L1 transfer). This tendency makes the pronunciation of these vowels not accurate, and produces the non-categories, such as [ah_ax], [aa_ao] and [ae_ay].

Similar observations exist in consonant inventories comparison of Cantonese and English. Due to the limitation of space, the consonant inventories of English and Cantonese are put in the Supplemental Material B. [26] analyzed the English consonants missing in Cantonese and divided them into classes according to different articulation absence. These absence may also result in the corresponding non-categories, as shown in Table 5.

There are three English voiced plosives ([b], [d] and [g]) missing from Cantonese, and often being pronounced similarly with the voiceless, unaspirated Cantonese plosives ([p], [t] and [k]). This results in the non-categories of [b_p], [d_t] and [g_k]. And also, [ch] is an English affricate missing in Cantonese, and it is often replaced with the aspirated alveolar Cantonese affricate [$ts^{h}$], which bears a resemblance to both English [ch] and [t]. Similarly, [f_v], [dh_l] and [s_z] are caused by the missing English fricatives [v], [dh] and [z] in Cantonese respectively. [r_w] is produced by the missing English approximant [r], which is often substituted with [w] or [l] in Cantonese. Finally, it is often acceptable to substitute [n] with [l] in colloquial Cantonese, for e.g. 你(you) [nei] pronounced as 理(logic) [lei] [26]. Hence, Cantonese learners often unconciously perform these substitutions in English, resulting in the non-category [l_n].