The NTNU System at the Interspeech 2020 Non-Native Children’s Speech
ASR Challenge

Hybrid DNN-HMM (e.g., CNN-TDNNF) acoustic models have shown to be significantly superior than the conventional HMM-based acoustic models that employ Gaussian mixture models (GMM) to characterize the emission probabilities of frame-level speech feature vectors being generated by each HMM state (denoted by GMM-HMM) on many ASR tasks. Hybrid DNN-HMM acoustic models still have to resort to GMM-HMM acoustic models to obtain good forced-alignment information for better estimating their corresponding neural network parameters. Therefore, the GMM-HMM acoustic model of our best system was training with the audio segments selected out from the speech training dataset with high recognition confidence scores generated by an existing hybrid DNN-HMM system. As we shall see later, the empirical ASR results confirm this intuitive data-cleansing therapy.

Due to the constraint posed by the closed-track competition, viz. only the speech and text corpora provided by the organizer could be used for the ASR system development, we thus set out to leverage different data-augmentation strategies based on label-preserving transformations, including both utterance- and word-level speed perturbation and spectrogram augmentation, to diversify and enrich the original speech training dataset, apart from the aforementioned data-cleansing operation. We anticipated that these data-augmentation strategies could further push the performance limit of our ASR system.

To alleviate the data-scarcity problem for acoustic modeling, a natural thought is to perform utterance-level speed perturbation [11]. It modifies the speaking rate of a speech utterance by resampling its waveform signal. Following the procedure described in [11], in this paper two additional copies of the original speech training data were created by perturbing the speaking rate of each training utterance to 0.9 times and 1.1 times of its original one, respectively. In this way, the training data had increased three-fold.

Furthermore, in initial experiments, we observed that the word-level speech of non-native children’s utterances exhibits high inter- and intra-speaker variabilities and thus tends to be unstable. To capitalize on this observation, we proposed a word-level speed perturbation method so as to make the resulting acoustic models better accommodate the intricate pronunciation phenomena inherent in non-native children’s speech. Word-level speech perturbation was conducted in a two-stage manner. At the first stage, word-level boundaries of the original training utterances were obtained with a baseline hybrid DNN-HMM ASR system. At the second stage, the speaking rate of each word segment was perturbed by randomly altering it to 0.9 times or 1.1 times of the original one. More specifically, one copy of the training dataset had 80% of its word segments increase their speaking rate to 1.1 times and 20% of its word segments reduce their speaking rate to 0.9 times of the original ones. Alternatively, another copy of the training dataset had 20% of its word segments increase their speaking rate to 1.1 times and 80% of its word segments reduce their speaking rate to 0.9 times of the original ones. To recap, the aforementioned utterance- and word-level speed perturbation procedures will generate four additional copies of training data, as schematically depicted in Figure 2. Note also here that, due to these augmentation operations will change in the lengths of the wave signals, the forced-alignment information of the speed-perturbed utterances were generated using the baseline hybrid DNN-HMM system.

Another line of research on training data augmentation for ASR acoustic modeling has focused on feature-space augmentation, which takes inspiration from the success of augmentation methods employed in the computer vision (CV) community, many of which augmented CV datasets by adding transformed sample instances along with their respective original labels [17-19]. The most celebrated feature-space augmentation method adopted for acoustic modeling is vocal tract length perturbation (VTLP) [20]. VTLP, which employs a linear warping transformation along the frequency bins, simulates the effect of altering the vocal tract lengths of speakers that produce the training utterances. Very recently, SpecAugment has drawn much attention from the ASR community, which treats the spectrogram of an utterance as an image, and in turn warps it along the time axis, mask blocks of consecutive frequency along the time axis bins and mask the whole frequency bins in short spans of time [10]. These operations collectively lead to considerable word error rate reductions on several benchmark tasks. Apart from the waveform-domain speed perturbation (viz. utterance- and word-level speed perturbation) mentioned previously in Section 2.2, SpecAugment was also applied to generate augmented acoustic training data. To this end, we made use of the component ‘spec-augment-layer’ of Kaldi toolkit [21], which consists only of the operations that mask blocks of consecutive frequency along the time axis bins and mask the whole frequency bins in short spans of time. This is probably because warping spectrogram along the time axis is conceptually similar to waveform-domain speed perturbation, but its costs a great amount of computation and does not get any significant improvement [10].

We evaluated our approaches to low-resourced non-native children’s English speech ASR on the TLT-school corpus [24] , while the baseline ASR systems was developed with the Kaldi toolkit [25] and the recipes released by organizer. The TLT-school corpus consists of English spoken responses collected from Italian school students between the ages of 9 to 16. Several intricate phenomena of non-native children’s speech, such as mispronounced, code-switched words and linguistic differences between children and adult speech, make this task much more challenging than before. The training set and development set consisted of 13,999 utterances from 340 speakers, and 562 utterances from 84 speakers, respectively. In addition, the evaluation set was composed of 578 utterances from another set of 84 speakers. A smaller-sized training set, which was used for quick tuning of the baseline settings. Table 1 shows some basic statistics of the TLT-school corpus.

Our first set of experiments on the development set is designed to analytically investigate the effectiveness of data cleansing (DC) and word-level pronunciation modeling (denoted by WPM), previously proposed in Sections 3 and 4, respectively. To this end, two disparate acoustic models, viz. TDNNF and CNN-TDNNF trained with the small-sized training dataset, are respectively employed as the default acoustic model. Three noteworthy points can be drawn from Table 2. First, the application of DC leads to a relative WER reduction of 12.4% (cf. Rows 1 and 2) as TDNNF is used as the acoustic model. Second, when DC is applied, CNN-TDNNF (stacking CNN with TDNNF) can further yield a relative WER reduction of 3.4% over that using TDNNF in isolation. Third, working in conjunction with WPM, the performance of CNN-TDNNF the based ASR system can be steadily improved, while using the full training dataset (cf. the last row of Table 2) instead of the small-sized training dataset further advances the performance. From now on, unless otherwise stated, we will adopt the model configuration determined in the last row of Table 2 for the following experiments.

In the second set of experiments, we turn to assess the impacts of different combinations of data augmentation methods, viz. spectrogram augmentation and speed perturbation (cf. Section 2), on the TLT-school task (viz. non-native children’s English speech ASR). Note here that, for speed perturbation, either utterance-level speed perturbation (denoted by USP) or word-level speed perturbation (denoted by WSP), or their synergy are used to expand the training dataset for acoustic modeling. The corresponding results on the development are shown in Table 3. As compared to the last row of Table 2, we can find that all different combinations of spectrogram augmentation and speed perturbation (cf. the first three rows of Table 3) can considerably boost the ASR performance, leading to a relative WER reduction of 6.6% when with the best combination setting. This results also confirm the merits of conducting data augmentation for resource-scarce ASR tasks, such as the TLT-school task studied in this paper. As a side note, if an additional second-pass lattice rescoring is further applied (with a proper combination of RNNLM and the word n-gram language model), the WER of our system on the development set can be further decreased to 18.86%.

In the last set of experiments, we report on the results of our final system submitted to the ASR challenge organizer. The final system performed an ensemble of the ASR systems previously evaluated in Tables 2 and 3. Specifically, the ASR results of all the abovementioned systems, in the form of word lattices, were first merged (unified) into a single word lattice with equal prior weights. We then conducted Minimum Bayes-Risk (MBR) decoding on the merged lattice, whose outputs were served as the results of our final ASR system. On a separate front, since it was allowed to make use of the label-agnostic evaluation dataset (viz. the corresponding reference transcripts were not provided), we thus went one step further to leverage the label-agnostic evaluation dataset for acoustic model training. That is, we conducted semi-supervised learning of the acoustic model by additionally using the unlabeled evaluation dataset and adopting the strategies proposed in [25] and [26]. As we can see in Table 4, our proposed system-ensemble approach (Row 1) can further improve the best WER results on the development dataset from 18.86% to 16.70%. Further, with the additional use of semi-supervised learning, though our best WER result on the development dataset was slightly degraded from 16.70% to 16.74%, such combination of the system-ensemble approach with semi-supervised learning achieved a WER result 17.59% on the evaluation when using our best ASR system configuration. Finally, Table 5 summarizes the final WER results of the participating teams on the evaluation dataset of the TLT-school Challenge.