An Investigation on Applying Acoustic Feature Conversion
to ASR of Adult and Child Speech

The overall workflow of acoustic feature conversion process includes two steps as illustrated in Figure 1. The first step is to perform adult-to-child feature transformation via a conversion model. The conversion approaches are categorized into two streams: signal processing based methods, e.g., formant modification (FM) and time-scale modification (TSM); DNN based methods, e.g., disentanglement-based AE and CycleGAN. After feature conversion, the second step is to train an ASR model using the converted speech feature set. The degree of improvement on recognition performance on child test speech is evaluated against systems without using converted features.

Voice conversion (VC) aims to modify content irrelevant information while preserving the linguistic content. The same idea is adopted for acoustic feature conversion except that the vocoder for converting spectrogram to waveform is not needed. AdaIN in [18] is a disentanglement-based auto-encoder network, which performs one-shot VC by separating content and speaker embeddings with instance normalization (IN). Three modules constitute the AdaIN. They are the content encoder $E_{c}$, the speaker encoder $E_{s}$ and the decoder $D$, respectively. The acoustic feature $\mathbf{x}$ is fed into $E_{c}$ and $E_{s}$ as input. $E_{c}$ generates a sequence of content embeddings $\mathbf{zc}$, while $E_{s}$ produces the speaker representation $\mathbf{zs}$. The decoder outputs $\mathbf{\hat{x}}$ which is intended to reconstruct $\mathbf{x}$ from $\mathbf{zc}$ and $\mathbf{zs}$. The IN is applied to the content encoder to remove speaker-related information, while the adaptive IN [27] is used to provide the global speaker information encoded by $E_{s}$ to the decoder. The whole network is trained to minimize the reconstruction loss $||D(\mathbf{zc},\mathbf{zs})-\mathbf{x}||_{1}$ in an unsupervised manner. The conversion is performed by feeding $\mathbf{zc}_{src}$ of the source speaker and $\mathbf{zs}_{tar}$ of the target speaker to the decoder, i.e., $D(\mathbf{zc}_{src},\mathbf{zs}_{tar})$.

As shown in Figure 2, the AdaIN network (in dashed block) is the core module of the DAE framework, which aims to perform adult-to-child conversion in the acoustic feature space. The subscript $A/C$ in a variable indicates that it represents the adult or the child domain. The solid-line arrows illustrate the reconstruction process in the training phase, while the dashed arrows refer to the conversion stage. $\mathbf{\overline{zs}}_{C}$ is the representation of child domain and calculated by averaging all speaker embeddings of child speech utterances. $\mathbf{x}_{A2C}$ denotes the converted acoustic feature, which is generated by replacing $\mathbf{zs}_{A}$ with $\mathbf{\overline{zs}}_{C}$. In the evaluation of ASR models trained by $\mathbf{x}_{A2C}$, to overcome the great mismatch between the real and generated speech features, the child test speech is also performed by $C2C$ conversion, and the corresponding generated feature is denoted as $\mathbf{x}_{C2C}$. The whole conversion process can be expressed as:

The original AdaIN network is trained in a fully unsupervised manner, i.e., only speech data are required. Nevertheless, the domain class label and median F0 value are used to facilitate a better conversion. A domain-critic module is built on top of $\mathbf{zc}$, which is adversarially trained to force content embeddings to be domain-invariant. Since the F0 distribution is found to be an important attribute to discriminate adult and child speech, the median F0 of each utterance is estimated and used to train the F0 classifier. Moreover, the matrix subspace projection (MSP [28]) is applied to the speaker embedding space. It transforms $\mathbf{zs}$ into a low-dimensional attribute space, adult versus child domain in this case, which is expected to make $\mathbf{zs}$ contain more discriminative domain information.

Acoustic feature conversion is performed to reduce mismatch between the two speech domains, namely adult speech and child speech. The adult speech ($A$) data are from a subset of the AISHELL1 corpus. The child speech ($C$) data are from the 2021 SLT CSRC (short for Children Speech Recognition Challenge) dataset. AISHELL1 [29] is an open-source dataset of Mandarin speech by adult speakers in reading style. It is intended and widely used for ASR research. The CSRC dataset [30] consists of two parts of child speech with different speaking styles. The first part, denoted by $C_{1}$, contains read speech. The second part, denoted by $C_{2}$, contains conversational speech. The test set of both $C_{1}$ and $C_{2}$ contain 2 hours of speech. The train data sets of AISHELL1 and CSRC are summarized as in Table 1.

80-dimensional log Mel spectrograms are extracted from raw audio with 25 ms window length and 10 ms hop length. All audio data are sampled at 16 kHz. The feature sets of $A$, $C_{1}$ and $C_{2}$ are pooled together where global mean and variance normalization (GMVN) is applied. The disentanglement-based AE network is trained with the normalized features. It is noted that the speech in $C_{2}$ exhibits significant mismatch with $C_{1}$ in the preliminary ASR experiment. This is related to the speaking styles difference, i.e., read speech vs conversational speech. Therefore, we consider adult-to-child conversion under the same speaking style. Specifically, $C_{1}$ is regarded as the target child domain of interest, meaning that there are totally three conversion types, namely $A2C_{1}$, $C_{1}2C_{1}$, and $C_{2}2C_{1}$.

The domain-critic module is implemented with a three-layer fully-connected (FC) network for domain classification on three classes, namely $A$, $C_{1}$, and $C_{2}$. The F0 classifier adopts a similar network structure to perform a 10-class task. The utterance-wide median F0 values are divided in 10 equal intervals covering the range from 100 Hz to 350 Hz. F0 estimation is implemented by the Parselmouth library. When applying MSP on the speaker embedding space, the attribute label is designed as a 2-dimensional vector, in which the first element represents the adult/child domain and the second element represents the read/conversational speaking style. The core AdaIN network follows the same Conv1D layers architecture as described in [18]. The channel dimension is 512. The variational regularization is not imposed on the content embeddings $\mathbf{zc}$ [31], though it is applied in the original AdaIN network by default. The model is trained with $segment\_length=128$, $batch\_size=128$ for 100k steps. The optimizer is Adam with $learning\_rate=0.0005$ [32].

In the conversion stage, the average of speaker embeddings from the $C_{1}$ development set (denoted as $\overline{\mathbf{zs}}_{C1}$) is computed to represent the target child domain. The adult-to-child speech feature conversion is performed by replacing the original speaker embedding with $\overline{\mathbf{zs}}_{C1}$. The decoded output will be de-normalized as the converted spectrogram.

Apart from the disentanglement-based AE framework, other conversion methods are evaluated in our experiments. The MaskCycleGAN network [33] is utilized to perform direct domain transformation from $A$ to $C_{1}$. Two generative models and two discriminative models are trained with 20k utterances for each domain. In terms of non-DNN methods, F0-based feature normalization conducts the formant modification by assuming a linear relation between F0 and formants on the Mel scale [26]. The target value of normalized F0 is empirically set to be 270 Hz. In addition, statistic matching on spectrum space (denoted as Stats) is evaluated [34], in which the set $A$’s feature is first normalized by its mean and standard deviation (std), and then de-normalized by the mean and std of the $C_{1}$ set. The Correlation Alignment (Coral [35]) is also experimented to minimize the domain shift by aligning the second-order statistics of $A$ and $C_{1}$ distributions. The only difference with the Stats method is that Coral uses the covariance instead of the std.

The speech recognition model used in our experiments adopts the joint CTC-attention architecture [3]. It consists of three components, namely the shared encoder, the attention decoder, and the CTC loss layer. The encoder comprises 12 Conformer layers and the decoder has 6 Transformer layers [36]. The input feature of ASR model is 80-dimensional log Mel spectrogram, either the original or the converted one. The ASR model is trained to minimize the weighted summation of the attention decoder loss and CTC loss, where the CTC loss weight is empirically set to be 0.3. The maximum number of epochs is set to be 150 for model convergence on 60-hour training data. The SpecAug [37] and GMVN are applied by default. The language model is disabled if not stated otherwise. The ASR experiments are conducted using the ESPnet toolkit [38].

The word error rate (WER) results of child test sets are given in the Table 2. The baseline model trained by original $A$ set attains $8.9\%$ WER on the adult test set. In $C_{1}$ and $C_{2}$, the performance of baseline model suffers from great mismatch even with language model (LM). The ASR models trained with converted acoustic features are compared against the baseline. Five feature conversion methods are evaluated in our experiments. The DAE based conversion model (DAE for short) achieves $0.5\%$ absolute recognition improvement on the $C_{1}$ test set, and $1.5\%$ on the $C_{2}$ test set. The best recognition performance can be attained by the F0 normalization (F0-norm). Not all of the conversion methods can lead to performance gain, e.g., the CycleGAN. Although, it is still surprising to observe that the approach of statistic matching takes no effect and even makes degradation. The mean statistics of the log Mel spectrogram with 80 channels are plotted in Figure 3, in which the Stats and Coral curves overlap with that of $C_{1}$. A reasonable formants up-scale is noted in F0-norm compared to the original $A$.

To investigate the efficacies of different network components in the DAE conversion model, an ablation study is carried out as shown in Table 3. The usage of domain critic and F0 classifier are represented by the symbol DAT and F0_clf. Having DAT on the content encoder is important for effective disentanglement. The role of MSP seems not to be as useful as F0_clf. Imposing variational regularization on the content encoder does not work. Besides, an experiment with vanilla DAE found that additional benefits ($29.1\%\xrightarrow{}28.8\%$) can be attained by training more steps (200k in this case). The limited performance improvements are noted in all cases, which may suggest the quantity of data to train DAE is insufficient.

Since paired speech data are not available, i.e., parallel utterances of the same content are not available from adult and child domains, spectral distance measures like the Mel cepstral distortion (MCD) are not applicable. An adult-child classification model is trained to distinguish the three domains, i.e., $A$, $C_{1}$ and $C_{2}$. We hypothesize that the converted features are obtained from a high-quality $A2C_{1}$ conversion process if they are classified into the $C_{1}$ domain. The percentages of different types of converted features classified as $C_{1}$ are shown as in Table 4. The CycleGAN appears to perform very well, having $100\%$ converted features classified as $C_{1}$. However, the ASR model trained with these features shows performance degradation on test speech from $C_{1}$. Generally, DNN-based conversion methods are able to generate a high percentage of features classified as $C_{1}$. This may be related to the robustness issue of deep classification model. The pearson correlation coefficient between the WERs and $C_{1}$ classification percentages is $0.07$.

In view of distinctive F0 levels in adult and child speech, the utterance-wide median F0 values of these two domains are estimated. The F0 distributions are visualized in Figure 4 using 100 utterances from different types of acoustic features, including the converted and the unconverted ones. The ASR is expected to have better performance if the F0 distribution of the converted features is close to that of the $C_{1}$. The 1D Wasserstein distance is adopted to measure the discrepancy of two F0 distributions [39], which are listed in the last column of Table 4. The pearson correlation coefficient with WERs is $0.83$.