Aphasic Speech Recognition using a Mixture of Speech Intelligibility Experts

Aphasia is an acquired language disorder that manifests itself in speech [1, 2]. In the U.S., approximately two million people are living with aphasia and more than 180,000 acquire it every year due to brain injury, most commonly from a stroke [3]. Aphasic speech can be difficult to understand, and as a result, persons with aphasia (PWAs) are typically characterized as having low speech intelligibility. Previous studies have shown that speech-language therapy activities have a positive effect on the communication abilities of PWAs [4, 5]. However, current speech-language analyses and speech therapy activities require in-clinic visits with a trained speech-language pathologist, which is both costly and time-consuming. Speech-based technology provides an attractive avenue for assisting medical professionals due to its low cost, high accessibility, and promising results for automating processes that relate to the assistance and analysis of those with disordered speech [6, 7, 8, 9, 10, 11]. Many of these automatic analyses rely on accurate speech transcriptions, which motivates the need for better automatic speech recognition (ASR).

There are several challenges associated with building robust ASR systems for aphasic speech, which include low speech intelligibility [12], high (inter- and intra-) speaker variability [13, 14], and relatively limited data [15]. Some artifacts of low speech intelligibility include halting speech, use of jargon, as well as various phone-level and word-level substitutions (paraphasias). These speech impairments can also be compounded by co-occurring motor control disorders such as apraxia of speech and dysarthria [16]. Additionally, in regards to machine learning efforts, it’s difficult to apply large, monolithic networks that traditionally take advantage of big data due to the low-resource domain of aphasic speech.

Prior works have focused on improving ASR for disordered speech by addressing the issue of speaker variability through speaker-selective model adaptation [15] and speaker embeddings [10]. In modeling speaker variability, it is likely that these models also capture speech intelligibility, as it is an underlying feature of speaker variability. Therefore, we believe that the success of these approaches demonstrates the potential for modeling speech intelligibility for disordered speech recognition.

In this work, we present a novel approach that explicitly models speech intelligibility in the acoustic model of an ASR system to provide more robust aphasic speech recognition. To accomplish this, we use an acoustic model architecture designed to handle the wide range of speech intelligibility present in aphasic speech. Specifically, we use a Mixture of Experts (MoE), deep neural network (DNN) acoustic model where each expert in the model focuses on specific severity classes defined using the speaker Aphasia Quotient (AQ) severity metric[17]. We introduce a speech intelligibility detector (SID), trained to detect severity levels of a given speech frame, to automatically estimate the contribution of each expert in our proposed acoustic model. Our results demonstrate that the proposed MoE acoustic model improves phone recognition performance over a traditional one-size-fits-all acoustic model across all speaker severity levels.

Disordered speech recognition is an active area of study due to its important role in automatic speech-based analyses and potential impact on remote health monitoring applications. The disordered speech recognition domain shares many challenges with the accented speech recognition domain. Several approaches such as joint modeling, multitask-learning, and MoEs have been proposed in the field of accented speech recognition to address these challenges  [18, 19, 20].

Le and Mower Provost improved aphasic speech recognition performance by adapting the acoustic model through appending input acoustic features with fixed-length speaker identity vectors (i-vectors) [10]. i-vectors reduce speaker variability in an acoustic model by providing information about general speaker characteristics (e.g., pronunciation patterns) [21]. Christensen et. al. showed that speaker selective training improves isolated word recognition on disordered speech [15]. The authors applied maximum a posteriori (MAP) adaptation to various HMM-GMM acoustic models with a select pool of speakers. They found that speaker-independent models trained on non-dysarthric speech were inferior to models trained on dysarthric speech for the task of disordered speech recognition, highlighting the importance of acoustic similarity in disordered speech recognition.

Findings from previous research suggest that careful consideration should be taken when training acoustic models for disordered speech applications as a high variation in speech intelligibility exists among speakers. One approach to address the high variation present in disordered speech is to train several independent acoustic models, one for each intelligibility group. However, the main challenge with training independent models is the limited amount of data available for each group. The proposed MoE acoustic model mitigates this challenge by allowing us to train a single network that explicitly decomposes the acoustic model into several expert acoustic models, where each expert is focused on a specific range of speech intelligibility. MoE based approaches in acoustic modeling have proven to be successful in accented speech recognition tasks by allowing acoustic models to handle several accents simultaneously [20]. To the best of our knowledge, we are the first to explore utilizing speech intelligibility in an acoustic model for aphasic speech recognition.

AphasiaBank is a large audiovisual database composed of multiple sub-databases, which contain interactions between PWAs and clinicians[22]. Sub-databases contain a mix of speakers with varying severity and aphasia type (i.e., Broca’s, Conduction, etc.). Some sub-databases also include healthy speakers who followed a similar elicitation protocol. The elicitation protocol consists of free speech (i.e., personal story, describing current day), semi-structured speech (i.e., picture description tasks), and procedural discourse (i.e. “How would you make a peanut butter and jelly sandwich?”). In this work, we focus on English only sub-databases containing at least four speakers.

The final dataset we use contains 25 sub-databases with 106,509 utterances and over 95 hours of speech. There are 537 speakers (237 healthy, 138 mild-aphasia, 115 moderate-aphasia, 46 severe-aphasia). Of these speakers, 241 are male and 210 are female, with the average age being 62.5$\pm 14.5$ years. Our metric for speech intelligibility is the WAB-R Aphasia Quotient (AQ), which is a measurement for aphasic severity [17]. AQ scores $\geq 75$ are considered mild, scores of 75-50 are considered moderate, and scores $<50$ are considered severe.

We randomly split the dataset into training, validation, and test partitions using a 70/5/25 rule. Our test set is formed by randomly selecting 25% of the data in each sub-database. In addition, our splits are speaker-independent and severity-stratified to ensure similar distributions of severity across the three partitions. All recordings were down-sampled to 16kHz in this work.

The architecture for the proposed MoE acoustic model is shown in Figure 1. The inputs to the model are acoustic features and the outputs are context-dependent triphone state probabilities (senones). The acoustic model consists of two parts: (1) shared initial layers used for feature extraction and (2) several expert networks each focusing on a pre-defined severity group. We fix the number of experts to four (healthy, mild, moderate, and severe) following the speech intelligibility levels defined in Section 3. The shared layers of the model are designed to capture broad phonetic representations applied across all severities, while the experts are designed to reduce speech intelligibility variation within their respective networks. All hidden layers are fully-connected, consist of 1024 nodes, and use a ReLU activation function. The shared layers of the model consist of four hidden layers and each expert is composed of two hidden layers and the final output layer applies a softmax function. Our baseline network is a fully connected DNN with 6 hidden layers.

In addition to the acoustic model, we also propose a speech intelligibility detector (SID) which predicts speech intelligibility levels, as quantified by the AQ severity metric, given the acoustic features and utterance-level speaker embeddings. The outputs from the SID are used as weights for computing the contribution of each expert in our acoustic model. We use speaker embeddings extracted from a pre-trained x-vector system [23] and apply principal component analysis (PCA) to reduce the dimensionality of these x-vectors from 512 to 32 to avoid over-fitting. x-vectors have previously been used to estimate speech intelligibility [24] due to their ability to encode many properties about a speaker (i.e. speaking style, speaking rate, etc.) [25, 26]. The SID consists of two hidden layers and a final output layer which applies a softmax function.

The output of the acoustic model at time frame $x$ is determined using the following equation:

where $y(x)$ are the final senone probabilities, $w_{i}(x)$ is the contribution of expert $i$ as computed by the SID, $M_{i}$ are the senone probabilities from expert $i$, and $N$ is the total number of experts in the model.

We use the Kaldi toolkit [28] for feature extraction and HMM-GMM training and use the pytorch-kaldi toolkit [29] for neural network training. The feature extraction pipeline involves the extraction of 13-dimensional mel frequency cepstral coefficient (MFCC) features, with cepstral mean and variance normalization (CMVN) spliced across 7 frames ($\pm$3). Frames are created using a sliding window of size 25 ms with a 10 ms shift rate. We apply Linear Discriminant Analysis (LDA) to reduce the dimension size to 40 and use Maximum Likelihood Linear Transformation (MLLT) to further decorrelate the features [30]. Lastly, we apply feature-space Maximum Likelihood Linear Regression (fMLLR) [31] as a means of speaker adaptation and use the resulting output as acoustic features for HMM-DNN training. The final inputs to the DNN acoustic models are 40-dimensional fMLLR acoustic features with a context window of 11 ($\pm$5) frames, for a total feature size of 440.

We train our networks with an SGD optimizer using an initial learning rate of 0.01. The learning rate is halved after each epoch when the performance on the validation set stagnates. We use early stopping to reduce overfitting. For evaluation, we use Phone Error Rate (PER), following the same approach as [10]. To assess the statistical significance of PER improvement between our models, we use a bootstrap estimation test provided by the Kaldi compute-wer-bootci tool [27]. This method computes probability of improvement (PoI), where PER is computed on $10^{4}$ sampled utterances for a given model in order to estimate the spread of PER around its mean. The difference in PER is then computed for each sample between the baseline and the proposed model being evaluated, providing an estimate of the probability of improvement.

In this work, we first show that modeling speech intelligibility in a MoE acoustic model improves phone recognition across all severities with the use of an oracle SID. We then demonstrate the importance of data assignment when training the MoE model. As we move away from the use of an oracle SID, we show how a neural SID can be trained using fMLLR and x-vector features to estimate severity classes. Lastly, we show that the outputs of the SID can be used to merge expert senone predictions at test time for improved MoE performance over a one-size-fits-all acoustic model.

In section 6.2.1 we suggest that a potential limitation of this work is in utilizing speaker-level speech intelligibility scores. In future work, we plan to explore smaller time-scale metrics for estimating speech intelligibility and how this can be used to improve the SID and ultimately our MoE acoustic model.

This material is based in part upon work supported by the Toyota Research Institute (“TRI”) and by the National Science Foundation Graduate Research Fellowship Program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF, TRI, or any other Toyota entity.