Refining Automatic Speech Recognition System for older adults

TL is a type of adaptation widely used in data-scarce scenarios aiming to reuse the learned knowledge of a base DNN model on a related target domain. The fundamental idea is tuning the weights of a pre-trained model. WTL and hidden linear TL (HLTL) are two well-studied ones. WTL initializes the target model with a pre-trained base model where both models share the exact same DNN architecture. Then, tune the entire or a portion of the DNN with a small learning rate [10]. HLTL builds upon a hypothesis where similar tasks share common low-level features. It considers the base model as part of the target model by using the base model as a feature extractor that makes the first process on the input data [11]. Other than these well-studied methods, Factorized Hidden Layer (FHL) models the pre-trained parameters with a linear interpolation of a set of bases [12, 13]. A key ingredient to a successful model adaptation is a well-trained base model from a similar task that has learned from a relatively large and yet diverse training samples [14].

Researchers utilized the attention mechanism to manage long sequence memory without forgetting [15]. Given an input vector, the mechanism aligns the input to each memory cell, and outputs the summarized memory. The following equations are the general operations of the mechanism:

	$\displaystyle score_{t,j}$	$\displaystyle=\langle\phi(q_{t}),\psi(k_{j})\rangle$
	$\displaystyle\alpha_{t,j}$	$\displaystyle=align(q_{t},k_{j})$
		$\displaystyle=\frac{score_{t,j}}{\sum_{j^{\prime}=1}^{length(V)}score_{t,j^{\prime}}}$
		$\displaystyle=softmax(score_{t,*})$
	$\displaystyle o_{t}$	$\displaystyle=\sum_{j=1}^{length(V)}\alpha_{t,j}*v_{j}$

where $q_{t}$ is the $t$th column of matrix $Q$. The vectors $k_{j}$ and $v_{j}$ are the $j$th columns in matrices $K$ and $V$, respectively. These two matrices represent the identity and the content of the memory. The symbols, $\phi$ and $\psi$, represent linear layers. The left graph in Figure 1 describes the general structure of this mechanism. The alignment is conditional to the input $k$. Although the conditional alignment shows its power in multiple researches [15, 16], a small dataset is not sufficient to train these additional layers in TL. In Section 3, we will introduce our conditional-independent attention mechanism which only requires a limited amount of additional parameters.

DS2 [17] is an end-to-end ASR system that leverages a recurrent neural network (RNN) for modeling the spectrogram to the sequence of 26 English letters. This architecture utilized convolution (Conv) layers for low-level information processing and uses bidirectional gate recurrent unit (bi-GRU) layers for high-level processing. The output layer is a linear layer. In the decoding process, the DS2 [17] utilizes a beam search approach [18] to search for the transcription with the highest probability based on the combination of probabilities from the DS2 model and a n-gram language model. The left graph of Fig 2 is the architecture that we used in our experiments.

We remove the alignment function from the standard attention mechanism and manually assign the attention to the outputs of GRU layers based on our knowledge about the model. The following equation is the operations of manual attention:

$\displaystyle O$

$\displaystyle=\sum_{n=1}^{3}\alpha_{n}*V_{n}$

where $V_{n}$ is the output of GRU_n and $\alpha_{n}$ is the attention that we manually assign to it. This format is the same as the weighted average of GRU outputs. This does not have additional parameters. Therefore it is ideal for evaluating the effectiveness of utilizing intermediate information. We name this mechanism as the manual attention mechanism (MAM).

To learn the attention from the target dataset, we apply a function to learn assigning attention in a conditional-independent fashion. The learnable attention mechanism can be described with following operations:

	$\displaystyle R$	$\displaystyle=\left[\begin{array}[]{c|c|c}R_{1}&R_{2}&R_{3}\end{array}\right]$
	$\displaystyle M$	$\displaystyle=\left[\begin{array}[]{c|c|c}M_{1}&M_{2}&M_{3}\end{array}\right]$
	$\displaystyle score$	$\displaystyle=R\cdot M$
	$\displaystyle\alpha_{*}$	$\displaystyle=softmax(score)$
	$\displaystyle O$	$\displaystyle=\sum_{n=1}^{3}\alpha_{n}*V_{n}$

where $score$ is a one-by-three matrix and $V_{n}$ is GRU_n’s output. The matrix $R$ is an one-by-$r$ non-negative matrix and is the additional parameter matrix. $R_{1}$, $R_{2}$ and $R_{3}$ are partitioned matrices of $R$. Each represents the importance of the output of the GRU layer with the identical subscript ID. Matrix $M$, which is a $r$-by-three binary matrix, summarize each partitioned matrix by summing up its elements. This matrix is fixed once we set the column sizes for all $R_{n}$ where $n\in\{1,2,3\}$. If $R_{n}$ contains more elements than others, it is more likely that GRU_n’s output receives higher attention than others. Thus, we view matrix $R$ as the matrix of representatives and matrix $M$ as the matrix of tally. By setting the column size of $R$’s partitioned matrices, we can gently encourage the mechanism to assign extra attention on layers which are important in our prior knowledge. For example, if we think GRU₁ should receive more attention, we can define the column sizes for $R_{1}$, $R_{2}$ and $R_{3}$ to be 2, 1 and 1, respectively. The $R$ and $M$ should be the following format:

	$\displaystyle R$	$\displaystyle=\left[\begin{array}[]{cc|c|c}ele_{1}&ele_{2}&ele_{3}&ele_{4}\end{array}\right]$
	$\displaystyle M$	$\displaystyle=\left[\begin{array}[]{c|c|c}1&0&0\\ 1&0&0\\ 0&1&0\\ 0&0&1\\ \end{array}\right]$

where $ele_{h}$ is a scaler for all $h\in\{1,2,3,4\}$. We name this mechanism as the learnable attention mechanism (LAM).

As our target speakers are seniors, we remove moderators’ utterances based on the given speaker labels of utterances in the manual transcription. We extract word-level timestamps using a force-alignment algorithm available in Gentle¹¹1https://github.com/lowerquality/gentle, which is an open-source software, for each senior’s utterance. We segment long utterances into multiple pieces that are less than 7 seconds long by utilizing the word-level timestamps. Finally, we removed all utterances that are less than 3 seconds.

For both validation and testing sets, we randomly select 2 MCI and 2 healthy participants from both genders and leave 53 participants for the training set. With 14 hours of transcribed speech, this splitting leaves about $12.7$ hours of audio recordings for the training purpose. The total recording duration in the validation set and testing set are $0.66$ and $0.56$ hours, respectively. We use the validation set to select hyperparameters (i.e., learning rate) and the testing set is only used for assessing the model performance.

We use the MAM at the AttenL and define the basic attention unit to be $1/6$. All $\alpha$s must be an integral multiple of the unit. We use M-$\alpha_{1}$-$\alpha_{2}$-$\alpha_{3}$ to present the setting of attention in an experiment. For example, if we assign all attention to GRU₁, the attention setting is M-6/6-0/6-0/6. In train process, we fine tune the modified model for 40 epochs.

In Fig 3, the PTM completely outperforms the base mode on senior domain. The top 5 settings, where we assign more than half attention to GRU₁’s output, outperform PTM. The M-$4/6$-$2/6$-$0/6$ achieves $1.58\%$ absolute improvements over the PTM. Since we use the pre-trained Linear₁, who used to receive GRU₁’s output only, GRU₁ is naturally strongly related to Linear₁. On the other hand, unreasonable settings, assigning small attention to the output of GRU₁, perform worse than PTM. This experiment brings us confidence on utilizing intermediate information.

We adopt the LAM at the AttenL to learn the attention from the target dataset. We use L-$r_{1}$-$r_{2}$-$r_{3}$ to specify the column sizes of partitioned matrices in $R$. We evaluate four settings: L-1-1-1, L-4-1-1, L-3-2-1 and L-5-4-0. We use the first one to evaluate LAM’s learning ability. The other settings are designed to assign more attention to GRU₁’s output. All experiments first train the AttenL for $5$ epochs while freezing other layers. Then, we reverse the freezing status and train the model for $40$ epochs. We try each setting for $5$ times to evaluate the influence of randomly initialization on additional parameters.

In Fig 4, random initialization dramatically influences L-1-1-1’s final outcome. On the contrary, we achieve more stable performance when applying prior knowledge through the column setting. This proves that setting the column sizes, based on prior knowledge, positively influences a tuned model’s performance. Another evidence comes from the comparison between L-4-1-1 and L-3-2-1. Although the total column size of $R$s are the same, models with L-4-1-1 perform more stable than the other. Both L-4-1-1 and L-5-4-0 outperform PTM. Our results are marginally worse than the optimal WER in Section 5.1, but we cannot exclude the negative influence from the small training set. Moreover, LAM can be transformed to conditional-dependent form, which is a flexibility that MAM does not have.