Multi-modal Automated Speech Scoring using Attention Fusion

An important aspect of scoring rubric is the delivery of a spoken response. It helps in measuring speech quality, pronunciation, fluency, stress, and intonation [5]. To capture this acoustic information of audio samples, we propose a Recurrent Convolutional Neural Network (RCNN) architecture. CNN is able to extract local features, whilst the RNN is able to summarise the long temporal information [11]. Such an architecture proves beneficial to us because of the nature of our data; which consists of long monologues (ranging from 60 to 120 seconds), and have long term interdependencies present in it.

We split the log-scaled mel spectrograms of the audio samples along the temporal dimension into frames of fixed size (discussed in Section 3.2). Each frame is passed to 5 sets of CNN consisting of 2 layers of 1D convolutions followed by max-pooling with the number of filters doubling after every set. Then, we apply global max pooling to each frame. The output vectors obtained from this network are then fed into the RNN component, which in our case is a BDLSTM allowing us to capture the sequential structure of audio, taking into account both past and future speech production.

It is also essential that the content of the spoken response is relevant to the topic and appropriate. Given the nature of our data, we use response-based content scoring, where we compare the similarity of response with previously scored responses. This approach is popular in both automated scoring of essays [6, 12] as well as speech scoring [7] as it allows capturing variations in responses comprehensively.

To achieve this, we generate transcripts of spoken responses using a non-native ASR system. These transcripts are pre-processed, and words are mapped to their word embeddings using an embedding layer. This embedding layer is initialized with 300-D glove embeddings [13] trained on Wikipedia and is optimized during training. Similar to the acoustic model, we use BDLSTM network here to capture sequential the structure of words and learn the development of content at different score levels.

In the past few years, attention mechanism [14] has achieved state-of-the-art on various natural language processing tasks. It enables weighting contextual information learned during each time step, allowing the model to determine which states to pay attention to. In our study, we use attention fusion [10] to combine features from the text and audio modality for grading spoken responses (see Figure 1). Given bi-directional temporal hidden states from audio model (BDRCNN), $h^{a}$, and text model (BDLSTM), $h^{t}$, we combine these states, $h^{m}=[h^{a},h^{t}]$ (where $[\cdot]$ denotes the concatenation of the state vectors) and compute context vector, $c^{m}$ as:

$$e_{t}=h_{t}^{m}w_{a};a_{t}^{m}=\frac{\exp(e_{t})}{\sum_{i=1}^{T}\exp(e_{i})};c^{m}=\sum_{t=1}^{T}a_{t}^{m}h_{t}^{m}$$

(1)

Here, $h_{t}^{m}$ is the multimodal representation of the lexical/acoustic cues at time $t$ and $w_{a}$ is the weight matrix for the attention layer. The multimodal attention importance scores for each time, $a_{t}^{m}$ is obtained by multiplying the representation $h_{t}^{m}$ with the weight matrix, $w_{a}$, followed by normalization to construct probability distribution across these cues. Finally, we calculate the context vector of cues, $c^{m}$, as a weighted summation over all time steps using multimodal attention importance scores as weights. This context vector is passed through a final dense layer before generating score for a input spoken response.

For uni-modal baselines, we applied attention weighting to individual BDRCNN and BDLSTM models (BDRCNNAttn [A] for audio and BDLSTMAttn [T] for text) and trained them separately for comparison.

In this study, we utilize data collected by Second Language Testing Inc. (SLTI) administrating Simulated Oral Proficiency Interview (SOPI) for L2 English speakers, majorly from the Philippines. Each examinee is presented with a form consisting of six prompts of varying difficulty levels on their computer screen, and their responses are recorded. These responses are then scored independently by two expert raters. A third expert rater resolves disagreements, if any. Both the prompts and rubrics are aligned with the guidelines of the Common European Framework of Reference (CEFR) [15]. To answer the questions, candidates require explanatory and argumentative abilities. Depending on their performance on these tasks, they are assessed to have English oral proficiency ranging from lowest grade, A2, to highest grade, C1 (with the maximum score obtainable being equal to the difficulty of the prompt). The examinees are mostly high school graduates of age 17 years and above. Most of the recorded audios have a size lesser than 120 seconds. Table 1 shares the other relevant statistics of the dataset. Dual-scoring with the intervention of third rater and robust rubric ensured that the scoring process is not biased towards irrelevant details like gender and features like text length. As a point of reference, the correlation between score and text length if 0.35 across all the prompts.

For each prompt, we stratify and split the responses into train, validation and test with the ratio $70:10:20$. We train the model for every prompt and report the quadratic kappa score obtained on the test set. For training the models, first, we downsample the audio responses to $16kHz$ and generate their log-scaled mel spectrograms. The number of mel bands is $128$, the length of the FFT window is $2048$ samples, and the hop length is $512$ samples. These spectrograms are then normalized and zero-padded to maximum response length. Finally, we split the output along the temporal dimension into frames of size $128\times 128$.

We use a Deep Speech 2 [16] based ASR system with trigram language model for transcription of these non-native responses. The ASR system is trained on approximately $1000$ hours of audio sampled from CommonVoice [17] and LibriSpeech dataset [18] and further fine-tuned on approximately $22$ hours of transcribed non-native spoken responses from our dataset. This ASR system achieves a word error rate of $16.63\%$ on approximately $5$ hours of non-native spoken responses.

We tokenize the transcripts using the spacy tokenizer [19] and lowercase them. Words not part of training data vocabulary are mapped to the unknown token and initialized with zero vector in the embedding layer. We treat the scoring of the responses as a regression problem. The CEFR scores ($N$ levels) associated with the responses are mapped to the range $[0,N-1]$ based on the increase in proficiency level. These scores are then normalized to the range of $[0,1]$ for training. While testing, we rescale the model output to the original score range and measure the performance.

For evaluating the models, we use the Quadratic Weighted Kappa (QWK) score as our primary metric. The QWK measures the agreement between two graders. It normally ranges from $0$ to $1$ and can also be negative if there is lesser agreement than expected by chance. To calculate the QWK score, a weight matrix $W$ of size $N\times N$ is constructed, where $N$ represents the number of classes, using the formula as mentioned in Eq. 2.

$$W_{i,j}=\frac{(i-j)^{2}}{(N-1)^{2}}$$

(2)

Next, we construct a confusion matrix $O$ of size $N\times N$, where $O_{ij}$ is equal to the number of speech responses that receive a grade $i$ by the human and a grade $j$ by the model.

Then we create the histogram matrix of expected grades $E$ by computing the outer product between the histogram vector of actual grades and the histogram vector of predicted grades, followed by a normalization which ensures that $E$ and $O$ have the same sum. Lastly, the QWK is obtained as in Eq. 3.

$$\kappa=1-\frac{\sum_{i,j}W_{i,j}O_{i,j}}{\sum_{i,j}W_{i,j}E_{i,j}}$$

(3)

We use the Adam optimization algorithm to minimize the mean squared error (MSE) loss over the training data (see Eq. 4).

$$MSE(y_{true},y_{pred})=\frac{1}{N}\sum_{i=1}^{N}(y_{true}^{i}-y_{pred}^{i})^{2}$$

(4)

where $N$ is the number of audio responses for a given prompt, and $y_{true}^{i}$ and $y_{pred}^{i}$ are the normalized human grade and model predicted grade for response $i$, respectively. To prevent overfitting of the model, we use dropout regularization. We train the model for a fixed number of epochs with early stopping on validation loss and select the model with the best QWK score on the validation set. We also optimize thresholds between each score for rounding off the predictions in order to maximize the QWK. This is done by defining the search space of thresholds between each possible score and optimizing for maximum QWK using the hyperopt package [20].

In Table 2, we compare MMAF model with two strong baselines: BDRCNNAttn [A] and BDLSTMAttn [T] models on the SOPI dataset. It shows that fusing both the modalities using attention gives the best scores across all the prompts. Overall, the improvement in average QWK score across prompts for MMAF when compared to BDLSTMAttn [T] and BDRCNNAttn [A] models is close to 8.2% and 42.6% respectively. Optimizing thresholds improves the scores by approximately 4.8% over text-only models and 36.5% over audio-only model. Since prompts (1, 2 and 6) have high class imbalance and QWK is not a reliable metric since it becomes too sensitive towards the under-represented classes [21, 22], we also track MSE for comparison. We observe the average MSE across prompts for MMAF model decreases by 5.5% and 18.3% when compared to BDLSTMAttn [T] and BDRCNNAttn [A] models respectively. Table 2 shows that the performance of all models for lower difficulty level prompts (prompt 1, 2 and 6) is poor compared to higher difficulty prompts (prompt 3, 4 and 5). The human-human agreement also shows similar performance. Table 5 shows the variation of attention scores across all the prompts. It is evident that attention weights prefer text over audio broadly in the ratio of 85:15. Due to the non-availability of any public dataset or model, we were not able to show the same results on those [7, 9].

For further analysis, we select two exemplar prompts: one from the lower difficulty range (prompt 1) and one from the higher difficulty (prompt 4). Table 5 shows that audio attention weights partially increase for low scored responses. By checking the samples, we could infer that one of the reasons for this is due to the lower intelligibility of audios for those items. This, in turn, caused ASR word error rate to go up nudging the model to partially shift the attention to audio over the text modality.
We also investigate the effect of audio variation on our model by performing two sets of experiments.

1. White noise with intact text: Replacing audio inputs to the model with white noise, we observe that the QWK drops across both prompt 1 and 4, by an average of 23.8% and 22% without and with threshold optimization respectively (Table 5).

2. Template native voice with intact text: Next, we replace the audio inputs to the model with the Text-to-Speech outputs of the transcriptions. We use the eSpeak [23] speech synthesizer for this experiment (see Table 5 for results). We observe an average drop in QWK of 27.2% for the model without threshold optimization and of 27% for the model with threshold optimization, across both the prompts. This performance drop can possibly be explained due to the fact that the model was originally trained on speech responses from L2 speakers, and hence performed poorly when tested on L1 speech.

Curiously, the model scored the speech with the native accented templatized audio consistently lower than the speech with white noise audio part. In our analysis, this was majorly due to two reasons: text-to-speech system did not produce much sound variation and that the generated voice was closer to a native speaker but was tested on a model trained for L2 speakers.

For checking the attention weights of the model on a more granular level, we manually analyzed some samples by plotting their attention weights. Figure 2 shows attention splits for one “Low B1” and one “Low B2” scored sample, both taken from prompt 4, which is a prompt based on the topic of smoking. We show two frames (or segmentations) for both the samples, one in each row. Each cell represents one frame of audio input and its corresponding transcription. The bar graph shows the respective audio and text attentions, normalized separately using min-max scaling across the entire response. We can observe that for filler words like “uh” and “um”, false starts and pauses, the model pays great attention to the audio. For content-rich fluent portions, the model chooses to pay significantly high attention to the transcripts. This is particularly visible in Figure 2 where we can see that model pays high attention to text for frames containing words like “no smoking”, which would be important to judge the response to the question.