Self-supervision and Learnable STRFs for Age, Emotion, and Country Prediction

Ephraim and Malah’s MMSE-STSA filtering algorithm is based the principle that perceptual quality is more closely tied to amplitude than phase. As such, the spectral amplitude is optimized via MMSE and combined with the noisy phase to reconstruct an enhanced signal.

The MMSE-STSA is derived starting with the expression for the expected value of the clean STSA given the noisy short-time Fourier transform (STFT). After a series of approximations and algebraic manipulations, the expression in Eq. 1 is achieved for high signal-to-noise ratios (SNR):

Here, $\hat{A}_{k}$ and $R_{k}$ are the estimated clean spectral amplitude and given noisy spectral amplitude at time frame $k$, respectively, and $\xi_{k}$ is the SNR at frame $k$. From this expression, it is evident that when SNR is high, the multiplicative gain applied to the noisy spectral amplitude resembles a Wiener filter. However, when SNR is low, the gain deviates from the Wiener filter, avoiding the high bias of the Wiener filter at low SNRs.

The MMSE-STSA algorithm was used here because of its versatility in handling different SNRs. It was also shown to leave colorless residual noise rather than musical noise, which may be advantageous for the downstream classifier.

To understand which preprocessing techniques would work best for the task at hand, we tested the data at multiple preprocessing stages. Specifically, we used the clipped baseline audio, un-clipped raw audio, DC normalized audio, denoised audio, and VAD-processed audio. The model architecture used here is the one provided by the challenge.

Table 1 shows the results achieved when openSMILE ComParE 2016 features are extracted from each version of the data. Scores are shown in terms of the evaluation metrics outlined for the ExVo-MultiTask track, including unweighted average recall (UAR) for country prediction, mean absolute error (MAE) for age estimation, concordance correlation coefficient (CCC) for emotion prediction, and the harmonic mean of these metrics ($S_{MTL}$). From the table we can observe that denoising the data is indeed important for the task and achieves good results. VAD degrades the performance, indicating that the silences in between the expressions actually carry useful information which, when removed, reduce the predictive capacity of the models.

Although the official challenge provided a few baseline utterance-level feature frontends, we wanted to explore the benefits of sequence-level feature frontends. We decided to use standard log-mel spectrogram feature frontend, a modulation-based feature frontend and a HuBERT-based feature frontend. The three feature frontends are described in further detail below.

HuBERT Encoder: The HuBERT (Hsu et al., 2021) encoder comprises a series of BERT-like transformer layers following a wav2vec2.0 feature extractor unit.

Conformer Encoder: Conformers are the state of the art on multiple speech tasks including speech recognition. Conformers comprise transformer like layers with macaron-style feed-forward layers, convolution, and self-attention. The initial layer of the conformer encoder consists of a 2D convolutional downsampling layer. We increase the input channels of the downsampling layer when STRF features are used.

Self-Attentive Pooling Decoder: Our decoder uses task-specific self-attentive pooling, followed by a series of fully-connected layers.

We use the mean absolute error (MAE) objective function to compute the loss for both the emotion and age prediction. For the country prediction, we use the negative log-likelihood objective function. The final loss used to optimize our multitask system is a weighted combination of the three loss terms. We tune the weights for each loss term based on the performance on the validation set.

Due to each loss having a different range, we use 1, 80, and 8 for $\alpha_{1}$, $\alpha_{2}$, and $\alpha_{3}$, respectively.

This research was conducted using the ESPnet framework (Watanabe et al., 2018). The code for the features and model architecture will be accessible here¹¹1https://github.com/espnet/espnet.

We hypothesize that the inter-relatedness between the tasks under consideration make multitask learning a viable option. We compared the use of task independent models and task dependent models to establish if this holds for the task. Table 2 reports results on the validation set for our models that use the HuBERT encoder for each of the three tasks, and a simple multitask setup.

We find that multitask learning improves performance on age regression and country classification, while retaining comparable emotion recognition performance. From these results, it appears that there is no clear dependence on country of origin and age for emotion prediction. This is likely to be more true for vocal bursts that do not contain spoken language, since expressive vocal bursts are likely similar across countries and age groups.

Table 3 summarizes the results on the validation set for each of the individual features, encoders and frontends. For the log-mel and STRF features, we only use the original unprocessed waveform provided by the challenge. For the HuBERT features, we use the original unprocessed waveform and also investigate the benefits of using denoised speech. On the validation set, all three features outperformed the challenge baseline. Interestingly, we also observed additional benefits when we used the HuBERT encoder with the denoised speech. We hypothesize that denoising the speech is beneficial since background noise decreases the intelligibility of speech. Furthermore, we investigated the impact of speed perturbations for this task and found that it slightly improves country prediction and emotion recognition while performing worse on age prediction.

In addition to analyzing the performance of the individual features, we explore whether the features provide complementary benefits. Since each of the features extract information from different domains, we believe there will be benefits from combining different feature scores. We discuss our score fusion results in the next section.

Table 4 summarizes the results on the test set for the fused scores from the individual features. For score fusion, we simply take a weighted combination of the predictions from the individual features. We used the weighted combination of the individual feature scores that provided the best results on the validation set. Interestingly, when we combine the HuBERT predictions with either the log-mel conformer or STRF conformer predictions, simply averaging the country and age predictions between the two gave us the best results on the validation set. For the emotion task, using the HuBERT emotion predictions without any fusion gave us the best results on the validation set. In our initial score fusion experiments, we found that fusing the HuBERT predictions with the STRF predictions improved the performance of the challenge score from .406 to .42 on the validation set. Although simple score fusion gave us improvement on the validation set, we found that the benefits on the test set were minimal. We believe that training an additional classifier to combine the scores could provide benefits on the test set. Additionally, we hypothesize more sophisticated deep learning-based fusion methods would be beneficial and generalize better on the test set.

From our best test score, we observe that the model obtains the highest CCC on awe, amusement, and surprise. This is likely because of how distinctive the expression of these emotions is, while emotions such as awkwardness and triumph are more confusable.