CochCeps-Augment: A Novel Self-Supervised Contrastive Learning Using Cochlear Cepstrum-Based Masking for Speech Emotion Recognition

To enhance the expressivity of the CCGRAM features in modelling speech in a SSL setting, we design our masking augmentations to operate along both the angle $\theta$ and quefrency axis $m$. In doing so, we encourage self-supervision to infer the masked frequency tones arising from particular angular positions along the cochlea and masked speech segments along the time domain, both individually and in a combined fashion. Therefore, in a given sample with two cepstral-augmented views, the model should be able to predict the missing tones given the presence of other, contextually and acoustically relevant tones. Hence, it becomes evident that the proposed CochCeps-Augment proxy task, is relevant both for generic speech and audio processing, as well as in the context of SER. This leads us to the following three augmentation transforms:

1. 1.

   Angle masking, applied along the angle $\theta$ axis, so that $\Phi$ distinct masks are applied. Each mask spans $\phi$ consecutive angle bands, i.e. $[\theta_{0},\theta_{0}+\phi)$, where $\phi$ is sampled randomly from a uniform distribution from 0 to $\Phi$, and $\theta_{0}$ is a random angle sampled from $[0,990-\phi)$.

2. 2.

   Quefrency masking, applied along the quefrency indexes $m$, so that $Q$ distinct masks are applied. Each mask spans $Q$ consecutive quefrency indexes, i.e. $[m_{0},m_{0}+q)$, where q is sampled randomly from a uniform distibution from 0 to $Q$, and $m_{0}$ is chosen from the $[0,M-m)$.

3. 3.

   Cepstral masking, defined as the simultaneous masking of both angle bands and quefrency indexes according to the above parameters.

Figure 2 shows an example of the application of our proposed augmentations to an input CCGRAM.

To take advantage of the masking benefits that CochCeps-Augment adds to the learning process of a SSL system, we adopt SimCLR as our pre-training framework [23]. In SimCLR, two augmentation transforms ($t\sim T$ and $t^{\prime}\sim T$) are sampled from the family of the proposed CochCeps-Augment transforms $T$. SimCLR promotes similarity between two augmented views of an input sample, designed to maintain invariance on information axes that are deemed redundant. Specifically, meaningful representations can be learned by maximizing the similarity between positive pairs (views of the same sample) and minimizing the similarity between negative pairs (views belonging to different samples) in a latent space. To that end, every sample CCGRAM $x$ is masked twice in order to produce two distinct augmented views, $x_{i}$ and $x_{j}$. By forward-propagating the two views through a $shared$ $encoder$ $f(.)$, two representations $h_{i}$ and $h_{j}$ are obtained, which are further projected through a $projector$ $g(.)$ to obtain the final representations $\mathbf{z}_{i}$ and $\mathbf{z}_{j}$.

The self-supervision without labels in SimCLR, is facilitated by the NT-Xent (normalized temperature-scaled cross-entropy) loss function, which is given by:

where $\mathbf{z}_{i}$ and $\mathbf{z}_{j}$ are the representations of augmented views $i$ and $j$, $\text{sim}(\mathbf{z}_{i},\mathbf{z}_{j})=\frac{\mathbf{z}_{i}\cdot\mathbf{z}_{j}}{\|\mathbf{z}_{i}\|\|\mathbf{z}_{j}\|}$ is the cosine similarity, $\tau$ is a temperature parameter, $\mathbf{I}[\cdot]$ is the indicator function and $N$ are the number of samples in a batch.

In the pre-training phase, the encoder $f(.)$ and the projector $g(.)$ are trained end-to-end and evaluated through the NT-Xent loss. In downstream evaluations, the projector is discarded and the encoder $f(.)$ is used as a feature extractor that yields the features $h_{i}$ and $h_{j}$.

The K-EmoCon database contains multi-modal recordings from 32 participants divided in 16 groups [20]. Participants engaged in a dyadic debate in English ( 10 minutes) during which multi-modal data acquisition took place. For emotional labeling, we adopt the self-rated arousal/valence (A/V) space quadrant scheme (see Table I), where the provided integer ratings in the range $[1,5]$, are binned to a specific quadrant based on their combined arousal and valence, yielding four classes: LowA/LowV (LALV), LowA/HighV (LAHV), HighA/LowV (HALV), HighA/HighV(HAHV).

Data collection in K-EmoCon was modelled as a natural conversational setting, hence various sources of noise are present. Moreover, the problem of source separation needs to be addressed in speech segments where participants and the referee’s voices overlap, as well as when heavy degradation occurs due to loud, non-conversation-related sounds. To that end, we design a combined pre-processing strategy which is mainly focused on source separation to obtain speaker-independent speech segments. The raw audio is first downsampled from the initial sampling rate 22.5 kHz to 16kHz, followed by max scaling. Silent segments are then removed with a Short-Time Fourier Transform energy thresholding [24]. We manually annotate speech segments identified as sound and based on their content, we apply one of two source separation techniques. For stationary noise, we apply the WTST-NST filter [25] to isolate the non-stationary speech. For speaker separation, we utilize a pre-trained on the Librimix dataset Sepformer model that outputs two speaker signals [26]. Finally, resulting speech segments are scaled again, and segmented into 3 second windows. To ensure uniform input to our deep learning models, we zero-pad segments that are shorter than 3s, and discard segments that are shorter than 1s.

We compute the CCGRAM of an input 3-second segment according to [18], using Hamming windows of 25ms duration with 50% overlap. We set the angle spacing between cochlear modes at $\theta=45\degree$, resulting in CCGRAMs of size 20x239.

Prior to any processing, we split the data into 4 distinct folds: pre-training (26 speakers), validation (2 speakers), fine-tuning (2 speakers) and test (2 speakers). For SSL pre-training we use the pre-training and validation folds, while for evaluation we use the pre-training, validation and test folds in the linear probing, and the fine-tuning, validation and test folds in the fine-tuning. We follow a speaker-independent approach, that relies on a 5-fold cross-validation scheme of K-EmoCon. Specifically, we design the folds in order to ensure that all speakers are present once at each fold, and that all folds contain only whole speakers.

For our augmentation pipeline, we first z-normalize all images in a fold with the mean value and standard deviation of that fold. Then, we generate two views of each input CCGRAM by randomly selecting one of the three masking transforms in the family of CochCeps-Augment for each view. The parameters $\Phi$ and $Q$ for the angle and quefrency masking are evaluated as 2 and 5, respectively. After masking, we resize the CCGRAM to a size of 239x239 with nearest-neighbors interpolation. We pre-train from scratch a ResNet18 to be used as the encoder $f(.)$ [27], where we slightly modify the input convolutional layer to accept single-channel intensity images. The projector $g(.)$ is implemented as a two-layered linear head with an output dimension of 256. For the NT-XEnt loss, we choose a temperature parameter of 0.07. For our implementation via PyTorch, we trained on a machine with four NVIDIA RTX6000 ADA GPUs, with a batch size of 64 for 1000 epochs on each pre-train fold. The LARS optimizer was used [28], with a starting learning rate of 0.1 and a weight decay parameter of 1e-6. A warm-up period of 10 epochs is followed by training with the cosine decay scheduler [29].

We evaluate the performance of our SSL approach through two schemes: linear probing and fine-tuning. It should be noted that in the evaluations, we do not apply CochCeps-Augment but only the z-normalization and resizing operations. Linear probing refers to the evaluation of the frozen encoder $f(.)$ through a simple linear head, that is used to perform the downstream task. Linear probing constitutes a simple evaluation method and allows us to understand the pure quality of the features that the encoder learned during the pre-training phase [1]. Moreover, to test the ability of the contrastive self-supervision to recover the CCGRAM-encoded speech information, we conduct a sanity check where the flattened CCGRAM is fed directly to the linear probe. We expect that this evaluation will reveal the information gain that our proposed pre-training approach contributes to the end result. From a technical perspective, this is attributed to the fact that flattening disrupts the spatio-temporal cochlear cepstral features. Fine-tuning on the other hand, utilizes a non-linear head for downstream classification, and refers to the classic process of fine-tuning the whole network, including the encoder. We train with linear probing on the same data that was used for the pre-training, and evaluate on the left-out test fold. For fine-tuning, we tune on the fine-tuning fold and evaluate on the test fold. On both settings, we train with a batch size of 16, for 50 epochs to avoid overfitting. The Adam optimizer is used [30] with a starting learning rate of 1e-4 for linear probing and 5e-6 for fine-tuning, with weight decay of 1e-6 and a cosine decay schedule. For performance evaluation, we use the weighted accuracy and weighted F1-score.