SigWavNet: Learning Multiresolution Signal Wavelet Network for Speech Emotion Recognition

In our method, we have centered our focus on a particular linear time-frequency transform, namely the wavelet transform. In SER and signal processing, the Uncertainty Principle reveals inherent limitations in simultaneously capturing the temporal and frequency details of a signal. This principle is mathematically grounded in the assertion that the localization of a function $f$ and its DFT, $Df$, cannot both be narrowly confined. Formally, this limitation is expressed in eq. (1):

In response to these challenges, the Wavelet Transformation emerges as a compelling solution within signal processing. Unlike the Fourier transform, wavelets provide a multi-scale decomposition of a signal that achieves localized analysis in both time (or space) and frequency. This duality is essential for effectively dissecting complex signals like speech, where emotional nuances are in both temporal and spectral dimensions. The mathematical foundation of the Wavelet Transformation is established on the principle of dilating and translating a mother wavelet $\psi(t)$ to capture signal characteristics across different scales as shown in eq. (3):

Specifically, we utilize the FDWT, which employs a set of wavelets designed to create an orthonormal family through their scaling and translation by powers of 2, as delineated in [47]. The core wavelet, or the ’mother wavelet’ denoted as $\psi$, is defined to have a zero mean and a unit norm. The variations of this wavelet function, through dilation and scaling, take the mathematical form in eq. (4):

In this context, wavelet functions serve as a band-pass filter bank, enabling the decomposition of a signal using this filter bank. As the wavelets are band-pass, we integrate a low-pass scaling function, $\phi$, which cumulatively represents all wavelets above a certain scale, $j$. The scaling function’s Fourier transform, $\hat{\phi}$, adheres to the relationship in eq. (6), with its phase being arbitrary [47]:

Utilizing these wavelets and the corresponding scaling function, the FDWT successively decomposes a signal $x$ into a coarser approximation $a$ (low-pass filtered version of $x$) and its detail coefficients $d$ (high-pass filtered version of $x$, also known as wavelet coefficients). With an orthonormal basis in $\mathbf{L}^{2}(\mathbb{R})$, such a decomposition is reversible. Importantly, due to the scale factor of 2 between levels and in translating coefficients, the decomposition at any level $l$ can be articulated as a function of the preceding approximation, sub-sampled by a factor of 2, and the base non-dilated wavelet.

The FDWT is computable via a fast-decimating algorithm, commonly referred to as the cascade algorithm. This is visually represented in the block diagram of a one-level FDWT cascade algorithm in Fig. 1, where $g$ represents the wavelet and $h$ the scaling function. We define two filters using eq. (7) and eq. (8):

Our method leverages a notable feature of the FDWT related to filter identification, grounded in the principles of the CQF bank, as elucidated in [48]. A key aspect of the CQF bank is its quadrature property, which ensures a symmetric response from the decomposition filters relative to the cutoff frequency, thereby imparting an antialiasing characteristic to the system. This is achieved by designing the filters in such a way that the wavelet function $g$ becomes the alternating flip of the scaling function $h$. This relationship is denoted in eq. (9) as:

The process of wavelet filter bank decomposition is integral to our method. It involves the computation of approximation and detail coefficients, $a$ and $d$ respectively, using the following eqs. (10) and (11):

A significant advantage of this algorithm is its requirement of only two filters (scaling and wavelet filters), as opposed to needing an entire filter bank. This efficiency is achieved without compromising the ability of the wavelet transform to effectively partition the signal into distinct frequency bands as defined by the wavelet functions. This methodical approach allows for a more precise and computationally efficient analysis of the signal’s frequency components, which is essential for accurate SER.

Continuing our methodological discussion, a crucial application of wavelet analysis in our framework is signal denoising. The foundational assumption in wavelet-based denoising is that structured signals, when decomposed under an appropriate wavelet basis, typically result in a sparse decomposition. This means they activate specific wavelet coefficients at particular times and levels of decomposition. Conversely, noise, which is inherently unstructured, tends to activate wavelet filters at any level, but usually with lower amplitudes. Therefore, denoising typically involves applying a hard thresholding function to the wavelet coefficients. However, determining the optimal thresholding parameters has historically been a complex challenge and a subject of extensive research.

In our implementation, we introduce a series of convolutional layers, $Conv_{h}$ and $Conv_{g}$, that function as learnable scaling (low-pass) and wavelet (high-pass) filters at each decomposition level. These layers facilitate the automatic extraction of meaningful and sparse representations from the raw speech waveform, tailored to the nuances of SER. To enhance the interpretability and effectiveness of the learnable FDWT, we initialize the $Conv_{h}$ and $Conv_{g}$ filters with a suitable wavelet, which provides an orthogonal wavelet basis designed for time-frequency analysis. This initialization helps to guide the network towards efficient signal decomposition while still allowing the filters to adapt optimally for the SER task during training.

Convolutional Operations and Learning Principle: In traditional FDWT, fixed filters are used to compute the approximation and detail coefficients at each decomposition level. In our learnable FDWT framework, however, we implement these filters as convolutional layers $Conv_{h}$ and $Conv_{g}$, initialized with wavelet filter coefficients but further refined through training. For an input signal $x[n]$, the low-pass (approximation) coefficients $a_{j+1}[p]$ and high-pass (detail) coefficients $d_{j+1}[p]$ at each level $j$ are calculated as follows:

Recursive Decomposition Process: The FDWT layer mirrors a recursive cascade structure across multiple levels $L$, each consisting of $Conv_{h}$ and $Conv_{g}$ filters that adapt to capture SER-relevant features. At each level, $a_{j+1}$ (low-pass result) and $d_{j+1}$ (high-pass result) are obtained through the convolution operations, where $a_{j+1}$ is passed to the next level for further decomposition. This recursive process captures frequency-specific features over time, resulting in a data-driven, multilevel wavelet decomposition optimized for the emotional features within the speech data.

CQF Property: To maintain orthogonality and ensure a coherent decomposition structure, we incorporate the CQF property between $Conv_{h}$ and $Conv_{g}$, simplifying the learning process while preserving wavelet theory principles. By deriving $Conv_{g}$ from $Conv_{h}$ as shown in eq. (14), the number of parameters is halved, reducing optimization complexity and enhancing learning efficiency as proven in the ablation study section V:

Data-Driven Adaptation to SER: The wavelet-initialized learnable FDWT filters provide a starting point, capturing basic time-frequency structure. However, during training, the filters adapt further to isolate frequency components particularly relevant for SER. By minimizing our supervised loss function, the model tunes $Conv_{h}$ and $Conv_{g}$ to emphasize emotional features, such as tone and pitch variations, that are integral to SER. Through this learning process, the FDWT layer achieves a multiband filter bank structure that analyzes the speech signal across different frequency bands in a manner that aligns with the MEL scale. This resemblance arises because, like the MEL scale, our FDWT layer emphasizes lower frequency components crucial for emotional tones, while the high-pass filters isolate finer details within the emotional cues.

In particular, our learnable FDWT focuses on decomposing only the low-frequency representation of the previous level at each stage, giving progressively more emphasis to variations in the low-frequency range of the speech signal. This approach reflects the MEL scale’s emphasis on perceptually important lower frequencies, where emotional nuances often reside, while still retaining high-frequency components as details at each level. Each level of the FDWT layer thus serves as a learnable, data-adapted filter bank that provides a nuanced analysis across emotional frequency bands, capturing both the broad tonal structure in the low frequencies and the finer, higher-frequency details essential for robust SER.

Comparison with Classical Convolution Layers: Learnable FDWT layers and classical convolution layers both use convolution operations but serve distinct purposes. FDWT layers act as a multiband filter bank inspired by wavelet theory, decomposing signals into frequency-specific components across multiple levels, with a focus on low-frequency variations crucial for emotional content. In contrast, classical convolution layers extract features across the entire signal without targeting specific frequency bands. FDWT layers adapt their filters to align with time-frequency localization, enabling detailed, frequency-sensitive analysis of emotional cues, unlike the broader feature extraction of classical convolutions.

Efficient, Lightweight Architecture: Our design also benefits from the hierarchical nature of wavelet decomposition, enabling efficient representation with fewer parameters, by constructing an $L$-level cascade network with only $2L$ convolutional filters (one pair of $Conv_{h}$ and $Conv_{g}$ per level). Since the limit for $L$ is set by the nearest second logarithm of the smallest training input size, we achieve a lightweight architecture that balances complexity with performance. This approach results in a significantly smaller and more efficient model than traditional deep networks, which often rely on millions of parameters.

To learn the correct hard-thresholding operation, the coefficients obtained are then processed by a specially designed LAHT layer/function. This layer is continuous and differentiable, mimicking a wavelet denoising operation, and feeds into the next block for further decomposition. This structure allows for a more nuanced and effective approach to signal denoising, enhancing the overall efficacy of the SER process.

In our architecture, the thresholding step is integrated to autonomously learn the optimal thresholding parameters, eliminating the need for it to be treated as a separate process. We introduce the novel LAHT activation function, crafted as a blend of two contrasting sigmoid functions. This function is distinguished by its asymmetry in both sharpness and bias parameters. Mathematically, the sigmoid function $S(x)$ is defined in eq. (15) as:

This module’s design features two principal distinctive elements: firstly, all convolution kernels, and secondly, both positive and negative hard thresholds, and sharpness factors are independent and asymmetrically learnable. Furthermore, our proposed sparsely connected deep neural network is capable of approximating representation systems that extend beyond the traditional wavelet representation system, offering greater generality and flexibility.

Each of the final low-pass and high-pass representations from all $L$ decomposition levels is then forwarded to the subsequent module, ensuring a seamless and efficient flow of information within our DL framework. This integration of asymmetric learnability and efficient multi-resolution signal representation is pivotal to enhancing the capability and adaptability of our system in SER.

The proposed architecture incorporates a 1D dilated CNN with a spatial attention mechanism and a Bi-GRU network with temporal attention to enhance feature extraction and sequence modeling for each frequency band, as shown in Fig. 5. The 1D dilated CNN efficiently captures local dependencies over varying time scales, while the spatial attention mechanism dynamically emphasizes the most relevant features across the input sequence. The Bi-GRU further models the sequential patterns by processing both forward and backward temporal dependencies, and the temporal attention layer identifies critical regions within the sequence that contribute most to emotion recognition. These components play a crucial role in complementing the proposed learnable FDWT layer. A detailed explanation of their design and implementation is provided in the Appendix.

As we can see in the general architecture of SigWavNet illustrated in Fig. 2, the obtained attention vectors are subsequently concatenated channel-wise, to construct a composite representation with dimensions $(batch\_size,levels+1,length)$, where ”$levels+1$” encompasses obtained frequency bands from high to low, and ”$length$” denotes the dimensionality of each attention vector. Within this composite representation, each ”$channel$” corresponds to an attention vector that stems from a specific frequency band of the original signal. This configuration coalesces the emotionally significant information harnessed from the obtained frequency spectrum of the speech signal, furnishing a rich, multifaceted feature set for the model’s ensuing stages.

Following the concatenation of attention vectors into a composite representation with dimensions $(batch\_size,levels+1,length)$, our architecture incorporates a Channel Weighting layer, a novel component designed to further refine the aggregated feature set. This layer introduces a mechanism to dynamically adjust the importance of each channel (or frequency band) in the composite representation, allowing the model to emphasize or de-emphasize specific frequency components based on their relevance to emotion recognition as formulated in eq. (17):

In this setup, ‘Channel Weighting‘ defines a learnable parameter ‘weights‘ for each channel, initially set to one, indicating equal importance across all channels. As the model undergoes training, these weights are adjusted to optimally scale the contribution of each channel based on how informative it is for the task at hand. This strategic weighting allows the model to leverage frequency-specific insights more effectively, enhancing its ability to discern and classify emotional states from speech signals with greater precision.

Subsequently, another single 1D dilated convolutional layer is integrated, equipped with Instance Normalization (IN) and the Leaky ReLU activation function. This layer is crucial for its output channels, which match the number of classes (emotions) in our dataset. It serves to extract features from attention vectors’ composite representation and generate feature maps corresponding to each emotion class. This design compels the network to learn distinct and accurate representations for each class in individual channels.

After the convolutional processing, the output is then directed to a Log Softmax layer for classification purposes. The Log Softmax layer, defined by eq. (18), offers significant benefits over the traditional Softmax function:

To seamlessly integrate the final 1D dilated CNN layer and the Log Softmax layer, our architecture employs a single Global Average Pooling (GAP) layer, as advocated in [49]. Unlike fully connected layers, the GAP layer averages the activations across the convolution output of the attention vectors. This process effectively condenses each feature map into a single scalar value, thereby reducing the dimensionality of the output while preserving essential information. The mechanism of the GAP layer is illustrated in Fig. 6.

The incorporation of GAP not only streamlines the network architecture by avoiding the complexity of fully connected layers but also contributes to reducing overfitting by minimizing the number of trainable parameters. This layer is particularly effective in summarizing the extracted features, ensuring that the subsequent Log Softmax layer receives a concise yet informative representation of the input for robust emotion classification.

The Interactive Emotional Dyadic Motion Capture IEMOCAP dataset [51] offers a rich multi-modal and multi-speaker collection with 12 hours of audiovisual data, including video, voice, facial motion, and text transcriptions. It features dyadic sessions of improvised and scripted interactions designed to evoke various emotions, structured across five sessions with pairs of male and female speakers, totaling 10039 utterances each, averaging 4.5 seconds, at a 16 KHz sampling rate. Emotions in IEMOCAP are categorized by multiple annotators into basic emotions like anger, happiness, sadness, and neutrality, and dimensionally represented by valence, activation, and dominance scales. Notably, the dataset has class imbalances, with emotions like ’disgust’, ’fear’, and ’surprise’ being less represented. Following prior research [52] [53], we exclude these classes and combine ’happy’ and ’excited’ into a single category to enhance dataset balance and research alignment.

The Berlin Emotional Speech Database (EMO-DB) [54] is a pivotal SER resource, featuring 535 recordings in German by 10 professional actors (5 male and 5 female), simulating seven emotions: ’anger’, ’boredom’, ’disgust’, ’anxiety/fear’, ’happiness’, ’sadness’, and ’neutral’. Actors read 10 short, emotionally neutral texts in different emotional states to ensure recording consistency and avoid emotional bias. Each recording lasts approximately 5 seconds and was made in a professional, soundproof studio setting, ensuring high audio quality for analysis. Widely used in emotion recognition research, EMO-DB is freely available for non-commercial use, offering a valuable tool for SER technology development.

The data presented in Table I illustrates the effectiveness of the SigWavNet model in discerning various emotional expressions within the IEMOCAP dataset. This model demonstrates good accuracy in identifying different emotions, as reflected in its precision, recall, and F1-score metrics for each emotion category. In particular, SigWavNet excels in detecting ’Neutral’ emotions with a high degree of precision (97%) and recall (93%), highlighting its capability to accurately identify this specific emotional state.

However, the model shows a noticeable precision-recall trade-off when it comes to emotions like ’Happy’ and ’Sad’. For ’Happy’, the model achieves a higher precision rate of 96.7% at the cost of a lower recall. Conversely, in recognizing ’Sad’ emotions, the model exhibits a higher recall rate of 95.4% but with reduced precision. This variation indicates the model’s differential sensitivity to various emotional expressions.

The macro and weighted average scores provide a holistic view of SigWavNet’s overall performance. The macro average suggests a balanced performance across all emotion categories, while the weighted average takes into account the frequency of each class within the dataset. With an overall accuracy rate of 84.8%, SigWavNet proves to be quite effective in the correct classification of emotions across the board. These findings underscore the robustness and potential applicability of SigWavNet in SER tasks, highlighting its capacity to distinguish and interpret a range of emotional states.

Fig. 8 presents a confusion matrix that offers an in-depth analysis of SigWavNet’s capabilities in discerning various emotional states, revealing both its good accuracy and areas where enhancement is needed. In identifying ’Angry’, the model achieves a good recognition rate of 80.91%. However, it tends to confuse ’Angry’ with ’Sad’ in 15.45% of cases, underscoring a challenge in differentiating these two emotions. Notably, there are no cases where ’Angry’ is mistaken for ’Neutral’, indicating a clear distinction the model makes between these two emotions. When it comes to recognizing ’Happy’ emotions, SigWavNet successfully identifies them 71.95% of the time. Yet, it incorrectly labels ’Happy’ as ’Angry’ in 21.95% of instances, suggesting a difficulty in differentiating between these positive and negative emotional expressions. While the model is quite adept at detecting ’Happy’ emotions, enhancing its recall could further improve its performance. SigWavNet excels in identifying ’Neutral’ emotions, boasting a high correct recognition rate of 92.98%. Its minimal misclassification with other emotional states demonstrates its strong capability of accurately distinguishing neutral expressions. The model also shows proficiency in recognizing ’Sad’ emotions, with a correct recognition rate of 95.37%. However, there are instances (4.63%) where ’Sad’ is misidentified as ’Neutral’. Despite this, the model’s high accuracy rate suggests that its predictions for ’Sad’ are generally reliable.

Overall, the confusion matrix in Fig. 8 provides a comprehensive evaluation of SigWavNet’s performance, emphasizing its strong suit in identifying ’Neutral’ and ’Sad’ emotions. While the model shows considerable effectiveness in certain emotional classifications, the observed misclassifications between ’Angry’ and ’Sad’, as well as ’Happy’ and ’Angry’, point to areas where the model could be refined. Enhancing the model’s ability to distinguish these specific emotional nuances could greatly improve its overall effectiveness.

Table II details the performance of our model, SigWavNet, on the IEMOCAP dataset, showcasing its superior SI accuracy and F1-score in comparison to a variety of state-of-the-art methods in the field. These comparative methods utilize diverse techniques for feature extraction and classification. For instance, [60] applies MFCC for feature extraction. In contrast, [61] expands on this by incorporating MFCC along with four different spectral representations (Mel-scaled spectrogram, chromagram, spectral contrast feature, and Tonnetz representation) of the same record. The research by [62] focuses on using delta and delta-deltas for feature extraction, while [63] employs a 3-dimensional Log-Mel spectrum approach. When it comes to classification strategies, these methodologies predominantly rely on DL models. Both [60] and [61] utilize CNN, whereas [64] integrates CNN with Bi-LSTM networks. The study by [62] introduces a 3-D attention-based convolutional RNN (ACRNN), merging CRNN with an attention mechanism. Meanwhile, [63] opts for dilated convolution networks as an alternative to traditional CNNs. Another approach is presented in [65], where the scattering transform is used for SER, offering stable feature representations that are robust to deformations and shifts in time and frequency and adept at capturing emotional cues in speech. Additionally, [66] proposes a framework that combines multi-layer wavelet sequence sets derived from wavelet packet reconstruction (WPR) with standard feature sets, utilizing an RNN based on an attention mechanism for SER.

Our model, SigWavNet, stands out with the highest accuracy and F1-score among these state-of-the-art methods on the IEMOCAP dataset, as indicated in Table II. This performance is likely attributable to SigWavNet’s capability to decompose the signal and amalgamate both local and temporally distributed features, resulting in a more effective representation of data for accurate SER.

Table III offers a thorough assessment of SigWavNet’s capacity to discern various emotions in the EMO-DB dataset, highlighting its good precision, recall, and F1-score metrics across different emotional states. For ’Anger’, the model exhibits exceptional precision (100%), signifying its high accuracy in predicting this emotion. Coupled with a recall of 92.3%, it indicates SigWavNet’s effectiveness in identifying most instances of ’Anger’, resulting in a well-balanced F1-score of 96%.

In recognizing ’Anxiety/Fear’, SigWavNet achieves notable precision (94.7%) and a slightly lower recall of 85.7%, leading to a strong F1-score of 90%. This demonstrates the model’s aptitude for accurately identifying ’Anxiety/Fear’ instances. The model also displays good accuracy in distinguishing ’Boredom’, with precision and recall rates of 95.5% and 87.5%, respectively. The resulting F1-score of 91.3% suggests that SigWavNet is adept at differentiating ’Boredom’ from other emotional expressions. SigWavNet’s recognition of ’Disgust’ shows a balanced performance, with a precision of 73.7% and a high recall of 93.3%, resulting in an F1-score of 82.4%. This indicates the model’s efficiency in detecting ’Disgust’ while minimizing false positives. For ’Happiness’, the model performs well, achieving a precision of 90.9% and a recall of 95.2%, with an F1-score of 93%, reflecting a balanced trade-off between precision and recall. In identifying ’Neutral’ emotions, SigWavNet excels with a precision of 91.3% and a recall of 87.5%, leading to a balanced F1-score of 89.4%. This shows the model’s proficiency in accurately identifying neutral states. ’Sadness’ recognition is also effective, with a precision of 76.2% and a recall of 88.9%, culminating in an F1-score of 82.1%, which suggests a good balance between precision and recall, highlighting the model’s capability in capturing ’Sadness’ accurately.

When considering the macro average, SigWavNet demonstrates an overall balanced performance with average precision, recall, and F1-score values of 88.9%, 90.1%, and 89.2%, respectively. The weighted average shows an overall high recognition performance with an accuracy of 90.1%. Overall, the metrics presented in Table III affirm that SigWavNet is a robust model for emotion recognition across a diverse range of emotions in the EMO-DB dataset.

Fig. 9 presents a confusion matrix that details SigWavNet’s ability to identify various emotions in the EMO-DB dataset. SigWavNet shows good performance in detecting ’Anger’, with a high correct recognition rate of 92.31%. The model has minimal errors in this category, misidentifying ’Anger’ as ’Disgust’ in only 5.13% of instances. When it comes to recognizing ’Anxiety/Fear’, the model achieves a success rate of 85.71%. However, there are instances where ’Anxiety/Fear’ is confused with ’Happiness’ (9.52%) and ’Disgust’ (4.76%).

In identifying ’Boredom’, SigWavNet maintains a solid performance, correctly recognizing this emotion in 87.50% of cases. It occasionally misclassifies ’Boredom’ as ’Neutral’ (4.17%) and ’Sadness’ (8.33%). The model’s proficiency in detecting ’Disgust’ is evident, with a high correct recognition rate of 93.33%. There is a small tendency to confuse ’Disgust’ with ’Neutral’ (6.67%), but no significant errors with other emotion classes, indicating a robust recognition ability.

SigWavNet also performs effectively in identifying ’Neutral’ emotions, attaining a correct recognition rate of 87.50%. Some instances of ’Neutral’ are misinterpreted as ’Disgust’ (4.17%) and ’Sadness’ (8.33%). The model performs well at recognizing ’Happiness’, with a high correct recognition rate of 95.24%. It shows specificity in this classification, with a minor misclassification rate of ’Happiness’ as ’Anxiety/Fear’ (4.76%) and no errors with other emotions. Regarding the recognition of ’Sadness’, SigWavNet achieves a correct recognition rate of 88.89%. Misclassifications occur with ’Sadness’ being identified as ’Boredom’ (5.56%) and ’Disgust’ (5.56%).

Overall, the confusion matrix in Fig. 9 offers a comprehensive view of SigWavNet’s performance across different emotional states in the EMO-DB dataset. The model exhibits a high correct recognition rate in most classes, particularly in ’Anger’, ’Disgust’, ’Happiness’, and ’Sadness’. The instances of misclassification provide insightful data, suggesting areas where the model can be further refined.

In our analysis, we compare SigWavNet’s SI performance with a selection of recent and significant studies. These studies represent a range of approaches and methodologies. The first study, [68], introduces EmoSTAR, a new emotional database, and conducts cross-corpus testing with the EMO-DB database. Utilizing the openSMILE toolkit for feature extraction and the WEKA tool for classification and feature selection, they achieve a test accuracy of 87.2% on EMO-DB. The second study, [69], explores the use of DL for SER, particularly employing CNNs on the EMO-DB dataset. By employing various spectral features for acoustic signal analysis, they report an average accuracy of 99.3% for binary classification and 76.4% for the seven-class EMO-DB dataset. The third method, detailed in [70], proposes an enhanced SER technique using Mel frequency magnitude coefficients for feature extraction. Testing their method across multiple databases, including EMO-DB, they report a notable accuracy of 81.5% using a multiclass SVM classifier. Additional approaches include [71], which uses the fast Continuous Wavelet Transform (fCWT) to augment Deep CNN (DCNN) for SER. The study in [73] presents a three-stage feature selection algorithm combining wavelet packet decomposition, statistical analysis, and an information gain (IG) ranking algorithm based on high IG entropy. In [72], a novel nonlinear multi-level feature generation model with a cryptographic shuffle box structure is employed. The approach in [74] combines wavelet threshold denoising with a multi-task learning framework, utilizing shared and private LSTM networks for SER.

These methods are summarized in Table IV, where we compare their SI performances on the EMO-DB dataset, highlighting the best results in bold. As per these comparisons, our proposed method, SigWavNet, surpasses these approaches, achieving an accuracy of 90.1% and an F1-score of 90.3%, demonstrating its superior performance in accurately recognizing emotions in speech.