On The Differences Between Song and Speech Emotion Recognition: Effect of Feature Sets, Feature Types, and Classifiers

The first evaluated feature set for emotional speech and song is Geneva Minimalistic Acoustic Parameter Set (GeMAPS) [3]. This feature set is a proposal to standardize acoustic features for voice and affective computing. Twenty-tree acoustic features, known as low-level descriptor (LLD), are chosen as minimalistic parameter set while the extended version contains added spectral and frequency related parameters and its functional. The extended version (eGeMAPS) consist of 88 parameters. This research used the minimalistic GeMAPS feature set due to its effectiveness compared to other features sets [3, 1].

The openSMILE toolkit [4] was used to extract 23 LLDs GeMAPS feature set for each time frame. That frame-based processing is conducted with 25 ms window length and 10 ms hop length resulting (523, 23) feature size for speech data and (633, 23) feature size for song data.

The second evaluated feature set is pyAudioAnalysis (pAA). pyAudioAnalysis was designed for general-purpose open-source Python library for audio signal analysis. The library provides a wide range of audio analysis procedures including: feature extraction, classification of audio signals, supervised and unsupervised segmentation, and content visualization [6]. Thirty-four LLDs are extracted on frame-based processing from this feature set. Those LLDs, along with the previous GeMAPS and next LibROSA feature sets, are shown in Table 1.

As the final feature set, we selected five LLDs from LibROSA feature extractor including MFCCs, chroma, mel spectrogram, spectral contrast and tonnetz. The number of features for each LibROSA LLD is shown in Table 1 with a total of 193 features. This number of features is chosen based on experiments. The detail of LibROSA version used in this experiment can be found in [8].

The traditional method to extract acoustic feature from a speech is done on a frame-based processing method, i.e., the aforementioned LLDs in each acoustic feature set. The higher level acoustic features can be extracted as statistical aggregation functions over LLDs on fixed-time processing, e.g., an average feature value of each 100 ms, 500 ms, 1 s, or per utterance. This high-level statistical functions (HSF) is intended to roughly describe the temporal variations and contours of the different LLDs over a fixed-time or an utterance [9]. In dimensional speech emotion recognition, this HSF feature performs better than LLDs, as reported in [2, 12, 10].

Schmitt and Schuller [12] evaluated mean and standard deviation (Mean+Std) of GeMAPS feature set and compared it with eGeMAPS and bag-of-audio-word (BoAW) representations of LLDs. The result showed that Mean+Std works best among the three. Based on this finding, we incorporated Mean+Std as HSFs from the previously explained three feature sets. The Mean+Std is calculated per utterance on each feature set resulting difference size/dimension for each feature set, i.e. 46-dimensional for GeMAPS, 68-dimensional for pyAudioAnalysis, and 386-dimensional for selected LibROSA features. Hence, two feature types are evaluated; LLD and HSF, from GeMAPS, pyAudioAnalysis and selected LibROSA features.

Four classifiers are evaluated: a dense network (or multi-layer perceptron, MLP), a long short-term memory (LSTM) network, a gated recurrent unit (GRU) network, and a convolution network. The brief explanations of those networks are described below.

1. 1.

   MLP: Three dense layers are stacked with 256 units and ReLU activation function for each layer. The last dense layer is flattened, and a dropout rate with probability 0.4 is added after it. The final layer is a dense layer with eight units for speech and six units for song with a softmax activation function.

2. 2.

   LSTM: Three LSTM layers are stacked with 256 units each and returned all values. The rest layers are the same as MLP classifier, i.e., a dropout layer with probability 0.4 and a dense layer with a softmax activation function.

3. 3.

   GRU: The GRU classifiers similar to LSTM. The LSTM stack is replaced by GRU stack without changing other parameters.

4. 4.

   Conv1D: Three 1-dimensional convolution networks are stacked with 256 units and a ReLU activation function for each layer. The filter lengths (strides) are 4, 8, 12 for first, second, and third convolution layers. The rest layers are similar to other classifiers.

Table 2 shows accuracy and UAR of different feature sets and feature types. On different feature types, LibROSA-based acoustic features perform best on both emotional song and speech data. On different feature sets, HSF features perform better than LLD, except on pyAudioAnalysis feature set for speech data. Only in that pyAudioAnalysis feature set LLDs of pyAudioAnalysis obtained better accuracy and UAR than its HSF. In overall evaluation, our proposal on using Mean+Std of LibROSA-based acoustic features perform best among six different features. This finding suggests that Mean+Std performs well not only on dimensional speech emotion recognition (regression task) but also on categorical song and speech emotion recognition.

To find in which emotion category our method performs best and worse, we performed confusion matrix presentation as shown in Figure 1 and 2. We found feature-specific patterns on the detection of each emotional song category. For instance, the GeMAPS features, both LLD and HSF, obtained the lowest recall on the sad category while LibROSA HSF obtained high recall on this sad category. The happy category which obtained low scores on LLDs of pyAudioAnalysis and LibROSA obtained improvement on its HSFs. The highest and lowest recall scores vary for each feature on both song and speech data. On the speech data (Figure 2), LLD features shows best recall scores on calm category while HSF features obtained better score on neutral category. On both song and speech data, it was found the HSF improved recall scores of LLDs features. Since each feature has a different highest recall score, it is interesting to combine those features sets to improve the current result for future research direction.

Although we found no remarkable different on the performance trends from the same features sets between emotional song and speech data, a significant difference maybe found on the using of specific acoustic features, e.g., F0 contour, spectral features and amplitude envelope, as reported in [11] for emotional singing voice.

For different classifiers, the accuracy and recall scores are presented in Table 3. On the previous table, the results was obtained using LSTM classifiers. In overall evaluation, the LSTM classifier obtained the best result among other three. However, GRU classifier shows better UAR on song data, i.e., on recognizing each emotion category. This result shows that recurrent-based classifiers (LSTM and GRU) performs better than MLP and 1-dimensional convolution network on classification of emotional song and speech. Similar to the previous Table 2, the song data showed higher scores than speech data. Since the song data contains fewer data (samples) and fewer emotion categories than speech data, it can be concluded that song is more emotional than speech. The result on the same classifier, same feature set, and same feature type supports this finding.