A Novel Exploitative and Explorative GWO-SVM Algorithm for Smart Emotion Recognition

Data collection is one of the most important steps for emotion detection. The definition of different emotions must be explicit in this phase. If the definition is not clear, confusion may occur among different emotions in the classification phase and the classification performance will be influenced negatively. However, emotions normally instantaneously occur and hold for a short period. The longer the period is, the more irrelevant data is included in ECG signals. Thus, it is hard to properly label the corresponding emotion class.

To avoid the aforementioned issue, we self-collect a dataset with high quality and a short period for each emotion, making sure accurate data collection and labeling processes. The ECG signals are recorded by a low-cost wearable IoT ECG patch, called iRealcare [38, 39, 40, 41, 42] with 128 Hz sampling rates. The data collected by the iRealcare IoT ECG sensor can be transmitted to a smartphone application (APP) via Bluetooth Low Energy (BLE) and then to a cloud. From the cloud, we can acquire the raw ECG signals. Signals are recorded for four emotions including happiness, tension, peacefulness, and excitement. Except for peacefulness, each emotion is generated based on an external environmental stimulus, which is similar to the published datasets stimulating subjects through audio or video [43]. The peacefulness describes the normal state, for which the ECG signals are recorded without any external stimulus. Signals for happiness, tension, and excitement are recorded when subjects watch comedies, watch thriller movies and do exercises, respectively. Generally, the record duration should be short as we discussed before. Therefore, the record time is in a range of 3.22-6.16 minutes for each emotion.

It should be noticed that we only record the period that subjects are actually in that emotion condition and ignore the transition period. Clearly, the definition of different emotions under this external stimulus setting is clear and subjects are easy to get into a specific emotion. Taking into account differences among different subjects, 5 subjects are involved and each subject is recorded with four emotion types. For each subject, there are 192-229 samples for peacefulness, 99-141 samples for excitement, 156-236 samples for happiness and 166-205 samples for tension. More information on the iRealcare database is shown in Table I and segmentation details are described in Section III-A2.

The dataset, accessible in [43], is comprised of recordings of 15 subjects (aged 24–35) watching video clips and doing public speaking and mental arithmetic tasks. The dataset is recorded with a wrist-based device (including the following sensors: photoplethysmography, accelerometer, electrodermal activity, and body temperature) and a chest-based device (including the following sensors: ECG, accelerometer, electromyogram, respiration, and body temperature). This dataset offers a fusion of physiological parameters to efficiently identify human emotions, as these represent the body’s instinctive reactions. However, it is not suitable for practical applications, and it may hinder subjects during daily life activities [23]. Therefore, in this paper, we only study single ECG channel signals for this dataset. The ECG signal is acquired from a RespiBAN Professional using a three-lead configuration with 700$Hz$ sampling rates. Three types of emotions (baseline, stress and amusement) are annotated by subjects [43]. Amusement condition signals are collected when subjects watch funny video clips. Stress condition signals are collected when subjects are asked to provide public speaking. Baseline condition signals are collected when subjects sit/stand at a table and read magazines. For each subject, there are 9 samples for amusement, 15-18 samples for stress, and 28-29 samples for baseline. The segmentation details are described in Section III-A2.

Normally, ECG signals are non-linear with low signal amplitudes. The frequency range of ECG signals is from 0.05$Hz$ to 100$Hz$ and the dynamic range is below 4$mV$ [44]. Thus, the collected ECG signals are susceptible to being disturbed by external factors such as interference. To acquire ECG signals with low interference, we conduct the pre-processing of the raw ECG signals. During the data collection and transmission stage, ECG signals are mainly affected by baseline drift, power line interference, and electrode contact noise. The baseline drift is caused by body movement and breathing. It can make the entire ECG signal shift down or up at the horizontal axis. The frequency of baseline drift is around 0.5$Hz$ and it will influence the analysis of ECG signals. The power-line interference is characterized by 50 or 60$Hz$, which can be caused by the electromagnetic field of nearby facilities and electromagnetic interference of the power lines. Since the iRealcare sensor used BLE instead of cables, the power-line interference will not affect the collected ECG signals from the sensor. The electrode contact noise is caused by the variance of impedance when the skin is stretched. This frequency is typically between 1 and 10$Hz$ [45].

For the first time, the X-GWO-SVM approach is proposed for ECG emotion identification in this work. The hyperparameter-free property of the proposed method provides a new way for radial basis function-based SVM (RBF-SVM) learning. In general, classifying the non-linearly separable data with RBF-SVM requires two hyperparameter which are considered—a penalty coefficient $C$ and a spacial parameter $\gamma$. The objective function of RBF-SVM with $C$ and $\gamma$ introduced is expressed in Eqs. (3) and (4):

where $P$ is the number of training samples; $\epsilon_{i}$ is a slack variable which is added to relax the constraints of linear SVM; $\mathbf{w}^{T}\Psi(\mathbf{x}_{i})+b$ is the decision function; $y_{i}$ is the class label; $\mathbf{x}_{i}$ is the sample; $C$ is the penalty parameter and it controls the trade-off between the size of the margin and the slack variable penalty; $\gamma$ is a spacial parameter which controls data distribution in a new feature space [51, 52]. Obviously, hyperparameter ($C$ and $\gamma$) tuning for RBF-SVM is necessary but complex. Thus, the proposed method can internally learn hyperparameters by emphasizing the importance of the $\alpha$ wolf and non-linearly updating coefficient vectors used in GWO. Moreover, this method has higher recognition accuracy than the existing GWO-SVM and PSO-SVM techniques for ECG emotion recognition use, and it can effectively avoid the algorithm falling into a local solution by increasing the exploration ability and speed up the convergence ability by increasing the exploitation ability.

Fig. 3 demonstrates our X-GWO-SVM method, which is inspired by the activity of grey wolves. There are 4 types of grey wolves, named alpha ($\alpha$), beta ($\beta$), delta ($\delta$), and omega ($\omega$), simulating the leadership hierarchy. These wolves continuously search for prey, the optimal solution in our case, and hunting (optimization) is guided by the fittest solution, second and third best solutions, $\alpha$, $\beta$ and $\delta$, respectively. The $\omega$ wolves follow these three wolves. A total number of search agents (wolves) is represented by $n$. $C_{o}$ and $\gamma_{o}$ are two elements of the searched optimal solution $\mathbf{\xi}_{1}$. The X-GWO-SVM method has 10 steps as described below.

1. 1.

   The X-GWO-SVM related parameters are initialized, i.e., maximum iteration $L$; the number of search agents $n$; positions of $\alpha$ ($\mathbf{\xi}_{1}$), $\beta$ ($\mathbf{\xi}_{2}$) and $\delta$ ($\mathbf{\xi}_{3}$); positions of search agents (wolves) ${\mathbf{\eta}_{1},\mathbf{\eta}_{2},...,\mathbf{\eta}_{i},...,\mathbf{\eta}_{n}}$. $\mathbf{\eta}_{i}\in\mathbb{R}^{d}$ and $\mathbf{\xi}_{i}\in\mathbb{R}^{d}$ are $d$-dimensional vectors. In this case, $d$ is equal to $2$, representing two optimal hyperparameters ($C$ and $\gamma$) required for search.

2. 2.

   If the current iteration time $t$ is less than the maximum iteration $L$, go to the subsequent steps; otherwise, proceed directly to step 9).

3. 3.

   For each agent, train RBF-SVM with current position elements $\eta_{i}=(C_{i},\gamma_{i})$.

4. 4.

   Predict trained RBF-SVM with the test set for each agent and output its loss as a fitness value based on Eq. (5):

   $$loss(\eta_{i})=\frac{1}{M}\sum_{i=1}^{M}(y_{i}-h_{i})^{2},$$

   (5)

   where $M$ is the number of test samples and $h_{i}$ represents the predicted value for the $i_{th}$ test sample.

5. 5.

   Sort all fitness values in ascending order and assign positions which have the corresponding top three fitness values as $\mathbf{\xi}_{1}$, $\mathbf{\xi}_{2}$ and $\mathbf{\xi}_{3}$, respectively. The mathematical expressions are

   	$\displaystyle\mathbf{\xi}_{1}(t)$	$\displaystyle=\operatorname*{arg\,max}_{\eta_{i}(t),i=1,...,n}loss(\eta_{i}(t)),$		(6)
   	$\displaystyle\mathbf{\xi}_{2}(t)$	$\displaystyle=\operatorname*{arg\,max}_{\eta_{i}(t);\eta_{i}(t)\neq\mathbf{\xi}_{1}(t)}loss(\eta_{i}(t)),$		(7)
   	$\displaystyle\mathbf{\xi}_{3}(t)$	$\displaystyle=\operatorname*{arg\,max}_{\eta_{i}(t);\eta_{i}(t)\neq\mathbf{\xi}_{1}(t),\mathbf{\xi}_{2}(t)}loss(\eta_{i}(t)).$		(8)

6. 6.

   Update exploration-exploitation regulation function $\phi(t)$ based on Eq. (9):

   $$\phi(t)=-2\frac{t-L+1}{-t+L}.$$

   (9)

7. 7.

   For each search agent, update its position $\mathbf{\eta}_{i}$ based on following equations:

   	$\displaystyle\mathbf{\eta}_{i}(t+1)$	$\displaystyle=\frac{1}{4}\mathbf{\xi}_{1}(t)+\frac{1}{4}\sum_{i=1}^{3}[\mathbf{\xi}_{i}(t)$		(10)
   		$\displaystyle-b_{i}(t)\odot|c_{i}(t)\odot\mathbf{\xi}_{i}(t)-\mathbf{\eta}_{i}(t)|],$
   	$\displaystyle b_{i}(t)$	$\displaystyle=2\phi(t)r_{i}(t)-\phi(t)\mathbf{1},\quad c_{i}(t)=2s_{i}(t),$		(11)

   where $\odot$ and $|\cdot|$ represent Hadamard product operation and element wise absolute value operations, respectively; $t$ is the iteration number; $\mathbf{1}\in\mathbb{R}^{2}$ and its elements are all ones; $b_{i}\in\mathbb{R}^{2}$ and $c_{i}\in\mathbb{R}^{2}$ are coefficient vectors. The coefficients $r_{i}\in\mathbb{R}^{2}$ and $s_{i}\in\mathbb{R}^{2}$ are random vectors, where elements are in the range 0 to 1.

8. 8.

   Accumulate iterative time and go back to step 2).

9. 9.

   Output the optimal parameters $\xi_{1}=(C_{o},\gamma_{o})$ and the trained SVM model.

10. 10.

    Calculate the classification accuracy of the model based on the test set and end the X-GWO-SVM algorithm.

We demonstrate improvements of the proposed X-GWO-SVM algorithm with respect to its exploration and exploitation ability in following two subsections.

The classification performance of various methods can be evaluated by standard statistical measurements: accuracy (ACC) and F1-score (F1), defined as

where TP (true positive) is the number of samples correctly predicted as the current class; TN (true negative) means the number of correctly predicted as other classes; FP (false positive) indicates the number of samples incorrectly detected as the current class; FN (false negative) denotes the number of samples incorrectly detected as other classes. Accuracy is the general measurement of the correctly predicted ratio of the total testing samples for each dataset, indicating the method’s capability to classify emotions correctly. The F1-score, on the other hand, more accurately captures the ideal model for the unbalanced class distribution. The goal is to maximize these two measures as representations of effective models.

As aforementioned in Section III-A3, determination of a proper number of extracted features is necessary. Therefore, we perform feature importance selection in the range of 20 to 135 with step size 5 under the X-GWO-SVM method for iRealcare dataset. Each simulation result is repeated 10 times for random selection of training and test samples.

Fig. 7 illustrates the accuracy versus the dimension of the feature under the X-GWO-SVM algorithm with 10-fold cross-validation for the iRealcare dataset. Clearly, the recognition accuracy displays the tendency to rise up at the beginning and decline in late. The highest mean accuracy is 93.37% located at the feature dimension equal to 95. Moreover, its corresponding box plot (filled with orange color) has relatively low variance, indicating the stability of this feature dimension. After getting the most discriminative result with feature dimension 95, we apply it to other comparison methods.

As we demonstrated the significance of exploration-exploitation regulation function $\phi(t)$ in Section III-B, various exploration-exploitation regulation functions are used in our experiments here to demonstrate that our choice of $\phi(t)$ used in the X-GWO-SVM algorithm is the best. Expressions on them are shown in Eqs. (15) to (19) and these exploration-exploitation regulation functions are plotted in Fig 6. It should be noticed that the conventional linear exploration-exploitation regulation function, a benchmark, is expressed in Eq. (15). Additionally, the one we proposed in the X-GWO-SVM in Eq. (9) is rewritten as $f_{\phi 4}(t)$ in Eq. (18).

Based on Fig. 6, we can observe that both $f_{\phi 2}$ and $f_{\phi 3}$ are deformed from the Sigmoid function, which dramatically decrease from 2 to 0 at the beginning and the end of the iteration, respectively. The function $f_{\phi 4}$ and function $f_{\phi 5}$ successively alleviate this decreasing trend on a basis of function $\frac{1}{x}$ and $\cos$, respectively.

To evaluate the effects of exploration-exploitation regulation function, we apply 10-fold cross-validation to the proposed X-GWO-SVM, varying exploration-exploitation regulation functions based on the aforementioned five functions. The evaluated results on exploration-exploitation regulation functions are shown in Table II for iRealcare dataset and Table III for WESAD dataset. Clearly, for iRealcare dataset, X-GWO-SVM combined with $f_{\phi 4}$ has the highest accuracy (93.37%) and F1-score (93.38%) among others. Moreover, the lowest variance and pretty low training time indicate its stability with low computation time. A similar conclusion can be derived for results on the WESAD dataset, where the highest accuracy (95.93%) and F1-score (95.56%) are from the combination of X-GWO-SVM with $f_{\phi 4}$. To this end, we have determined the optimal feature dimension—95, and exploration-exploitation regulation function—$f_{\phi 4}$. Therefore, later evaluations of the proposed X-GWO-SVM are based on these two settings.

Despite the restricted number of subjects in the iRealcare dataset, the number of samples for each subject is sufficient since the time of data collection for each emotion is sufficient. It is true that the WESAD dataset contains a larger number of subjects; however, the sample length required for this dataset to achieve high accuracy, which is 14000, drastically reduces the actual number of samples, for example, 9 samples for each subject on amusement, 15-18 samples for each subject on stress, and 28-29 samples for each subject on the baseline. On the contrary, the sample length required for the iRealcare dataset to achieve high accuracy, which is only 200. Therefore, the number of samples for the iRealcare dataset is much larger than the one in the WESAD dataset.

The reason for the caused aforementioned situation might come from the way of giving external stimulus and recording data. For the iRealcare dataset, ECG signals for happiness, tension, and excitement are recorded when subjects watch comedies, watch thriller movies and do exercises, respectively. It should be noticed that emotions normally instantaneously occur and hold for a short period. Therefore, we only record the period that subjects are actually in that emotion condition and ignore the transition period. Clearly, the definition of different emotions under this external stimulus setting is clear and subjects are easy to get into a specific emotion. However, for the WESAD dataset, amusement condition signals are collected when subjects watch funny video clips; stress condition signals are collected when subjects are asked to provide public speaking and mental arithmetic tasks; baseline condition signals are collected when subjects sit/stand at a table and read magazines. In fact, subjects tend to take some time to transfer from one emotion condition to the other. However, such a transition period is also recorded in the WESAD dataset. Thus, the sample length need to be long enough to make sure not just the transition period is included. The shorter the time, the more probable it is that only transitional periods will be included in the sample.

To sum up, in spite of the fact that the iRealcare dataset has limited subjects, the actual number of samples is much larger than the one in the WESAD dataset. Moreover, due to the exclusive emotion transition period for the WESAD dataset, the selection of sample length for the iRealcare dataset is more flexible than the WESAD dataset. We use the widely-used WESAD dataset as a benchmark for further comparison to validate our proposed X-GWO-SVM algorithm.

We study the impact of exploration-exploitation regulation functions on emotion recognition performance. For this study, we select possible base functions that control the significance of each exploration-exploitation regulation function as listed in Eqs. (15) to (19). Table II and Table III show the emotion recognition performance for five exploration-exploitation regulation functions on the iRealcare dataset and WESAD dataset, respectively. This analysis provides in-depth insight into the effect of the exploration-exploitation regulation functions associated with the emotion recognition outcome. Furthermore, this analysis helps us narrow down the most suitable exploration-exploitation regulation function in order to achieve the best performance.

As we mentioned in Section III-B, the declining rate of the exploration-exploitation regulation function at the beginning and end with respect to the iteration time represents the exploration and exploitation ability of the proposed X-GWO-SVM. From Table II and Table III, we notice that for the exploration-exploitation regulation function $f_{\phi 3}$, where the function is formed on a basis of the sigmoid function, the model performance on emotion recognition is poor since it is under-explored and under-exploited. However, for those exploration-exploitation regulation functions lying above the benchmark function $f_{\phi 1}$, the model shows significantly better performance. Interestingly, the performance drops when exploration-exploitation regulation functions decline too fast ($f_{\phi 2}$) or too slow ($f_{\phi 5}$). The function $f_{\phi 4}$ gives the highest performance for emotion recognition compared to others since it has the most suitable diverging and converging performance to the X-GWO-SVM algorithm. Moreover, the fact that $f_{\phi 4}$ outperformed other functions for both datasets is also indicative of its stability.

In summary, the analysis above shows that for all the exploration-exploitation regulation functions, when the declining rate of the function at the beginning is too large or too small, for example, $f_{\phi 3}$ or $f_{\phi 2}$, emotion recognition accuracy drops due to the under-exploration or over-exploration. This results in the X-GWO-SVM more easily falling into local solutions. Similarly, when the declining rate of the function at the end is too large or too small, $f_{\phi 2}$ or $f_{\phi 3}$, the performance also drops due to the under-exploitation or over-exploitation in such cases becomes too difficult for the algorithm to properly find the global solution. Hence, we conclude that there is a trade-off between exploration and exploitation for the exploration-exploitation regulation functions associated with the proposed X-GWO-SVM algorithm, for which the proper exploration-exploitation regulation function $f_{\phi 4}$ is applied resulting in avoiding falling into local solutions.

This section discusses the performance of X-GWO-SVM for emotion recognition in terms of reliability, stability, and efficiency.

The possible limitation of the current study would be that we only investigate four exploration-exploitation regulation functions, i.e., $f_{\phi 2}$, $f_{\phi 3}$, $f_{\phi 4}$, and $f_{\phi 5}$. Moreover, the dataset we collected is still insufficient and other existing published datasets, e.g., AMIGOS [27], Augsburg Biosignal Toolbox (AuBT) [54], etc., have not been verified by the proposed X-GWO-SVM algorithm. In future work, we will use other exploration-exploitation regulation functions for the proposed algorithm to explore their effectiveness in emotion recognition. Additionally, more published datasets will be examined by our method.

Besides, through the results and conclusions reported in  [29], we also observed that deep learning is competitive in emotion recognition, which may further improve the performance of our proposed strategy. Thus in our future work, we will try to find an effective deep learning method and embedded GWO methods to further improve emotion recognition performance.