A Spontaneous Driver Emotion Facial Expression (DEFE) Dataset for Intelligent Vehicles

To select the most effective video-audio clips, two research assistants (1 male and 1 female) reviewed more than 500 video-audio clips and conducted the preliminary screening. They were asked to select video-audio clips that lasted 30-120 seconds and contained content to elicit a single target emotion, including a negative emotion (anger), a positive emotion (happy), and a neutral state. Another two research experts (1 male and 1 female) with rich experience in driver emotions analysis evaluated each selected video-audio clip. A consensus of the two experts decided the selections of the video-audio clips.

The selected video-audio clips are mainly based on Chinese real-life scenarios and events, such as aggressive driving and chatting. Other video-audio clips selection criteria include: 1) the video background should not be too dark, 2) the clip should contain complete speech segments, and 3) there is only one wanted expressing emotion in the clip. Accordingly, we selected 18 video-audio clips and checked them further in subjective annotation session.

The web-based subjective emotion annotation experiment was conducted to evaluate the video-audio clips. For each participant, the 18 video clips were displayed in a random order, and there was a relatively long break time (3 minutes) between every two clips to avoid interference from the previous one. After watching each video-audio clip, participants finished two questionnaires based on their true feelings, namely the self- assessment manikin (SAM) [13] and the differential emotion scale (DES)[12]. SAM uses non-verbal graphical representations to assess the arousal, valence, and dominance level. The study in [13] has concluded the effectiveness of SAM. We adopted a 9-point scale (1 = ”not at all”, 9 = ” extremely ”) SAM [13] in our study for evaluation. DES is used to assess the different component of emotions, which consists of ten basic emotions. In this study, we used a 9-point scale DES (1= ”not at all”, 9= ” extremely ”) [12] to assess the intensity of each self-reported emotional dimension. None of the clips was evaluated twice by the same participant, and at least 35 assessments were collected for each video.

Three video-audio clips were selected by comprehensively considering the SAM and DES results. Firstly, we normalized the variables by calculating the Z-scores and then conducted a cluster analysis using the K-means algorithm to identify the clusters of emotions based on the SAM data. The clustering results showed that a total of three emotion categories were generated, which corresponded to the positive emotion (happy), negative emotion (anger), and neutral, respectively. The video-audio clip whose rating was closest to the extreme corner of each quadrant was selected and marked as the representative video-audio clip of the cluster [21].

Moreover, we selected video-audio clips for each emotional category based on the following scores of the DES data: 1) Hit rate was defined as the proportion of participants who chose the target emotion. 2) The intensity value was defined as the average score of target emotions. 3) The success index represented the sum of the two Z-scores, which were obtained by normalizing the hit rate and intensity values. Next, video-audio clips with the highest success index were selected from each emotion category, and representative video-audio clips were also selected according to the SAM data for verification. It should be noted that the neutral video-audio clip was only selected based on the clustering results. Eventually, as shown in Table 3, three of the most effective videos were selected for driver facial expression data collection experiment.

The experimental procedure and the video content shown to the participants were approved by Chongqing University Cancer Hospital Ethics Committee, China. Participants and data from participants were treated according to the Declaration of Helsinki. The participants were also informed that they had the right to quit the experiment at any time. The video recordings of the participants were included in the dataset only after they gave written consent for the use of their videos for research purpose. A few participants were also agreed to use their face images in research articles.

Sixty Chinese participants(47 males and 13 females) with aging from 19 to 56 years old (mean [M] = 27.3 years, standard deviation [SD] = 7.7. Years) were recruited to participate in this experiment from Shapingba District, Chongqing, China. Each participant had a valid driving license with at least one year of driving experience (average [M] = 5.5 years, standard deviation [SD] = 5.8, range = 1-30 years). All the participants had normal or corrected to normal vision (36 participants wear glasses) and normal hearing ability. The presence of occlusion such as glasses is a significant research challenge of facial expression recognition; hence participants wearing glasses were included to evaluate the robustness of emotion recognition. All the participants signed the consent form to participate in the study and received 100 CNY as financial reimbursement for their participation.

The experiments were carried out in a fix-based driving simulator (Figure.1(b)) with illumination-controlled (RDS2000, Real-time technology SimCreator, Ann Arbor, Michigan, USA). Figure.1(d) shows the front view, which was presented using three projectors, and the rear view was displayed using three LCD screens (one for the rearview in the vehicle and two for the left and right rear views). Another two LCD screens were used to display the dashboard and central stackf. The ambient noise and sound of the engine were presented through two speakers. The vibration of the vehicle was simulated through a woofer under the driver’s seat. For the presentation of stimulus without changing the internal environment of the driving simulator, as shown in Figure.1(e), we used a 20-inch central stack screen ($1,280\times 1,024$, 60Hz) to display the video-audio stimulus materials. A stereo Bluetooth speaker (Xiaomi) was used to play the audio, and the audio volume was set to a relatively loud level. However, each participant was asked before the experiment whether the volume was comfortable and adjusted when necessary for clear hearing. During the experiment, as shown in Figure.1(a), the participants’ faces were continuously imaged with a visual camera. The visual face camera was an HD Pro Webcam C920 (Logitech, Newark, CA.) with a resolution of $1,920\times 1,080$ pixels, collecting data at a frame rate of 30 fps. Also, an iPad (Apple) was used for participant self-reported emotion. Figure.1(c) shows the overall data collection experiment setup.

Two driving scenarios on highways were realized in the simulator. The reason for setting these two scenarios is to minimize the impact of complex driving scenarios on driver performance. The first was a practice scenario (PD) to help participants familiarize themselves with the simulator before the experiment. As shown in Figure.2(a), the practice scenario was an 8km straight section of a four-lane highway with two for each driving direction. The participants were asked to drive on the right lane with the speed changes in the range of 80km/h – 50km/h – 100km/h. The second scenario is an emotional driving (ED) scenario. As shown in Figure.2(b), the emotional driving scenario was a 3km straight section of the same highway with a posted speed limit of 80km/h. The participants were asked to drive on the right lane with speed around 80km/h.

To obtain drivers’ ED data, we designed an experimental protocol about 45 minutes driving. The protocol was composed of one PD, followed by three ED. ED driving included angry driving (AD), happy driving (HD) and neutral driving (ND). Figure.3 presented details of the protocol. Before the experiment, each participant signed a consent form and filled out a basic information questionnaire (gender, age, driving age). Next, they were provided with a set of instructions to inform them of the experimental protocol and the definition of different scales used for self-reported emotions. Then, the participants were required to drive a 10-minute PD to help them get familiar with the operation and motion performance of the driving simulator. After a short break following PD, the participants started the three EDs. The corresponding emotion was induced by watching the selected video-audio clip at the beginning of each ED, following by driving with emotion. At the end of each ED session, the participant was required to report his/her self-evaluated emotion level using SAM and DES. There was a 3 minutes break between each two EDs. During the entire experiment, if the participants felt any discomfort, they could withdraw from the experiment at any time.

To identify the emotion experienced by participants, we employed self-reported scales for subjective assessment of emotions. After each driving task, the participants were asked to assess their emotional experience while driving using SAM and DES. The SAM and DES scales were presented to participants by an iPad. In SAM, the valence scale ranged from unhappy to happy, the arousal scale ranged from calm to stimulation, and the dominance scale ranged from submissive (or ”without control”) to dominant (or ”under control, empowered ”). In DES, there were ten emotion dimensions, and each dimension evaluated the intensity of emotions from ”not at all” to ” extremely ”. Each dimension of the SAM scale and the DES scale is represented from one to nine by a Likert scale. If the self-assessments from participants were not consistent with the induced target emotions, we would use the participants’ self-reported data as the ground truth to label the facial video data.

In this section, we described the processing of driver facial expression data. First, we labelled the facial expression data of 60 drivers according to their self-reported emotion and removed the ED data that was not successfully induced. Second, we reported how to split data for driver emotion recognition, including splitting effective video clips from the original data and extracting driver facial expression.

During data collection, each participant completed three ED sessions with average recording data of 405s. Also, we compiled the self-reported data for each participant. As shown in Figure.4, the numbers of successfully induced emotional drivers were 52, 56, and 56 for the anger, happy, and neutral driving, respectively. Participants’ self-reported data were used as the ground truth to label driver facial expression data.

As per [48] and [49], the facial expression video sequences 15s after drivers started driving were clipped as the most effective data. Face detection and alignment in driving environments are challenging due to various poses, illuminations and occlusions (glasses). MTCNN (Multi-task Cascaded Convolutional Networks) is a cascade structure based on deep learning, which is relatively accurate when detecting faces in multiple pose angles and in unconstrained scenes [50]. Hence, we used MTCNN to track and extract driver face data from each video frame. After extracting driver face expression data, we obtained a total of 17,310 image frames of driver faces with 64*64 pixel.

Therefore, the created dataset contains facial expression videos and images from 60 drivers with the ground truth of dimensional emotion (valence, arousal and dominance) and discrete emotion (emotion categories and its intensity). A few examples of the dataset images are provided in Figure.5, which shows that drivers’ facial expressions varied with the types of emotion, but the variation was weak in some cases during driving, for example, the difference between AD and ND was tiny. Most video clips were challenging to observe peak expressions, and we also observed that the change of emotion with driving duration was weak, and this phenomenon is probably because the facial expression of emotion was affected by driving tasks.

In this section, we introduced two different types of protocols for driver emotion recognition based on facial expression data. (1) To investigate driver emotion classification results based on the dimensional emotion model, we proposed three different nine-classification problems: valence, arousal, and dominance. To this end, the SAM scores of participants were used as the ground truth. Each classification (valence, arousal, dominance) on these scales was divided into nine levels (1 = ”not at all”, 9 = ”extremely”). (2) To study driver emotion classification results based on the discrete emotion model, we proposed a three-emotion classification protocol, namely anger, happiness, and neutral. Besides, we discussed the intensity recognition for anger and happy emotions, respectively. To this end, the DES scores were taken as the ground truth. Each emotion(anger and happiness) intensity was divided into 5 levels (5 = ”no emotion”, 9 = ”maximum intensity”).

It should be noted that the above approach can lead to unbalanced classes for some participants and scales. In light of this, we included F1 scores in order to report reliable results. The F1 score is a commonly used metric in classification tasks, which considers both precision (P) and recall (R) of the model. It quantifies the correct prediction of the positive samples. When categories are unbalanced, the F1 score will be attenuated [51]. We additionally used accuracy as another metric. Accuracy quantifies how well the classification correctly identifies or excludes conditions, and it is robust to unbalanced data.

Both the traditional and the deep learning methods for emotion recognition tasks were included in this study. As the most effective traditional method in most classification tasks [19], SVM (Support Vector Machine) was selected to be implemented by the sklearn toolbox with a linear kernel. As for the deep learning-based classification methods, Xception [52] was applied. The Xception network has been widely adopted in emotion recognition tasks, and many state-of-the-art emotion recognition networks are developed based on the Xception network[53][54]. For the network, the loss function can be expressed as:

Equation1 where $\hat{y}$ is the prediction and $y$ is the ground truth. The above deep learning method used the same training strategy. First, it employed Adam optimizer [55], which has a learning rate of $10^{-3}$ and a weight decay of $10^{-6}$ for training. Second, image augmentations, including random horizontal flips, random crop, and random rotation, were applied on-the-fly to increase the amount of training images effectively. SVM was applied with Intel R CoreTM i5-dual-core CPU. Xception was used with TITAN XP.

Apart from the emotion recognition results for the proposed dataset, we also selected the DEAP [21] and CK + [19] datasets which were collected in static life scenarios as the comparison datasets. The DEAP dataset consists of 32 participants. Each participant watched 40 1-minute long video-audio chips as the emotional stimulus while recording facial videos and physiological signals. There are 40 trials recorded per participant, each corresponding to one emotion elicited by one video-audio chip. After watching each video, the participants were asked to assess their real emotions from five dimensions: valence, arousal, dominance, liking and familiarity. The rating ranges from 1 (weakest) to 9 (strongest), except liking and familiarity, which rating from 1 to 5. Facial videos from 22 of the participants were also recorded at the same time. This paper adopted the 22 facial videos in this dataset and investigated the emotion classification results based on the dimensional emotion model for comparison. The CK + dataset consists of 123 participants. This dataset was posed and spontaneous by multiple participants whose facial expressions started from neutral to the peak. In the CK + dataset, 327 sequences have discrete emotion labels including neutral, sadness, surprise, happiness, fear, anger, contempt and disgust. This paper selected the neutral, anger and happy sequences in this dataset to compare the emotion classification results based on discrete emotion models.

Table 4 shows the average accuracies and F1 scores (average F1 scores for nine classes) for each rating scale (valence, arousal and dominance) when using protocol one on DEFE. We compared the performances of SVM and Xception on the DEFE dataset. In general, the accuracies when using Xception method were at least $30\%$ higher than the accuracy when using SVM. The highest classification accuracy for valence, arousal, and dominance achieved $86.00\%$, $91.54\%$, and $88.17\%$, respectively, when using Xception. In terms of the F1 scores, the highest scores for valence, arousal, and dominance were: $83.73\%$, $91.76\%$, and $79.55\%$ respectively, when using Xception. In addition to the emotion recognition results on the DEFE dataset, Table VI also shows the comparison results on the DEAP dataset when using the same recognition algorithms. The results show that the DEFE dataset had higher recognition accuracies and F1 scores than the DEAP dataset, which may be because the participants’ faces were affixed with electrode pads for physiological signals collection in the DEAP dataset, which affected the facial expression recognition results.

Similarly, Table 5 shows the average accuracies and F1 scores for the emotion categories (anger, happiness, and neutral) when using protocol two. We also compared the classification results when using SVM, and Xception in Table V. The results show that both the highest classification accuracy ($90.34\%$) and the highest F1 scores ($90.21\%$) were obtained when using Xception. Apart from the emotion recognition results on DEFE, Table V also presented the comparison results on the CK + dataset when using the same recognition algorithms. The results show that the recognition results of the CK + dataset were higher than that of DEFE dataset.

Moreover, Table 5 shows the average accuracies and F1 scores of the intensity classification results on anger and happiness emotions when using protocol two with different algorithms. Five classes of the intensity of anger and happiness were classified based on facial expression data. The results show that the highest classification accuracies for angry and happy driving intensity were $97.60\%$ and $97.88\%$, respectively. The highest F1 scores for angry and happy intensity were $97.88\%$ and $97.59\%$, respectively. It should be noted that in recognition of emotion intensity, we did not compare the results with other datasets, because there was currently no spontaneous facial expression datasets with emotional intensity labels.

The comparison results in this section show that there is a difference in human facial expression between DEFE and CK+. Due to the influence of driving tasks in driving scenarios, facial expressions of drivers may be suppressed when they experience emotional states. Hence, it is necessary to discuss further the difference between human facial expressions in dynamic driving scenarios and static life scenarios.

In this section, we conducted a differential analysis of the facial expressions between dynamic driving and static life conditions by comparing the DEFE and JAFFE datasets. The static life dataset, Japanese Female Facial Expression (JAFFE) dataset[14], was selected as a baseline. It was posed by 10 East-Asian females with seven emotion expressions (happy, anger, disgust, fear, sad, and neutral). Each female had two to four examples for each emotion. In total, there are 213 grayscale facial expression images in this dataset.

Given the East-Asian cultural background with small difference, the JAFFE dataset was the most optimal control group for our DEFE dataset because of the excluded most cultural bias[56]. Since DEFE only include two emotions (anger and happiness), we also selected anger and happiness expressions from JAFFE for analysis. Meanwhile, gender differences may affect the results so that we removed the male drivers from the initial DEFE dataset.

Each participant’s facial expressions were evaluated by observing subtle changes in facial features. The Facial Action Coding System (FACS) [58] is a systematic approach to describe what a face looks like when facial muscle movements have occurred. There are 44 coded facial muscle movements, namely Action Units (AUs), in FACS according to the presence and intensity of facial movements. Ekman et al. further proposed that facial emotion expressions could be coded as a combination of several AUs. Figure.6 (a) and (b) display the common FACS [57] codes for anger and happiness, respectively, and Figure.6 (c) presents the AUs descriptions for anger and happiness. In this study, the AU codes for anger (AU 4, 5, 7 and 23) and happiness (AU 6 and 12) were used as the basic units for differential analysis.

We utilized OpenFace [59], a facial expression analysis toolkit, to detect the presence of AUs. When an AU was detected, we coded it as 1 and otherwise 0. Due to video enable to capture enriched data, DFEE contained more facial expression information compared than JAFFE. In the end, the number of observations of happy and anger expressions in JAFFE was 61. DEFE, as a video dataset, had 10020 and 6660 number of observations of happy and angry expressions.

To analyze the differences of AU presence between dynamic driving and static life conditions, we conducted a statistical analysis to investigate the presence of AUs in the two datasets. Given the same emotions in both datasets, we should not observe a statistical difference if the facial expressions were similar between dynamic driving and static life conditions. Meanwhile, the average difference between the two datasets may not fully reflect emotional changes. Instead, it may be led by the baseline difference of two datasets.

Hence, to study the relationship of these AUs to anger and happiness in the two datasets, a logit regression was performed on the two datasets separately with happiness coded as 1 and anger as 0. If the relationship coefficients of AUs had differences in the two datasets, it could be concluded that some AUs performed differently between dynamic driving and static life scenarios. It should be noted that positive coefficient means the AU is related to happiness and negative coefficient means the AU is related to anger.

Statistical analysis results of AUs presence are shown in Table 6. For happiness, the results show that AU6 and AU12 movements could be observed in both JAFFE and DEFE. However, compared with JAFFE, the presence frequencies of AU 6 and AU 12 in DEFE were significantly lower (p<0.01). For anger, the results show that AU4, AU5 and AU23 movements could be observed in both JAFFE and DEFE, and there are significant differences (p<0.01). Besides, we found that AU7 related to anger from DEFE did not appear in the anger expressions from JAFFE. Sample images of facial expressions in JAFFE and DEFE with labelled AUs as shown in Figure.7.

Compared with JAFFE, DEFE had a lower presence frequency on AU4, AU5, AU6, and AU12, especially AU4, which is highly related to anger, had a slight presence frequency in DEFE. The results may be caused by the main driving task, which requires concentration during driving, and the concentration may decrease the presence of AUs near eyes. On the other hand, the presence frequencies of AU7 and AU23 were lower in JAFFE, which maybe because of the difficulties to express negative emotions in Japanese culture [60].

The logit regression results are shown in Table 7. According to our regression results, in JAFFE, for happiness, the coefficients of AU6 and AU12 were consistent with the results from FACS [57], which means AU6 and AU12 were related to happiness. However, only the results of AU12 are significant (p<0.01). For anger, the coefficients of AU4, AU5, and AU23 were consistent with the results from FACS [57], which means AU4, AU5, and AU23 were related to happiness. The results of AU4, AU5, and AU23 are significant (p<0.01). Interestingly, AU7 (lid tightener) presence shows that AU7 was related to happiness which was different from previous researches[57]. In DEFE, only the result of AU4 was significant(0.01<p<0.05), and the coefficient was consistent with the research in FACS, indicating that AU4 had a significant predictive ability for anger. Other AUs were not observed with significant results.

Overall, for AUs presence, AU4 (Brow Lowerer), AU5 (Upper Lid Raiser), AU6 (Cheek Raiser), AU7 (Lid Tightener), AU12 (Lip Corner Puller), AU23 (Lip Tightener) are significant differences between dynamic driving and static life scenarios. The presence of AU4, AU5, AU6 and AU12 are higher in static life scenarios, indicating that AU AU4, AU5, AU6 and AU12 in dynamic driving scenarios are affected by the main driving tasks, which suppresses the facial expression of the driver ’s emotions. Meanwhile, the presence of AU7 and AU23 is higher in dynamic driving scenarios, which may be because Japanese culture suppresses the expression of negative emotions [60]. As for logit regression results, there are also significant differences between dynamic driving and static life scenarios. For anger, the results in dynamic driving scenarios show that only AU4 is significantly related to anger, while in static life scenarios AU4, AU5, and AU23 are all significantly related to anger. For happiness, the logistic regression results in dynamic driving scenarios show that there is no significant correlation between AUs and happiness, but the results in static life scenarios show that AU12 is significantly related to happiness. These significant differences were most likely due to the main driving tasks, which reduced the frequency and amplitude of facial muscle movements. Due to the limitation of JAFFE data amount, these results may require further investigations.